A method for dynamically constructing a dataset based on NLP and a controllable shared API

Abstract

This invention discloses a method for dynamically constructing datasets and controllable shared APIs based on Natural Language Processing (NLP), belonging to the field of computer software technology. The method includes: receiving a natural language query input from a user; calling an NLP engine to parse the natural language query and generate an intermediate representation; dynamically generating executable query logic based on the intermediate representation and executing it in a secure sandbox environment to obtain the original result set; analyzing the original result set, automatically inferring and constructing a structured dataset, and dynamically creating a data model based on its metadata; dynamically generating API endpoints at runtime, wherein the API endpoints include Uniform Resource Locator (URL) paths, endpoint functions bound to the data model, and automatically injected authentication dependencies; registering the dynamically generated API endpoints to a routing table and returning access information. This invention achieves end-to-end automated generation from natural language requirements to standardized APIs with a unified authentication mechanism, lowering the development threshold for data services and ensuring production-grade API security.

Description

Technical Field

[0001] This invention relates to the field of computer software technology, specifically to artificial intelligence, natural language processing, database querying, and automatic API service generation technology, and more specifically to a method for dynamically constructing datasets and controllable shared APIs based on NLP. Background Technology

[0002] With the rapid expansion of the high-speed railway network and the continuous advancement of railway informatization, the amount of data accumulated during high-speed rail operations is growing exponentially. This data extensively covers business areas such as passenger transport, rolling stock, electrical engineering, locomotive maintenance, power supply, disaster monitoring, human resources, and finance, and has become a core asset for the railway system's refined management, predictive maintenance of equipment, and optimization of operational strategies. Enterprises and organizations have an increasingly urgent need for efficient data utilization, and business personnel frequently need to extract specific data from various databases and share it externally via APIs to support real-time decision-making, mobile applications, and third-party system integration.

[0003] The current process, from proposing business requirements to obtaining a secure and usable API, presents significant technical challenges. Transforming data requirements described in natural language into a standardized, programmable API essentially requires mapping across three levels: first, mapping the semantics of the requirements to database query language; second, mapping the raw data to a structured data model; and third, mapping the data model to a secure and managed API service.

[0004] Existing technologies exhibit fragmentation when processing the above mappings, resulting in the following persistent pain points: High technical barriers: Business personnel typically lack professional database query skills and need to "translate" their business requirements into SQL statements before handing them over to technical staff for implementation, or submit API development work orders to the development team. This process is not only costly in terms of communication, but also prone to ambiguity and misunderstandings during requirement transmission, leading to rework and extended development cycles.

[0005] Low development efficiency: After receiving requirements, developers need to manually write data query logic (such as SQL statements), design API routing structures, define request and response data models, integrate authentication and authorization code, and configure API documentation. This work is largely repetitive, and the burden of code modification and maintenance is heavy, especially when data requirements change frequently.

[0006] Complex access control: Enterprise-level APIs must be integrated into a unified identity authentication and access control system. For each new API, developers must manually integrate JWT authentication, Auth2 workflows, or API key verification, and configure corresponding role-based access control policies. Because each API is developed independently, access control configurations are prone to oversights or inconsistencies, posing security vulnerabilities to the system.

[0007] Poor consistency: APIs developed by different developers or at different times are difficult to unify in terms of interface style, parameter naming, error handling, and response format, which increases the integration difficulty for front-end callers and makes later maintenance difficult. Summary of the Invention

[0008] The purpose of this invention is to provide a method for dynamically constructing datasets and controllable shared APIs based on NLP, which solves the technical problem in the prior art that the natural language data requirements cannot be directly and automatically converted into standardized APIs with unified authentication mechanisms, resulting in low development efficiency and security risks.

[0009] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: A method for dynamically constructing datasets and a controllable shared API based on NLP includes the following steps: Step 1: Receive the natural language query input by the user; Step 2: Call the Natural Language Processing (NLP) engine to parse the natural language query and generate an intermediate representation, which includes at least one of the following: data source, filtering conditions, aggregation fields, and sorting rules; Step 3: Based on the intermediate representation and the pre-configured data source connection information, dynamically generate executable query logic, and execute the executable query logic in a secure sandbox environment to obtain the original result set from the data source; Step 4: Analyze the original result set, automatically infer and construct a structured dataset containing field names and data types, and dynamically create the corresponding data model based on the metadata of the structured dataset; Step 5: Dynamically generate an API endpoint at runtime. The API endpoint includes a Uniform Resource Locator path, an endpoint function bound to the data model, and automatically injected authentication dependencies for verifying the identity and permissions of the caller. Step 6: Register the dynamically generated API endpoint to the routing table of the API gateway or web application framework, and return the access information of the API endpoint to the user.

[0010] Furthermore, step 2, which involves calling a Natural Language Processing (NLP) engine to parse the natural language query and generate an intermediate representation, also includes: Based on a pre-built domain dictionary and thesaurus, the extracted filtered entities and indicator fields are ambiguously resolved and mapped to generate a mapping relationship that precisely matches the data source metadata. When multiple candidate mappings exist, the matching confidence score is calculated based on the user's historical query preferences and the current query context, and the mapping with the highest confidence score is selected.

[0011] Specifically, step 4, which involves dynamically creating a corresponding data model based on the metadata of the structured dataset, includes: Based on the inferred data type, numerical range, string length, and null value constraint of each field, a data model class containing type annotations, field validators, serialization methods, and JSONSchema documents is generated using the dynamic class creation mechanism or metaclass programming of the programming language. The data model class is compiled and loaded into memory at runtime, and is used for real-time type validation and automated documentation generation of the returned data from subsequent API calls.

[0012] Furthermore, step 5, which dynamically generates an API endpoint at runtime, includes: By using a dependency injection container, pre-defined authentication dependencies are injected into the code implementation of endpoint functions as constructor parameters or closure capture variables, making the authentication logic an integral part of the endpoint functions. The authentication dependency is executed first when the endpoint function is called, and its execution result is passed to the subsequent data retrieval logic as the permission context. Any attempt to bypass the authentication dependency and directly call the data retrieval logic will not be possible due to the function scope limitation.

[0013] Compared with the prior art, the present invention has the following beneficial effects: This invention transforms user-input natural language queries directly into callable, standardized APIs by constructing an automated pipeline driven by an intermediate representation. It automates the entire process, from semantic parsing and query logic generation to data model abstraction and API service encapsulation, eliminating the need for manual SQL writing or server-side code. This significantly lowers the technical barrier to data service development and drastically shortens the cycle from business requirement submission to API deployment.

[0014] This invention changes the traditional API security mechanism's "post-configuration, manual integration" model. During the dynamic generation of API endpoints, pre-defined, standardized authentication dependencies are forcibly injected into the endpoint functions as an unavoidable component. This ensures that every automatically generated API possesses unified authentication and permission checking capabilities from its inception, eliminating the need for individual configuration by developers. This fundamentally prevents security vulnerabilities caused by human error and achieves native binding of security policies to the API lifecycle.

[0015] This invention supports the dynamic creation, registration, and provision of entirely new API endpoints based on a single natural language query while the application service is running without restarting. The dynamically generated APIs possess complete routing, data models, business logic, and documentation, and can be immediately invoked by clients. This breaks the traditional static "development-deployment-release" delivery model, enabling data services to quickly respond to business changes and improving system agility and resource utilization.

[0016] This invention automatically infers structured metadata from query result sets, dynamically generates a data model with strict type annotations, and binds it to API endpoints. This ensures the standardization of API returned data in terms of format, type, and constraints. Simultaneously, it utilizes the programming language's type system to validate output data at runtime, avoiding call errors caused by data type mismatches or inconsistent formats, thus improving the reliability and maintainability of the API.

[0017] This invention automatically incorporates dynamically generated APIs into a unified routing system and interface documentation specifications. All APIs adhere to the same authentication model, error handling mechanism, and documentation format. Developers can view, debug, and manage all automatically generated APIs through a unified entry point, significantly improving the consistency and observability of enterprise-level API governance and reducing the complexity of front-end integration.

[0018] This invention provides a fully automated solution from requirements to services by deeply integrating natural language understanding, data querying, model abstraction and security mechanisms, achieving a systemic breakthrough in reducing development barriers, ensuring security baselines and improving delivery efficiency. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained from these drawings without creative effort.

[0020] Figure 1 This is a diagram of the overall architecture of the present invention.

[0021] Figure 2 This is a flowchart illustrating the overall process of dynamically constructing datasets and controllable shared APIs based on NLP in this invention. Detailed Implementation

[0022] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the spirit or scope of the embodiments of the invention. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0023] The following is in conjunction with the appendix Figures 1-2 The embodiments of the present invention will be described in detail below.

[0024] Example 1: This embodiment discloses a method for dynamically constructing datasets and controllable shared APIs based on NLP. See [link / reference] Figure 1 The specific system architecture used in this embodiment is as follows: 1. Natural Language Understanding and Task Execution Layer: It receives natural language descriptions from user input. It utilizes a pre-trained NLP model for intent recognition and semantic parsing. The parsed results are then converted into an intermediate representation that explicitly describes the data source, filtering conditions, aggregation fields, sorting rules, etc.

[0025] Based on the intermediate representation and pre-configured data source connection information, executable code is automatically generated. Parameterized SQL queries are preferred, but for complex logic, Pandas DataFrame operations or Python processing scripts can be generated.

[0026] Secure execution: Execute the generated code in a secure sandbox environment, connect to the database or data platform, and obtain the raw data result set.

[0027] 2. Dataset Construction and Encapsulation Layer: Scans the data result set obtained in the first layer and automatically infers the name, data type (string, integer, date, etc.) and possible constraints of each field.

[0028] Automatically create Pydantic data models: Dynamically generate corresponding Pydantic model classes based on inferred metadata. This model will serve as the response body model for the API, ensuring type safety and serializability of the output data.

[0029] 3. Secure API Auto-Generation Layer: Application scaffolding generation: Dynamically create routes, dependencies, and endpoint functions that conform to the FastAPI specification within the current application's runtime environment.

[0030] Route creation: Create a unique API endpoint path for this request (e.g., / api / aut_dataset / {dataset_id}).

[0031] Authentication and permission injection: The system pre-sets a standard set of authentication dependencies (using Bearer Tken's JWT verification).

[0032] These authentication dependencies are automatically injected as function parameters into dynamically generated API endpoint functions. This means that any call to that endpoint must be made through unified authentication.

[0033] It can be further integrated with access control policies (for example, binding the generated dataset_id to user roles) to achieve row-level or column-level data access control.

[0034] Endpoint function logic encapsulation: The internal logic of the generated endpoint functions is fixed. S1. Invoke the authentication dependency to verify the user's identity and permissions.

[0035] S2. Based on the request parameters (if any) and the user identity, execute the query logic generated and cached in step S1.

[0036] S3. Use the Pydantic model generated in the second layer to serialize and return the result.

[0037] S4. Returns a standard JSN response.

[0038] Documentation automation: Utilizing the FastAPI mechanism, the endpoint's description, parameters, response model, and other information are automatically registered in the system's penAPI documentation for front-end developers to review and debug.

[0039] The detailed implementation process of dynamically constructing datasets and controllable shared APIs based on NLP is as follows: Phase 1: Natural Language Parsing and SQL Generation; Step 1: The user sends a natural language query; Users describe their data requirements in natural language via a web interface, command-line tool, or API client. For example: "Display the top 10 products by sales revenue last month, sorted in descending order of sales revenue, including product name, sales revenue, and growth rate information." Upon receiving the query, basic format validation and standardization are performed.

[0040] Specifically as follows: 1) Users submit query requests via web interface, chatbot, or API endpoint; 2) The system records request logs, including request ID, timestamp, IP address, user agent, and other information; 3) The system verifies the request format and user permissions (whether they have the right to create a new dataset).

[0041] Step 2: The system calls the NLP engine to parse the intent and semantics. The preprocessed natural language query is sent to a dedicated NLP processing service. This service is responsible for understanding the deep intent and semantic structure of the query.

[0042] The detailed implementation is as follows: 1) Preprocessing: Text cleaning (removal of special characters, standardization of formatting); Word segmentation (Chinese and English word segmentation); Part-of-speech tagging; 2) Intent recognition (using a fine-tuned BERT model): Input: The segmented sequence; Output: Intent classification probability distribution; Supported intents: * SELECT_QUERY (40%); * AGGREGATIN (30%); * TIME_SERIES (20%); * CMPARISN (10%); 3) Named Entity Recognition (NER): Identify entity types: time, amount, product, department, etc.; Entity example: * "First Quarter of 2024" → {"type":"TIME_RANGE", "value": "2024-01-01 t2024-03-31"}; * "1 million" → {"type": "AMUNT", "value": 1000000, "unit": "RMB"}; 4) Semantic Role Labeling (SRL): Identify predicate-argument structures; Determine the core predicates and parameters of the query; 5) Database pattern matching: Use vector similarity to match user terms with database fields; Example of a matching matrix:

[0043] Step 3: The NLP engine returns an intermediate representation (IR); NLP engines use the following techniques to parse queries: 1. Intent recognition: Classified into "data query", "report generation", "predictive analysis", etc.; 2. Entity extraction: Recognizing time, indicators, dimensions, filtering conditions, etc.; 3. Semantic understanding: Constructing a deep meaning representation of the query; An example of an Intermediate Representation (IR) format is shown below: Step 4: The system calls the SQL generator to generate parameterized SQL; Specifically, the structured intermediate representation is passed to the SQL generation module, which combines the data source metadata (table structure, field types, and relationships) to generate an executable SQL query.

[0044] SQL generation algorithm process: Inputs: IR semantic graph, database schema, user context; Output: Parameterized SQL statement, parameter mapping; 1. Table selection strategy: The main table is determined based on entity recognition; Add related tables based on the connection relationship; Apply the minimum set principle; 2. Field mapping: Map fields in the semantic graph to actual database fields; Handling field aliases and calculated fields; 3. Conditional generation: Convert the filter conditions into a WHERE clause; All user input values ​​are replaced with parameter placeholders (:p1, :p2...); Generate a parameter mapping dictionary; 4. Connection generation: Generate the JIN clause based on the jin information in the semantic graph; Handling implicit joins (through foreign key relationships); 5. Aggregation and Grouping: Detect aggregation functions (SUM, AVG, CUNT, etc.); Generate a GRUP BY clause; 6. Sorting and pagination: Detection and sorting requirements; Add default pagination (LIMIT 1000); 7. Privilege Injection: Add tenant-based filtering conditions; Add row-level security policy conditions; 8. SQL optimization: Rewrite queries to improve performance; Check for potential full table scans; Step 5: The SQL generator returns the SQL and parameters; The SQL generator outputs parameterized SQL statements and security parameters to prevent SQL injection attacks.

[0045] Phase Two: Secure Query Execution; Step 6: Execute the query (with permission check); The secure execution engine receives the SQL and parameters and performs the following security checks: 1. Permission verification: Check the user's access permissions to the target table; 2. Data anonymization: Apply anonymization rules based on user roles; 3. Query limitations: Limit the maximum number of rows returned, execution time, etc. 4. Audit Log: Records all query operations; The secure execution engine workflow is as follows: 1. Session initialization: - Create a database connection (obtained from the connection pool); - Set session-level parameters (time zone, character set, etc.); - Application session-level permissions; 2. Dynamic permission verification: a) Character verification: - Check if the user role includes "sales_viewer"; - Verify that the character is within its validity period; b) Data access control verification: - Check user access permissions to the sales and products tables; - Verify field-level permissions (whether sales_amunt is accessible); c) Application of row-level security policies: - Automatically add the WHERE condition: AND department_id IN (Departments accessible to the user); - Apply data anonymization rules; 3. Query rewriting: - Add an audit field; - Application performance tips (such as USE INDEX); - Limit the maximum number of rows returned; 4. Pre-execution checks: - SQL syntax validation; - Prevent Cartesian product warnings; - Resource usage estimates; 7. The secure execution engine executes queries on the database; The engine establishes a database connection and executes a parameterized query: Connection pool management: reusing database connections; Transaction management: Ensures the atomicity of queries; Performance monitoring: Record query execution time; The execution process is as follows: async def execute_query(cnnectin, sql, parameters, cntext): # 1. Preparation Statements stmt = await cnnectin.prepare(sql) # 2. Bind parameters (to prevent SQL injection) bund_params = sanitize_parameters(parameters) # 3. Set execution timeout (30 seconds) timeut_task = asynci.create_task(stmt.execute(**bund_params)) # 4. Monitor execution progress try: result = await asynci.wait_fr(timeut_task, timeut=30.0) except asynci.TimeutErrr: # Log timeout query lg_query_timeut(sql, cntext) raise QueryTimeoutErrr("Query execution timed out") # 5. Retrieve the result set rws = await result.fetchall() clumn_names = [desc[0] fr desc in result.descriptin] return QueryResult( rws=rws, clumns=clumn_names, rw_cunt=len(rws), executin_time=result.get_executin_time() ) Step 8: The database returns the original data; The database executes queries and returns raw result sets, typically in a two-dimensional table format.

[0046] The returned data structure is as follows: Raw data representation: [ ["Product A", 1250000, 45], ["Product B", 1100000, 38], ["Product C", 980000, 52], ... ] Column metadata: [ {"name": "prduct_name", "type": "VARCHAR", "length": 100}, {"name": "ttal_sales", "type": "DECIMAL", "precisin": 15, "scale":2}, {"name": "transactin_cunt", "type": "INTEGER"} ] Step 9: Return a safe result set; The engine applies security rules to the raw data: Row-level security: Filters rows that users do not have permission to access; Column-level anonymization: masking sensitive information (such as mobile phone numbers, ID cards); Data watermarking: Add tracking watermarks to sensitive data; The post-processing workflow is as follows: 1. Data anonymization: - Apply desensitization rules based on field sensitivity; - Example: product_id may be hashed; 2. Standardized format: - Date and time formatted as IS 8601; - Numeric types are formatted as standard JSN numbers; - Empty values ​​are treated as null; 3. Add audit information: - Query execution time; - Data generation timestamp; - Row count; 4. Quality Inspection: - Check data consistency; - Validate data types; - Detect outliers; The final result set structure is as follows: { "data": [ { "prduct_name": "Product A", "ttal_sales": 1250000.00, "transactin_cunt": 45 } ], "metadata": { "query_id": "qry_20240520_001", "generated_at": "2024-05-20T10:30:00Z", "rw_cunt": 25, "clumn_cunt": 3, "executin_time_ms": 120, "data_hash": "a1b2c3d4e5f6", "security_level": "cnfidential" }, "schema": { "fields": [ { "name": "prduct_name", "type": "string", "nullable": false, "max_length": 100 }, { "name": "ttal_sales", "type": "decimal", "precisin": 15, "scale": 2, "nullable": false } ] } } Phase 3: Dataset Construction and Model Generation; Step 10: Structuring and Type Inference Data structuring analyzes the patterns in the result set and performs intelligent structure optimization: Nested structure detection: Converting flat data into a hierarchical structure; Data type inference: Inferring the exact data type from a data value; Data normalization: standardizing date formats, units, etc.; The type inference algorithm is as follows: def infer_clumn_types(data, clumn_names): inferred_schema = {} fr i, clumn_name in enumerate(clumn_names): clumn_data = [rw[i] fr rw in data] # Sampling Analysis (for Large Datasets) sample = clumn_data[:1000] if len(clumn_data) > 1000 else clumn_data # Application type detection rules type_scres = { "integer": scre_integer_type(sample), "flat": scre_flat_type(sample), "blean": scre_blean_type(sample), "datetime": scre_datetime_type(sample), "categrical": scre_categrical_type(sample), "string": scre_string_type(sample) } # Select the type with the highest score inferred_type = max(type_scres, key=type_scres.get) # Calculate statistical information stats = cmpute_statistics(clumn_data, inferred_type) # Inference Constraints cnstraints = infer_cnstraints(clumn_data, inferred_type) inferred_schema[clumn_name] = { "type": inferred_type, "python_type": map_t_python_type(inferred_type), "nullable": Nne in clumn_data, "unique_values": len(set(clumn_data)), "statistics": stats, "cnstraints": cnstraints, "sensitive": is_sensitive_field(clumn_name) } return inferred_schema

[0047] Step 11: Return the structured dataset; Output Enhanced Data Structures: The dataset object is as follows: class StructuredDataset: def __init__(self, data, schema, metadata): self.id = generate_dataset_id(metadata["query_id"]) self.name = generate_dataset_name(metadata) self.descriptin = generate_dataset_descriptin(metadata) self.data = data # Structured data (a list of lists or a list of dictionaries) self.schema = schema # Field schema self.metadata = { **metadata, "created_at": datetime.utcnw(), "versin": "1.0", "quality_scre": calculate_quality_scre(data, schema) } self.cache_key = f"dataset:{self.id}" def t_jsn_serializable(self): return { "id": self.id, "name": self.name, "descriptin": self.descriptin, "rw_cunt": len(self.data), "schema": self.schema, "sample_data": self.data[:5], # The first 5 lines are for example "metadata": self.metadata }

[0048] Step 12: Generate the Pydantic model; Dynamically generate Pydantic models based on data structures, supporting: Type validation: Strict data type checking; Document generation: Automatically generate field descriptions; penAPI compatibility: Ensure compatibility with the Swagger / penAPI specification; The dynamic model generation algorithm is as follows:

[0049]

[0050]

[0051]

[0052] return mdel_class Step 13: Return the dynamic model class; Dynamically create Pydantic model classes.

[0053] The model class example is as follows:

[0054]

[0055]

[0056] Phase Four: Secure API Generation; Step 14: Generate permission dependencies; Create permission verification logic based on the context of the original query: Resource permissions: Verify user access permissions to data entities; Operation permissions: Verify whether a user can perform a specific operation; Data range: Restricts users to access only a specific range of data; The permission dependency generation algorithm is as follows:

[0057]

[0058]

[0059]

[0060]

[0061] return permissin_dependency Step 15: Return to the permission check function; The FastAPI dependencies are generated as follows:

[0062]

[0063] Step 16: Register the API endpoint; The route manager receives all components and configures the API endpoints: Path generation: Generate meaningful URL paths based on query semantics; HTTP method selection: Choose GET or PST based on the operation type; Parameter definition: Define query parameters and path parameters; Route Registry; Step 17: Add routes to FastAPI; Use APIRuter from FastAPI to dynamically register routes.

[0064] The process of adding a route is as follows: 1. The route registry calls the app.add_api_rute() method. 2. FastAPI internals: a) Verify routing parameters (path, method, etc.) b) Register routes to the routing table c) Update the penAPI documentation d) Register dependency injection 3. Create a route handler: - Generate a unique operation ID - Create a request handler chain - Registration Middleware 4. Update API documentation: - Add a new endpoint to the penAPI schema - Generate interactive documents - Update route index Step 18: Confirm that the route has been registered The FastAPI application updates the routing table, the penAPI documentation updates automatically, and a registration success confirmation is returned.

[0065] Step 19: Return API endpoint information to the user; Returns detailed information about the API.

[0066] The final response is as follows:

[0067]

[0068]

[0069] Phase 5: API Calls and Responses; Step 20: The user invokes the newly generated API; Users invoke the newly generated API endpoint via an HTTP client, carrying the authentication token and parameters.

[0070] Example of an HTTP request:

[0071] Steps 21-22: Verify JWT and permissions; The API endpoint first executes the permission dependency, verifying the validity of the JWT token and the user's permissions.

[0072] After successful permission verification, a context object containing information such as user identity, role, and data scope is returned.

[0073] Authentication and Authorization Process:

[0074]

[0075]

[0076]

[0077]

[0078]

[0079]

[0080] Steps 23-24: Cache check; Generate cache keys based on query characteristics and check caching systems such as Redis.

[0081] If the cache is hit and the data has not expired, return the cached data directly; otherwise, return null and trigger a database query.

[0082] Cache strategy implementation:

[0083]

[0084]

[0085]

[0086] Steps 25-26: Retrieve the latest data (when the cache is not hit); When the cache is not hit, execute the original SQL query to retrieve the latest data: Parameter binding: Combines user context with query parameters; Query optimization: Add data range filtering conditions; Connection management: Use database connection pooling; The data service returns the query results, and the system updates the cache at the same time.

[0087] Data acquisition process:

[0088]

[0089]

[0090]

[0091] Steps 27-28: Response Processing Detailed description: Post-processing of the data: Pagination: Pagination based on the page parameter; Data transformation: formatted into a response model; Additional calculations: Calculated fields requested by the client; Returns a data structure that conforms to the Pydantic model, ensuring type safety.

[0092] Response processor:

[0093]

[0094]

[0095]

[0096]

[0097]

[0098]

[0099] Step 29: Return a JSN response; Detailed description: Ultimately, a standardized API response is returned.

[0100] Final response structure:

[0101]

[0102]

[0103] This invention enables the automatic generation of secure, high-performance APIs from natural language, significantly reducing the development threshold for data APIs while ensuring system security and maintainability.

[0104] This invention is the first to deeply integrate two independent technical fields, "natural language to query" and "application scaffolding to secure API," forming an end-to-end automated pipeline. It proposes to forcibly and automatically inject predefined, unified security dependencies into the endpoint logic during dynamic API generation. This is the core design to ensure production-grade security for generated APIs, distinguishing it from manual integration or post-configuration. Without restarting existing applications, it dynamically creates, registers, and manages API endpoints and their underlying data query logic and models, achieving true "real-time service."

[0105] Example 2: Based on the natural language parsing and SQL generation in Example 1, this example further refines the NLP engine's parsing process for natural language queries, focusing on resolving semantic ambiguity between business terms and database fields, and improving the accuracy and robustness of intermediate representation generation.

[0106] First, during the system initialization phase, a domain dictionary and thesaurus are automatically built based on the metadata of all configured data sources.

[0107] In practice, the system scans the table structure, field names, field comments, and data dictionaries of each data source to extract core business terms. For example, it extracts the term "sales revenue" from the field name "sales_amount" and the term "product" from the field comment "product name". Subsequently, based on a general thesaurus (such as WordNet) and a domain-specific corpus, each term is expanded with synonyms to generate an expanded thesaurus. For example, "sales revenue" is expanded to "sales income", "sales amount", and "turnover", and "product" is expanded to "goods", "items", and "SKU". Simultaneously, the system assigns an initial matching weight to each term and its synonyms: exact matching of field names has the highest weight, followed by matching of field comments, and then matching of synonyms.

[0108] Upon receiving a natural language query from the user, the NLP engine performs intent recognition and named entity recognition to extract elements such as metric fields, filter entities, and time conditions from the query. Based on this, it performs ambiguity resolution and field mapping for each extracted entity. The specific steps are as follows: Candidate field generation: For each entity, the domain dictionary is queried to obtain all possible candidate database fields. For example, the entity "sales" may correspond to fields in multiple tables, such as sales.amount, orders.total_price, and products.revenue.

[0109] Context-aware similarity calculation: The entire natural language query is encoded into a context vector using a Transformer-based pre-trained language model (such as BERT), while each candidate field and its corresponding table description text (including table name, field name, and comments) are encoded into a field vector. The cosine similarity between the query vector and each field vector is calculated as the semantic matching score.

[0110] User historical preference weighting: The system records each user's historical query behavior, including the field mappings actually selected and used in each query. A user preference model is built based on this historical data, giving extra weight to field mappings frequently used by the same user. For example, if user A frequently queries "sales amount" and ultimately uses `sales.amount`, the confidence level of that mapping will be increased in subsequent queries.

[0111] Confidence fusion and ranking: The semantic matching score, user preference weight, and synonym weight are weighted and fused to obtain the final confidence score of each candidate mapping. The candidate mapping with the highest confidence score is selected as the actual mapping field for that entity.

[0112] Intermediate Representation Generation: The mapping results are combined with other extracted information to form a structured intermediate representation (IR), and the mapping relationship and confidence level of each field are explicitly recorded in the IR. For example, the IR may contain the following information: Metric fields: Original term "sales revenue", mapped field "sales.amount", confidence level 0.98, aggregation method "sum"; Dimension field: Original term "product", mapped field "products.name", confidence level 0.95; Time condition: Original term "last month", mapped field "sales.date", time range expression.

[0113] Through the above-described ambiguity resolution and mapping process, this embodiment significantly improves the accuracy of NLP parsing, avoids query errors caused by terminology ambiguity or differences in field naming from different data sources, and makes the generated intermediate representation closer to the user's true intent, laying a solid foundation for subsequent query logic generation.

[0114] Example 3: Based on the three-stage dataset construction and model generation in Example 1, this example further refines the creation process of the dynamic data model. It utilizes the metaclass mechanism of the programming language to generate a data model class with complete type annotations, validation logic, and documentation descriptions, ensuring the type security and standardization of API response data.

[0115] After obtaining the original result set through secure query execution, the system first performs structuring and type inference on the result set, a process that is more in-depth and detailed than that in Example 1. For each column of data, the system not only infers the basic data type (string, integer, floating-point number, date and time, etc.), but also extracts the following statistical constraint information: Numerical range: For integers and floating-point numbers, calculate the minimum, maximum, average, and standard deviation to determine the boundary conditions for subsequent validation.

[0116] String length: For string types, calculate the maximum length, minimum length, and average length, and check if it contains a specific pattern (such as email address, mobile phone number, ID card number, etc.) so that regular expression validation can be generated later.

[0117] Null value ratio: This counts the proportion of null values ​​(NULL) in the column, determining whether the field can be nullable.

[0118] Uniqueness: Check if the field value is unique. If it is unique, a unique constraint can be added at the model level (although the API level usually does not enforce uniqueness, it can be used for documentation hints).

[0119] Enumerated values: If the number of possible values ​​for a field is limited (such as status fields "pending", "in process", "completed"), extract all possible values ​​as enumeration constraints.

[0120] For example, for the product_name field, the inferred result is of type string, maximum length 100, not empty, and the value is not enumerated; for the total_sales field, the type is decimal, minimum value 0, maximum value 10^7, precision 15, decimal places 2, and not empty.

[0121] Subsequently, based on the inferred field metadata, the system generates a data model class at runtime using the metaclass or dynamic class creation mechanism of the programming language (such as Python's type metaclass or Pydantic's dynamic model API). The generation process includes the following steps: Construct field definitions: Iterate through the inferred schema and generate a definition for each field that includes type annotations, default values ​​(None if nullable), validator parameters (such as ge, le, max_length, regex), and field description information.

[0122] Validator method generation: For fields requiring custom validation (such as date format validation, enumeration value validation, and regular expression pattern matching), the system dynamically generates corresponding validator methods and adds them to the class dictionary. These validators will be executed automatically during data instantiation to ensure that the data conforms to business rules.

[0123] Add configuration metadata: Generate an internal configuration class to set the title, description, sample data, and JSON Schema extension information of the data model. This information will be used for the automatic generation of OpenAPI documentation later.

[0124] Creating the model class: The model class is dynamically created by calling the metaclass constructor and immediately loaded into memory. This model class inherits from the base model class (such as Pydantic's BaseModel), and therefore has serialization, deserialization, and verification capabilities.

[0125] The generated model class is not only used for type validation of the API response body, but also automatically bound to the API endpoint. When a client calls this endpoint, the returned raw data is first converted into a model class instance, and Pydantic automatically performs type conversion and validation. If the data does not conform to the model definition (e.g., excessively long strings, out-of-bounds numeric values, or invalid enumeration values), the system immediately throws a validation error and returns a clear error message to prevent the front end from receiving incorrectly formatted or invalid data. Furthermore, because the model class contains JSON Schema metadata, the FastAPI framework automatically merges it into the global OpenAPI documentation, ensuring that the generated API endpoint has complete request / response documentation and supports real-time interaction and debugging with Swagger UI and ReDoc.

[0126] This embodiment achieves fully dynamic generation of the data model through metaclass programming. The model not only includes type annotations, but also embeds statistical constraints and business rules inferred from actual data, making the API response have strict type safety and data standardization, which is superior to the static models based only on the library table structure or the simple data encapsulation lacking runtime verification in the prior art.

[0127] Example 4: Based on the Phase 4 security API generation in Example 1, this example further refines the injection mechanism of authentication dependencies, ensuring that the authentication logic becomes an integral part of the endpoint function, fundamentally eliminating the possibility of bypassing authentication, and realizing the native binding of security policies and API lifecycle.

[0128] During the system initialization phase, a set of standardized authentication dependency components are predefined and registered, including: User authentication component: responsible for extracting JWT tokens from HTTP request headers, verifying the token's signature and validity period, and parsing user identity information (such as user ID, role, tenant, etc.).

[0129] Permission check component: Based on user identity and the dataset ID to be accessed, check whether the user has permission to read the dataset, and can further refine permissions to the field level.

[0130] Rate limiting component: Based on user ID and API endpoint path, it checks whether the user's access frequency exceeds a preset threshold to prevent abuse.

[0131] These components exist as callable objects and are uniformly registered in the dependency injection container for use when dynamically generating endpoint functions later.

[0132] During the dynamic generation of API endpoints, the system uses closure capture technology to directly "solidify" authentication dependencies into the code implementation of endpoint functions, rather than dynamically resolving them at the time of invocation through the framework's dependency injection declarations (such as FastAPI's Depends).

[0133] The specific steps are as follows: Obtain an instance of the authentication component: Retrieve the callable object (i.e., function or method) of the authentication component mentioned above from the dependency injection container.

[0134] Define an endpoint function closure: In the code that generates the endpoint function, capture the obtained authentication component objects as free variables in the closure environment. Then, write the internal logic of the endpoint function, which explicitly calls these captured authentication components and checks their results. For example, at the beginning of the function body, call the user authentication component, rate limiting component, and permission check component in sequence; only if all checks pass will the subsequent data retrieval logic continue to execute.

[0135] Data retrieval logic encapsulation: Data retrieval logic (such as reading from the cache or re-executing the query) is also defined within the same closure and is not exposed externally. Any call to the endpoint must go through the preceding authentication steps and cannot directly access the data retrieval part.

[0136] Returning endpoint functions: The generated endpoint functions are returned as closures, which fully embed authentication and data retrieval logic. Since the authentication components are captured through closures, they are determined at function definition time and cannot be replaced or bypassed at runtime.

[0137] For example, the generated endpoint function logic is as follows (described in natural language, without code): Function begins: Calls the user authentication component, extracts the token from the request and verifies it; if it fails, it returns a 401 Unauthorized Error.

[0138] Invoke the rate limiting component to check the current user's access frequency to this endpoint. If the limit is exceeded, return a 429 Too Many Requests error.

[0139] Call the permission check component to verify whether the user has permission to access the target dataset. If not, return a 403 Forbidden error.

[0140] If all authentications pass, data is retrieved from the cache based on request parameters (such as pagination and field filtering); if the cache is not hit, the original query is re-executed to retrieve the latest data.

[0141] The acquired data is serialized using a dynamically generated data model, and a standard JSON response is returned.

[0142] Because the authentication logic is written directly inside the endpoint function, any network request to that endpoint will trigger these authentication steps. Even if an attacker obtains a reference to the endpoint function object in some way (which is extremely difficult in a runtime environment), they cannot skip the internal authentication calls because the data retrieval logic is not exposed separately. Furthermore, due to the scope limitations of closures, the captured authentication components cannot be modified or replaced externally.

[0143] This embodiment uses closure capture technology to "embed" authentication dependencies into the code body of endpoint functions, achieving native binding of security policies and API lifecycles. Compared to the external interception mode of traditional API gateways (relying on network-level filters or middleware) or dependency injection declaration mode (relying on the framework to resolve dependencies at call time), this method fundamentally eliminates the possibility of authentication bypass due to configuration errors, code vulnerabilities, or framework defects, providing a higher level of security for dynamically generated APIs. Simultaneously, all access is forcibly logged in audit logs to ensure traceability.

[0144] This invention achieves accurate mapping from natural language terms to database fields by constructing a domain dictionary, calculating contextual semantic similarity, and learning user historical preferences. It solves the semantic gap and ambiguity between business language and the underlying data model, and significantly improves the accuracy and adaptability of NLP parsing.

[0145] Meanwhile, this invention achieves adaptive generation of API response structures and runtime type safety by using metaclass programming to dynamically generate data model classes containing type annotations, field validators, and JSON Schema through in-depth statistical analysis of the result set. This avoids calling errors caused by data type mismatch or constraint violation, and improves the reliability and maintainability of the API.

[0146] This invention injects authentication dependencies directly into the internal implementation of endpoint functions through closure capture technology, making the authentication logic an integral part of the function. This achieves native binding of security policies and API lifecycle, fundamentally eliminating the risk of authentication bypass and providing stricter security guarantees than external gateway interception.

[0147] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention.

[0148] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. It should be noted that any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for dynamically constructing datasets and controllable shared APIs based on NLP, characterized in that, Includes the following steps: Step 1: Receive the natural language query input by the user; Step 2: Call the Natural Language Processing (NLP) engine to parse the natural language query and generate an intermediate representation, which includes at least one of the following: data source, filtering conditions, aggregation fields, and sorting rules; Step 3: Based on the intermediate representation and the pre-configured data source connection information, dynamically generate executable query logic, and execute the executable query logic in a secure sandbox environment to obtain the original result set from the data source; Step 4: Analyze the original result set, automatically infer and construct a structured dataset containing field names and data types, and dynamically create the corresponding data model based on the metadata of the structured dataset; Step 5: Dynamically generate an API endpoint at runtime. The API endpoint includes a Uniform Resource Locator path, an endpoint function bound to the data model, and automatically injected authentication dependencies for verifying the identity and permissions of the caller. Step 6: Register the dynamically generated API endpoint to the routing table of the API gateway or web application framework, and return the access information of the API endpoint to the user.

2. The method for dynamically constructing datasets and controllable shared APIs based on NLP according to claim 1, characterized in that, Step 2 involves calling a Natural Language Processing (NLP) engine to parse the natural language query and generate an intermediate representation, including: performing intent recognition on the natural language query to determine the query type; performing named entity recognition on the natural language query to extract time conditions, indicator fields, and filter entities; and mapping the identified query type, time conditions, indicator fields, and filter entities to predefined data source metadata to form the intermediate representation.

3. The method for dynamically constructing datasets and controllable shared APIs based on NLP according to claim 1, characterized in that, Step 3, which dynamically generates executable query logic, includes: converting the filtering conditions and indicator fields in the intermediate representation into parameterized database query statements; replacing the user input values ​​involved in the natural language query with query parameter placeholders and generating a corresponding parameter mapping table to prevent injection attacks.

4. The method for dynamically constructing datasets and controllable shared APIs based on NLP according to claim 1, characterized in that, Step 3, which involves executing the executable query logic in a secure sandbox environment and obtaining the original result set from the data source, further includes: before executing the query, dynamically adding row-level security filtering conditions and column-level data anonymization rules to the executable query logic based on the user's identity information.

5. The method for dynamically constructing datasets and controllable shared APIs based on NLP according to claim 1, characterized in that, In step 4, a corresponding data model is dynamically created based on the metadata of the structured dataset. Specifically, according to the inferred data type of each field, a data model containing multiple type annotation fields is generated using the dynamic class creation mechanism of the programming language. The data model is used to perform type verification and serialization on the returned data of subsequent API calls.

6. The method for dynamically constructing datasets and controllable shared APIs based on NLP according to claim 1, characterized in that, Step 5, which dynamically generates an API endpoint at runtime, includes: generating a unique endpoint function whose internal logic encapsulates: calling the authentication dependency to obtain the permission context, executing or obtaining the result of the executable query logic based on the permission context and request parameters, and serializing the result using the data model; and allocating a unified resource locator path to the endpoint function.

7. The method according to claim 1 or 6, characterized in that, The authentication dependency automatically injected in step 5 is a pre-built, standardized security component used to force all calls to the API endpoint to pass unified authentication and permission checks, including verifying token validity, user roles, and access permissions to the structured dataset.

8. The method for dynamically constructing datasets and controllable shared APIs based on NLP according to claim 1, characterized in that, After registering the dynamically generated API endpoint to the routing table of the API gateway or Web application framework in step 6, the method further includes: automatically updating the interface documentation of the API gateway or Web application framework so that the newly generated API endpoint, its data model, and calling parameters can be viewed and debugged in the online documentation.

9. The method for dynamically constructing datasets and controllable shared APIs based on NLP according to claim 1, characterized in that, Following step 6, the following is also included: Receive a call request for the API endpoint; The authentication dependency is used to verify the identity token carried in the call request and obtain the permission context; If the verification passes, the executable query logic is retrieved from the cache or re-executed according to the query parameters in the call request to obtain the latest data; The acquired data is serialized using the data model, and a formatted JSN data response is returned.

Images