NL2SQL model-oriented standardized test method and system

Abstract

The invention provides an NL2SQL model-oriented standardized test method and system. The method comprises the following steps of: obtaining business demand data, generating a visual test case through a visual mind mapping tool, and converting the visual test case into structured data to obtain an initial test case set; semantic splitting is carried out on the initial test case to obtain a keyword set, and a keyword-field mapping relation is established through multiple similarity verification. And expanding, writing and recombining the initial test case based on the mapping relation to form an expanded test case set. And inputting the extended test case set into the to-be-tested NL2SQL model to generate a corresponding SQL statement set, and finally realizing standardized evaluation of model performance through comparative analysis. According to the method and the system, the design efficiency and the standardization of the test case are improved through a visual mind mapping tool and multiple similarity verification, the accuracy and the robustness of the model in an actual business scene are accurately evaluated, and a reliable basis is provided for iterative optimization and quality assurance of the NL2SQL model.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence testing technology, specifically relating to a standardized testing method and system for NL2SQL models. Background Technology

[0002] With the widespread application of Natural Language to Structured Query Language (NL2SQL) technology in intelligent data analysis, effectively evaluating and ensuring the accuracy and robustness of NL2SQL models in real-world business scenarios has become a key issue restricting its large-scale deployment. Existing technologies for testing NL2SQL models face the following challenges: test case construction heavily relies on human experience, resulting in low efficiency and insufficient standardization; it is difficult to systematically generate semantically equivalent but diverse test texts, leading to insufficient evaluation of the NL2SQL model's generalization ability; the mapping relationship between test cases and underlying database table fields in business requirement data is ambiguous, making it difficult to automatically verify the semantic correctness of generated SQL statements; and there is a lack of a fully automated toolchain from business requirements to test execution, resulting in high testing costs and difficulty in scaling. These problems restrict the reliability verification and continuous iterative optimization of NL2SQL models in real-world, complex business scenarios. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this invention provides a standardized testing method and system for NL2SQL models to solve the aforementioned problems. This method and system improve the efficiency and standardization of test case design through visual mind mapping tools and multiple similarity verification, accurately evaluate the accuracy and robustness of the model in actual business scenarios, and provide a reliable basis for the iterative optimization and quality assurance of NL2SQL models.

[0004] To address the aforementioned technical problems, this invention provides a standardized testing method for NL2SQL models, comprising the following steps: Obtain business requirement data; The business requirement data is input into a visual mind mapping tool, which then generates visual test cases based on the business requirement data. Based on the visualized test cases and preset SQL structure rules, feature transformation is performed to obtain structured data; An initial test case set is obtained based on the structured data and business requirement data; Based on the initial test case set, semantic segmentation is performed to obtain a keyword set; Based on the keyword set and business requirement data, multiple similarity verifications are performed to establish a keyword-field mapping relationship; Based on the keyword-field mapping relationship and the initial test case set, the test case set is expanded and reorganized to obtain an extended test case set; The extended test case set is input into the NL2SQL model to be tested, so that the NL2SQL model generates a set of SQL statements based on the extended test case set; Based on the SQL statement set and extended test case set, standardized testing was performed to obtain the model performance evaluation results.

[0005] In the above solution, business requirement data is transformed into visual test cases using a visual mind mapping tool. Combined with pre-defined SQL structure rules, feature transformation and structured data processing are completed to construct a precise initial test case set. Further semantic segmentation yields a keyword set, and multiple similarity verifications establish a highly reliable keyword-field mapping relationship set. This set is then expanded and reorganized to generate a large-scale, high-coverage, and semantically consistent extended test case set. This solution integrates test case design, generation, and verification into a closed-loop process, significantly improving the efficiency, standardization, and maintainability of test case design for NL2SQL models. Finally, by inputting the extended test case set into the NL2SQL model under test, automated comparison testing of the generated SQL statement set is performed. This accurately evaluates the accuracy and robustness of the NL2SQL model in real-world business scenarios, providing a reliable basis for the iterative optimization and quality assurance of the NL2SQL model.

[0006] Furthermore, the step of inputting the business requirement data into a visual mind mapping tool, so that the visual mind mapping tool generates visual test cases based on the business requirement data, includes: The business requirement data is input into a visual mind mapping tool so that the tool can construct several central theme nodes, each of which carries a natural language query text. Based on the central theme node, construct query field branches and query condition branches; Based on the query field branch, several first-level child nodes are created; wherein each first-level child node corresponds to a database table field to be queried; Based on the query condition branch, several second-level child nodes are created; wherein each second-level child node is used to define a query condition key-value pair, the key in the query condition key-value pair is a database table field, and the value is a query condition constraint; The visual test cases are generated based on the central theme node, query field branches, query condition branches, several first-level child nodes, and several second-level child nodes.

[0007] In the above scheme, a central theme node carries the natural language query text, which is then expanded into query field branches and query condition branches. By mapping first-level child nodes to database table fields and defining query condition key-value pairs in second-level child nodes, the core components of SQL are clearly separated and visually defined, making complex query logic readily apparent and lowering the design threshold. Simultaneously, the direct mapping between nodes and database fields ensures the accuracy and standardization of test cases, effectively supporting the expression of complex query scenarios and providing a reliable input foundation for subsequent structured data processing and the generation of extended test case sets. This ensures the rigor and efficiency of the NL2SQL model testing process from the outset.

[0008] Furthermore, the step of performing feature transformation based on the visualized test cases and preset SQL structure rules to obtain structured data includes: Based on the preset SQL structure rules, the visual test cases are traversed to obtain the central theme node, query field branches, and query condition branches; Obtain the natural language query title based on the central theme node; Text extraction is performed based on the query field branches to obtain a list of query fields; Based on the query condition branches, key-value pairs are parsed to obtain the query condition set; The structured data is obtained based on the natural language query title, query field list, and query condition set.

[0009] In the above solution, the process of traversing visual test cases based on preset SQL structure rules, extracting natural language query titles, parsing query field lists and query condition sets, and integrating them into unified structured data not only fully preserves the original query logic and semantic integrity but also eliminates human transcription errors through rule-based parsing, ensuring data format standardization and consistency. The resulting structured data provides high-quality, directly processable input for subsequent initial test case set construction and extended test case set generation, thereby effectively improving the automation, processing efficiency, and reliability of the entire testing process.

[0010] It should be noted that the preset SQL structure rules can be specifically set as follows: This includes decomposing the SQL statement into a header section and a condition section, where the header corresponds to the query fields and the condition corresponds to the query conditions. The structured data can be in JSON format, containing a title, a list of query fields, and condition key-value pairs. The structured data can be directly generated by parsing the visual test case file using a software development kit (SDK) of a visual mind mapping tool, thus achieving automated conversion from graphical test cases to structured descriptions.

[0011] Furthermore, the step of obtaining an initial test case set based on the structured data and business requirement data includes: Obtain the business target database based on the aforementioned business requirement data; Based on the query field list, query conditions, business target database, and preset SQL structure rules, a standard SQL statement set is obtained; Natural language query text is obtained based on the structured data; The initial test case set is obtained by establishing a mapping relationship between the natural language query text and the standard SQL statement set.

[0012] The above solution generates a standard SQL statement set based on preset SQL structure rules, combined with a list of query fields, a set of query conditions, and a business target database. This set is then linked to natural language query text to establish a mapping relationship, constructing an initial test case set. This process directly ensures the accurate correspondence between natural language queries and standard SQL statements, eliminating errors and inefficiencies associated with manually written SQL, and forming standardized initial test cases. This solution provides an accurate and reliable input foundation for subsequent semantic segmentation, multiple similarity verification, and the generation of extended test case sets, thereby guaranteeing the quality and consistency of the starting point of the testing process.

[0013] It should be noted that the standard SQL statement set is assembled according to the query field list and query condition set, combined with the database table names and field names, and in accordance with SQL syntax specifications. The initial test case set can be in the form of a file containing natural language query text and its corresponding standard SQL statement, such as an Excel file, where each line represents a test case, including the test case title, natural language text, and expected SQL statement.

[0014] Further, the semantic segmentation based on the initial test case set to obtain a keyword set includes: Obtain natural language query text based on the initial test case set; The natural language query text is segmented to obtain a sequence of semantic units; Based on the semantic unit sequence, semantic filtering is performed to obtain the core semantic unit sequence; A keyword set is obtained based on the core semantic unit sequence.

[0015] The above scheme obtains natural language query text based on an initial test case set, performs word segmentation to obtain a sequence of semantic units, and then extracts the core semantic unit sequence through semantic filtering to generate a precise keyword set. This process directly filters out auxiliary words and noise with no query value, achieving automated conversion from complex natural language to core semantic units and reducing the subjectivity and error of manual parsing. This scheme ensures that the obtained keyword set accurately matches the true query intent, providing a clean and accurate input foundation for subsequent multiple similarity verifications and the construction of keyword-field mapping relationships, thereby guaranteeing the reliability and consistency of the testing method at the semantic understanding level.

[0016] It should be noted that the word segmentation process can employ existing word segmentation tools, such as the jieba word segmenter, to split natural language text into semantic units using a preset word segmentation algorithm. The semantic filtering process aims to remove stop words, interjections, and other words without query intent, while retaining core nouns, verbs, numerical values, and operators that reflect the query intent, such as "product name," "amount," "greater than," and "September 2025."

[0017] Furthermore, the step of performing multiple similarity verifications based on the keyword set and business requirement data to establish a keyword-field mapping relationship set includes: Data metadata is obtained from the target database based on the business requirement data, and the data metadata includes several field description texts; For each keyword in the keyword set, perform the following operations: Based on the keyword and the descriptive text of several fields, semantic similarity processing is performed to obtain an initial semantic similarity set; Based on the initial similarity set and the preset first threshold, obtain the initial candidate field set corresponding to the keyword; Based on the initial candidate field set and the preset semantic disambiguation algorithm, semantic disambiguation is performed to obtain a weighted candidate field set; Based on the weighted candidate field set, a target mapping field is obtained so that a mapping relationship is established between the keyword and the target mapping field; Obtain a set of keyword-field mapping relationships based on the mapping relationships corresponding to several keywords.

[0018] The above solution involves obtaining data metadata from a target database based on business requirements, performing semantic similarity processing on each keyword in the keyword set and the field description text, and then filtering an initial candidate field set using a preset first threshold. A weighted candidate field set is then obtained through a preset semantic disambiguation algorithm to establish a keyword-field mapping relationship set. This process directly and effectively distinguishes semantically similar but functionally different fields, eliminates the uncertainty of single-round matching, significantly improves the uniqueness and accuracy of the mapping relationship, and ensures the syntactic and semantic correctness of subsequent SQL generation. The established mapping relationship set provides a reliable structured foundation for test case expansion and reorganization, thereby ensuring the rigor and automation level of the testing process at the semantic understanding level.

[0019] It should be noted that the semantic similarity processing can be implemented based on the cosine similarity calculation of word vectors. Specifically, keywords and field description text are converted into word vectors respectively, and the cosine angle between them is calculated. The closer the similarity is to 1, the more similar the semantics. The preset first threshold is used to filter candidate fields that are initially associated. The semantic disambiguation algorithm is used to resolve ambiguities that may arise when multiple candidate fields are semantically similar. Its specific implementation may include calculating the semantic homogeneity between candidate fields and performing cluster evaluation. Based on the evaluation results, it is decided whether to use weighted similarity calculation or combine table structure information for final filtering.

[0020] Further, the semantic disambiguation is performed based on the initial candidate field set and the preset semantic disambiguation algorithm to obtain a weighted candidate field set. The semantic disambiguation algorithm specifically includes: Perform semantic similarity acquisition processing on any two initial candidate fields in the initial candidate field set to obtain a semantic homogeneity matrix; Based on the semantic homogeneity matrix and the preset clustering homogeneity algorithm, homogeneity evaluation is performed to obtain the semantic homogeneity evaluation result; If the semantic homogeneity assessment result is less than a preset second threshold, then the following steps are performed: For each keyword in the keyword set, a weighted semantic similarity processing is performed based on the keyword and several field description texts to obtain a weighted semantic similarity set; The weighted candidate field set is constructed based on the weighted semantic similarity set and the corresponding descriptive text of several fields.

[0021] In the above scheme, the semantic disambiguation algorithm first calculates a semantic homogeneity matrix for the initial candidate field set, then performs semantic homogeneity evaluation using a preset clustering homogeneity algorithm. Next, when the evaluation result is less than a preset second threshold, weighted semantic similarity processing is performed on the keywords and field description text to construct a weighted candidate field set. This process directly and effectively distinguishes database fields with similar meanings but different functions, overcoming the limitations of traditional single-threshold matching. This scheme provides a reliable decision-making basis for accurately determining the target mapping field, thereby ensuring the robustness and high accuracy of the semantic parsing stage in the NL2SQL model testing process.

[0022] It should be noted that the clustering homogenization algorithm is used to calculate the overall semantic similarity of the candidate field set. One specific implementation is to calculate the average similarity between all candidate field pairs as a homogenization index H. The closer the H value is to 1, the more semantically similar the candidate fields are; the closer it is to 0, the greater the difference. When the H value is less than a preset second threshold (e.g., 0.7), it indicates that the candidate fields have significant functional differences, and weighted semantic similarity processing is required. This weighted semantic similarity processing involves introducing weight coefficients to adjust the initial similarity. These weight coefficients can be dynamically determined based on factors such as the historical query frequency of the field; the higher the query frequency, the greater the weight, thereby strengthening the association probability of frequently used fields in the secondary calculation.

[0023] Furthermore, the step of performing homogeneity evaluation based on the semantic homogeneity matrix and a preset clustering homogeneity algorithm to obtain semantic homogeneity evaluation results also includes: If the semantic homogeneity evaluation result is greater than or equal to a preset second threshold, then the following steps are performed: Obtain the target database table structure information based on the aforementioned business requirement data; Based on the initial candidate field set, obtain the database table and data type corresponding to each candidate field; For each keyword in the keyword set, a structured filtering is performed based on the database table corresponding to each candidate field, the data type corresponding to each candidate field, the target database table structure information, and the keyword to obtain the target mapping field corresponding to the keyword. The weighted candidate field set is constructed based on the target mapping fields corresponding to several keywords.

[0024] In the above scheme, semantic homogeneity assessment is completed based on a semantic homogeneity matrix and a preset clustering homogeneity algorithm. The semantic homogeneity matrix quantifies the semantic similarity between any two initial candidate fields, while the clustering homogeneity algorithm comprehensively judges the overall semantic consistency of the fields. The magnitude of the assessment result is positively correlated with the degree of semantic convergence between the initial candidate fields. When the assessment result is greater than or equal to a preset second threshold, it indicates that the semantic differences between the initial candidate fields are minimal, falling into the category of highly semantically consistent fields. At this point, the scheme switches to using the target database table structure information, the database tables corresponding to the candidate fields, and their data types for structured screening, avoiding the limitations of simple semantic calculations and ensuring the accuracy and reliability of the mapping decision.

[0025] Furthermore, the expansion and reorganization based on the keyword-field mapping relationship and the initial test case set to obtain an expanded test case set includes: Obtain a natural language text set based on the initial test case set; For each natural language text in the natural language text set, a synonym replacement operation is performed based on the keyword-field mapping relationship and the natural language text to obtain the first variant query text; The natural language text is input into a preset semantic expansion model, so that the semantic expansion model performs semantic expansion based on the natural language text to obtain a second variant query text. Based on the first variant query text and the second variant query text, obtain the variant query text set corresponding to the natural language text; For each variant query text in the variant query text set, an SQL statement is constructed based on the keyword-field mapping relationship, the preset SQL structure rules and the variant query text to obtain the standard SQL statement corresponding to the variant query text. Based on several variant query texts and corresponding standard SQL statements, obtain an extended test sample set corresponding to the natural language text; The extended test case set is obtained based on an extended test sample set corresponding to several natural language texts.

[0026] The above solution employs a dual process: generating the first variant query text through synonym substitution based on the natural language text set of the initial test case set, and simultaneously generating the second variant query text through a semantic expansion model. This directly ensures the diversity and semantic equivalence of test cases. Furthermore, by generating standard SQL statements from all variant texts according to a unified mapping relationship and preset SQL structure rules, semantic biases and syntax errors inherent in manual expansion are effectively avoided, ensuring the accuracy of the expanded test cases. This solution automates the production of test cases from seed cases to a large-scale expanded test case set, providing the NL2SQL model with comprehensive and accurately labeled test data, significantly enhancing the evaluation effectiveness of the model's robustness and generalization ability.

[0027] It should be noted that the preset semantic expansion model can be an interface service based on a large language model, such as calling a specific MCP (Model Context Protocol) service via HTTP. This service receives the original natural language text and the desired number of expansions, and returns multiple sets of semantically equivalent but differently expressed expanded texts. For each variant query text, based on the established keyword-field mapping relationship, the keywords in the text are replaced with the corresponding database field names, and combined with SQL syntax rules to assemble a standard SQL statement, thereby ensuring that each natural language variant has a unique and accurate corresponding expected SQL result.

[0028] It should be noted that the step of inputting the extended test case set into the NL2SQL model to be tested, so that the NL2SQL model generates a set of SQL statements based on the extended test case set, may specifically include: submitting the natural language query text in the extended test case set to the model one by one or in batches through the software development kit or application programming interface provided by the NL2SQL model, and receiving the SQL statements returned by the NL2SQL model to be tested, thus forming the set of SQL statements generated by the model.

[0029] This invention also provides a standardized testing system for NL2SQL models, comprising: The business requirement acquisition module is used to acquire business requirement data; The visual test case generation module is used to input the business requirement data into the visual mind mapping tool, so that the visual mind mapping tool can generate visual test cases based on the business requirement data. The feature conversion module is used to perform feature conversion based on the visualized test cases and preset SQL structure rules to obtain structured data; The initial test case generation module is used to obtain an initial test case set based on the structured data and business requirement data. An extended test case generation module is used to perform semantic segmentation based on the initial test case set to obtain a keyword set; perform multiple similarity verifications based on the keyword set and business requirement data to establish a keyword-field mapping relationship; and expand and reorganize the keyword-field mapping relationship and the initial test case set to obtain an extended test set. The model testing module is used to input the extended test set into the NL2SQL model to be tested, so that the NL2SQL model generates a set of SQL statements based on the extended test set; and to perform normalization tests based on the set of SQL statements and the extended test set to obtain the model performance evaluation results.

[0030] The above solution integrates modules for business requirement acquisition, visual test case generation, feature conversion, initial test case generation, extended test case generation, and model testing. These modules operate collaboratively based on unified rules and data flow. Through a closed-loop process from business requirements to visual test cases, structured data, and initial and extended test case sets, the scattered manual testing steps are transformed into automated pipeline operations, significantly improving testing efficiency and standardization. Simultaneously, relying on visual design, structured parsing, and semantic expansion technologies, it ensures comprehensive semantic coverage of test cases and accurate expected results, providing a reliable quality assessment method for the NL2SQL model and strongly supporting model performance optimization and iterative deployment. Attached Figure Description

[0031] Figure 1 This is a schematic diagram of a standardized testing method for the NL2SQL model provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of a standardized test system architecture for the NL2SQL model provided in an embodiment of the present invention. Detailed Implementation

[0032] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0033] Please see Figure 1 This embodiment provides a standardized testing method for NL2SQL models, including the following steps: Step S1: Obtain business requirement data; Step S2: Input the business requirement data into the visual mind mapping tool so that the visual mind mapping tool can generate visual test cases based on the business requirement data; Step S3: Perform feature transformation based on the visualized test cases and preset SQL structure rules to obtain structured data; Step S4: Obtain an initial test case set based on the structured data and business requirement data; Step S5: Perform semantic segmentation based on the initial test case set to obtain a keyword set; Step S6: Perform multiple similarity verifications based on the keyword set and business requirement data to establish a keyword-field mapping relationship; Step S7: Based on the keyword-field mapping relationship and the initial test case set, expand and reorganize to obtain an expanded test case set; Step S8: Input the extended test case set into the NL2SQL model to be tested, so that the NL2SQL model generates a set of SQL statements based on the extended test case set; Step S9: Perform standardized testing based on the SQL statement set and extended test case set to obtain the model performance evaluation results.

[0034] In this embodiment, a visual mind mapping tool is used to transform business requirement data into visual test cases. Combined with preset SQL structure rules, feature transformation and structured data processing are completed to construct a precise initial test case set. Further semantic segmentation is used to obtain a keyword set, and multiple similarity verifications are used to establish a highly reliable keyword-field mapping relationship set. This set is then expanded and reorganized to generate a large-scale, high-coverage, and semantically consistent extended test case set. This embodiment integrates test case design, generation, and verification into a closed-loop process, significantly improving the efficiency, standardization, and maintainability of test case design for NL2SQL models. Finally, by inputting the extended test case set into the NL2SQL model to be tested, automated comparison testing of the generated SQL statement set is performed, accurately evaluating the accuracy and robustness of the NL2SQL model in real-world business scenarios, providing a reliable basis for the iterative optimization and quality assurance of the NL2SQL model.

[0035] In one embodiment, a normalized testing method for NL2SQL models is provided, comprising the following steps: Obtain business requirement data and preset SQL structure rules; Based on the aforementioned business requirement data and the SQL structure rules, corresponding test case files are designed and generated using the visual mind mapping tool XMind. It should be noted that the SQL structure rules summarize SQL statements into a structure of SELECT [query condition] FROM [table name] CONDITION, which can be further abstracted into a combination of a query header and a query condition. The XMind tool is used to transform the natural language-based business requirements into a structured test case design diagram according to these structure rules. The software development kit (SDK) of the XMind tool is used to parse the test case design diagram file and generate the corresponding JSON (JavaScript Object Notation, lightweight data interchange format) file. The JSON file is parsed to extract the corresponding natural language text use cases (titles) and expected SQL statements. A mapping relationship between use case names and SQL statements is established and written into an Excel file. Based on the EXCEL file, the openpyxl or pandas tools are used to read and process it to obtain the natural language text use cases and their corresponding expected SQL statements; further, by combining Word2Vec (word vector model) and Model Context Protocol (MCP) calls, the text use cases are semantically expanded to generate multiple semantically similar text use cases, and they are associated with the expected SQL statements of the source text use cases to form an extended test case set; Input the natural language text test cases from the extended test case set into the NL2SQL model to be tested, and obtain the SQL statement results returned by the model by calling the model's SDK or Application Programming Interface (API); The SQL statements generated by the model and the expected SQL statements are executed in the database of the system under test, and the consistency of the execution results is compared to determine the accuracy of the SQL generated by the NL2SQL model.

[0036] It should be noted that this embodiment combines business requirements with SQL structure rules through a visual mind mapping tool, achieving standardized design and structured transformation of test cases. By parsing the mind map file and using table files for intermediate storage, efficient extraction and management of test data are achieved. Furthermore, semantic-level expansion of text test cases is achieved through word vector models and protocol calls, generating large-scale, high-coverage, and semantically consistent extended test case sets, effectively enhancing the diversity of test scenarios. Finally, the model output is verified by comparing execution results, accurately evaluating the accuracy and robustness of the NL2SQL model in real-world business scenarios.

[0037] Furthermore, the standardized test case design method described in this embodiment is also applicable to complex business scenarios involving multi-table join queries.

[0038] Furthermore, the step of inputting the business requirement data into a visual mind mapping tool, so that the visual mind mapping tool generates visual test cases based on the business requirement data, includes: The business requirement data is input into a visual mind mapping tool so that the tool can construct several central theme nodes, each of which carries a natural language query text. Based on the central theme node, construct query field branches and query condition branches; Based on the query field branch, several first-level child nodes are created; wherein each first-level child node corresponds to a database table field to be queried; Based on the query condition branch, several second-level child nodes are created; wherein each second-level child node is used to define a query condition key-value pair, the key in the query condition key-value pair is a database table field, and the value is a query condition constraint; The visual test cases are generated based on the central theme node, query field branches, query condition branches, several first-level child nodes, and several second-level child nodes.

[0039] In this embodiment, a central topic node carries the natural language query text, expanding into query field branches and query condition branches. By mapping first-level child nodes to database table fields and defining query condition key-value pairs in second-level child nodes, the core components of SQL are clearly separated and visually defined, making complex query logic readily apparent and lowering the design threshold. Simultaneously, the direct mapping between nodes and database fields ensures the accuracy and standardization of test cases, effectively supporting the expression of complex query scenarios and providing a reliable input foundation for subsequent structured data processing and the generation of extended test case sets. This ensures the rigor and efficiency of the NL2SQL model testing process from the outset.

[0040] Furthermore, the step of performing feature transformation based on the visualized test cases and preset SQL structure rules to obtain structured data includes: Based on the preset SQL structure rules, the visual test cases are traversed to obtain the central theme node, query field branches, and query condition branches; Obtain the natural language query title based on the central theme node; Text extraction is performed based on the query field branches to obtain a list of query fields; Based on the query condition branches, key-value pairs are parsed to obtain the query condition set; The structured data is obtained based on the natural language query title, query field list, and query condition set.

[0041] In this embodiment, the process of traversing visual test cases based on preset SQL structure rules, extracting natural language query titles, parsing query field lists and query condition sets, and integrating them into unified structured data not only fully preserves the original query logic and semantic integrity but also eliminates human transcription errors through rule-based parsing, ensuring data format standardization and consistency. The resulting structured data provides high-quality, directly processable input for subsequent initial test case set construction and extended test case set generation, thereby effectively improving the automation, processing efficiency, and reliability of the entire testing process.

[0042] Furthermore, the step of obtaining an initial test case set based on the structured data and business requirement data includes: Obtain the business target database based on the aforementioned business requirement data; Based on the query field list, query conditions, business target database, and preset SQL structure rules, a standard SQL statement set is obtained; Natural language query text is obtained based on the structured data; The initial test case set is obtained by establishing a mapping relationship between the natural language query text and the standard SQL statement set.

[0043] In this embodiment, a standard SQL statement set is generated based on preset SQL structure rules, combined with a list of query fields, a set of query conditions, and a business target database. This set is then linked to natural language query text to establish a mapping relationship, constructing an initial test case set. This process directly ensures the accurate correspondence between natural language queries and standard SQL statements, eliminating errors and inefficiencies associated with manually written SQL, and forming standardized initial test cases. This embodiment provides an accurate and reliable input foundation for subsequent semantic segmentation, multiple similarity verification, and the generation of extended test case sets, thereby ensuring the quality and consistency of the starting point of the testing process.

[0044] Further, the semantic segmentation based on the initial test case set to obtain a keyword set includes: Obtain natural language query text based on the initial test case set; The natural language query text is segmented to obtain a sequence of semantic units; Based on the semantic unit sequence, semantic filtering is performed to obtain the core semantic unit sequence; A keyword set is obtained based on the core semantic unit sequence.

[0045] In this embodiment, the natural language query text is obtained based on an initial test case set. After word segmentation, a sequence of semantic units is obtained. Then, a semantic filter is used to extract the core semantic unit sequence to generate a precise keyword set. This process directly filters out auxiliary words and noise with no query value, achieving automated conversion from complex natural language to core semantic units and reducing the subjectivity and error of manual parsing. This embodiment ensures that the obtained keyword set accurately matches the true query intent, providing a clean and accurate input foundation for subsequent multiple similarity verifications and the construction of keyword-field mapping relationships, thereby guaranteeing the reliability and consistency of the testing method at the semantic understanding level.

[0046] Furthermore, the step of performing multiple similarity verifications based on the keyword set and business requirement data to establish a keyword-field mapping relationship set includes: Data metadata is obtained from the target database based on the business requirement data, and the data metadata includes several field description texts; For each keyword in the keyword set, perform the following operations: Based on the keyword and the descriptive text of several fields, semantic similarity processing is performed to obtain an initial semantic similarity set; Based on the initial similarity set and the preset first threshold, obtain the initial candidate field set corresponding to the keyword; Based on the initial candidate field set and the preset semantic disambiguation algorithm, semantic disambiguation is performed to obtain a weighted candidate field set; Based on the weighted candidate field set, a target mapping field is obtained so that a mapping relationship is established between the keyword and the target mapping field; Obtain a set of keyword-field mapping relationships based on the mapping relationships corresponding to several keywords.

[0047] In this embodiment, data metadata is obtained from a target database based on business requirements data. Semantic similarity processing is performed on each keyword in the keyword set and the field description text. An initial candidate field set is then selected using a preset first threshold. A weighted candidate field set is obtained through a preset semantic disambiguation algorithm to establish a keyword-field mapping relationship set. This process directly and effectively distinguishes semantically similar but functionally different fields, eliminating the uncertainty of single-round matching, significantly improving the uniqueness and accuracy of the mapping relationship, and ensuring the syntactic and semantic correctness of subsequent SQL generation. This embodiment provides a reliable structured foundation for test case expansion and reorganization, thereby ensuring the rigor and automation level of the testing process at the semantic understanding level.

[0048] Further, the semantic disambiguation is performed based on the initial candidate field set and the preset semantic disambiguation algorithm to obtain a weighted candidate field set. The semantic disambiguation algorithm specifically includes: Perform semantic similarity acquisition processing on any two initial candidate fields in the initial candidate field set to obtain a semantic homogeneity matrix; Based on the semantic homogeneity matrix and the preset clustering homogeneity algorithm, homogeneity evaluation is performed to obtain the semantic homogeneity evaluation result; If the semantic homogeneity assessment result is less than a preset second threshold, then the following steps are performed: For each keyword in the keyword set, a weighted semantic similarity processing is performed based on the keyword and several field description texts to obtain a weighted semantic similarity set; The weighted candidate field set is constructed based on the weighted semantic similarity set and the corresponding descriptive text of several fields.

[0049] In this embodiment, the semantic disambiguation algorithm first calculates a semantic homogeneity matrix for the initial candidate field set, then performs semantic homogeneity evaluation using a preset clustering homogeneity algorithm. If the evaluation result is less than a preset second threshold, weighted semantic similarity processing is performed on the keywords and field description text to construct a weighted candidate field set. This process directly and effectively distinguishes database fields with similar meanings but different functions, overcoming the limitations of traditional single-threshold matching. This embodiment provides a reliable decision-making basis for accurately determining the target mapping field, thereby ensuring the robustness and high accuracy of the semantic parsing stage in the NL2SQL model testing process.

[0050] Furthermore, the step of performing homogeneity evaluation based on the semantic homogeneity matrix and a preset clustering homogeneity algorithm to obtain semantic homogeneity evaluation results also includes: If the semantic homogeneity evaluation result is greater than or equal to a preset second threshold, then the following steps are performed: Obtain the target database table structure information based on the aforementioned business requirement data; Based on the initial candidate field set, obtain the database table and data type corresponding to each candidate field; For each keyword in the keyword set, a structured filtering is performed based on the database table corresponding to each candidate field, the data type corresponding to each candidate field, the target database table structure information, and the keyword to obtain the target mapping field corresponding to the keyword. The weighted candidate field set is constructed based on the target mapping fields corresponding to several keywords.

[0051] In this embodiment, semantic homogeneity assessment is performed based on a semantic homogeneity matrix and a preset clustering homogeneity algorithm. The semantic homogeneity matrix quantifies the semantic similarity between any two initial candidate fields, while the clustering homogeneity algorithm comprehensively judges the overall semantic consistency of the fields. The magnitude of the assessment result is positively correlated with the degree of semantic convergence between the initial candidate fields. When the assessment result is greater than or equal to a preset second threshold, it indicates that the semantic differences between the initial candidate fields are minimal, falling into the category of highly semantically consistent fields. At this point, the approach switches to using the target database table structure information, the database tables corresponding to the candidate fields, and their data types for structured filtering, avoiding the limitations of simple semantic calculations and ensuring the accuracy and reliability of the mapping decision.

[0052] In one embodiment, a normalized testing method for NL2SQL models is provided, comprising the following steps: Obtain sample natural language queries to be converted; Based on the natural language query example samples, semantic segmentation is performed to obtain a keyword set; Based on the keyword set and the metadata of the database, multiple similarity verifications are performed to establish a keyword-field mapping relationship set.

[0053] Specifically, the semantic segmentation based on the natural language query example sample to obtain the keyword set includes: Input the natural language query example into the jieba word segmenter (a Chinese word segmentation tool); It should be noted that the word segmenter uses its built-in word segmentation algorithm to split natural language text into semantic units; from the split semantic unit sequence, the core semantic units that express the database query intent, such as "user name" and "order amount", are extracted; Based on the extracted core semantic units, a set of keywords corresponding to the example sample is generated.

[0054] Specifically, the step of performing multiple similarity verifications based on the keyword set and the metadata of the database to establish a keyword-field mapping relationship set includes: The descriptive text of each field is obtained based on the metadata of the database; For each keyword in the keyword set, perform the following double association operation: First-level association: Calculate the semantic similarity of the text based on the word vector of the keyword and the word vector of the description text of each field's remarks; It should be noted that the text similarity calculation uses a cosine similarity algorithm based on word vectors. Let the word vector corresponding to the keyword be... The word vectors corresponding to the database field remarks information are Then the formula for calculating cosine similarity is: The cosine similarity The closer to 1, the more similar the meaning.

[0055] A preset first similarity threshold μ (e.g., 0.7) is set when... When μ is greater than or equal to μ, a preliminary mapping relationship between the keyword and the corresponding field is established, forming an initial candidate field set.

[0056] The second layer of association (semantic disambiguation): Homogeneity evaluation and verification are performed on the initial candidate field set. This step further includes: Construct a vector set of candidate field annotation information: Extract the annotation description text of each candidate field in the initial candidate field set and convert it into the corresponding word vector to form a vector set. , where t is the number of candidate fields; Calculate the semantic homogeneity matrix: Calculate the pairwise similarity between the above candidate field memo information vectors to form a similarity matrix M: in, For vectors and The cosine similarity.

[0057] Homogeneity assessment: Based on the semantic homogeneity matrix M, a clustering homogeneity index H is calculated to quantify the overall semantic similarity of candidate field annotation information. The calculation formula is as follows: . The value range is [0,1]. The closer the value is to 1, the more similar the semantics are between the candidate fields. The closer the value is to 0, the greater the difference is.

[0058] Ambiguity determination and screening: If the homogeneity index H is less than the preset second threshold θ (e.g., 0.7), then the candidate fields are determined to have significant functional differences (heterogeneous case). In this case, a second-order weighted similarity calculation of keywords and notes is performed on each candidate field: in, The weighting coefficient is determined by the frequency of this field's occurrence in historical queries, and the calculation formula is as follows: α is the adjustment coefficient (usually taken as 0.5). Let be the query frequency of the i-th field. Based on the quadratic weighted similarity results, interference items are removed, and the target mapping fields are selected.

[0059] If the homogeneity index H is greater than or equal to a preset second threshold θ, the candidate fields are determined to be highly homogeneous in semantics. At this point, further structured filtering is performed using database table structure information (such as the table name and data type of the field) to determine the final target mapping field from highly similar candidate fields.

[0060] Based on the target mapping field finally determined for each keyword in the keyword set, establish the keyword-field mapping relationship set.

[0061] It should be noted that this embodiment significantly reduces manual intervention by introducing an automated keyword decomposition and dual association verification process. The first layer of association achieves preliminary mapping based on word vector similarity, while the second layer of association distinguishes between "heterogeneous" and "highly homogeneous" scenarios through homogeneity assessment, and performs precise disambiguation using either secondary weighted similarity calculation or a combination of table structure information. This embodiment effectively improves the uniqueness and accuracy of the mapping relationship between keywords and database fields, providing a reliable core technical guarantee for generating high-quality, diverse NL2SQL test datasets.

[0062] For example, taking the keyword "regional output value" as an example, after the first layer of association, three initial candidate fields are obtained: I1: Field name "regional_gdp", remarks "the final result of the production activities of all resident units in a certain region within a certain period (regional GDP), numeric type"; I2: Field name "area_industry_output", remarks "total output value of industrial sectors within a specific area, numeric type"; I3: Field name "regional_asset_value", remarks "Total value assessment of all assets in a certain region, numerical type".

[0063] The pairwise cosine similarity between the candidate field annotation information vectors is calculated to form matrix M. The homogeneity index H is calculated to be approximately 0.517, which is less than the preset second threshold θ (0.7), thus entering the heterogeneity processing flow. By performing a weighted quadratic similarity calculation on each candidate field, candidate fields with relatively low semantic relevance (such as I3) can be effectively eliminated, thereby accurately identifying the target database field that best matches "regional output value".

[0064] In another implementation, for each keyword in the keyword set, weighted semantic similarity processing is performed based on the keyword and several field description texts to obtain a weighted semantic similarity set, including: For each candidate field in the initial candidate field set, obtain its historical query frequency; Calculate the weight coefficient corresponding to the candidate field based on the historical query frequency; The initial semantic similarity between the candidate field and the keyword is calculated by weighting the weight coefficients to obtain the weighted semantic similarity.

[0065] It should be noted that the weighting coefficients The calculation formula is: in, This refers to the historical query frequency of this candidate field. The maximum historical query frequency among all initial candidate fields is α, which is a preset adjustment coefficient (usually 0.5).

[0066] Specifically, obtaining the target mapping field based on the weighted candidate field set includes: Based on the calculated weighted semantic similarity of each candidate field, the maximum value is determined; Based on the preset secondary similarity filtering rules, candidate fields that meet the conditions are filtered from the weighted candidate field set; If a candidate field that meets the criteria is unique, then it is determined as the target mapping field for that keyword. If the candidate fields that meet the conditions are not unique, the final target mapping field will be determined according to the preset priority rules (such as giving priority to fields with higher weighted semantic similarity).

[0067] It should be noted that the secondary similarity filtering rule can be set as follows: retain candidate fields whose weighted semantic similarity is not lower than a certain percentage (e.g., 60%) of the highest weighted semantic similarity. That is, set a secondary similarity filtering threshold. ,in, This is a proportionality coefficient (e.g., 0.6). The maximum value among the weighted semantic similarities of all candidate fields is retained. Candidate fields are selected for final evaluation. The aforementioned secondary similarity filtering rule aims to exclude interference fields that, although passing the initial threshold screening, are significantly lower than the optimal option after weighted comparison.

[0068] For example, continuing with the previous keyword "regional output value" and its three initial candidate fields I1, I2, and I3. Assume their historical query frequencies are known as: I1 (high frequency, 2000 times), I2 (medium frequency, 1000 times), and I3 (low frequency, 500 times), with an adjustment coefficient α = 0.5. The weights are then calculated as follows: Based on the initial semantic similarity obtained from the first layer of association (assuming they are 0.95, 0.8, and 0.7 respectively), calculate the weighted second-order similarity: Assuming the proportionality coefficient β = 0.6, then the secondary similarity screening threshold is... After comparison, Therefore, field I3 was removed. Among the remaining fields I1 and I2, I1 had the highest weighted similarity, so I1 was ultimately determined as the target mapping field for the keyword "regional output value".

[0069] In this embodiment, by introducing historical query frequency as a weighting factor and setting secondary similarity filtering rules, candidate fields in heterogeneous scenarios are more finely differentiated. This embodiment not only considers the static semantic association between keywords and fields but also dynamically incorporates the historical preferences of business queries, making the final keyword-field mapping relationship closer to actual application scenarios and further improving the accuracy and practicality of the mapping relationship.

[0070] Furthermore, the expansion and reorganization based on the keyword-field mapping relationship and the initial test case set to obtain an expanded test case set includes: Obtain a natural language text set based on the initial test case set; For each natural language text in the natural language text set, a synonym replacement operation is performed based on the keyword-field mapping relationship and the natural language text to obtain the first variant query text; The natural language text is input into a preset semantic expansion model, so that the semantic expansion model performs semantic expansion based on the natural language text to obtain a second variant query text. Based on the first variant query text and the second variant query text, obtain the variant query text set corresponding to the natural language text; For each variant query text in the variant query text set, an SQL statement is constructed based on the keyword-field mapping relationship, the preset SQL structure rules and the variant query text to obtain the standard SQL statement corresponding to the variant query text. Based on several variant query texts and corresponding standard SQL statements, obtain an extended test sample set corresponding to the natural language text; The extended test case set is obtained based on an extended test sample set corresponding to several natural language texts.

[0071] In this embodiment, a dual process is employed: generating the first variant query text through synonym substitution based on the natural language text set of the initial test case set, relying on keyword-field mapping relationships; and simultaneously generating the second variant query text through a semantic expansion model. This directly ensures the diversity and semantic equivalence of the test cases. Furthermore, by generating standard SQL statements from all variant texts according to a unified mapping relationship and preset SQL structure rules, semantic biases and syntax errors inherent in manual expansion are effectively avoided, ensuring the accuracy of the expanded test cases. This embodiment achieves automated production from seed test cases to large-scale expanded test cases, providing the NL2SQL model with broadly covered and accurately labeled test data, significantly enhancing the evaluation effectiveness of the model's robustness and generalization ability.

[0072] In one embodiment, the process of expanding and reorganizing the test case set based on the keyword-field mapping relationship and the initial test case set to obtain an expanded test case set includes the following steps: Obtain a natural language text set based on the initial test case set; For each natural language text in the natural language text set, perform semantic expansion and restructuring and structured SQL generation operations, specifically including: Based on the keyword-field mapping relationship and the natural language text, synonym replacement and recombination operations are performed to obtain the first variant query text; It should be noted that this operation is achieved by replacing keywords with synonyms or similar words that have established mapping relationships in the natural language text. For example, "amount" is replaced with "fee" or "price" while maintaining the basic structure of the original sentence, thereby generating a new query text with similar semantics.

[0073] The natural language text is input into a preset semantic expansion model to obtain the second variant query text; It should be noted that the semantic expansion model can be implemented by calling the Model Context Protocol (MCP) interface service. This service can perform sentence-level semantic expansion on the input natural language text, generating multiple texts that are semantically equivalent to the original text but have different expressions, thereby expanding the semantic coverage dimension of the sample.

[0074] Based on the first variant query text and the second variant query text, construct a variant query text set corresponding to the natural language text.

[0075] For each variant query text in the variant query text set, an SQL statement is constructed based on the keyword-field mapping relationship, the preset SQL structure rules and the variant query text to obtain the standard SQL statement corresponding to the variant query text. It should be noted that the construction process is as follows: based on the preset SQL syntax structure rules, the SQL statement is decomposed into two core modules: the query header and the query conditions; the database table field names associated with the keyword-field mapping relationship are filled into the header module of the SQL statement; combined with the query intent of the variant query text, the synonym relationship between the expanded sample and the original example is mapped to the conditions module of the SQL statement, and the corresponding condition logic is supplemented; finally, a standard SQL statement conforming to the syntax specification is assembled.

[0076] Based on all variant query texts corresponding to the natural language text and their respective generated standard SQL statements, an extended test sample set corresponding to the natural language text is constructed.

[0077] The extended test case set is obtained by summarizing the extended test sample set corresponding to all natural language texts.

[0078] In this embodiment, by combining rule-based replacement based on keyword mapping and intelligent expansion based on semantic models, the generated natural language variant text is effectively guaranteed to possess both rich diversity and semantic consistency with the original intent. Furthermore, a unified and structured SQL construction process ensures that each variant text generates syntactically correct and semantically matched standard SQL statements. This embodiment achieves automated generation from a small number of seed test cases to a large-scale, high-quality extended test case set, providing a reliable data foundation for comprehensively evaluating the accuracy and robustness of the NL2SQL model.

[0079] Please see Figure 2 This embodiment also provides a standardized testing system for the NL2SQL model, including: The business requirement acquisition module is used to acquire business requirement data; The visual test case generation module is used to input the business requirement data into the visual mind mapping tool, so that the visual mind mapping tool can generate visual test cases based on the business requirement data. The feature conversion module is used to perform feature conversion based on the visualized test cases and preset SQL structure rules to obtain structured data; The initial test case generation module is used to obtain an initial test case set based on the structured data and business requirement data. An extended test case generation module is used to perform semantic segmentation based on the initial test case set to obtain a keyword set; perform multiple similarity verifications based on the keyword set and business requirement data to establish a keyword-field mapping relationship; and expand and reorganize the keyword-field mapping relationship and the initial test case set to obtain an extended test set. The model testing module is used to input the extended test set into the NL2SQL model to be tested, so that the NL2SQL model generates a set of SQL statements based on the extended test set; and to perform normalization tests based on the set of SQL statements and the extended test set to obtain the model performance evaluation results.

[0080] In this embodiment, modules for business requirement acquisition, visual test case generation, feature conversion, initial test case generation, extended test case generation, and model testing are integrated. These modules operate collaboratively based on unified rules and data flow. Through a closed-loop process from business requirements to visual test cases, structured data, and initial and extended test case sets, the scattered manual testing steps are transformed into automated pipeline operations, significantly improving testing efficiency and standardization. Simultaneously, relying on visual design, structured parsing, and semantic expansion technologies, comprehensive semantic coverage of test cases and accurate expected results are ensured, providing a reliable quality assessment method for the NL2SQL model and strongly supporting model performance optimization and iterative deployment.

[0081] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A standardized testing method for NL2SQL models, characterized in that, Includes the following steps: Obtain business requirement data; The business requirement data is input into a visual mind mapping tool, which then generates visual test cases based on the business requirement data. Based on the visualized test cases and preset SQL structure rules, feature transformation is performed to obtain structured data; An initial test case set is obtained based on the structured data and business requirement data; Based on the initial test case set, semantic segmentation is performed to obtain a keyword set; Based on the keyword set and business requirement data, multiple similarity verifications are performed to establish a keyword-field mapping relationship; Based on the keyword-field mapping relationship and the initial test case set, the test case set is expanded and reorganized to obtain an extended test case set; The extended test case set is input into the NL2SQL model to be tested, so that the NL2SQL model generates a set of SQL statements based on the extended test case set; Based on the SQL statement set and extended test case set, standardized testing was performed to obtain the model performance evaluation results.

2. The standardized testing method for NL2SQL models according to claim 1, characterized in that, The step of inputting the business requirement data into a visual mind mapping tool, so that the visual mind mapping tool generates visual test cases based on the business requirement data, includes: The business requirement data is input into a visual mind mapping tool so that the tool can construct several central theme nodes, each of which carries a natural language query text. Based on the central theme node, construct query field branches and query condition branches; Based on the query field branch, several first-level child nodes are created; wherein each first-level child node corresponds to a database table field to be queried; Based on the query condition branch, several second-level child nodes are created; wherein each second-level child node is used to define a query condition key-value pair, the key in the query condition key-value pair is a database table field, and the value is a query condition constraint; The visual test cases are generated based on the central theme node, query field branches, query condition branches, several first-level child nodes, and several second-level child nodes.

3. The standardized testing method for NL2SQL models according to claim 1, characterized in that, The step of performing feature transformation based on the visualized test cases and preset SQL structure rules to obtain structured data includes: Based on the preset SQL structure rules, the visual test cases are traversed to obtain the central theme node, query field branches, and query condition branches; Obtain the natural language query title based on the central theme node; Text extraction is performed based on the query field branches to obtain a list of query fields; Based on the query condition branches, key-value pairs are parsed to obtain the query condition set; The structured data is obtained based on the natural language query title, query field list, and query condition set.

4. The standardized testing method for NL2SQL models according to claim 3, characterized in that, The process of obtaining an initial test case set based on the structured data and business requirement data includes: Obtain the business target database based on the aforementioned business requirement data; Based on the query field list, query conditions, business target database, and preset SQL structure rules, a standard SQL statement set is obtained; Natural language query text is obtained based on the structured data; The initial test case set is obtained by establishing a mapping relationship between the natural language query text and the standard SQL statement set.

5. The standardized testing method for NL2SQL models according to claim 1, characterized in that, The semantic segmentation based on the initial test case set to obtain a keyword set includes: Obtain natural language query text based on the initial test case set; The natural language query text is segmented to obtain a sequence of semantic units; Based on the semantic unit sequence, semantic filtering is performed to obtain the core semantic unit sequence; A keyword set is obtained based on the core semantic unit sequence.

6. The standardized testing method for NL2SQL models according to claim 4, characterized in that, The step involves performing multiple similarity verifications based on the keyword set and business requirement data to establish a keyword-field mapping relationship set; including: Data metadata is obtained from the target database based on the business requirement data, and the data metadata includes several field description texts; For each keyword in the keyword set, perform the following operations: Based on the keyword and the descriptive text of several fields, semantic similarity processing is performed to obtain an initial semantic similarity set; Based on the initial similarity set and the preset first threshold, obtain the initial candidate field set corresponding to the keyword; Based on the initial candidate field set and the preset semantic disambiguation algorithm, semantic disambiguation is performed to obtain a weighted candidate field set; Based on the weighted candidate field set, a target mapping field is obtained so that a mapping relationship is established between the keyword and the target mapping field; Obtain a set of keyword-field mapping relationships based on the mapping relationships corresponding to several keywords.

7. A standardized testing method for NL2SQL models according to claim 6, characterized in that, The semantic disambiguation is performed based on the initial candidate field set and a preset semantic disambiguation algorithm to obtain a weighted candidate field set. The semantic disambiguation algorithm specifically includes: Perform semantic similarity acquisition processing on any two initial candidate fields in the initial candidate field set to obtain a semantic homogeneity matrix; Based on the semantic homogeneity matrix and the preset clustering homogeneity algorithm, homogeneity evaluation is performed to obtain the semantic homogeneity evaluation result; If the semantic homogeneity assessment result is less than a preset second threshold, then the following steps are performed: For each keyword in the keyword set, a weighted semantic similarity processing is performed based on the keyword and several field description texts to obtain a weighted semantic similarity set; The weighted candidate field set is constructed based on the weighted semantic similarity set and the corresponding descriptive text of several fields.

8. A standardized testing method for NL2SQL models according to claim 7, characterized in that, The process of evaluating homogeneity based on the semantic homogeneity matrix and a preset clustering homogeneity algorithm to obtain semantic homogeneity evaluation results also includes: If the semantic homogeneity evaluation result is greater than or equal to a preset second threshold, then the following steps are performed: Obtain the target database table structure information based on the aforementioned business requirement data; Based on the initial candidate field set, obtain the database table and data type corresponding to each candidate field; For each keyword in the keyword set, a structured filtering is performed based on the database table corresponding to each candidate field, the data type corresponding to each candidate field, the target database table structure information, and the keyword to obtain the target mapping field corresponding to the keyword. The weighted candidate field set is constructed based on the target mapping fields corresponding to several keywords.

9. A standardized testing method for NL2SQL models according to claim 1, characterized in that, The process of expanding and reorganizing the test case set based on the keyword-field mapping relationship and the initial test case set to obtain an expanded test case set includes: Obtain a natural language text set based on the initial test case set; For each natural language text in the natural language text set, a synonym replacement operation is performed based on the keyword-field mapping relationship and the natural language text to obtain the first variant query text; The natural language text is input into a preset semantic expansion model, so that the semantic expansion model performs semantic expansion based on the natural language text to obtain a second variant query text. Based on the first variant query text and the second variant query text, obtain the variant query text set corresponding to the natural language text; For each variant query text in the variant query text set, an SQL statement is constructed based on the keyword-field mapping relationship, the preset SQL structure rules and the variant query text to obtain the standard SQL statement corresponding to the variant query text. Based on several variant query texts and corresponding standard SQL statements, obtain an extended test sample set corresponding to the natural language text; The extended test case set is obtained based on an extended test sample set corresponding to several natural language texts.

10. A standardized testing system for NL2SQL models, characterized in that, include: The business requirement acquisition module is used to acquire business requirement data; The visual test case generation module is used to input the business requirement data into the visual mind mapping tool, so that the visual mind mapping tool can generate visual test cases based on the business requirement data. The feature conversion module is used to perform feature conversion based on the visualized test cases and preset SQL structure rules to obtain structured data; The initial test case generation module is used to obtain an initial test case set based on the structured data and business requirement data. An extended test case generation module is used to perform semantic segmentation based on the initial test case set to obtain a keyword set; perform multiple similarity verifications based on the keyword set and business requirement data to establish a keyword-field mapping relationship; and expand and reorganize the keyword-field mapping relationship and the initial test case set to obtain an extended test set. The model testing module is used to input the extended test set into the NL2SQL model to be tested, so that the NL2SQL model generates a set of SQL statements based on the extended test set; Based on the SQL statement set and extended test set, normalization tests are performed to obtain the model performance evaluation results.

Images