A data processing method and device, electronic equipment and storage medium

Abstract

A data processing method and device, electronic equipment and storage medium are disclosed. The method comprises: obtaining a problem to be processed, performing semantic recognition on the problem to be processed, determining entity information and multi-dimensional label information of the problem to be processed, the multi-dimensional label information being used to represent multi-dimensional attributes of the problem to be processed, the multi-dimensional attributes of the problem to be processed including at least one of a business field, a data type and data hierarchical information; matching in a preset vector database based on the problem to be processed, the entity information and the multi-dimensional label information, to determine a plurality of candidate database schema information, the candidate database schema information being used to represent business attributes of a to-be-queried object; determining target database schema information based on the problem to be processed and the plurality of candidate database schema information; and determining a query statement based on the target database schema information and the problem to be processed, thereby achieving accurate generation of the query statement.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a data processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] Structured query statements are widely used in data querying, analysis, and decision support, and are an important tool for interacting with databases.

[0003] In existing technologies, the Text-to-SQL method is commonly used to automatically convert user-input natural language text into executable structured query statements. However, in practical applications on real-world data platforms, the Text-to-SQL solution suffers from low accuracy because database table structures often use Chinese Pinyin abbreviations or meaningless encoding names, and data definition language statements lack necessary comments. This is because the structured query statements cannot accurately parse the positional relationship between the natural language text and the target data in the database. Summary of the Invention

[0004] This invention provides a data processing method, apparatus, electronic device, and storage medium to achieve accurate generation of query statements.

[0005] According to one aspect of the present invention, a data processing method is provided, comprising: The process involves acquiring a problem to be processed, performing semantic recognition on the problem, and determining the entity information and multi-dimensional label information of the problem. The multi-dimensional label information is used to characterize the multi-dimensional attributes of the problem to be processed, which include at least one of the following: business domain, data type, and data layering information. Based on the problem to be processed, entity information, and multi-dimensional label information, the system performs matching in a pre-set vector database to determine multiple candidate database pattern information. The candidate database pattern information is used to characterize the business attributes of the object to be queried. The target database schema information is determined based on the problem to be addressed and multiple candidate database schema information. The query statement is determined based on the target database schema information and the problem to be addressed.

[0006] According to another aspect of the present invention, a data processing apparatus is provided, comprising: The semantic recognition module is used to acquire the problem to be processed, perform semantic recognition on the problem to be processed, and determine the entity information and multi-dimensional label information of the problem to be processed. The multi-dimensional label information is used to characterize the multi-dimensional attributes of the problem to be processed. The multi-dimensional attributes of the problem to be processed include at least one of business domain, data type and data layering information. The candidate database pattern information determination module is used to match the problem to be processed, entity information and multi-dimensional label information in a preset vector database to determine multiple candidate database pattern information. The candidate database pattern information is used to characterize the business attributes of the object to be queried. The target database schema information determination module is used to determine the target database schema information based on the problem to be processed and multiple candidate database schema information. The query statement determination module is used to determine the query statement based on the target database schema information and the problem to be processed.

[0007] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: At least one processor; and A memory that is communicatively connected to at least one processor; wherein, The memory stores a computer program that can be executed by at least one processor, such that the at least one processor can perform the data processing method of any embodiment of the present invention.

[0008] According to another aspect of the present invention, a computer-readable storage medium is provided, which stores computer instructions for causing a processor to execute and implement the data processing method of any embodiment of the present invention.

[0009] The technical solution of this invention obtains the problem to be processed, performs semantic recognition on the problem to be processed, and determines the entity information and multi-dimensional label information of the problem to be processed. The multi-dimensional label information is used to characterize the multi-dimensional attributes of the problem to be processed. The multi-dimensional attributes of the problem to be processed include at least one of business domain, data type, and data layering information, providing comprehensive data support for subsequent analysis and processing. Based on the problem to be processed, entity information, and multi-dimensional label information, a matching is performed in a preset vector database to determine multiple candidate database pattern information. The candidate database pattern information is used to characterize the business attributes of the object to be queried, realizing the filtering of candidate database pattern information, which can effectively reduce the amount of computation in subsequent processing, thereby improving processing efficiency and accuracy. Based on the problem to be processed and multiple candidate database pattern information, the target database pattern information is determined, providing accurate and reliable data support for subsequent analysis and processing. Based on the target database pattern information and the problem to be processed, the query statement is determined, which solves the problem of low accuracy of query statements in the prior art, making the query statement fit the database table, thereby improving the accuracy and usability of the query statement and providing an accurate data foundation for subsequent data queries.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a flowchart of a data processing method provided in an embodiment of the present invention; Figure 2 This is a flowchart of another data processing method provided in an embodiment of the present invention; Figure 3 This is a flowchart of another data processing method provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a data processing device provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0013] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0014] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0015] Figure 1 This is a flowchart of a data processing method provided in an embodiment of the present invention. This embodiment is applicable to the generation of query statements. The method can be executed by a data processing device, which can be implemented in hardware and / or software. This data processing device can be configured in the electronic device provided in this embodiment of the present invention. The electronic device can be a server, computer, or mobile terminal, such as a mobile phone or tablet computer. Figure 1 As shown, the method specifically includes the following steps: S110. Obtain the problem to be processed, perform semantic recognition on the problem to be processed, and determine the entity information and multi-dimensional label information of the problem to be processed. The multi-dimensional label information is used to characterize the multi-dimensional attributes of the problem to be processed. The multi-dimensional attributes of the problem to be processed include at least one of business domain, data type and data layering information.

[0016] The application scenarios of this invention include, but are not limited to, medical and financial scenarios. For example, the specific application scenario of this invention can be an outpatient data query scenario. The query to be processed is a natural language query. The query to be processed can be obtained through an interactive device. For example, a user inputs a query request in natural language form through the text input interface of an interactive device, using this natural language query request as the query to be processed. The user can input this request through physical input devices and virtual input devices. Physical input devices include, but are not limited to, physical mice and physical keyboards, while virtual input devices include, but are not limited to, virtual keyboards. Semantic recognition of the query to be processed can be achieved through a semantic recognition model. For example, the query to be processed is input into a trained semantic recognition model for semantic recognition, obtaining entity information and multi-dimensional label information of the query to be processed. The semantic recognition model includes, but is not limited to, neural network models; for example, the semantic recognition model can be a large language model. The entity information refers to the business entities in the query to be processed, including, but not limited to, the query object. The multi-dimensional label information is a classification identifier for the query to be processed, used to characterize the multi-dimensional attributes of the query to be processed. Multi-dimensional attributes include at least one of business domain, data type, and data layering information. The business domain includes, but is not limited to, the financial domain, the medical domain, the e-commerce domain, and the manufacturing domain. The data type includes, but is not limited to, transaction detail data and transaction summary data. Data layering information includes, but is not limited to, raw data layer, detailed data layer, summary data layer, and application data layer.

[0017] Specifically, users input natural language queries through the text input interface of the interactive device using a physical keyboard, treating these queries as questions to be processed. The questions are then input into a trained semantic recognition model for semantic recognition, yielding entity information and multi-dimensional label information for the questions, providing comprehensive data support for subsequent analysis and processing.

[0018] S120. Based on the problem to be processed, entity information and multi-dimensional label information, match them in the preset vector database to determine multiple candidate database pattern information. The candidate database pattern information is used to characterize the business attributes of the object to be queried.

[0019] The vector database is a storage structure for vectorized descriptions of pre-built data models. Optionally, the vector database includes multiple database schema information sets; these schema sets include first-level, second-level, and third-level vectorized descriptions, with the first-level descriptions having a lower granularity than the second-level, and the second-level descriptions having a lower granularity than the third-level. The database schema information represents the business attributes of the business data. These attributes include, but are not limited to, business domain, data type, and data layering information. The first-level vectorized descriptions characterize the overall business overview of the business data, including but not limited to vector features corresponding to the business data name and attributes. The second-level vectorized descriptions characterize the data structure and business purpose of the business data, including but not limited to vector features corresponding to the names and annotations of key fields. The third-level vectorized descriptions characterize the structural attributes and data rules of the business data, including but not limited to feature vectors corresponding to field content, field type, and primary / foreign key relationships. It should be noted that the fineness of first-level vectorized description information is lower than that of second-level vectorized description information, and the fineness of second-level vectorized description information is lower than that of third-level vectorized description information.

[0020] Candidate database schema information consists of data representing the table structure, field information, and corresponding business attributes of the object to be queried. Business attributes include at least one of the following: business domain, data type, and data hierarchy information. Candidate database schema information can be determined based on the problem to be processed, entity information, and multi-dimensional label information. For example, the problem to be processed, entity information, and multi-dimensional label information can be converted into vectors. A similarity search can be performed in a pre-defined vector database, and database schema information with similarity scores greater than or equal to a similarity threshold can be selected as candidate database schema information.

[0021] Specifically, the problem to be processed, entity information, and multi-dimensional label information are converted into vectors. Similarity is searched in a preset vector database, and database pattern information corresponding to similarities greater than or equal to a similarity threshold is selected as candidate database pattern information. This process effectively reduces the computational load of subsequent processing, thereby improving processing efficiency and accuracy.

[0022] Optionally, the construction process of the vector database is as follows: Obtain the metadata information of the database tables. The metadata information is used to characterize the multi-dimensional attributes of the database tables, including at least one of the following: business domain, data type, and data layering information; perform fine-grained hierarchical classification on the metadata information to obtain first-level, second-level, and third-level descriptive information; perform structured processing on the first-level, second-level, and third-level descriptive information to obtain first-level vectorized descriptive information, second-level vectorized descriptive information, and third-level vectorized descriptive information; construct the vector database based on the first-level, second-level, and third-level vectorized descriptive information.

[0023] In this context, database tables are the structures for storing business data. Metadata information describes the attributes of database tables, characterizing their multi-dimensional attributes. These multi-dimensional attributes include at least one of the following: business domain, data type, and data hierarchy information. For example, metadata information may include the database table's name, comments, type, and business domain. The metadata information of database tables can be obtained from a metadata information database. For instance, a preset number of database tables' metadata information can be randomly selected from the metadata information database, which stores metadata information for multiple database tables.

[0024] Fine-grained hierarchical classification of metadata information can be achieved through a fine-grained hierarchical model. For example, metadata information can be input into a trained fine-grained hierarchical model to obtain first-level, second-level, and third-level descriptive information. The fine-grained hierarchical model includes, but is not limited to, neural network models. The choice of model is based on requirements, and this invention does not impose any limitations. First-level descriptive information characterizes the overall business overview of the business data in the database table, including but not limited to the table name and business attributes. Second-level descriptive information characterizes the data structure and business purpose of the business data in the database table, including but not limited to the names of key fields and their corresponding comments. Third-level descriptive information characterizes the structural attributes and data rules of the business data in the database table, including but not limited to field content, field types, and primary / foreign key relationships.

[0025] Level 1 vectorized description information is used to characterize the overall business overview of business data, including but not limited to vectorized business data names and vectorized business attributes. Level 2 vectorized description information is used to characterize the data structure and business purpose of business data, including but not limited to vectorized key field names and corresponding vectorized comments. Level 3 vectorized description information is used to characterize the structural attributes and data rules of business data, including but not limited to vectorized field content, vectorized field types, and vectorized primary and foreign key relationships. It should be noted that the granularity of Level 1 vectorized description information is lower than that of Level 2, and the granularity of Level 2 vectorized description information is lower than that of Level 3.

[0026] The structuring of primary, secondary, and tertiary descriptive information can be achieved through a structured processing model. For example, the primary, secondary, and tertiary descriptive information can be input into a trained structured processing model for structuring, resulting in primary, secondary, and tertiary vectorized descriptive information, respectively. The structured processing model includes, but is not limited to, neural network models; for example, it can be an embedding model. The choice of structured processing model is based on requirements, and this invention does not impose any limitations.

[0027] Vector databases can be constructed based on the first-level, second-level, and third-level vectorized description information corresponding to database tables. For example, the first-level, second-level, and third-level vectorized description information corresponding to multiple database tables can be integrated to construct a vector database.

[0028] Specifically, metadata information from a predetermined number of database tables is randomly selected from the metadata information database. This metadata information is then input into a trained fine-grained hierarchical model for fine-grained hierarchical classification, yielding first-level, second-level, and third-level descriptive information. These first-level, second-level, and third-level descriptive information are then input into a trained structured processing model for structured processing, yielding first-level, second-level, and third-level vectorized descriptive information. Finally, the first-level, second-level, and third-level vectorized descriptive information corresponding to multiple database tables are integrated to construct a vector database. This achieves hierarchical semantic storage of the database tables from coarse to fine, providing comprehensive and standardized data support for subsequent analysis and processing.

[0029] Optionally, the method further includes: performing semantic enhancement processing on the metadata information to obtain semantically enhanced metadata information; performing fine-grained hierarchical classification on the semantically enhanced metadata information to obtain first-level description information, second-level description information, and third-level description information; performing structuring processing on the first-level description information, second-level description information, and third-level description information to obtain structured vectors corresponding to the first-level description information, second-level description information, and third-level description information, respectively; and constructing a vector database based on the structured vectors.

[0030] Semantic enhancement of metadata information can be achieved through thesaurus and terminology database. For example, using a thesaurus, the thesaurus associated with the metadata information is obtained, the semantic similarity between the metadata information and multiple words in the thesaurus is calculated, and the semantic similarity is filtered based on a pre-set first semantic similarity threshold. Candidate words with semantic similarity greater than or equal to the first semantic similarity threshold are identified and added to the metadata information as supplementary descriptions, thus obtaining semantically enhanced metadata information. Similarly, using a terminology database, the terminology database associated with the metadata information is obtained, the semantic similarity between the metadata information and multiple terms in the terminology database is calculated, and the semantic similarity is filtered based on a pre-set second semantic similarity threshold. Candidate terms with semantic similarity greater than or equal to the second semantic similarity threshold are identified and added to the metadata information as supplementary descriptions, thus obtaining semantically enhanced metadata information. For example, metadata information is input into a large language model, and the prompt "Please generate business description information for the following metadata information" is entered. The business description information generated by the large language model is obtained. The business description information is used as semantically enhanced metadata information. The business description information represents the business domain, data type and data layer information corresponding to the metadata information, thereby realizing semantic enhancement of the metadata information.

[0031] The process of fine-grained grading of semantically enhanced metadata information to obtain first-level, second-level, and third-level descriptive information is as follows: the fine-grained grading model trained with semantically enhanced metadata information is used for fine-grained grading to obtain first-level, second-level, and third-level descriptive information. The fine-grained grading model includes, but is not limited to, neural network models. The fine-grained grading model is selected according to the requirements, and this invention does not impose any restrictions.

[0032] Specifically, the semantic similarity between metadata information and multiple words in the thesaurus is calculated. Based on a pre-set first semantic similarity threshold, the semantic similarity is filtered to determine candidate words with semantic similarity greater than or equal to the first threshold. The metadata information is then semantically expanded and enhanced based on these candidate words to obtain semantically enhanced metadata information. The semantically enhanced metadata information is then used to perform fine-grained hierarchical model training, resulting in first-level, second-level, and third-level descriptive information. These first-level, second-level, and third-level descriptive information are then input into a trained structured processing model for structured processing, resulting in first-level, second-level, and third-level vectorized descriptive information. Finally, the first-level, second-level, and third-level vectorized descriptive information corresponding to multiple database tables are integrated to construct a vector database. By semantically enhancing the metadata information, the semantic richness and expressive completeness of the metadata information are improved, thereby achieving hierarchical semantic storage of the database tables from coarse to fine, providing comprehensive and standardized data support for subsequent analysis and processing.

[0033] S130. Determine the target database schema information based on the problem to be processed and multiple candidate database schema information.

[0034] The target database schema information refers to the database schema information among the candidate database schema information that matches the problem to be processed. The target database schema information can be determined based on the problem to be processed and multiple candidate database schema information. For example, the semantic similarity between the problem to be processed and multiple candidate database schema information is calculated separately, and the candidate database schema information corresponding to a semantic similarity greater than or equal to a pre-set third semantic similarity threshold is used as the target database schema information. It should be noted that there can be one or multiple target database schema information.

[0035] Specifically, the semantic similarity between the problem to be processed and multiple candidate database pattern information is calculated separately. Candidate database pattern information with semantic similarity greater than or equal to a pre-set third semantic similarity threshold is used as target database pattern information, providing accurate and reliable data support for subsequent analysis and processing.

[0036] S140. Determine the query statement based on the target database schema information and the problem to be processed.

[0037] The query statement is a structured data query instruction that matches the problem to be processed. The query statement can be determined based on the target database pattern information and the problem to be processed. For example, the target database pattern information and the problem to be processed are input into a trained query statement generation model for processing to obtain the query statement. The query statement generation model includes, but is not limited to, neural network models; for example, a large language model. The query statement generation model is selected according to requirements, and this invention does not impose any limitations.

[0038] Specifically, the target database schema information and the problem to be processed are input into the trained query statement generation model for processing to obtain query statements. This makes the query statements fit the database tables, thereby improving the accuracy and usability of the query statements and providing an accurate data foundation for subsequent data queries.

[0039] Optionally, the method further includes: searching in a preset business database based on the query statement to obtain query results, the query results including business data that matches the problem to be processed.

[0040] The query results are the matching results retrieved from the business database using a query statement. The query results include business data that matches the problem to be addressed. The process of searching the preset business database based on the query statement to obtain the query results is as follows: The data tables and fields in the preset business database are filtered using the query statement; the business data in the filtered data tables is integrated and summarized to obtain the business data that matches the problem to be addressed. This allows for quick and accurate retrieval of business data that meets the query requirements, efficiently completing the data query and providing real and effective data support for subsequent decision-making.

[0041] The technical solution of this embodiment obtains the problem to be processed, performs semantic recognition on the problem to be processed, and determines the entity information and multi-dimensional label information of the problem to be processed. The multi-dimensional label information is used to characterize the multi-dimensional attributes of the problem to be processed. The multi-dimensional attributes of the problem to be processed include at least one of business domain, data type, and data layering information, providing comprehensive data support for subsequent analysis and processing. Based on the problem to be processed, entity information, and multi-dimensional label information, a matching process is performed in a preset vector database to determine multiple candidate database pattern information. The candidate database pattern information is used to characterize the business attributes of the object to be queried, realizing the filtering of candidate database pattern information, which can effectively reduce the amount of computation in subsequent processing, thereby improving processing efficiency and accuracy. Based on the problem to be processed and multiple candidate database pattern information, the target database pattern information is determined, providing accurate and reliable data support for subsequent analysis and processing. Based on the target database pattern information and the problem to be processed, the query statement is determined, so that the query statement fits the database table, thereby improving the accuracy and usability of the query statement and providing an accurate data foundation for subsequent data queries.

[0042] Figure 2 This is a flowchart of another data processing method provided by an embodiment of the present invention. This embodiment is a refinement of the above embodiments. Based on the foregoing embodiments, it provides a detailed explanation of matching the problem to be processed, entity information, and multi-dimensional label information in a preset vector database to determine multiple candidate database pattern information. For specific implementation details, please refer to the technical solution of this embodiment. Technical terms that are the same as or corresponding to those in the above embodiments will not be repeated here. Figure 2 As shown, the method specifically includes the following steps: S210. Obtain the problem to be processed, perform semantic recognition on the problem to be processed, and determine the entity information and multi-dimensional label information of the problem to be processed. The multi-dimensional label information is used to characterize the multi-dimensional attributes of the problem to be processed. The multi-dimensional attributes of the problem to be processed include at least one of business domain, data type and data layering information.

[0043] S220. Convert the problem to be processed, entity information, and multi-dimensional label information into first semantic features; determine first similarity information based on the first semantic features and first-level vectorized description information; filter the vector database based on the first similarity information to obtain multiple candidate database pattern information, which is used to characterize the business attributes of the object to be queried.

[0044] The first semantic feature is the feature information obtained by feature transformation of the problem to be processed, entity information, and multi-dimensional label information. The first semantic feature can be obtained based on the problem to be processed, entity information, and multi-dimensional label information. For example, the problem to be processed, entity information, and multi-dimensional label information can be simultaneously input into a trained semantic transformation model for semantic transformation to obtain the first semantic feature. The semantic transformation model includes, but is not limited to, a neural network model. The semantic transformation model can be set according to requirements, and this invention does not impose any limitations.

[0045] The first similarity information is data characterizing the degree of semantic matching between the first semantic feature and the first-level vectorized description information. The first similarity information can be determined based on the first semantic feature and the first-level vectorized description information. For example, the first semantic feature and the first-level vectorized description information are input into a trained semantic similarity determination model for processing to obtain the first similarity information. The semantic similarity determination model includes, but is not limited to, neural network models and mathematical models. The semantic similarity determination model is selected according to requirements, and this invention does not impose any limitations.

[0046] The process of filtering in the vector database based on the first similarity information to obtain multiple candidate database pattern information is as follows: According to the pre-set fourth semantic similarity threshold, the vector database is filtered to select the database pattern information corresponding to the first similarity information in the vector database that is greater than or equal to the fourth semantic similarity threshold. The database pattern information corresponding to the first similarity information in the vector database that is greater than or equal to the fourth semantic similarity threshold is used as candidate database pattern information.

[0047] It should be noted that the process of filtering in the vector database based on the first similarity information to obtain multiple candidate database pattern information can also be as follows: sort the multiple first similarity information in descending order, select the database pattern information corresponding to the first similarity information that ranks first in the vector database, and use the first preset number of database pattern information in the vector database as candidate database pattern information.

[0048] Specifically, the problem to be processed, entity information, and multi-dimensional label information are simultaneously input into a trained semantic transformation model for semantic transformation to obtain the first semantic feature. The first semantic feature and the first-level vectorized description information are then input into a trained semantic similarity determination model for processing to obtain the first similarity information. Multiple first similarity information are sorted in descending order, and a preset number of database pattern information corresponding to the first similarity information at the top of the vector database are selected. The preset number of database pattern information at the top of the vector database is used as candidate database pattern information, which realizes the preliminary screening of database pattern information, effectively reduces the amount of computation in subsequent processing, and thus helps to improve processing efficiency and accuracy.

[0049] Optionally, the database pattern information carries weight labels, which are used to characterize the importance of the database pattern information; the method further includes: correcting the first similarity information based on the weight corresponding to the weight label to obtain corrected first similarity information; and filtering in a vector database based on the corrected first similarity information to obtain multiple candidate database pattern information.

[0050] The database schema information carries weight labels, which characterize the importance of the database schema information. It should be noted that different database schema information carries different weight labels. The process of correcting the first similarity information based on the weights corresponding to the weight labels is as follows: the first similarity information is weighted according to the weights corresponding to the weight labels, and the weighted first similarity information is used as the corrected first similarity information. Candidate database schema information can also be obtained by filtering from the vector database based on the corrected first similarity information. For example, multiple corrected first similarity information are sorted in descending order, and a predetermined number of database schema information corresponding to the top-ranked corrected first similarity information in the vector database are selected as candidate database schema information.

[0051] Specifically, the first similarity information is weighted according to the weight corresponding to the weight label, and the weighted first similarity information is used as the corrected first similarity information. The multiple corrected first similarity information are sorted in descending order, and the database pattern information corresponding to the top-ranked preset number of corrected first similarity information in the vector database is selected. The top-ranked preset number of corrected database pattern information in the vector database is used as candidate database pattern information. By correcting the first similarity information, the efficiency of the initial screening of database pattern information can be improved, the computational load of subsequent processing can be effectively reduced, and thus the processing efficiency and accuracy can be improved.

[0052] S230. Determine the target database schema information based on the problem to be processed and multiple candidate database schema information.

[0053] Optionally, the database schema information also includes second-level vectorized description information and third-level vectorized description information, wherein the granularity of the first-level vectorized description information is lower than that of the second-level vectorized description information, and the granularity of the second-level vectorized description information is lower than that of the third-level vectorized description information.

[0054] The second-level vectorized description information is used to characterize the data composition and business purpose of the business data, including but not limited to the vector features corresponding to the names and annotation information of key fields. The third-level vectorized description information is used to characterize the structural attributes and data rules of the business data, including but not limited to the feature vectors corresponding to field content, field type, and primary / foreign key relationships. It should be noted that the granularity of the first-level vectorized description information is lower than that of the second-level vectorized description information, and the granularity of the second-level vectorized description information is lower than that of the third-level vectorized description information.

[0055] Optionally, the target database pattern information is determined based on the problem to be processed and multiple candidate database pattern information, including: converting the problem to be processed into a second semantic feature; determining second similarity information based on the second semantic feature, second-level vectorized description information and third-level vectorized description information; and filtering multiple candidate database pattern information based on the second similarity information to obtain the target database pattern information.

[0056] The second semantic feature is the feature information obtained by feature transformation of the problem to be processed. For example, the problem to be processed is input into a trained semantic transformation model for semantic transformation to obtain the second semantic feature. The semantic transformation model includes, but is not limited to, a neural network model. The semantic transformation model is set according to the requirements, and this invention is not limited thereto. The second similarity information is data representing the degree of semantic matching between the problem to be processed and the second-level vectorized description information and the third-level vectorized description information. The second similarity information can be determined based on the second semantic feature, the second-level vectorized description information, and the third-level vectorized description information. For example, the second-level vectorized description information and the third-level vectorized description information are concatenated to obtain concatenated vectorized description information. The second semantic feature and the concatenated vectorized description information are input into a trained semantic similarity determination model for processing to obtain the second similarity information. The semantic similarity determination model includes, but is not limited to, a neural network model and a mathematical model. The semantic similarity determination model is selected according to the requirements, and this invention is not limited thereto.

[0057] The process of filtering multiple candidate database pattern information based on the second similarity information to obtain the target database pattern information is as follows: sort the multiple second similarity information in descending order, select the candidate database pattern information corresponding to the top-ranked preset number of second similarity information, and use the candidate database pattern information corresponding to the top-ranked preset number of second similarity information as the target database pattern information.

[0058] Specifically, the problem to be processed is input into a trained semantic transformation model for semantic transformation to obtain the second semantic feature. The second-level vectorized description information and the third-level vectorized description information are concatenated to obtain the concatenated vectorized description information. The second semantic feature and the concatenated vectorized description information are then input into a trained semantic similarity determination model for processing to obtain the second similarity information. Multiple second similarity information are sorted in descending order, and a preset number of candidate database pattern information corresponding to the top-ranked second similarity information are selected as the target database pattern information. This achieves fine-grained screening of multi-level database pattern information, which can effectively improve the adaptability of the target database pattern information and the problem to be processed, thereby improving the accuracy of the query statement.

[0059] S240. Determine the query statement based on the target database schema information and the problem to be processed.

[0060] Based on the above embodiments, the method further includes: performing synonym augmentation on the entity information of the problem to be processed to obtain the synonym-augmented entity information; converting the problem to be processed, the synonym-augmented entity information, and multi-dimensional label information into first semantic features; determining first similarity information based on the first semantic features and first-level vectorized description information; filtering in a vector database based on the first similarity information to obtain multiple candidate database pattern information; determining target database pattern information based on the problem to be processed and the multiple candidate database pattern information; and determining a query statement based on the target database pattern information and the problem to be processed.

[0061] The process of synonym augmentation of entity information in the problem to be processed to obtain synonym augmented entity information is as follows: Calculate the semantic similarity between entity information and multiple entities in a pre-set entity database; filter the semantic similarity between entity information and multiple entities in the pre-set entity database based on a pre-set fourth semantic similarity threshold; determine candidate entities corresponding to semantic similarity greater than or equal to the fourth semantic similarity threshold; and perform synonym augmentation on the entity information in the problem to be processed based on the candidate entities to obtain synonym augmented entity information. By performing synonym augmentation on entity information, the accuracy of database schema information can be improved.

[0062] For example, see Figure 3 , Figure 3 This is a flowchart of another data processing method provided in an embodiment of the present invention.

[0063] The technical solution of this embodiment obtains the problem to be processed, performs semantic recognition on the problem to be processed, and determines the entity information and multi-dimensional label information of the problem to be processed. The multi-dimensional label information is used to characterize the multi-dimensional attributes of the problem to be processed. The multi-dimensional attributes of the problem to be processed include at least one of business domain, data type, and data layering information, providing comprehensive data support for subsequent analysis and processing. The problem to be processed, entity information, and multi-dimensional label information are converted into first semantic features. Based on the first semantic features and first-level vectorized description information, first similarity information is determined. Based on the first similarity information, multiple candidate database pattern information is obtained by filtering in a vector database. The candidate database pattern information is used to characterize the business attributes of the object to be queried, realizing the preliminary screening of database pattern information, which can effectively reduce the amount of computation in subsequent processing, thereby improving processing efficiency and accuracy. Based on the problem to be processed and multiple candidate database pattern information, target database pattern information is determined, providing accurate and reliable data support for subsequent analysis and processing. Based on the target database pattern information and the problem to be processed, a query statement is determined, so that the query statement fits the database table, thereby improving the accuracy and usability of the query statement and providing an accurate data foundation for subsequent data queries.

[0064] Figure 4 This is a schematic diagram of the structure of a data processing device provided in an embodiment of the present invention. Figure 4 As shown, the device includes a semantic recognition module 310, a candidate database pattern information determination module 320, a target database pattern information determination module 330, and a query statement determination module 340.

[0065] The semantic recognition module 310 is used to acquire the problem to be processed, perform semantic recognition on the problem to be processed, and determine the entity information and multi-dimensional label information of the problem to be processed. The multi-dimensional label information is used to characterize the multi-dimensional attributes of the problem to be processed. The multi-dimensional attributes of the problem to be processed include at least one of business domain, data type, and data layering information. The candidate database pattern information determination module 320 is used to match the problem to be processed, entity information, and multi-dimensional label information in a preset vector database to determine multiple candidate database pattern information. The candidate database pattern information is used to characterize the business attributes of the object to be queried. The target database pattern information determination module 330 is used to determine the target database pattern information based on the problem to be processed and multiple candidate database pattern information. The query statement determination module 340 is used to determine the query statement based on the target database pattern information and the problem to be processed.

[0066] The technical solution of this embodiment employs a semantic recognition module to acquire the problem to be processed, perform semantic recognition on the problem, and determine the entity information and multi-dimensional label information of the problem. The multi-dimensional label information is used to characterize the multi-dimensional attributes of the problem, including at least one of business domain, data type, and data layering information, providing comprehensive data support for subsequent analysis and processing. A candidate database pattern information determination module is used to match the problem, entity information, and multi-dimensional label information in a preset vector database to determine multiple candidate database pattern information. The candidate database pattern information characterizes the business attributes of the object to be queried, achieving the filtering of candidate database pattern information, effectively reducing the computational load of subsequent processing, and thus improving processing efficiency and accuracy. A target database pattern information determination module is used to determine the target database pattern information based on the problem and multiple candidate database pattern information, providing accurate and reliable data support for subsequent analysis and processing. A query statement determination module is used to determine the query statement based on the target database pattern information and the problem to be processed, ensuring that the query statement fits the database table, thereby improving the accuracy and usability of the query statement and providing an accurate data foundation for subsequent data queries.

[0067] Based on the above embodiments, optionally, the vector database includes multiple database schema information; the database schema information includes first-level vectorized description information.

[0068] Optionally, the candidate database pattern information determination module 320 is further configured to: convert the problem to be processed, entity information, and multi-dimensional label information into first semantic features; determine first similarity information based on the first semantic features and first-level vectorized description information; and filter in the vector database based on the first similarity information to obtain multiple candidate database pattern information.

[0069] Optionally, the database schema information also includes second-level vectorized description information and third-level vectorized description information, wherein the granularity of the first-level vectorized description information is lower than that of the second-level vectorized description information, and the granularity of the second-level vectorized description information is lower than that of the third-level vectorized description information.

[0070] Optionally, the target database pattern information determination module 330 is further configured to: convert the problem to be processed into a second semantic feature; determine second similarity information based on the second semantic feature, second-level vectorized description information and third-level vectorized description information; and filter multiple candidate database pattern information based on the second similarity information to obtain the target database pattern information.

[0071] Optionally, the database schema information carries weight labels, which are used to characterize the importance of the database schema information.

[0072] Optionally, the candidate database pattern information determination module 320 is further configured to: correct the first similarity information based on the weight corresponding to the weight label to obtain the corrected first similarity information; and filter the vector database based on the corrected first similarity information to obtain multiple candidate database pattern information.

[0073] Optionally, the vector database includes multiple database schema information; the database schema information includes first-level vectorized description information, second-level vectorized description information, and third-level vectorized description information, wherein the fineness of the first-level vectorized description information is lower than that of the second-level vectorized description information, and the fineness of the second-level vectorized description information is lower than that of the third-level vectorized description information.

[0074] Optionally, the device further includes a vector database construction module, used for: acquiring metadata information of database tables, the metadata information being used to characterize the multi-dimensional attributes of the database tables, the multi-dimensional attributes of the database tables including at least one of business domain, data type, and data layering information; performing fine-grained hierarchical classification on the metadata information to obtain first-level description information, second-level description information, and third-level description information; performing structured processing on the first-level description information, second-level description information, and third-level description information to obtain first-level vectorized description information, second-level vectorized description information, and third-level vectorized description information; and constructing a vector database based on the first-level vectorized description information, second-level vectorized description information, and third-level vectorized description information.

[0075] Optionally, the vector database construction module is also used for: semantically enhancing the metadata information to obtain semantically enhanced metadata information; fine-grained hierarchical classification of the semantically enhanced metadata information to obtain first-level description information, second-level description information, and third-level description information; structuring the first-level description information, second-level description information, and third-level description information to obtain structured vectors corresponding to the first-level description information, second-level description information, and third-level description information, respectively; and constructing a vector database based on the structured vectors.

[0076] Optionally, the device also includes a query result acquisition module, used to: search in a preset business database based on the query statement to obtain query results, which include business data that matches the problem to be processed.

[0077] The data processing apparatus provided in this embodiment of the invention can execute a data processing method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of executing the method.

[0078] Figure 5This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0079] like Figure 5 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0080] Multiple components in electronic device 10 are connected to input / output (I / O) interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of monitors, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0081] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as a data processing method.

[0082] In some embodiments, a data processing method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via read-only memory (ROM) 12 and / or communication unit 19. When the computer program is loaded into random access memory (RAM) 13 and executed by processor 11, one or more steps of a data processing method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform a data processing method by any other suitable means (e.g., by means of firmware).

[0083] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transferring data and instructions to the storage system, the at least one input device, and the at least one output device.

[0084] A computer program for implementing a data processing method of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer program causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer program can be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0085] This invention also provides a computer-readable storage medium storing computer instructions for causing a processor to execute a data processing method, the method comprising: The process involves: acquiring the problem to be processed; performing semantic recognition on the problem to determine its entity information and multi-dimensional label information; the multi-dimensional label information characterizing the multi-dimensional attributes of the problem, which include at least one of business domain, data type, and data layering information; matching the problem, entity information, and multi-dimensional label information against a pre-defined vector database to determine multiple candidate database schema information, which characterizes the business attributes of the object to be queried; determining the target database schema information based on the problem and the multiple candidate database schema information; and determining the query statement based on the target database schema information and the problem.

[0086] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0087] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0088] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0089] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0090] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements a data processing method according to any embodiment of the invention.

[0091] In implementing the computer program product, computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0092] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0093] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A data processing method, characterized in that, include: The process involves acquiring a problem to be processed, performing semantic recognition on the problem to be processed, and determining the entity information and multi-dimensional label information of the problem to be processed. The multi-dimensional label information is used to characterize the multi-dimensional attributes of the problem to be processed, and the multi-dimensional attributes of the problem to be processed include at least one of business domain, data type, and data layering information. Based on the problem to be processed, the entity information, and the multi-dimensional label information, a matching is performed in a preset vector database to determine multiple candidate database pattern information, which is used to characterize the business attributes of the object to be queried. The target database schema information is determined based on the problem to be addressed and the multiple candidate database schema information. The query statement is determined based on the target database schema information and the problem to be processed.

2. The method according to claim 1, characterized in that, The vector database includes multiple database schema information; the database schema information includes first-level vectorized description information. The process involves matching the problem to be processed, the entity information, and multi-dimensional label information in a preset vector database to determine multiple candidate database pattern information, including: The problem to be processed, the entity information, and the multi-dimensional label information are converted into first semantic features; First similarity information is determined based on the first semantic feature and the first-level vectorized description information; Based on the first similarity information, the vector database is filtered to obtain the multiple candidate database pattern information.

3. The method according to claim 2, characterized in that, The database schema information also includes secondary vectorized description information and tertiary vectorized description information, wherein the granularity of the primary vectorized description information is lower than that of the secondary vectorized description information, and the granularity of the secondary vectorized description information is lower than that of the tertiary vectorized description information. The step of determining the target database schema information based on the problem to be processed and the multiple candidate database schema information includes: The problem to be processed is converted into a second semantic feature; The second similarity information is determined based on the second semantic feature, the second-level vectorized description information, and the third-level vectorized description information; The target database pattern information is obtained by filtering the multiple candidate database pattern information based on the second similarity information.

4. The method according to claim 2, characterized in that, The database schema information carries a weight label, which is used to characterize the importance of the database schema information. The method further includes: The first similarity information is corrected based on the weight corresponding to the weight label to obtain the corrected first similarity information; Based on the corrected first similarity information, the vector database is filtered to obtain the multiple candidate database pattern information.

5. The method according to claim 1, characterized in that, The vector database includes multiple database schema information; the database schema information includes first-level vectorized description information, second-level vectorized description information, and third-level vectorized description information, wherein the fineness of the first-level vectorized description information is lower than that of the second-level vectorized description information, and the fineness of the second-level vectorized description information is lower than that of the third-level vectorized description information. The construction process of the vector database is as follows: Obtain metadata information of a database table, wherein the metadata information is used to characterize the multi-dimensional attributes of the database table, and the multi-dimensional attributes of the database table include at least one of business domain, data type and data layering information; The metadata information is then finely classified to obtain first-level description information, second-level description information, and third-level description information; The first-level description information, the second-level description information, and the third-level description information are processed in a structured manner to obtain the first-level vectorized description information, the second-level vectorized description information, and the third-level vectorized description information; The vector database is constructed based on the first-level vectorized description information, the second-level vectorized description information, and the third-level vectorized description information.

6. The method according to claim 5, characterized in that, The method further includes: The metadata information is semantically enhanced to obtain semantically enhanced metadata information; The semantically enhanced metadata information is then subjected to fine-grained hierarchical classification to obtain the first-level description information, the second-level description information, and the third-level description information; The first-level description information, the second-level description information, and the third-level description information are processed in a structured manner to obtain the structured vectors corresponding to the first-level vectorized description information, the second-level vectorized description information, and the third-level vectorized description information, respectively. The vector database is constructed based on the structured vectors.

7. The method according to claim 1, characterized in that, The method further includes: The query statement is used to search the preset business database to obtain query results, which include business data that matches the problem to be solved.

8. A data processing apparatus, characterized in that, include: A semantic recognition module is used to acquire a problem to be processed, perform semantic recognition on the problem to be processed, and determine the entity information and multi-dimensional label information of the problem to be processed. The multi-dimensional label information is used to characterize the multi-dimensional attributes of the problem to be processed. The multi-dimensional attributes of the problem to be processed include at least one of business domain, data type and data layering information. The candidate database pattern information determination module is used to match the problem to be processed, the entity information and multi-dimensional label information in a preset vector database to determine multiple candidate database pattern information, wherein the candidate database pattern information is used to characterize the business attributes of the object to be queried. The target database pattern information determination module is used to determine the target database pattern information based on the problem to be processed and the multiple candidate database pattern information. The query statement determination module is used to determine the query statement based on the target database schema information and the problem to be processed.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the data processing method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the data processing method according to any one of claims 1-7.

Images