Structured data processing method and device based on large model and server

Abstract

The invention discloses a structured data processing method and device based on a large model and a server, and relates to the technical field of data processing. And extracting a structured data set from the collected to-be-processed metadata, and importing the structured data set into the database. A data query text in a natural language description form input by a user is analyzed through a large model, semantic representation data is automatically obtained, and a structured query language is generated. And searching in a pre-constructed database by using the structured query language to obtain a query result. Through the combination of the large model and the database, the structured data processing capability of the database is utilized, the data accuracy is improved, the user query intention is converted into the database query statement, and the query result is quickly retrieved from the database, so that the research and development work of a front-end application program is omitted, the structured query language processing efficiency is improved, and the user experience is improved. And the input cost is reduced.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a structured data processing method, device and server based on a large model. Background Art

[0002] With the rapid development of information technology, big data has been deeply integrated into all walks of life and has become an indispensable resource for all walks of life. Data is diverse in form and complex in structure, covering multiple types such as structured, semi-structured and unstructured. How to extract the structured data required by users from massive and diverse data has become an important challenge currently faced.

[0003] At present, the relevant technology is to develop application interfaces by R&D personnel to meet the structured data required by users. However, user needs are constantly changing. In order to cope with the changing needs, R&D personnel need to develop application interfaces and debug them from time to time, resulting in low efficiency in processing structured data and high investment costs. Summary of the invention

[0004] The present application provides a structured data processing method, device and server based on a large model, so as to at least solve the problems of low efficiency and high investment cost in processing structured data in related technologies.

[0005] The present application provides a structured data processing method based on a large model, comprising:

[0006] Receiving a data query request sent by a user; wherein the data query request includes a data query text and a data result display method;

[0007] According to the data query text, obtain the text data to be parsed;

[0008] The text data request to be parsed is input into the big model, so that the big model performs the following steps:

[0009] Determine whether there is text data to be parsed in the cache;

[0010] If it is determined that the text data to be parsed does not exist in the cache, obtaining semantic representation data according to the text data to be parsed;

[0011] Based on the semantic representation data, identify the key data required to generate structured query language;

[0012] The key data is structured and converted according to the preset template to obtain structured key data;

[0013] Generate structured query language corresponding to the semantic representation data based on structured key data;

[0014] Use structured query language to search in the pre-built database and obtain the query results corresponding to the data query request;

[0015] The query results are sent to the user end in accordance with the data result display method, so that the user end can display the query results.

[0016] The present application also provides a structured data processing device based on a large model, comprising:

[0017] The receiving module is used to receive a data query request sent by a user terminal; wherein the data query request includes a data query text and a data result display method;

[0018] The acquisition module is used to query the text according to the data and obtain the text data to be parsed;

[0019] The input module is used to input the text data request to be parsed into the large model, so that the large model performs the following steps:

[0020] The input module includes:

[0021] A determination unit, used to determine whether there is text data to be parsed in the cache;

[0022] A first acquisition unit, configured to acquire semantic representation data according to the text data to be parsed if it is determined that the text data to be parsed does not exist in the cache;

[0023] An identification unit, used for identifying key data required for generating a structured query language according to semantically represented data;

[0024] A second acquisition unit is used to perform structured conversion on the key data according to a preset template to obtain structured key data;

[0025] A generating unit, used for generating a structured query language corresponding to the semantic representation data according to the structured key data;

[0026] A retrieval module is used to search in a pre-built database using a structured query language to obtain a query result corresponding to the data query request;

[0027] The sending module is used to send the query results to the user end in accordance with the data result display mode, so that the user end can display the query results.

[0028] The present application also provides a server, comprising: a memory for storing a computer program; and a processor for implementing the steps of any one of the above-mentioned large model-based structured data processing methods when executing the computer program.

[0029] The present application also provides a computer-readable storage medium, in which a computer program is stored, wherein when the computer program is executed by a processor, the steps of any of the above-mentioned structured data processing methods based on large models are implemented.

[0030] The present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned large model-based structured data processing methods when executed by a processor.

[0031] Through this application, a structured data set is extracted from the metadata to be processed, and the structured data set is imported into the database. The ability of the database to process structured data is used to improve the accuracy of the data. The data query text in the form of natural language description input by the user is analyzed through a large model, and the semantic representation data is automatically obtained to generate a structured query language. Using the structured query language, a search is performed in a pre-built database to obtain the query results. Through the combination of the large model and the database, the user's query intention is converted into a database query statement, and the query results are quickly retrieved from the database, omitting the research and development work of the front-end application, improving the efficiency of processing the structured query language, and reducing the investment cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0033] Figure 1 A schematic diagram of a scenario of a structured data processing method based on a large model provided in an embodiment of the present application;

[0034] Figure 2 A schematic diagram of a flow chart of a structured data processing method based on a large model provided in an embodiment of the present application;

[0035] Figure 3 A schematic diagram of the structure of a large model-based structured data processing device provided in an embodiment of the present application;

[0036] Figure 4 A schematic diagram of the structure of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0038] It should be noted that, in the description of this application, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. The terms "first", "second", etc. in this application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0039] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0040] First, the terms involved in this application are explained:

[0041] Structured Query Language: It is a standard programming language used to manage and operate relational databases. It can access and process databases, including data insertion, query, update and deletion.

[0042] In order to solve the problem that the demand for structured data required by users in the prior art is constantly changing, R&D personnel are required to develop application interfaces and debug them from time to time, resulting in low efficiency in processing structured data and high investment costs. The embodiment of the present application proposes the following technical concept: the inventor has thought of a method for combining a database and a large model. A structured data set is extracted from the metadata to be processed, and the structured data set is imported into the database. The data query text in the form of a natural language description input by the user is analyzed by the large model, semantic representation data is automatically obtained, and a structured query language is generated. Using the structured query language, a search is performed in a pre-built database to obtain query results. Through the combination of the large model and the database, the user's query intention is converted into a database query statement, and the query results are quickly retrieved from the database, omitting the research and development work of the front-end application, improving the efficiency of processing the structured query language, and reducing the investment cost.

[0043] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0044] In conjunction with the specific application environment architecture or specific hardware architecture on which the execution of the large model-based structured data processing method depends, the specific application environment architecture or specific hardware architecture is described herein.

[0045] refer to Figure 1 , Figure 1 A schematic diagram of a scenario of a structured data processing method based on a large model provided in an embodiment of the present application. The server provided in this embodiment includes: a receiving device 101 , a processor 102 and a display device 103 .

[0046] It is understandable that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the structured data processing method based on the large model. In other feasible implementations of the present application, the above architecture may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently, which can be determined according to the actual application scenario and is not limited here. Figure 1 The components shown may be implemented in hardware, software, or a combination of software and hardware.

[0047] In a specific implementation process, the receiving device 101 may be an input / output interface or a communication interface, and may receive a data query request sent by a user terminal.

[0048] The processor 102 may perform a series of processing on the data query text in the data query request to obtain a query result.

[0049] The display device 103 can be used to display the query results.

[0050] It should be understood that the above-mentioned processor can be implemented by the processor reading instructions in the memory and executing the instructions, or it can be implemented by a chip circuit.

[0051] In addition, the network architecture and business scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Ordinary technicians in this field can know that with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0052] Figure 2 A flow chart of a structured data processing method based on a large model provided in an embodiment of the present application is shown in FIG. Figure 2As shown in the figure, an embodiment of the present application provides a method for processing structured data based on a large model, and the method is described in detail as follows:

[0053] S201: Receive a data query request sent by the client; the data query request includes a data query text and a data result display method.

[0054] S202: Obtain the text data to be parsed according to the data query text.

[0055] Specifically, delete the noise data in the data query text to obtain the initial text data to be parsed; split the initial text data to be parsed into multiple words; remove the stop words in the multiple words to obtain the text data to be parsed.

[0056] In this embodiment, once the user inputs a data query request on the front-end interface of the client and submits it, the data query text will be immediately captured. Denoise, tokenize, and remove stop words from the data query text. The data query text may contain various irrelevant characters and noises. Delete the noise data in the data query text to obtain the initial text data to be parsed; through the tokenization technology, split the initial text data to be parsed into meaningful lexical units to obtain multiple words. In the tokenization result, there are some words that do not contribute substantially to the subsequent analysis, that is, stop words, such as "de", "le", and "zai", etc. Although these words appear frequently in the data query text, they do not carry information themselves.

[0057] Optionally, the tokenization technology includes but is not limited to dictionary-based tokenization, statistics-based tokenization, and deep learning tokenization methods, etc.

[0058] S203: Input the text data request to be parsed into the large model, so that the large model performs the following steps.

[0059] Optionally, the large model supports the fusion analysis of structured data and unstructured data, provides a unified data processing framework, and enhances the comprehensiveness of the analysis.

[0060] Specifically, step S203 includes S2031~S2035:

[0061] S2031: Determine whether the text data to be parsed exists in the cache.

[0062] Specifically, perform a hash calculation on the text data to be parsed to obtain the hash identifier of the text data to be parsed; determine whether the hash identifier exists in the cache; if the hash identifier does not exist in the cache, it is determined that the text data to be parsed does not exist in the cache.

[0063] In this embodiment, if the hash identifier exists in the cache, it is determined that the text data to be parsed exists in the cache; obtain the structured query language corresponding to the hash identifier from the cache.

[0064] In this embodiment, the cache stores key-value pairs, where the key is the text data to be parsed and the value is the structured query language corresponding to the text data to be parsed. If there is a hash identifier corresponding to the text data to be parsed, it means that the text data to be parsed and the structured query language corresponding to the text data to be parsed exist in the cache, and the corresponding structured query language is directly obtained without regenerating the structured query language.

[0065] S2032: If it is determined that the text data to be parsed does not exist in the cache, semantic representation data is obtained according to the text data to be parsed.

[0066] Specifically, according to the text data to be parsed, the semantic representation data is obtained, and the formula is:

[0067]

[0068] In the formula, Representation semantics represents data, Represents the text data to be parsed. Represents the parameters of the large model; It indicates the probability of obtaining semantic representation data when the text data to be parsed and the parameters of the large model are input; Indicates the semantic representation data with the highest probability of selection.

[0069] In this embodiment, the large model generates multiple semantic representation data based on the text data to be parsed and the parameters of the large model, and finds the semantic representation data with the highest probability from all the semantic representation data as the final semantic representation data.

[0070] S2033: Identify key data required to generate structured query language based on the semantic representation data.

[0071] Among them, key data include keywords, operation types and query conditions.

[0072] Specifically, step S2033 includes Sa~Se:

[0073] Sa: Segment the semantic representation data into words to obtain multiple semantic representation words.

[0074] Sb: Perform part-of-speech tagging on multiple semantic representation words to obtain multiple tagged semantic representation words.

[0075] In this embodiment, the part of speech is marked for each semantic representation word to determine the categories such as noun, verb, adjective and adverb. Through the part of speech tagging, the role of the semantic representation word in the sentence can be preliminarily determined. For example, a noun may be a table name, a field name or an entity in a condition, and a verb may represent an operation type.

[0076] Sc: Identify keywords from multiple annotated semantic representation words based on pre-built database keywords.

[0077] In this embodiment, the pre-constructed database keywords include all table names and field names. The annotated semantic representation words are compared with the pre-constructed database keywords, and if there is a match, it is identified as a table name or field name.

[0078] Sd: Identify the operation type from multiple annotated semantic representation words based on a pre-built list of operation types.

[0079] In this embodiment, the pre-built operation type list includes query, insert, update, delete, etc., which is used to determine the type of operation that the user wants to perform. When the user input includes a query, the operation type can be identified as a query.

[0080] Se: According to the pre-built query term lexicon, the query terms are identified from multiple annotated semantic representation words.

[0081] In this embodiment, the pre-built query condition word library includes words such as "greater than", "less than", "equal to", "include", "maximum" and "minimum" that represent conditional relationships or limited ranges. The annotated semantic representation words are matched with the pre-built query condition word library to identify the query conditions used to construct the structured query language.

[0082] S2034: Structural conversion is performed on the key data according to a preset template to obtain structured key data.

[0083] Optionally, the preset template can be JSON, Schema, Pydantic model, etc.

[0084] S2035: Generate a structured query language corresponding to the semantic representation data based on the structured key data.

[0085] Specifically, according to the structured key data, the formula for generating the structured query language corresponding to the semantic representation data is:

[0086]

[0087] In the formula, stands for Structured Query Language, Representation semantics represents data, Represents structured key data, Indicates the parameters of the generated structured query language function. represents the probability of generating structured query language when the semantic representation data, structured key data and parameters of the function generating structured query language are input; It represents the structured query language with the highest selection probability.

[0088] In this embodiment, the formula steps for generating the structured query language are as follows:

[0089] SQL="Operation+" "+Columns+" FROM "+Table+" WHERE "+Conditions"

[0090] Among them, Operation indicates the operation type, Columns indicates the field name to be queried, Table indicates the table name to be queried, and Conditions indicates the query conditions.

[0091] For example, the user input is: What is the best speccpu score published by manufacturer A? The identified key data include speccpu, score, manufacturer A, and best. The table name is speccpu, the field name is score, the operation type is query, and the query conditions are manufacturer A and best. The generated structured query language is SELECT max(score) FROM speccpuWHERE manufacturer="A".

[0092] In this embodiment, multiple structured query languages ​​are generated based on input semantic representation data, structured key data and parameters of a structured query language generation function, and a structured query language with the highest probability is found from all structured query languages ​​as the final structured query language.

[0093] In this embodiment, after the structured query language is generated, a hash calculation is performed on the text data to be parsed to obtain a hash identifier. According to the hash identifier, the text data to be parsed and the structured query language are saved in a cache in the form of a key-value pair.

[0094] S204: Using structured query language, search in a pre-built database to obtain a query result corresponding to the data query request.

[0095] Specifically, the structured query language is used to search in the pre-built database to obtain the query result corresponding to the data query request. The formula is:

[0096]

[0097] In the formula, Indicates the query result. Represents a pre-built database, stands for Structured Query Language, Represents a database execution function.

[0098] In this embodiment, the database execution function sends the structured query language to a pre-built database for execution to obtain the query result.

[0099] Optionally, an efficient protocol or interface can be designed to achieve seamless connection between large models and databases.

[0100] Specifically, the steps of pre-building the database include Sf~Sh:

[0101] Sf: Collect metadata to be processed.

[0102] In this embodiment, metadata to be processed is widely collected, and these data come from a wide range of sources, including but not limited to various websites, etc. Taking statistical server performance data as an example, the data may come from the official website of the server manufacturer, etc. The collection process needs to adapt to the formats and interfaces of different data sources. When grabbing metadata from a web page, it is necessary to parse the HTML or XML structure, etc.

[0103] Sg: Extract structured datasets from metadata to be processed.

[0104] In this embodiment, the data formats of the metadata to be processed collected from different data sources may vary greatly. In order to facilitate subsequent unified processing, the data needs to be converted into structured data. For example, the time format may be different in different data sources, some are YYYY-MM-DD HH:MM:SS, and some are timestamp formats. All time data needs to be unified into one format, and other data types such as numbers, strings, etc. are also structured.

[0105] In this embodiment, the metadata to be processed needs to be cleaned and deduplicated before extracting the structured data set. The metadata to be processed often contains noise data, such as irrelevant special characters and garbled characters, which will interfere with subsequent analysis. These noises need to be identified and removed; duplicate data may appear during the collection process, which will occupy storage space and affect the accuracy and efficiency of data analysis. It is necessary to find duplicate records and retain unique copies.

[0106] In this embodiment, in order to ensure the timeliness and freshness of the data, the metadata to be processed is refreshed daily to ensure that the latest data acquisition situation can be reflected so that subsequent steps can understand and process the data based on accurate metadata.

[0107] Sh: Create an initial database and import the structured data set into the initial database to obtain a pre-built database.

[0108] In this embodiment, the database type is selected, the table structure is designed, and the index is created to create an initial database. The initial database also provides data retrieval functions, such as query, classification, statistics, and sorting, so that data access becomes fast and convenient. According to the characteristics of the structured data set and the query mode, the optimized index is dynamically generated. The database also provides query performance monitoring and tuning functions.

[0109] Optionally, the database type includes but is not limited to MySQL, PostgreSQL, and Oracle.

[0110] In this embodiment, the structured data set is stored in the initial database in the form of fields according to the designed table structure, thereby achieving smooth connection and processing of data flow.

[0111] In this embodiment, encryption and desensitization need to be performed during data transmission and processing to ensure data security.

[0112] S205: Send the query result to the user terminal according to the data result display method, so that the user terminal can display the query result.

[0113] Optionally, the data results are presented in forms including tables, charts, and natural language answers.

[0114] In summary, a structured data set is extracted from the metadata to be processed, and the structured data set is imported into the database. According to the data query text in the form of natural language description entered by the user, the text data to be parsed is obtained, and the text data to be parsed is analyzed by the big model to automatically obtain the semantic representation data, and then generate the structured query language. Using the structured query language, search in the pre-built database to obtain the query results. By combining the big model with the database, the ability of the database to process structured data is used to improve the accuracy of the data; the distributed computing ability of the big model is used to process large-scale structured data in parallel, significantly improving the processing speed. The user's query intention is converted into a database query statement, and the query results are quickly retrieved from the database, which omits the research and development work of the front-end application, improves the efficiency of processing the structured query language, and reduces the investment cost. In addition, the big model automatically completes data processing and generates structured query language without human intervention, further reducing the cost of human investment.

[0115] Optionally, before searching the pre-built database using structured query language in step S204 and obtaining the query result corresponding to the data query request, an access control mechanism is added, as described in detail as follows:

[0116] S301: Obtain the user identifier carried in the data query request.

[0117] S302: Obtain the user role according to the user identifier.

[0118] In this embodiment, users corresponding to different user IDs have different roles, such as administrator, common user, auditor, etc. Each role has different responsibilities and operation scopes. For example, the administrator role may have all operation permissions for all tables; common users may only be able to query data in some tables; auditors may only be able to view audit log tables.

[0119] S303: Determine whether the user has permission to execute structured query language according to the user role.

[0120] In this embodiment, in a pre-built database, access rights to various resources in the database are defined for different roles, including operations such as query, insert, update, and delete.

[0121] Specifically, the structured query language is checked according to the user role. The check content includes whether the user has the execution permission of the operation type and whether the user has the permission to access the tables and fields involved in the structured query language. If the user role has the execution permission of the operation type, it is further determined whether the user has the permission to access the tables and fields involved in the structured query language. If the user role has the access permission, the user has the permission to execute the structured query language.

[0122] If there is no execution permission for the operation type and no permission to access the tables and fields involved in the structured query language, the structured query language will be refused to execute and a prompt message indicating insufficient permissions will be returned to the user.

[0123] In summary, adding an access control mechanism can prevent unauthorized access and increase data security by judging whether a user has the authority to execute structured query language based on the user role.

[0124] Based on the above embodiment, in this embodiment, the optimization of the generated structured query language is introduced, which is described in detail as follows:

[0125] S401: Calculate query complexity of structured query language.

[0126] Specifically, the formula for calculating the query complexity of the structured query language is:

[0127]

[0128] In the formula, represents the query complexity of the structured query language, Indicates the number of tables involved in the structured query language query. Indicates the weight of the table; Indicates the number of fields involved in the structured query language query. Indicates the weight of the field; Indicates the number of conditions in a SQL query. Indicates the weight of the condition.

[0129] In this embodiment, query complexity is a quantitative value used to measure the difficulty and resource consumption of executing structured query language. The higher the complexity, the more computing resources and time are needed to complete the query.

[0130] In this embodiment, the more tables involved in the structured query language query, the more complex the table join operation that the database needs to perform when executing the query, which may cause query performance to deteriorate. For example, a structured query language needs to obtain data from two tables, then The more fields a structured query language query involves, the more data the database needs to read and process, which will increase the complexity of the query. For example, if the query needs to obtain three fields, then The more conditions involved in the structured query language query, the more judgments and comparisons the database needs to perform when filtering data, and the complexity of the query will increase accordingly. For example, the query condition is "age greater than 25 years old and department is Department B", there are two conditions here, .

[0131] In this embodiment, , and :Respectively, they are the weights of tables, fields, and conditions. These weights are adjusted according to specific business scenarios, reflecting the relative importance of the number of tables, fields, and conditions to the query complexity in that scenario. For example, in a business with high requirements for table association, The value of may be set to a larger value. In some scenarios where conditional filtering is the main focus, The weight may be higher.

[0132] S402: Optimizing the structured query language according to the query complexity to obtain an optimized structured query language.

[0133] In this embodiment, according to the query complexity, the structured query language can be optimized and a more efficient execution method can be selected, thereby improving data processing efficiency and reducing query execution time and resource consumption. For example, if it is found that the complexity of a certain structured query language is high and is mainly caused by too many tables involved, it can be considered to split or merge the tables to reduce the query complexity.

[0134] Optionally, the database design can also be optimized according to the query complexity, such as adjusting the table structure, creating appropriate indexes, etc.

[0135] In summary, query complexity can be used to evaluate the complexity of different structured query languages, thereby optimizing structured query languages ​​and improving data processing efficiency.

[0136] Optionally, after the query result is sent to the user end, the query result feedback information can be provided to the user end, as detailed below:

[0137] S501: receiving query result feedback information sent by a user terminal.

[0138] In this embodiment, the query result feedback information is cleaned to remove duplicate, invalid or incomplete query result feedback information. For example, some users' incorrectly filled or randomly filled content, as well as the same feedback information submitted by the same user multiple times, are screened and processed.

[0139] In this embodiment, the query result feedback information is classified according to different topics or types, such as performance issues and data accuracy. At the same time, each feedback is labeled for subsequent analysis and processing. For example, the query result feedback from the user is inaccurate and is labeled as the data accuracy category. The query speed feedback from the user is slow and is labeled as the performance problem category.

[0140] S502: Optimize the large model and / or the pre-built database according to the result feedback information.

[0141] In this embodiment, for the optimization of the large model, data related to the large model is sorted out according to the results marked in step S501. For example, if the user feedback shows that the answer of the large model in certain specific fields is inaccurate, the correct data and examples in the relevant fields can be collected to expand and optimize the training data set of the large model. Use the sorted feedback data to fine-tune the large model. Adjust the parameters of the large model so that it can better handle the questions and needs raised by users. For example, in response to the problems of natural language understanding reported by users, the semantic understanding part of the large model is fine-tuned. After using the feedback data to train the large model, an effect evaluation is performed. By comparing the performance of the large model on related tasks before and after training, it is determined whether the training is effective. For example, compare whether the accuracy and recall of the large model in handling common problems in user feedback have been improved.

[0142] In this embodiment, for the optimization of the database, the data related to the database is sorted out according to the results marked in step S501. For example, if the user feedbacks that the query speed is slow, it is necessary to analyze the index structure and query statements of the database to find out the reasons for the slow query. The structure of the database is optimized, such as adjusting the table structure, adding or deleting indexes, etc. For example, if it is found that the data redundancy of a table is high, the table can be split or merged to improve the efficiency of data storage and query.

[0143] In summary, by collecting query result feedback information sent by the user end and optimizing the big model and / or the pre-built database according to the query result feedback information, the performance of the big model and the pre-built database can be improved, thereby improving the user experience.

[0144] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method.

[0145] Figure 3 The schematic diagram of the structure of the large model-based structured data processing device provided in the embodiment of the present application. Figure 3 As shown, the embodiment of the present application also provides a structured data processing device based on a large model, including: a receiving module 301, an acquisition module 302, an input module 303, a retrieval module 304 and a sending module 305. The input module 303 includes a judgment unit 3031, a first acquisition unit 3032, an identification unit 3033, a second acquisition unit 3034 and a generation unit 3035.

[0146] The receiving module 301 is used to receive a data query request sent by a user terminal; wherein the data query request includes a data query text and a data result display method;

[0147] An acquisition module 302 is used to acquire text data to be parsed according to a data query text;

[0148] The input module 303 is used to input the text data request to be parsed into the large model, so that the large model performs the following steps:

[0149] Wherein, the input module 303 includes:

[0150] A judging unit 3031 is used to judge whether there is text data to be parsed in the cache;

[0151] A first acquisition unit 3032 is used to acquire semantic representation data according to the text data to be parsed if it is determined that the text data to be parsed does not exist in the cache;

[0152] The identification unit 3033 is used to identify key data required for generating structured query language according to the semantic representation data;

[0153] The second acquisition unit 3034 is used to perform structured conversion on the key data according to a preset template to obtain structured key data;

[0154] A generating unit 3035, configured to generate a structured query language corresponding to the semantic representation data according to the structured key data;

[0155] The search module 304 is used to search the pre-built database using structured query language to obtain the query result corresponding to the data query request;

[0156] The sending module 305 is used to send the query result to the user terminal according to the data result display method, so that the user terminal can display the query result.

[0157] In a possible implementation, the structured data processing device based on the large model further includes: a construction module. The construction module includes:

[0158] A collection unit, used for collecting metadata to be processed;

[0159] An extraction unit, used for extracting a structured data set from the metadata to be processed;

[0160] The creation unit is used to create an initial database and import the structured data set into the initial database to obtain a pre-built database.

[0161] In a possible implementation, the acquisition module 302 includes:

[0162] The deletion unit is used to delete the noise data in the data query text to obtain the initial text data to be parsed.

[0163] The segmentation unit is used to segment the initial text data to be parsed into multiple words.

[0164] The elimination unit is used to eliminate stop words from the multiple words to obtain the text data to be parsed.

[0165] In a possible implementation, the first acquisition unit 3032 acquires semantic representation data according to the text data to be parsed, and the formula is:

[0166]

[0167] In the formula, Representation semantics represents data, Represents the text data to be parsed. Represents the parameters of the large model; It indicates the probability of obtaining semantic representation data when the text data to be parsed and the parameters of the large model are input; Indicates the semantic representation data with the highest probability of selection.

[0168] In a possible implementation, the key data includes keywords, operation types, and query conditions;

[0169] Accordingly, the identification unit 3033 includes:

[0170] A word segmentation subunit, used to segment the semantic representation data to obtain multiple semantic representation words;

[0171] A tagging subunit, used for performing part-of-speech tagging on the plurality of semantic representation words to obtain a plurality of tagged semantic representation words;

[0172] The first recognition subunit is used to recognize keywords from a plurality of annotated semantic representation words according to pre-constructed database keywords.

[0173] A second identification subunit is used to identify the operation type from the multiple annotated semantic representation words according to the pre-built operation type list;

[0174] The third recognition subunit is used to recognize the query condition from the multiple annotated semantic representation words according to the pre-constructed query condition word library.

[0175] In a possible implementation, the generating unit 3035 generates a structured query language corresponding to the semantic representation data according to the structured key data, and the formula is:

[0176]

[0177] In the formula, stands for Structured Query Language, Representation semantics represents data, Represents structured key data, Indicates the parameters of the generated structured query language function. represents the probability of generating structured query language when the semantic representation data, structured key data and parameters of the function generating structured query language are input; It represents the structured query language with the highest selection probability.

[0178] In a possible implementation, the determining unit 3031 includes:

[0179] A hash calculation subunit is used to perform hash calculation on the text data to be parsed and obtain a hash identifier of the text data to be parsed;

[0180] A judgment subunit, used to judge whether a hash identifier exists in the cache;

[0181] The first determination subunit is used to determine that there is no text data to be parsed in the cache if there is no hash identifier in the cache.

[0182] In a possible implementation, the structured data processing device based on the large model further includes: a judgment module, the judgment module including:

[0183] A second determination subunit is used to determine that there is text data to be parsed in the cache if there is a hash identifier in the cache;

[0184] The acquisition subunit is used to obtain the structured query language corresponding to the hash identifier from the cache.

[0185] In a possible implementation, the structured data processing device based on the large model further includes: a first optimization module, the first optimization module includes:

[0186] A complexity calculation subunit, used to calculate the query complexity of the structured query language;

[0187] The first optimization subunit is used to optimize the structured query language according to the query complexity to obtain an optimized structured query language.

[0188] In a possible implementation, the complexity calculation subunit calculates the query complexity of the structured query language using the formula:

[0189]

[0190] In the formula, represents the query complexity of the structured query language, Indicates the number of tables involved in the structured query language query. Indicates the weight of the table; Indicates the number of fields involved in the structured query language query. Indicates the weight of the field; Indicates the number of conditions in a SQL query. Indicates the weight of the condition.

[0191] In a possible implementation, the search module 304 uses structured query language to search in a pre-built database to obtain a query result corresponding to the data query request, and the formula is:

[0192]

[0193] In the formula, Indicates the query result. Represents a pre-built database, stands for Structured Query Language, Represents a database execution function.

[0194] In a possible implementation, the structured data processing device based on the large model further includes: a second optimization module, the second optimization module includes:

[0195] A receiving subunit, used for receiving query result feedback information sent by the user end;

[0196] The second optimization subunit is used to optimize the large model and / or the pre-built database according to the result feedback information.

[0197] For the description of the features in the embodiments corresponding to the structured data processing device based on the big model, please refer to the relevant description of the embodiments corresponding to the structured data processing method based on the big model, which will not be repeated here.

[0198] Figure 4 This is a schematic diagram of the structure of the server provided in the embodiment of the present application. Figure 4 As shown, the server provided in this embodiment includes: at least one processor 401 and a memory 402. Optionally, the server also includes a communication component 403. The processor 401, the memory 402 and the communication component 403 are connected via a bus.

[0199] In the specific implementation process, at least one processor 401 executes the computer execution instructions stored in the memory 402, so that at least one processor 401 executes the above-mentioned large model-based structured data processing method embodiment.

[0200] The specific implementation process of the processor 401 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0201] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the application may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0202] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (NVM), such as at least one disk storage.

[0203] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of this application is not limited to only one bus or one type of bus.

[0204] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned large model-based structured data processing method embodiments when running.

[0205] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0206] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any one of the above-mentioned embodiments of the structured data processing method based on a large model are implemented.

[0207] An embodiment of the present application also provides another computer program product, including a non-volatile computer-readable storage medium, the non-volatile computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, implementing the steps in any of the above-mentioned large model-based structured data processing method embodiments.

[0208] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0209] The above is a detailed introduction to a structured data processing method based on a large model provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and its core idea of ​​the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A structured data processing method based on a large model, characterized in that: include: Receive a data query request sent by a user; The data query request includes a data query text and a data result display method; According to the data query text, obtain the text data to be parsed; The text data request to be parsed is input into the big model, so that the big model performs the following steps: Determine whether the text data to be parsed exists in the cache; If it is determined that the text data to be parsed does not exist in the cache, acquiring semantic representation data according to the text data to be parsed; identifying key data required to generate structured query language based on the semantic representation data; The key data is structured according to a preset template to obtain structured key data; Generating a structured query language corresponding to the semantic representation data according to the structured key data; Using the structured query language, searching in a pre-built database to obtain a query result corresponding to the data query request; The query result is sent to the user end according to the data result display method, so that the user end displays the query result.

2. The method according to claim 1, characterized in that: Before receiving the data query request sent by the user terminal, the method further includes: Collect metadata to be processed; extracting a structured data set from the metadata to be processed; An initial database is created, and the structured data set is imported into the initial database to obtain a pre-constructed database.

3. The method according to claim 1, characterized in that The step of obtaining the text data to be parsed according to the data query text includes: Deleting noise data in the data query text to obtain initial text data to be parsed; Segmenting the initial text data to be parsed into multiple words; Stop words are eliminated from the multiple words to obtain text data to be parsed.

4. The method according to claim 1, characterized in that: The formula for obtaining semantic representation data according to the text data to be parsed is: In the formula, Representation semantics represents data, represents the text data to be parsed, Represents the parameters of the large model; Indicates the probability of obtaining the semantic representation data when the text data to be parsed and the parameters of the large model are input; Indicates the semantic representation data with the highest probability of selection.

5. The method according to claim 1, characterized in that The key data includes keywords, operation types and query conditions; Accordingly, identifying key data required for generating structured query language according to the semantic representation data includes: Segmenting the semantic representation data to obtain a plurality of semantic representation words; Performing part-of-speech tagging on the multiple semantic representation words to obtain multiple tagged semantic representation words; According to the pre-constructed database keywords, identifying the keywords from the multiple annotated semantic representation words; According to a pre-built operation type list, identifying the operation type from the multiple annotated semantic representation words; The query condition is identified from the plurality of annotated semantic representation words according to a pre-constructed query condition word library.

6. The method according to claim 1, characterized in that The formula for generating the structured query language corresponding to the semantic representation data according to the structured key data is: In the formula, stands for Structured Query Language, represents the semantic representation data, represents the structured key data, Indicates the parameters of the generated structured query language function. Indicates the probability of generating the structured query language when the semantic representation data, the structured key data and the parameters of the function for generating the structured query language are input; It represents the structured query language with the highest selection probability.

7. The method according to claim 1, characterized in that The determining whether the to-be-parsed text data exists in the cache comprises: Performing hash calculation on the text data to be parsed to obtain a hash identifier of the text data to be parsed; Determine whether the hash identifier exists in the cache; If the hash identifier does not exist in the cache, it is determined that the text data to be parsed does not exist in the cache.

8. The method according to claim 7, characterized in that After determining whether the hash identifier exists in the cache, the method further includes: If the hash identifier exists in the cache, determining that the text data to be parsed exists in the cache; The structured query language corresponding to the hash identifier is obtained from the cache.

9. The method according to claim 1, characterized in that: After obtaining the query result corresponding to the data query request, the method further includes: Calculating the query complexity of the structured query language; The structured query language is optimized according to the query complexity to obtain an optimized structured query language.

10. The method according to claim 9, characterized in that The formula for calculating the query complexity of the structured query language is: In the formula, represents the query complexity of the structured query language, represents the number of tables involved in the structured query language query, Indicates the weight of the table; represents the number of fields involved in the structured query language query, Indicates the weight of the field; represents the number of conditions in the structured query language query, Indicates the weight of the condition.

11. The method according to any one of claims 1 to 10, characterized in that: The formula for using the structured query language to search in a pre-built database to obtain the query result corresponding to the data query request is: In the formula, Indicates the query result. Represents a pre-built database, represents the structured query language, Represents a database execution function.

12. The method according to any one of claims 1 to 10, characterized in that: After sending the query result to the user terminal according to the data result display method so that the user terminal displays the query result, the method further includes: Receiving query result feedback information sent by the user terminal; The large model and / or the pre-built database are optimized according to the result feedback information.

13. A server, characterized in that: include: Memory for storing computer programs; A processor, used to implement the steps of the large model-based structured data processing method as described in any one of claims 1 to 12 when executing the computer program.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the large model-based structured data processing method as described in any one of claims 1 to 12.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the large model-based structured data processing method as described in any one of claims 1 to 12 are implemented.

Images