Structured data conversion method based on RAG

Abstract

The invention provides an RAG-based structured data conversion method in the technical field of artificial intelligence. The method comprises the steps of S1, creating a question template configuration library, a cue word template configuration library and a code association configuration library; s2, simplifying the natural language query statement through NLP to obtain a simplified query statement; s3, performing intention recognition on the simplified query statement to obtain an intention recognition result, and matching a question category of the simplified query statement based on the intention recognition result; step S4, matching a question template from a question template configuration library on the basis of the simplified query statement through an RAG technology, and matching a cue word template from a cue word template configuration library on the basis of a question category; s5, generating cue words based on the question template and the cue word template, and inputting the cue words into the created large model to obtain conversion data; and S6, encoding the converted data through the encoding association configuration library to obtain structured data. The method has the advantages that the accuracy, expansibility and efficiency of structured data conversion are greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a structured data conversion method based on RAG. Background Art

[0002] With the rapid development of artificial intelligence, artificial intelligence has been widely used in various fields. One of the applications is to use natural language to directly perform corresponding tasks, making the interaction more concise. This process requires converting unstructured natural language (text data) into structured data. Structured data is easier to search, analyze and store, which is crucial for application scenarios such as intelligent questioning, data analysis, and automated processing. However, the existing structured data conversion methods have the following problems:

[0003] 1. Complex text data may contain implicit information, ambiguous information, professional terms, and non-standard terms. The traditional methods have low conversion accuracy when facing such text data. 2. When facing new or specific field text data conversion into structured data, it is necessary to re-develop or perform a large amount of parameter fine-tuning and training on the model created based on the neural network, which is time-consuming and resource-consuming. 3. Although the large model used for structured data conversion performs well in processing complex text data, a large amount of parameter fine-tuning or prompt word testing will face performance bottlenecks such as high computing resource consumption and slow generation speed, which in turn affects the efficiency of structured data conversion.

[0004] Therefore, how to provide a RAG-based structured data conversion method to improve the accuracy, scalability and efficiency of structured data conversion has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a structured data conversion method based on RAG, so as to improve the accuracy, scalability and efficiency of structured data conversion.

[0006] The present invention is implemented as follows: a structured data conversion method based on RAG, comprising the following steps:

[0007] Step S1, creating a question template configuration library, a prompt word template configuration library and a code association configuration library;

[0008] Step S2: obtaining an input natural language query statement, and simplifying the natural language query statement by using NLP technology to obtain a simplified query statement;

[0009] Step S3: performing intent recognition on the simplified query statement through a TextCNN model to obtain an intent recognition result, and matching the question category of the simplified query statement based on the intent recognition result;

[0010] Step S4: using RAG technology, matching a question template from a question template configuration library based on the simplified query statement, and matching a prompt word template from a prompt word template configuration library based on the question category;

[0011] Step S5: generating corresponding prompt words based on the question template and the prompt word template;

[0012] Step S6, creating a large model for structured data conversion, setting the output format of the large model, and inputting each of the prompt words into the large model to obtain conversion data;

[0013] Step S7: Encode the converted data through the encoding association configuration library to obtain structured data.

[0014] Furthermore, in step S1, the question template configuration library, the prompt word template configuration library and the encoding association configuration library are all vector databases.

[0015] Furthermore, in step S1, the question template configuration library is used to store question templates, and the question templates are the corresponding relationships between questions and answers;

[0016] The prompt word template configuration library is used to store prompt word templates, and the prompt word template is a prompt word output format;

[0017] The encoding association configuration library is used to store encoding rules, and the encoding conversion and mapping are performed on the conversion data according to the encoding rules.

[0018] Furthermore, the step S2 is specifically as follows:

[0019] A natural language query statement input by a user is obtained through a visual interface, indicator entities in the natural language query statement are identified through NLP technology, and the indicator entities in the natural language query statement are replaced by pre-set and associated replacement words, thereby simplifying the natural language query statement to obtain a simplified query statement.

[0020] Furthermore, the step S3 is specifically as follows:

[0021] The simplified query statement is feature extracted by a pre-trained TextCNN model to obtain text features, intent recognition is performed based on each of the text features to obtain an intent recognition result, and the question category of the simplified query statement is matched from a preset question category configuration library based on the intent recognition result.

[0022] Furthermore, the step S4 is specifically as follows:

[0023] Through the RAG technology, the most relevant question template is traversed and matched from the question template configuration library based on the simplified query statement, and the most relevant prompt word template is traversed and matched from the prompt word template configuration library based on the question category.

[0024] Furthermore, the step S5 is specifically as follows:

[0025] The question template and the prompt word template are pieced together to generate corresponding prompt words.

[0026] Furthermore, the step S6 is specifically as follows:

[0027] A large model for structured data conversion is created based on a deep neural network, an output format of the large model is set, and after pre-training the large model, each of the prompt words is input into the large model to obtain conversion data.

[0028] Furthermore, the step S7 is specifically as follows:

[0029] The characters in the conversion data are mapped through the encoding association configuration library, and then the conversion data is encoded and converted to obtain structured data, and the structured data is displayed in real time through a visual interface.

[0030] The advantages of the present invention are:

[0031] By creating a question template configuration library, a prompt word template configuration library and an encoding association configuration library, the acquired natural language query statements are simplified through NLP technology to obtain simplified query statements, the simplified query statements are subjected to intent recognition through the TextCNN model to obtain intent recognition results, and the question categories of the simplified query statements are matched based on the intent recognition results; then, through RAG technology, the question templates are matched from the question template configuration library based on the simplified query statements, the prompt word templates are matched from the prompt word template configuration library based on the question categories, and the corresponding prompt words are generated based on the question templates and the prompt word templates; then, a large model for structured data conversion is created, the output format of the large model is set, and each prompt word is input into the large model to obtain conversion data, and finally, the conversion data is encoded through the encoding association configuration library to obtain structured data; since the RAG technology combines information related to the current context in the retrieval process , effectively improving the accuracy of matching. The big model has the ability to understand and generate complex contexts. The prompt words can guide the big model to generate more accurate conversion data. By combining RAG technology, big models and prompt words, the accuracy of structured data conversion is greatly improved; when facing new or field-specific text data (natural language query statements), only the question template configuration library and the prompt word template configuration library need to be adjusted to generate matching prompt words for structured data conversion, without the need to retrain and fine-tune the parameters of the big model, thereby greatly improving the scalability of structured data conversion; by adjusting the question template configuration library and the prompt word template configuration library to optimize the prompt words, the retraining and parameter fine-tuning of the big model are avoided, and the number of iterations in the process of structured data generation is reduced. Combined with the simplification of natural language query statements by NLP technology, the efficiency of structured data conversion is greatly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The present invention will be further described below in conjunction with embodiments with reference to the accompanying drawings.

[0033] Figure 1 It is a flow chart of a structured data conversion method based on RAG of the present invention. DETAILED DESCRIPTION

[0034] The technical solution in the embodiments of the present application has the following overall ideas: by combining RAG technology, large models and prompt words, the accuracy of structured data conversion is improved; by adjusting the question template configuration library and the prompt word template configuration library to optimize the prompt words, there is no need to retrain the large model and fine-tune the parameters, so as to improve the scalability of structured data conversion; by adjusting the question template configuration library and the prompt word template configuration library to optimize the prompt words, there is no need to retrain the large model and fine-tune the parameters, and the number of iterations in the process of structured data generation is reduced, and the simplification of natural language query statements by combining NLP technology is used to improve the efficiency of structured data conversion.

[0035] Please refer to Figure 1 As shown, a preferred embodiment of a structured data conversion method based on RAG of the present invention comprises the following steps:

[0036] Step S1, creating a question template configuration library, a prompt word template configuration library and a code association configuration library;

[0037] Step S2: obtaining an input natural language query statement, and simplifying the natural language query statement by using NLP technology to obtain a simplified query statement;

[0038] Step S3: performing intent recognition on the simplified query statement through a TextCNN model to obtain an intent recognition result, and matching the question category of the simplified query statement based on the intent recognition result;

[0039] Step S4: using RAG technology, matching a question template from a question template configuration library based on the simplified query statement, and matching a prompt word template from a prompt word template configuration library based on the question category;

[0040] Step S5: generating corresponding prompt words based on the question template and the prompt word template;

[0041] Step S6, creating a large model for structured data conversion, setting the output format of the large model, and inputting each of the prompt words into the large model to obtain conversion data;

[0042] Step S7: Encode the converted data through the encoding association configuration library to obtain structured data.

[0043] During specific implementation, the large model is continuously optimized through the structured data.

[0044] In the step S1, the question template configuration library, the prompt word template configuration library and the encoding association configuration library are all vector databases, which can ensure the efficiency of retrieval and matching.

[0045] In step S1, the question template configuration library is used to store question templates, which are corresponding relationships between questions and answers, so as to facilitate rapid extraction of question templates in different scenarios to generate prompt words, thereby improving the accuracy of prompt word generation;

[0046] The prompt word template configuration library is used to store prompt word templates, and the prompt word template is a prompt word output format;

[0047] The coding association configuration library is used to store coding rules, and the conversion data is coded and mapped according to the coding rules to ensure the consistency and accuracy of the data. For example, the coding rules are set as: T: statistical time, C: city, P: province, time format corresponding to: Y: yyyy, M: yyyyMM, D: yyyyMMdd.

[0048] The step S2 is specifically as follows:

[0049] A natural language query statement input by a user is obtained through a visual interface, indicator entities in the natural language query statement are identified through NLP technology, and the indicator entities in the natural language query statement are replaced by pre-set and associated replacement words, thereby simplifying the natural language query statement to obtain a simplified query statement.

[0050] The step S3 is specifically as follows:

[0051] The simplified query statement is feature extracted by the pre-trained TextCNN model to obtain text features, and the intent recognition is performed based on each of the text features to obtain the intent recognition result, and the problem category of the simplified query statement is matched from the preset problem category configuration library based on the intent recognition result. For example, the simplified query statement "Find the 5G traffic revenue and total traffic revenue of ** City in March 23" belongs to the problem category of intelligent question category A.

[0052] The TextCNN model has high accuracy and efficiency in handling simple natural language processing tasks.

[0053] The step S4 is specifically as follows:

[0054] Through RAG (Retrieval Augmented Generation) technology, the most relevant question template is traversed and matched from the question template configuration library based on the simplified query statement, that is, the most relevant sample is matched, and the most relevant prompt word template is traversed and matched from the prompt word template configuration library based on the question category.

[0055] RAG technology is a natural language processing method that combines information retrieval and text generation. It assists text generation by retrieving information related to the current context, thereby improving the accuracy and relevance of the generated text. A vector database is a database specifically used to store and query vector data. It supports efficient multi-dimensional vector queries and is often used to store word vectors and sentence vectors.

[0056] By combining RAG technology and vector database, it is possible to quickly and accurately retrieve text fragments that are most similar to the current context, thereby generating richer and more accurate text, and quickly extract and identify privatized indicator knowledge in specific fields.

[0057] The step S5 is specifically as follows:

[0058] The question template and the prompt word template are pieced together to generate corresponding prompt words.

[0059] For example, if the simplified query statement is "query indicator 1 and indicator 2 of ** city in March 23", the relevant question template is:

[0060] 1. Find the index 1 and index 2 of ** city in May 2023#result: T>=20230501&D@T<=20230531&D@C=** city;

[0061] 2. Query the index 1 and index 2 of ** city on March 2, 22#result: T>=20220302&M@C=** city;

[0062] Combine the question categories of the intelligent questioning to find the corresponding prompt word template to generate a prompt word input model.

[0063] The step S6 is specifically as follows:

[0064] A large model for structured data conversion is created based on a deep neural network, an output format of the large model is set, and after pre-training the large model, each of the prompt words is input into the large model to obtain conversion data.

[0065] For example, the output format is set to use separators (such as #, @, &, ~) to separate keywords of different dimensions or different meanings, and the levels of the separators are distinguished.

[0066] Assume that the output of the large model is:

[0067] A#T>=20230301&D@T<=20230331&D@C=** City

[0068] First split “#” to obtain A and T>=20230301&D@T<=20230331&D@C=** City, where A represents a type A problem; split “@” again to obtain T>=20230301&D, T<=20230331&D, and C=** City, and get three conditional values; finally split “&” to obtain T>=20230301 and D, T<=20230331 and D, and get the format information under the time condition.

[0069] Natural language query statements are simplified through NLP technology, and conversion data is generated based on prompt words, that is, the input and output are simplified, so that the number of tokens input and output of the large model is greatly reduced, thereby improving the response performance of the large model and reducing the consumption of computing power.

[0070] For example, if the input natural language query is "query the proportion of 5G traffic revenue and total traffic revenue in ** city in March 2023", NLP technology is used to identify indicator entities (5G traffic revenue, total traffic revenue) and replace them with indicator 1 and indicator 2 respectively, so we simplify the natural language query to: "query indicator 1 and indicator 2 of ** city in March 2023". The conversion data output by the big model is "A#T>=20230301&D@T<=20230331&D@C=** city".

[0071] The step S7 is specifically as follows:

[0072] The characters in the conversion data are mapped through the encoding association configuration library, and then the conversion data is encoded and converted to obtain structured data, and the structured data is displayed in real time through a visual interface.

[0073] For example, "T>=20230301&D@T<=20230331&D@C=** City" is encoded through the encoding association configuration library and converted into structured data:

[0074] {

[0075] Category: Class A

[0076] Condition 1: Statistical time >= 20240301, time format: yyyyMMdd

[0077] Condition 2: Statistical time <= 20240331, time format: yyyyMMdd

[0078] Condition 3: City = ** City

[0079] }

[0080] In summary, the advantages of the present invention are:

[0081] By creating a question template configuration library, a prompt word template configuration library and an encoding association configuration library, the acquired natural language query statements are simplified through NLP technology to obtain simplified query statements, the simplified query statements are subjected to intent recognition through the TextCNN model to obtain intent recognition results, and the question categories of the simplified query statements are matched based on the intent recognition results; then, through RAG technology, the question templates are matched from the question template configuration library based on the simplified query statements, the prompt word templates are matched from the prompt word template configuration library based on the question categories, and the corresponding prompt words are generated based on the question templates and the prompt word templates; then, a large model for structured data conversion is created, the output format of the large model is set, and each prompt word is input into the large model to obtain conversion data, and finally, the conversion data is encoded through the encoding association configuration library to obtain structured data; since the RAG technology combines information related to the current context in the retrieval process , effectively improving the accuracy of matching. The big model has the ability to understand and generate complex contexts. The prompt words can guide the big model to generate more accurate conversion data. By combining RAG technology, big models and prompt words, the accuracy of structured data conversion is greatly improved; when facing new or field-specific text data (natural language query statements), only the question template configuration library and the prompt word template configuration library need to be adjusted to generate matching prompt words for structured data conversion, without the need to retrain and fine-tune the parameters of the big model, thereby greatly improving the scalability of structured data conversion; by adjusting the question template configuration library and the prompt word template configuration library to optimize the prompt words, the retraining and parameter fine-tuning of the big model are avoided, and the number of iterations in the process of structured data generation is reduced. Combined with the simplification of natural language query statements by NLP technology, the efficiency of structured data conversion is greatly improved.

[0082] Although the specific implementation modes of the present invention are described above, those skilled in the art should understand that the specific implementation modes described are only illustrative and are not intended to limit the scope of the present invention. Equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. A structured data conversion method based on RAG, characterized in that: The steps include: Step S1, creating a question template configuration library, a prompt word template configuration library and a code association configuration library; Step S2: obtaining an input natural language query statement, and simplifying the natural language query statement by using NLP technology to obtain a simplified query statement; Step S3: performing intent recognition on the simplified query statement through a TextCNN model to obtain an intent recognition result, and matching the question category of the simplified query statement based on the intent recognition result; Step S4: using RAG technology, matching a question template from a question template configuration library based on the simplified query statement, and matching a prompt word template from a prompt word template configuration library based on the question category; Step S5: generating corresponding prompt words based on the question template and the prompt word template; Step S6, creating a large model for structured data conversion, setting the output format of the large model, and inputting each of the prompt words into the large model to obtain conversion data; Step S7: Encode the converted data through the encoding association configuration library to obtain structured data.

2. The RAG-based structured data conversion method according to claim 1, characterized in that: In the step S1, the question template configuration library, the prompt word template configuration library and the encoding association configuration library are all vector databases.

3. The RAG-based structured data conversion method according to claim 1, characterized in that: In the step S1, the question template configuration library is used to store question templates, and the question template is a correspondence between questions and answers; The prompt word template configuration library is used to store prompt word templates, and the prompt word template is a prompt word output format; The encoding association configuration library is used to store encoding rules, and the encoding conversion and mapping are performed on the conversion data according to the encoding rules.

4. The RAG-based structured data conversion method according to claim 1, characterized in that: The step S2 is specifically as follows: A natural language query statement input by a user is obtained through a visual interface, indicator entities in the natural language query statement are identified through NLP technology, and the indicator entities in the natural language query statement are replaced by pre-set and associated replacement words, thereby simplifying the natural language query statement to obtain a simplified query statement.

5. The RAG-based structured data conversion method according to claim 1, characterized in that: The step S3 is specifically as follows: The simplified query statement is feature extracted by a pre-trained TextCNN model to obtain text features, intent recognition is performed based on each of the text features to obtain an intent recognition result, and the question category of the simplified query statement is matched from a preset question category configuration library based on the intent recognition result.

6. The RAG-based structured data conversion method according to claim 1, characterized in that: The step S4 is specifically as follows: Through the RAG technology, the most relevant question template is traversed and matched from the question template configuration library based on the simplified query statement, and the most relevant prompt word template is traversed and matched from the prompt word template configuration library based on the question category.

7. The RAG-based structured data conversion method according to claim 1, characterized in that: The step S5 is specifically as follows: The question template and the prompt word template are pieced together to generate corresponding prompt words.

8. The RAG-based structured data conversion method according to claim 1, characterized in that: The step S6 is specifically as follows: A large model for structured data conversion is created based on a deep neural network, an output format of the large model is set, and after pre-training the large model, each of the prompt words is input into the large model to obtain conversion data.

9. The RAG-based structured data conversion method according to claim 1, characterized in that: The step S7 is specifically as follows: The characters in the conversion data are mapped through the encoding association configuration library, and then the conversion data is encoded and converted to obtain structured data, and the structured data is displayed in real time through a visual interface.

Images