Method for converting structured data into unstructured text to realize knowledge base based on AI
By obtaining structured data and generating unstructured description statements, establishing association relationships and vector formats, the problem of structured data conversion in the existing technology is solved, efficient and accurate data understanding and real-time updates are achieved, and traditional application systems are enhanced. NLP capabilities are in line with the trends of the times and reduce investment costs.
Patent Information
- Application Number
- CN202411216017.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-02
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-09-02
AI Technical Summary
The prior art is difficult to convert structured data into unstructured data, and it is impossible to understand the correspondence and numerical relationships between data. It requires a large amount of data to retrain the model and the effect is not good.
通过获取结构化数据,预处理并生成非结构化描述语句,创建唯一标识符,建立关联关系,将数据转换为向量格式,使用大模型问答知识库进行计算,实时更新非结构化文本,实现结构化数据的转换和更新。
It realizes efficient and accurate structured data understanding and conversion, detects data changes in real time, automatically updates the knowledge base, enhances the NLP semantic understanding function of traditional application systems, quickly follows up on technological development and reduces investment costs.
Smart Images

Figure CN118733711B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing technology, and specifically to a method for converting structured data into unstructured text to realize a knowledge base based on AI. Background Art
[0002] Existing technologies usually convert unstructured data into structured data, usually to solve the retrieval, transmission and storage problems of unstructured data, but do not solve the problem of converting structured data into unstructured data. For structured data, it is impossible to understand the correspondence and numerical relationship between structured data. It is necessary to analyze and extract structured files, and the model needs to be retrained and fine-tuned. The amount of data required is large, and the model effect cannot be guaranteed. Summary of the invention
[0003] The purpose of the present invention is to provide a method for converting structured data into unstructured text to realize a knowledge base based on AI, so as to solve the problems raised in the above background technology.
[0004] To achieve the above object, the present invention provides the following technical solution: A method for converting structured data into unstructured text to realize a knowledge base based on AI, comprising the following steps:
[0005] Step S1: obtaining data in a structured format in a production system and preprocessing it, and further obtaining the header, field, and the relationship between the header, field, and each record information in the structured data;
[0006] Step S2: according to the relationship between the header, the field and each record information, an unstructured description statement is set, and each record in the structured data is generated into an unstructured description text according to the unstructured description statement format;
[0007] Step S3: Create a unique identifier for each unstructured description text and create an association relationship with the data in the production system;
[0008] Step S4: Create a large model question-answering knowledge base, process the unstructured description text with the association relationship established in step S3 into the vector format in the knowledge base, process the question asked by the large model into a vector, and calculate it with the text vector in the knowledge base, match the vector to the corresponding unstructured text based on the similarity between the two, and input the question and unstructured text into the large model to obtain the answer;
[0009] Step S5: If the record data in the structured data in the production system changes, the corresponding unstructured text will be updated and re-imported into the large model knowledge base to update the current record data; if the header or field of the structured data changes, the description text of all records will be constructed and updated to the large model knowledge base; if the statistical data content or statistical values in the system change, all automatically generated statistical data can be synchronized to the AI knowledge base in real time or by setting corresponding rules, and the original text will be overwritten to form a new knowledge base.
[0010] Preferably, the structured data includes tabular data and database tables; when the collected data is a scanned copy, the specific steps for obtaining the tabular data in the scanned copy are: first identify the position of the table line, then extract the position of each cell in the table and identify the text information in each cell, and output it according to the table structure; the relationship between the field and each record information refers to the correspondence between the table header and the field and the data record.
[0011] Preferably, the specific steps of preprocessing the data in the production system in step S1 are as follows:
[0012] Step S11: respectively obtain the table header and the field information in the database table;
[0013] Step S12: Arrange the header and field information in the first row, and arrange each data record in rows to generate a unified format;
[0014] Step S13: Delete the duplicate data in the data table with the unified format generated above, filter the table data with the unified format, obtain the primary key value of each table and sort them according to the primary key value.
[0015] Preferably, the specific process of generating unstructured description text according to the description statement format for each record in the structured data in step S2 is as follows:
[0016] Step S21: combining multiple fields into a descriptive text according to the relationship between the header, the field and each record information;
[0017] Step S22: using large language model generation technology to process the combined text into natural sentences or paragraphs, and maintaining the field information of the original table;
[0018] Step S23: extracting information of each column field in the table, and statistically analyzing the data in each column field;
[0019] Step S24: Process different types of data. If the data is of text type, the number of occurrences of each text is counted after deduplication. If the data is of numerical type, information about the field data is obtained, including the sum, mean, standard deviation, and median of the field data.
[0020] Step S25: Combine the processed data into descriptive text.
[0021] Preferably, in step S23, the length of each text segment does not exceed a set threshold, and the vector of text content that is less than the set length is filled with the value 0.
[0022] Preferably, the overall descriptive text file of the current table file generated in step S2 is named according to the table name.
[0023] Preferably, the specific steps of processing the unstructured description text into a knowledge base in step S4 are as follows:
[0024] Step S41: Obtain and read the descriptive text file generated by each table;
[0025] Step S42: automatically segmenting the file using a text vectorization model, and vectorizing the segmented text;
[0026] Step S43: Match the text vector generated by each table with the table file name, save them in the vector file library respectively, and generate a knowledge base of the text vector, the corresponding file name, and the content.
[0027] Preferably, the specific steps of processing the questions asked by the large model in step S4 into vectors are as follows:
[0028] Step S51: obtaining questions about the table content to be understood, generating a question vector of the table, wherein zeros are padded according to a set threshold length;
[0029] Step S52: Calculate the similarity between the question vector of the table and the knowledge base text vector;
[0030] Step S53: obtaining the most similar text vector, and obtaining the text content corresponding to the most similar text vector;
[0031] In step S54: the text content in the above step S53 and the question in step S51 are input into the large language model.
[0032] Compared with the prior art, the present invention has the following beneficial effects:
[0033] (1) The present invention innovates an efficient and accurate method for understanding data and table content. Users can convert structured data into unstructured text through simple statement logic settings;
[0034] (2) The present invention detects data changes in real time through rules generated by unstructured sentences and converts the changed data into unstructured text, and overwrites the unstructured text before the change, thereby realizing automatic updating of the knowledge base text;
[0035] (3) The present invention converts databases into unstructured texts and forms into unstructured texts, so that traditional application systems and table files can quickly add NLP semantic understanding functions and realize AI artificial intelligence, so that users' traditional application systems can quickly keep up with the trend, which is both in line with the times and reduces investment. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] Figure 1 is a flow chart of the present invention; DETAILED DESCRIPTION
[0037] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. Example
[0038] Combination Figure 1 As shown, the present invention provides a method for converting structured data into unstructured text to realize a knowledge base based on AI, comprising the following steps:
[0039] Step S1: Obtain and preprocess the data in the structured format in the production system. The structured data includes tabular data and database tables. The specific process of preprocessing the data is as follows: Step S11: Obtain the table header and field information in the database table respectively; Step S12: Arrange the table header and field information in the first row, and arrange each data record in rows to generate a unified format; Step S13: Delete the duplicate data in the data table generated in the unified format, filter the tabular data with a unified format, obtain the primary key value of each table and sort them according to the primary key value; After preprocessing the data, further obtain the table header, field, and the relationship between the table header, field and each record information in the structured data, wherein the relationship between the field and each record information refers to the corresponding relationship between the table header and the field. For example: if a column of data in a row of data is a number, if you do not look at the table header or field, you will not know the meaning of the number. If you know that the column of data described in the table header represents salary, then this column of data is salary data;
[0040] Step S2: According to the relationship between the table header, fields and each record information, set the unstructured description statement, and generate the unstructured description text for each record in the structured data according to the unstructured description statement format, wherein the unstructured description statement refers to describing a record in a certain table by using the table header plus text. Specifically, the process of generating the unstructured description text for each record in the structured data according to the description statement format is as follows: S21: According to the relationship between the table header, fields and each record information, combine multiple fields into a descriptive text; S22: Use the large language model generation technology to process the above combined text into natural sentences or paragraphs, and keep the original table field information; S23: extract the information of each column field in the table, statistically analyze the data in each column field, wherein the length of each paragraph of text in step S23 does not exceed the set threshold, and the vector of text content that is less than the set length is filled with the value 0; S24: process different types of data, if the data is of text type, then count the number of occurrences of each text after deduplication, if the data is of numerical type, obtain information about the field data, the field data information includes the sum, mean, standard deviation, and median of the field data; S25: combine the processed data into a descriptive text, and the overall descriptive text file of the current table file generated at the same time is named according to the table name. For example, if it is a salary table, the header and the text corresponding to the header need to be manually matched, and a text description is used, describing that the basic salary for month xx is xx yuan, deducting social security and provident fund xx yuan, and deducting tax xx yuan, then the actual salary received is xx yuan;
[0041] Step S3: Create a unique identifier for each unstructured description text and create an association relationship with the data in the production system;
[0042] Step S4: Create a large model question-and-answer knowledge base, process the unstructured description text with the association established in step S3 into the vector format in the knowledge base, process the questions asked by the large model into vectors, and calculate them with the text vectors in the knowledge base, match the vector to the corresponding unstructured text according to the similarity between the two, and input the large model questions according to the questions and unstructured text to obtain answers, wherein the specific steps of processing the unstructured description text into the knowledge base are as follows: S41: Obtain and read the descriptive text file generated by each table; S42: Use the text vectorization model to automatically segment the file, and vectorize the segmented text; S43: Match the text vector generated by each table with the table file name, and save them in the vector file library respectively, to generate a knowledge base of text vectors, corresponding file names, and contents;
[0043] S5: When the record data in the structured data in the production system changes, the corresponding unstructured text is updated and re-imported into the large model knowledge base to update the current record data, and the corresponding knowledge base is updated; if the header or field of the structured data changes, all the description texts of the records are constructed and updated in the large model knowledge base, and the knowledge base is updated synchronously; if the statistical data content or the statistical value of the system changes, the automatically generated statistical data can be synchronized to the AI knowledge base in real time or by setting corresponding rules, and the original text is overwritten to form a new knowledge base. The specific steps for processing the questions asked by the large model into vectors are as follows: S51: Obtain questions about the table content to be understood, generate a question vector for the table, wherein 0 is padded according to a set threshold length; S52: calculate the similarity between the question vector for the table and the text vector for the knowledge base; S53: obtain the most similar text vector, and obtain the text content corresponding to the most similar text vector; in S54: input the text content in the above step S53 and the question in step S51 into a large language model, so that the database can be converted into unstructured text, and the form can be converted into unstructured text, so that the traditional application system and table file can quickly increase the NLP semantic understanding function, realize AI artificial intelligence, and enable the user's traditional application system to quickly keep up with the trend, which is both in line with the sense of the times and reduces investment.
[0044] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments are to be regarded as exemplary and non-limiting in all respects, and the scope of the invention is to be defined by the appended claims rather than by the foregoing description, and it is intended that all variations falling within the meaning and range of equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be regarded as limiting the claim to which it relates.
Claims
1. Methods for converting structured data into unstructured text to realize knowledge base based on AI, It is characterized in that The following steps are involved: Step S1: obtain the data in the structured format in the production system and pre-process it, and further obtain the header, field, and the relationship between the header, field and each record information in the structured data; the specific steps of pre-processing the data in the production system are as follows: Step S11: obtain the table header and field information in the database table respectively; Step S12: arrange the header and field information in the first row, and arrange each data record in rows to generate a unified format; Step S13: delete the duplicate data in the data table with the unified format generated above, filter the table data with a unified format, obtain the primary key value of each table and sort according to the primary key value; Step S2: according to the relationship between the header, the field and each record information, an unstructured description statement is set, and each record in the structured data is generated into an unstructured description text according to the unstructured description statement; The specific process of generating unstructured description text according to unstructured description statements for each record in structured data is as follows: Step S21: Combine multiple fields into a descriptive text according to the relationship between the header, the field and each record information; Step S22: Use the large language model generation technology to process the above combined text into a natural sentence or paragraph, and keep the field information of the original table; Step S23: Extract the information of each column field in the table, and statistically analyze the data in each column field; Step S24: Process different types of data. If the data is of text type, count the number of occurrences of each text after deduplication. If the data is of numerical type, obtain information about the field data, which includes the sum, mean, standard deviation and median of the field data; Step S25: Combine the processed data into a descriptive text; Step S3: Create a unique identifier for each unstructured description text and create an association relationship with the data in the production system; Step S4: Create a large model question-answering knowledge base, process the unstructured description text with the association relationship established in step S3 into the vector format in the knowledge base, process the question asked by the large model into a vector, and calculate it with the text vector in the knowledge base, match the vector to the corresponding unstructured text based on the similarity between the two, and input the question and unstructured text into the large model to obtain the answer; Step S5: If the record data in the structured data in the production system changes, the corresponding unstructured text is updated and re-imported into the large model knowledge base to update the current record data; If the header or field of structured data changes, all the description texts of the records are constructed and updated in the large model knowledge base; If the content or value of statistical data in the system changes, the corresponding rules can be set in real time to synchronize the text of all automatically generated statistical data to the AI knowledge base, and overwrite the original text to form a new knowledge base.
2. According to claim 1, the method for converting structured data into unstructured text based on AI to realize a knowledge base, Features: The structured data includes tabular data and database tables; when the collected data is a scanned copy, the specific steps for obtaining the tabular data in the scanned copy are: first identify the position of the table line, then extract the position of each cell in the table and identify the text information in each cell, and output it according to the table structure; the relationship between the field and each record information refers to the correspondence between the table header and the field and the data record.
3. According to the method for converting structured data into unstructured text to realize knowledge base based on AI according to claim 1, Features: In step S23, the length of each text segment does not exceed a set threshold, and the vector of text content that is less than the set length is filled with a value of 0.
4. According to claim 1, the method for converting structured data into unstructured text based on AI to realize a knowledge base, Features: The overall descriptive text file of the current table file generated in step S2 is named according to the table name.
5. According to claim 1, the method for converting structured data into unstructured text based on AI to realize a knowledge base, It is characterized in that The specific steps of processing the unstructured description text into a knowledge base in step S4 are as follows: Step S41: Obtain and read the descriptive text file generated by each table; Step S42: automatically segmenting the file using a text vectorization model, and vectorizing the segmented text; Step S43: Match the text vector generated by each table with the table file name, save them in the vector file library respectively, and generate a knowledge base of the text vector, the corresponding file name and content.
6. According to claim 1, the method for converting structured data into unstructured text based on AI to realize a knowledge base, It is characterized in that The specific steps of processing the questions asked by the large model into vectors in step S5 are as follows: Step S51: obtaining questions about the table content to be understood, generating a question vector of the table, wherein zeros are padded according to a set threshold length; Step S52: Calculate the similarity between the question vector of the table and the knowledge base text vector; Step S53: obtaining the most similar text vector, and obtaining the text content corresponding to the most similar text vector; In step S54: the text content in the above step S53 and the question in step S51 are input into the large language model.
