RAG-based universal structured acquisition method and system for unstructured data

Abstract

The invention relates to the technical field of artificial intelligence and data engineering, in particular to an unstructured data universal structured acquisition method and system based on RAG. The method comprises the following steps: firstly, constructing a knowledge base for storing a table structure and field information of a service table; carrying out index identification and extraction on the input unstructured text by utilizing the large model; secondly, searching and matching a plurality of most similar business fields in a knowledge base aiming at each extracted index by utilizing a search enhancement generation technology; based on the input unstructured text and the most similar business field information of each index, assisting a large language model to elect a most matched business field of the index in a specific business scene; and finally, storing the index value of the index in a knowledge base in combination with the most matched business field. The universal data acquisition method for the unstructured data is suitable for index extraction universal for all service lines in the field, can automatically align and screen service fields in a specific service scene and store the service fields in the specific service scene, and realizes the universal data acquisition method for the unstructured data.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and data engineering technology, and in particular to a general structured collection method and system for unstructured data based on RAG. Background Art

[0002] We have entered the data age, where data can be categorized into structured, semi-structured, and unstructured data. Unstructured data is complex and difficult to manage, and it's prone to data silos. However, unstructured data (such as financial reports, research report summaries, and announcements) often contains a wealth of key indicators (e.g., net profit, debt-to-asset ratio, and risk exposure).

[0003] Metric extraction can transform the disordered information in unstructured text into computable, relatable, and reusable data metric assets, thereby enabling data-driven decision-making. However, traditional metric extraction methods and their limitations are as follows: 1) Rule-driven methods, such as regular expressions, have high business customization costs and require independent design of matching rules for different business lines (such as finance, bonds, and macroeconomics in the financial sector). This makes maintenance difficult, and changes in data formats require frequent rule adjustments, resulting in poor generalization capabilities. 2) Small model-driven methods, such as BERT+CRF (Conditional Random Fields), require large amounts of labeled data for model training. This has limited model generalization capabilities, making it difficult to apply across businesses, and the cost of model updates is high. 3) Large model prompt-driven methods, which require designing specific indicator extraction prompts for different businesses. The prompt optimization process is complex and the results are unstable. Furthermore, if general metric extraction is required, the prompts will become too long as the number of prompts for specific businesses increases, resulting in resource consumption and even exceeding the capabilities of the large model. Summary of the Invention

[0004] In order to overcome the problem of the lack of a universal indicator extraction method for unstructured data in the above-mentioned prior art, the present invention proposes a universal structured collection method for unstructured data based on RAG, which can be used for extracting common indicators for various business lines in a field, and can automatically align and screen business fields in specific business scenarios and store them in the database, thereby realizing a universal data collection method for unstructured data.

[0005] The present invention proposes a universal structured collection method for unstructured data based on RAG, which first constructs a knowledge base to store business data tables and metadata of business fields; The first language model is used to extract indicators for the text to be collected. Each indicator is associated with an indicator name, indicator value, indicator type and confidence score; Search and match each indicator in the knowledge base to obtain one or more most similar business fields for each indicator; The pre-trained second language model is used to select a best-matching business field based on the indicator and its corresponding most similar business field, and the matching degree and matching basis are marked. The valid best-matching business field is selected, and the matching degree of the valid best-matching business field is greater than the set matching threshold. The indicator of the retained valid best-matching business field is marked as a valid indicator. Valid indicators are verified, valid indicators that pass the verification are retained as business indicators, and business indicators are stored in the knowledge base in combination with the most matching business fields corresponding to the business indicators.

[0006] Preferably, the metadata of the business field includes the field name, business semantic description, data type, range constraint, and business table name; the field name, business semantic description, range constraint, and business table name of the business indicator respectively adopt the field name, business semantic description, range constraint, and business table name of the corresponding most matching business field, and the data type of the business indicator adopts its target data; the target data is the object that meets the range constraint among the indicator value and standard field value of the business indicator; The standard field value of the business indicator is obtained by converting the indicator value into a standardized numerical value according to the data type that best matches the business field.

[0007] Preferably, the verification of valid indicators includes value range verification and type verification; the passing condition of the value range verification is: the indicator value or standard field value of the valid indicator meets the value range constraint of the most matching business field of the valid indicator; the passing condition of the type verification is: the target data of the valid indicator is consistent with the data type of its most matching business field; the target data is the object that meets the value range constraint in the indicator value and standard field value of the business indicator.

[0008] Preferably, valid indicators that fail the verification but have a confidence score greater than or equal to a set confidence threshold are manually checked, and if the check passes, the business indicators are added to the text to be collected.

[0009] Preferably, each business field in the knowledge base is also associated with a semantic vector of metadata; a hybrid retrieval matching method is used to search for the most similar business field of each indicator in the knowledge base, and first, the keyword similarity and semantic similarity between the indicator name and each business field in the knowledge base are calculated for each indicator; then, a weighted calculation is performed on the keyword similarity and semantic similarity, and the obtained weighted sum is used as the similarity between the indicator and the semantic business field, and then the most similar business field of the indicator is obtained according to the size or sorting of the similarity.

[0010] Preferably, the method for calculating the keyword similarity between the indicator and the business field in the knowledge base is: first, a retrieval model is used to obtain the correlation score between the indicator name and the field name of the business field, and the correlation score between the indicator name and the business semantic description of the business field; then the two correlation scores are weighted and summed to obtain the keyword similarity, and the sum of the weights of the two correlation scores is 1.

[0011] Preferably, the method for calculating the semantic similarity between the indicator and the business field in the knowledge base is: first, use a pre-trained vector representation model to vector represent the indicator name to obtain the semantic vector of the indicator name; then use the cosine formula to calculate the similarity between the semantic vector of the indicator name and the semantic vector of the business field in the knowledge base as the semantic similarity.

[0012] Preferably, each business indicator of the text to be collected is stored in a business table corresponding to the knowledge base in an SQL structure.

[0013] The present invention proposes a universal structured collection system for unstructured data based on RAG, including a memory and a processor, wherein a computer program is stored in the memory, the processor is connected to the memory, and the processor is used to execute the computer program to implement the universal structured collection method for unstructured data based on RAG.

[0014] The present invention proposes a storage medium storing a computer program, which is used to implement the universal structured acquisition method for unstructured data based on RAG when executed.

[0015] The advantages of the present invention are: (1) The present invention first constructs a knowledge base that stores the table structure and field information of the business table; then uses a large model to identify and extract indicators from the input unstructured text (such as financial reports, research reports, announcements, etc.); then uses retrieval enhancement generation technology to search and match multiple most similar business fields in the knowledge base for each extracted indicator; based on the input unstructured text and the most similar business field information of each indicator, the large language model is assisted to select the most matching business field for the indicator in a specific business scenario; finally, the indicator value of the indicator itself is stored in the knowledge base in combination with the most matching business field. On the one hand, the present invention utilizes the advantages of the large language model in semantic understanding and context modeling to effectively improve the efficiency and accuracy of indicator extraction; on the other hand, combined with the indicator interpretation information in the knowledge base, it greatly improves the matching efficiency of indicators and business fields, thereby improving the versatility and accuracy of indicator extraction; combined with the automatic verification of the most matching business field, it can realize the rapid and accurate storage of business indicators, thereby realizing the conversion of unstructured data to structured data, realizing the effective collection of unstructured data, and improving data utilization.

[0016] (2) The essence of the present invention is an intelligent collection and conversion method for unstructured data to structured data based on the retrieval enhancement generation (RAG) technology. By building a unified knowledge base to integrate cross-business field names, definitions, types and value domain information, combined with large model open domain extraction and hybrid retrieval matching technology, the full process of unstructured text (such as financial reports, research reports, announcements) to structured data is realized. The present invention can build a knowledge base of database table structure information for businesses in specified fields (such as finance, demographics, medical reports, education yearbooks, etc.), and enhance the matching ability of the large model for indicators and business fields based on retrieval matching items; dynamically associate the knowledge base through RAG (retrieval enhancement generation) to adapt to any data source format and business type without the need for customized development; reduce the workload of manual annotation and rule design, and greatly improve data extraction efficiency. The present invention is suitable for the collection and storage of various unstructured data. By building a knowledge base in a specified field, it is conducive to improving the matching efficiency and accuracy of the indicators of the text to be collected and the business fields in the knowledge base, and improving the effect of indicator extraction and field mapping of the text to be collected.

[0017] (3) The present invention adopts a hybrid strategy of metadata + vector storage to build a unified knowledge base, which can integrate the table structure, field definition and indicator logic of multiple business scenarios for a specified field, so that a system can support multiple businesses at the same time. For example, in the financial field, it can support the data collection needs of different financial businesses such as finance, bonds, and macroeconomics, significantly reducing the cross-business adaptation cost and having strong versatility. The present invention combines a hybrid retrieval strategy with the open domain extraction technology of a large language model to support the automated processing of multi-format unstructured data such as PDF, scanned documents, HTML, and tables. It does not require preset data templates, format rules or prompt words, and can dynamically adapt to complex semantic expressions and data forms in the field, effectively improving the data collection efficiency of new business scenarios and greatly reducing the business launch cycle.

[0018] (4) The present invention realizes automatic data cleaning and conflict resolution based on the business rules preset in the knowledge base (such as value range constraints and unit specifications). By introducing confidence scores, it can be combined with a multi-source confidence weighted fusion algorithm to significantly reduce the proportion of manual intervention before data is stored in the database, and effectively improve the accuracy of indicator data extraction.

[0019] (5) This method uses dynamic weighting to fuse keyword similarity and semantic similarity, effectively improving the matching accuracy between indicator names and knowledge base business fields. It also implements data type conversion and automatic verification based on knowledge base preset rules, significantly reducing the cost of manual intervention. This method supports multiple input formats, including PDF, scanned documents, HTML, and tables, and is suitable for complex business scenarios such as banking and bonds. It significantly improves the efficiency, transparency, accuracy, and versatility of extracting structured indicator data from unstructured data. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a flow chart of a general structured acquisition method for unstructured data based on RAG proposed by the present invention; Figure 2 This is a data flow framework diagram in a general structured acquisition method for unstructured data based on RAG proposed by the present invention. DETAILED DESCRIPTION

[0021] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0022] like Figure 1 、 Figure 2 As shown, the present invention proposes a universal structured collection method for unstructured data based on RAG, which includes the following steps.

[0023] Step 1: Build a knowledge base to store business data tables and metadata and multimodal semantic vectors of business fields; Let F represent the metadata set of business fields stored in the business data table structure in the knowledge base; V represents the multimodal semantic vector of the business field in the business data table, and the knowledge base is denoted as G={F,V}; ; in, Indicates the k A metadata collection of business fields, including field name, business semantic description, data type, value range constraint and business table name, 1≤ k ≤ K , K is the total number of business fields; V ; in, Indicates the k The semantic vector of the metadata of a business field can be obtained by encoding the concatenated string of the field name, business semantic description and data type.

[0024] The knowledge base in this embodiment supports keyword indexing, inverted indexing, and vector indexing, specifically using an Elasticsearch database. Specifically, the inverted index can be used to store business names and business semantic descriptions; the HNSW algorithm is used to construct an approximate nearest neighbor index to store the semantic vectors of business fields.

[0025] Step 2: Use the first language model to perform semantic extraction on the collected text paragraphs, and remove duplicates from the extracted indicators to obtain the indicator extraction results of the text paragraphs. The indicator extraction results include one or more indicators, and each indicator is associated with an indicator name, indicator value, indicator type, and confidence score.

[0026] The confidence score can be set specifically according to the data source; for example, the data source can be set to multiple credit levels, and the confidence scores corresponding to the different levels can be set.

[0027] Suppose we extract key financial indicators from a listed company's annual report (PDF). The segmented document is: "xxx's net profit for the year is 120 million yuan xxx". This step extracts the following indicator information: Indicator name: Net profit Index value: 120 million yuan Metric type: Float Confidence level: 0.95 During specific implementation, it is necessary to define the input segment length constraint of the first largest language model. For texts whose length is not greater than the input segment length constraint, they can be directly input into the first largest language model for indicator extraction; for texts whose length is greater than the input segment length constraint, they need to be divided into multiple text paragraphs that meet the input segment length constraint, and then the text paragraphs are input into the first largest language model for indicator extraction.

[0028] The text segmentation method can adopt sliding window segmentation. In this way, the text paragraphs can be defined by text representation, sliding window size, sliding step and window sequence number, which facilitates the splicing of subsequent text processing results.

[0029] In addition, each paragraph of the text can be divided into text paragraphs by using paragraph segmentation. If the paragraph length is still greater than the input segment length constraint, the paragraphs greater than the input segment length constraint are further divided by using sliding window segmentation.

[0030] Step 3: obtain multiple most similar business fields of each indicator in the text to be collected through mixed search matching; thereby obtaining all the most similar business fields corresponding to the text to be collected; In specific implementation, the most similar business fields of each indicator can be screened according to the similarity threshold; or the K most similar business fields with the greatest similarity can be screened for each indicator; thereby obtaining JK most similar business fields corresponding to the text to be collected; J is the total number of indicators extracted from the text to be collected, and K is the set value.

[0031] In this step, the keyword and semantic similarity between each indicator and each business field in the knowledge base is calculated. Then, a weighted sum of the keyword and semantic similarities is calculated, and the weighted sum is used as the similarity between the indicator and the business field. The sum of the weights for keyword and semantic similarity is equal to 1. For each indicator, the K business fields with the greatest similarity are identified as the most similar business fields.

[0032] The method for calculating the keyword similarity between indicators and business fields is as follows: first, the retrieval model is used to obtain the correlation score between the indicator name and the field name of the business field, as well as the correlation score between the indicator name and the business semantic description of the business field; then the two correlation scores are weighted and summed to obtain the keyword similarity, and the sum of the weights of the two correlation scores is 1.

[0033] When the retrieval model uses BM25, the calculation process of keyword similarity can be expressed as follows: S lex (e j ,f)=α×BM25(e j ,χ name )+(1-α)×BM25(e j ,χ desc ); f∈F; Among them, f represents the business field in the knowledge base, χ name Indicates the field name of the business field f, χ desc Represents the business semantic description of the business field f; α is the weight, e j is the jth index extracted from the text to be collected; BM25(e j ,χ name ) represents the e calculated by the BM25 retrieval model j and χ name The correlation score of BM25(e j ,χ desc ) represents the e calculated by the BM25 retrieval model j and χ desc 's relevance score.

[0034] The semantic similarity between indicators and business fields is calculated as follows: First, a pre-trained vector representation model, such as Embedding, is used to vectorize the indicator name to obtain the semantic vector of the indicator name. Then, the cosine formula is used to calculate the similarity between the semantic vector of the indicator name and the semantic vector of the business field in the knowledge base as the semantic similarity. The semantic vector of the business field is extracted from the set V in the knowledge base G.

[0035] Step 4: Use the pre-trained second largest language model, such as LLM, to select a best-matching business field based on the indicator and its corresponding K most similar business fields, and mark the matching degree and matching basis; select the valid best-matching business field based on the matching degree; and mark the indicator of the valid best-matching business field as the valid indicator.

[0036] In specific implementations, the output of the second-largest language model (LLM) can be defined as {metric name, metric value; metadata of the most matching business field, matching degree, and matching basis}. This allows the output of the second-largest language model (LLM) corresponding to each metric to be directly fed into the next stage for metric storage. The input of the second-largest language model (LLM) can be defined as {text to be collected, metric, and the K most similar business fields corresponding to the metric}, facilitating the mapping of the collected text to the metric.

[0037] The method of filtering the valid best-matching business fields according to the matching degree is as follows: retain the best-matching business fields and the corresponding valid indicators whose matching degree is greater than the set matching threshold; delete the indicators corresponding to the best-matching business fields whose matching degree is less than the matching threshold; thereby obtaining less than or equal to J valid indicators, where J is the total number of indicators extracted from the text to be collected.

[0038] Matching screening can effectively shield invalid indicators extracted by the first language model, thereby improving the accuracy of parsing the collected text and reducing the amount of redundant data.

[0039] Step 5: Perform value range check and type check on valid indicators; Perform standardized numerical conversion on the indicator value according to the data type of the most matching business field of the valid indicator to obtain the standard field value of the indicator; The pass condition for range check is: the indicator value or standard field value of the valid indicator meets the range constraint of the most matching business field of the valid indicator; the indicator value and standard field value that meet the range constraint are recorded as the target data of the indicator; The passing conditions for type verification are: the target data of the valid indicator is consistent with the data type of its most matching business field; the target data is the object that meets the value range constraints among the indicator value of the business indicator and the standard field value.

[0040] In this step, for valid indicators that fail the verification, they will be directly discarded only when their confidence scores are less than the set confidence threshold. For valid indicators that fail the verification but have confidence scores greater than or equal to the set confidence threshold, manual verification is performed to improve the accuracy of data analysis and avoid data loss caused by overfitting of the algorithm.

[0041] Step 6: Obtain all valid indicators that have passed verification of the text to be collected as business indicators, and generate a metadata set for the business indicators by combining the best-matching business fields and target data corresponding to the business indicators. Specifically, the field names, business semantic descriptions, value range constraints, and business table names in the metadata set of the business indicators use the field names, business semantic descriptions, value range constraints, and business table names of the best-matching business fields, respectively. The data types in the metadata set of the business indicators use the target data. That is, when the indicator value of a business indicator meets the value range constraint of the most matching business field, the indicator value is used as the data type of the business indicator; when the standard field value of a business indicator meets the value range constraint of the most matching business field, the standard field value is used as the data type of the business indicator.

[0042] Step 7: Store the metadata of each business indicator of the text to be collected into the corresponding business table.

[0043] The following describes the general structured collection method for unstructured data based on RAG in conjunction with specific embodiments.

[0044] Example 1 Taking the indicator "net profit" in financial reports as an example, the metadata of this business field in the database is as follows: field name (NetProfit), business semantic description (annual total profit after tax, in 10,000 yuan), data type (Float), value range constraint (>=0), and business table noun (financial statement). The above information is concatenated into the text "NetProfit; annual total profit after tax, in 10,000 yuan; Float; financial statement" and then vectorized using the Embedding model to obtain a multimodal vector for the business field.

[0045] In this embodiment, a dual-channel retrieval index is constructed in the knowledge base, in which the field names and business semantic descriptions in the metadata are stored using inverted index technology to support keyword indexing; the business semantic description is stored using approximate nearest neighbor index technology based on the HNSW algorithm, so as to create a mapping in Elasticsearch that supports vector retrieval and supports the setting of vector fields with field type Type=dense_vector.

[0046] In this example, the BM25 retrieval model is used to calculate the correlation score between the indicator name, field name, and business semantic description. The weights of field name and business semantic description are set to 0.3 and 0.7, respectively. Taking the indicator "net profit" and the business field "NetProfit; annual after-tax profit, unit: 10,000 yuan; Float; financial statement" in the knowledge base as an example, the keyword similarity between the two is calculated as follows: First, BM25 is used to calculate the correlation score E1 between the indicator name "Net Profit" and the field name of the business field "NetProfit". BM25 is also used to calculate the correlation score E2 between the indicator name "Net Profit" and the business semantic description of the business field "Annual total profit after tax, unit is 10,000 yuan". Then, 0.3E1+0.7E2 is calculated as the keyword similarity between the indicator "Net Profit" and the business field "Net Profit; Annual total profit after tax, unit is 10,000 yuan; Float; Financial Statement" in the knowledge base.

[0047] In this implementation, the weighted sum of keyword similarity and semantic similarity is used as the similarity between the indicator and the business field; the sum of the weight of keyword similarity and the weight of semantic similarity is 1. The calculation formula of the weight of keyword similarity is as follows: ; in, Indicates the weight of keyword similarity; TF(.) indicates the frequency of a specific word in a document; TF(indicator name, field name) indicates the frequency of the word "indicator name" appearing in "field name"; Indicates the setting constant, which is used to control the influence of TF value on the final result; The larger the value, the more sensitive the function value is to the change of TF value, or the TF value has a greater impact on the final weight. The greater the impact.

[0048] In this embodiment, for the indicator "net profit", through mixed search and matching, the three most similar business field sets finally returned are: [NetProfit, TotalIncome, GrossProfit].

[0049] For example, to determine the final mapping of the indicator "net profit" from the candidate fields [NetProfit, TotalIncome, GrossProfit], the following input needs to be given to the second largest language model, LLM: Prompt: Task: Analyze the correlation between indicators and business fields in context and select the only business field that best suits the current context.

[0050] Financial text: "xxx's net profit for the year is 120 million yuanxxx" Top K business field metadata: [ [Field Name: NetProfit, Semantic Description: Annual after-tax profit, unit: 10,000 yuan, Type: Float, Value Range: >= 0; Business Table Noun: Financial Statement], [Field Name: TotalIncome, Semantic Description: Clearly covers all sources of income (such as operating income and investment income), avoiding confusion with net profit, Type: Float, Value Range: >= 0; Business Table Noun: Financial Statement], [Field Name: GrossProfit, Semantic Description: Emphasizes "gross revenue minus direct costs" to distinguish it from net profit (after-tax profit), Type: Float, Value Range: >= 0; Business Table Noun: Financial Statement] ] Finally, the large model LLM2 is required to determine the optimal business field based on the context and provide information such as the matching degree and matching basis. The complete output is as follows: Indicator name: Net profit Best matching business field: NetProfit Match: 0.98 Matching basis: "Net Profit" and "Net Profit" both represent after-tax profit and have the same unit... Since valid indicators need to be screened based on the matching degree, a matching threshold needs to be set. The specific setting can be as needed. For example, in this embodiment, it is set to 0.5. In this way, the matching degree of "Net Profit" and "Net Profit" reaches 0.98, and "Net Profit" is retained as a valid indicator together with "Most Matching Business Field: Net Profit". Assume that in another task, the most matching business field for the indicator "Net Profit" is "TotalIncome", and the matching degree between the two is 0.2. Then the indicator "Net Profit" that matches "TotalIncome" is discarded.

[0051] When verifying the valid indicator "Net profit = 120 million yuan", first perform a range check to convert 120 million yuan into the standard field value 12000.0 based on the "unit: 10,000 yuan", and check whether it meets the range constraint of the business field "NetProfit" (">=0"). Then, perform a type check to confirm that 12000.0 is of type Float, which is consistent with the data type "Float" of the business field "NetProfit". Finally, add an exception handling process. If the range check fails (such as "-5 million yuan"), but the confidence level is 0.95, manual verification is triggered. If the confidence level is lower than 0.5, it is directly discarded.

[0052] In this embodiment, the extracted indicator "net profit" is finally matched with the business field "NetProfit" to generate a storage statement such as: INSERT INTO FinancialReport (NetProfit) VALUES (12000.0), which means that the net profit of 120 million from the financial report is stored in the knowledge base.

[0053] Example 2 This example uses the financial sector as an example. The data collection objects include: 10 regional economic reports (average page length 29, each with more than 19,000 words) as the experimental dataset; and a golden annotation set of 365 key regional economic indicators is constructed as the evaluation benchmark. That is, the knowledge base contains the business fields corresponding to these 365 key regional economic indicators.

[0054] In this example, the following four methods are used to convert 10 regional economic reports in the experimental dataset into structured data and insert them into the knowledge base. The models in each method are pre-trained and then tested directly on the experimental dataset: ①Rule-driven, using regularized expressions; ② Small model driven, based on BERT+CRF training text conversion model; ③ Large model prompt word driven type; ④ The present invention proposes a general structured acquisition method for unstructured data based on RAG (hereinafter referred to as the present invention method).

[0055] This example briefly compares data extraction accuracy and recall rate, and the results are shown in Table 1 below: Table 1: Comparison of experimental results ; Experimental results show that the method of the present invention has significant effects on several key performance indicators: Compared with rule-driven methods, the processing efficiency of the method of the present invention is significantly improved, effectively alleviating the bottleneck of traversal retrieval, and overcoming the problems of insufficient alias support, limited generalization ability, the need for manual customization of rules, and long time to traverse the entire text of the rule-driven method; Compared with small model methods, the method of the present invention significantly reduces the reliance on manual annotation; Compared with large model prompt word engineering, the method of the present invention avoids context overruns while improving processing efficiency through text segmentation, iterative extraction, and hybrid retrieval strategies, reducing the average processing time for a single report to approximately 2 minutes and 45 seconds. Moreover, the method of the present invention, with the help of a three-stage mechanism of open domain extraction-knowledge base matching-large model selection, achieved an accuracy rate of 89.7% in the experimental data set, far exceeding other methods, and providing an effective solution for end-to-end automated extraction of long texts, multiple indicators, and high alias complexity scenarios.

[0056] Of course, it will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, but also encompasses the same or similar structures that can be implemented in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims, not the foregoing description, and it is intended that all variations that fall within the meaning and range of equivalents of the claims be encompassed within the present invention. Any reference signs in the claims should not be construed as limiting the claim to which they relate.

[0057] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

[0058] The technology, shape, and structure not described in detail in the present invention are all well-known technologies.

Claims

1. A general structured acquisition method for unstructured data based on RAG, characterized in that: First, build a knowledge base to store business data tables and metadata of business fields; The first language model is used to extract indicators for the text to be collected. Each indicator is associated with an indicator name, indicator value, indicator type and confidence score; Search and match each indicator in the knowledge base to obtain one or more most similar business fields for each indicator; The pre-trained second largest language model is used to select the most matching business field based on the indicator and its corresponding most similar business field, and the matching degree and matching basis are marked; the effective most matching business field is selected, and the matching degree of the effective most matching business field is greater than the set matching threshold; The pointer that retains the most valid matching business field is designated as a valid pointer; Valid indicators are verified, valid indicators that pass the verification are retained as business indicators, and business indicators are stored in the knowledge base in combination with the most matching business fields corresponding to the business indicators.

2. The RAG-based universal structured acquisition method for unstructured data according to claim 1, characterized in that: The metadata of a business field includes the field name, business semantic description, data type, range constraint, and business table name. The field name, business semantic description, range constraint, and business table name of a business indicator use the field name, business semantic description, range constraint, and business table name of the corresponding most matching business field, respectively. The data type of a business indicator uses its target data. The target data is the indicator value of the business indicator and the object that meets the range constraint among the standard field values. The standard field value of the business indicator is obtained by converting the indicator value into a standardized numerical value according to the data type that best matches the business field.

3. The RAG-based universal structured acquisition method for unstructured data according to claim 2, characterized in that: Verification of valid indicators includes range verification and type verification; the passing condition for range verification is that the indicator value or standard field value of the valid indicator meets the range constraint of the most matching business field of the valid indicator; the passing condition for type verification is that the target data of the valid indicator is consistent with the data type of its most matching business field.

4. The RAG-based universal structured acquisition method for unstructured data according to claim 3, characterized in that: For valid indicators that fail the verification but have a confidence score greater than or equal to the set confidence threshold, manual verification is performed. If the verification passes, the business indicators of the text to be collected are added.

5. The RAG-based universal structured acquisition method for unstructured data according to claim 1, characterized in that: Each business field in the knowledge base is also associated with a semantic vector of metadata; a hybrid retrieval matching method is used to search for the most similar business field of each indicator in the knowledge base. First, the keyword similarity and semantic similarity between the indicator name and each business field in the knowledge base are calculated for each indicator; then the keyword similarity and semantic similarity are weighted, and the obtained weighted sum is used as the similarity between the indicator and the business field, and then the most similar business field of the indicator is obtained according to the size or sorting of the similarity.

6. The RAG-based universal structured acquisition method for unstructured data according to claim 5, characterized in that: The method for calculating the keyword similarity between the indicator and the business field in the knowledge base is as follows: first, the retrieval model is used to obtain the correlation score between the indicator name and the field name of the business field, and the correlation score between the indicator name and the business semantic description of the business field; then the two correlation scores are weighted and summed to obtain the keyword similarity, and the sum of the weights of the two correlation scores is 1.

7. The RAG-based universal structured acquisition method for unstructured data according to claim 5, characterized in that: The method for calculating the semantic similarity between indicators and business fields in the knowledge base is as follows: first, a pre-trained vector representation model is used to vectorize the indicator name to obtain the semantic vector of the indicator name; then the cosine formula is used to calculate the similarity between the semantic vector of the indicator name and the semantic vector of the business field in the knowledge base as the semantic similarity.

8. The RAG-based universal structured acquisition method for unstructured data according to claim 1, characterized in that: Each business indicator of the text to be collected is stored in the business table corresponding to the knowledge base in the SQL structure.

9. A universal structured acquisition system for unstructured data based on RAG, characterized in that: The invention comprises a memory and a processor, wherein a computer program is stored in the memory, the processor is connected to the memory, and the processor is used to execute the computer program to implement the universal structured acquisition method for unstructured data based on RAG as described in any one of claims 1 to 8.

10. A storage medium, characterized in that: A computer program is stored, and when the computer program is executed, it is used to implement the universal structured acquisition method for unstructured data based on RAG according to any one of claims 1 to 8.

Images