Data standardization method based on large model
Through the data standardization method based on large models, a data standard knowledge base is built and regular expressions and code is generated, which solves the problem of cumbersome and high cost of multi-source non-standard data processing, realizes efficient and convenient data standardization, and improves the degree of automation and the quality of data standardization.
Patent Information
- Application Number
- CN202410933988.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-12
- Publication Date
- 2024-11-22
AI Technical Summary
The existing data standardization methods are cumbersome to process multi-source non-standard data, with high cost and low automation, making it difficult to meet the company's needs for convenience, low cost and high automation.
Using a large model-based data standardization method, by building a data standard knowledge base, unstructured data is converted into an easy-to-retrieve format, using the large model to generate regular expressions and code, realizing data standardization, combining iterative data cleaning until it reaches the specified level. Threshold.
It significantly improves the efficiency and quality of data standardization, reduces labor costs, enhances the understanding and adaptability of large models to data standards, and provides higher transparency and interpretability.
Smart Images

Figure CN119003583A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing technology, and more specifically to a data standardization method based on a large model. Background Art
[0002] In the digital economy era, due to the rapid development of information technologies such as the Internet and big data, the amount of global data is growing exponentially. With the integration of Internet applications and corporate business, most companies are facing problems such as data silos, increased data management costs, and the inability to realize data monetization. How to activate the potential of data elements and make full use of the value of data assets to improve the governance efficiency of enterprises has become a problem that companies have to think about. Data standardization aims to convert heterogeneous data formats within a single column into a unified data format for better data processing and analysis. Data standardization is an important link in the field of data processing technology and the key to improving data availability. It ensures the consistency of data between various business domains and information systems, which is crucial for effective data mining, data analysis, and decision-making.
[0003] Traditional enterprise data standardization relies on extensive human-computer interaction, requiring data engineers to use programming languages such as Python and SQL to write specific scripts and formulate specific rules. Therefore, data standardization has become a "nightmare" for many data engineers, because they need to write corresponding conversion scripts for each special case and each column data type in the data, and there is no one-size-fits-all solution. When faced with big data with large scale and diversity, it may cause problems of high costs and lack of flexibility. In recent years, companies have begun to try to use a variety of data middle platforms and ETL products, or use machine learning technologies such as neural networks and rule matching to execute standardization processes, but this process still requires a lot of manpower and lacks security and transparency.
[0004] With the continuous advancement of artificial intelligence, the development of big models provides a good opportunity for fast and efficient data standardization. Big models refer to neural network models composed of billions or even trillions of parameters. These models are trained from massive amounts of unlabeled text data, can process large data sets, and have very strong learning and generalization capabilities. In recent years, with the increase in the scale of big models, they have initially acquired the ability to plan and solve complex tasks. With rich adaptation technologies, big models have acquired excellent understanding, generation, planning and learning capabilities, and have been widely used in natural language processing, information retrieval, recommendation systems, multimodal perception, knowledge graph reasoning, intelligent agents and other fields, covering machine translation, sentiment analysis, information extraction, video generation, data analysis, content recommendation and other tasks, and have been implemented in industries such as medical care, education, finance, justice and scientific research. Faced with the huge scale of enterprise data and the complexity of data processing tasks, big models have gradually been explored and used in data processing-related tasks, with extraordinary potential. Users only need to provide raw data and then make demands through natural language to obtain converted standard data, thereby significantly improving the efficiency and quality of data standardization.
[0005] In summary, in operational practice, enterprises face non-standard data from multiple sources and need more convenient, lower-cost, and more automated data standardization methods. How to use large models to achieve better solutions has become an issue that needs to be explored urgently.
[0006] Therefore, it is necessary to propose a data standardization method based on large models to solve the above problems. Summary of the invention
[0007] The purpose of the present invention is to solve the problems that the existing data standardization methods are relatively complicated, costly and have a low degree of automation for non-standard data from multiple sources.
[0008] In order to achieve the above-mentioned purpose, the present invention specifically adopts the following technical solutions:
[0009] A data standardization method based on a large model comprises the following steps:
[0010] Step S1: Build a data standard knowledge base. First, process the data standard documents, convert these unstructured PDF or Word documents into key-value pairs that are easier to search, and save them in CSV or JSON format, which greatly improves the retrieval efficiency and accuracy of the large model.
[0011] Step S2: Retrieve data standards. The user interacts with the large model, obtains the data column name to be standardized based on the original data input by the user, and retrieves the data standards related to it in the data standard knowledge base;
[0012] Step S3, generate a regular database, use the relevant data standards obtained in step S2 as the context of the large model, and provide a sample display for the large model, encapsulate it into a unified prompt template, use the context learning ability of the large model to generate regular expressions that conform to the data rules, and finally form a regular database of "data column name: regular expression";
[0013] Step S4, data standardization, first query the regular expression corresponding to the data column name to be standardized, screen and filter the database according to the expression, then randomly sample the non-standard data to provide samples, and put the non-standard data, regular expressions and samples into the big model together according to the prompting method of the thinking chain, guide the big model to analyze the error cause of the non-standard data, complete the writing and execution of the corresponding code, and the code written by the big model will be saved in the memory unit to improve the memory ability and context understanding ability of the big model;
[0014] Step S5, iterative data standardization, using an iterative data cleaning method, the data after a round of standardization conversion in step S4 is sent back to the beginning of step S4, and the data that does not match the regular expression is re-screened and filtered. Then, based on the resampled non-standard data, the large model is used to analyze the non-standard data, generate code, and perform data conversion operations until the proportion of non-standard data finally reaches the specified threshold.
[0015] Furthermore, the step S1 specifically includes the following steps:
[0016] Step S11, read the data standard document line by line, organize it according to the title, and organize it into a JSON or CSV standard file in the key-value pair format of "rule name: rule content";
[0017] Step S12: Use a transformation neural network or other model to generate embedding vectors corresponding to the key and value for the sorted standard files, and store them to build a data standard knowledge base.
[0018] Furthermore, the step S2 specifically includes the following steps:
[0019] Step S21, access the input original data to obtain the fields that need to be converted into data standardization, that is, the data column names;
[0020] Step S22: perform data enhancement for the data field, generate a group of fields with similar semantics, embed these fields, and obtain their vector representations;
[0021] Step S23: According to the vector representation, a vector-based retrieval algorithm is used to search in the data standard knowledge base to obtain multiple data standards with the closest similarity.
[0022] Furthermore, the step S3 specifically includes the following steps:
[0023] Step S31, randomly selecting a data column of the original data, manually formulating a regular expression according to the standard data agreed in the data standard knowledge base, thereby creating a task example;
[0024] Step S32, constructing a unified prompt word, which is composed of a regular expression generating task description, a task example and the data standard retrieved in step S2;
[0025] Step S33: search the column names of the original data one by one to generate corresponding regular expressions, and finally form a regular database in the form of "data column name: regular expression".
[0026] Furthermore, the step S4 specifically includes the following steps:
[0027] Step S41, interacting with the database, querying the regular expression database according to the column name of the data to be standardized, and obtaining the regular expression of its standard data;
[0028] Step S42: Screen the column of data according to the regular expression, and filter out the data rows that do not match the regular expression, thereby obtaining a smaller non-standard data set;
[0029] Step S43: sampling the non-standard data, randomly selecting a plurality of samples, and generating samples written in standardized codes by manual annotation;
[0030] Step S44, construct a prompt template that conforms to the thinking chain paradigm, the prompt template consists of a system task description, a user input description, a user input sample, and a code generation sample, and guides the large model to gradually analyze the error causes of non-standard data through the reasoning steps of the sample, and implements code writing and execution accordingly;
[0031] Step S45: Save the code written by the large model into the memory unit to improve the memory capacity and context understanding ability of the large model.
[0032] Furthermore, the step S5 specifically includes the following steps:
[0033] Step S51: Perform a new round of screening and filtering on the data that has undergone the previous round of standardization conversion according to the regular expression to obtain a non-standard data set with a gradually reduced data size;
[0034] Step S52, resampling the non-standard data set, and automatically filling the samples into the prompt template;
[0035] Step S53: According to the prompt template, the large model is used to perform problem judgment, problem analysis and problem processing on the new non-standard data set one by one, generate data standardization code, and perform data conversion operations until the proportion of non-standard data to the original data reaches a set threshold.
[0036] Compared with the prior art, the present invention has the following beneficial effects:
[0037] 1. The present invention realizes semi-automatic data standardization based on a large model, reduces the labor cost of manual data screening and code writing, and provides a more convenient, more efficient and faster data standardization solution.
[0038] 2. By combining external data standards, the present invention enhances the deep understanding of data standards by the big model and can introduce domain knowledge in the reasoning process, which not only improves the accuracy of data standardization, but also has the generalization ability to adapt to different fields and different business scenarios.
[0039] 3. The present invention introduces thought chain and retrieval enhancement generation technology, decomposes the data standardization task into several simple and efficient subtasks, and combines the data standard knowledge base to provide the problem analysis process and reasoning results of non-standard data, thereby enhancing the transparency and interpretability of data standardization.
[0040] 4. The present invention pre-processes data standard documents, improves the retrieval capability of large models, and supports a variety of structured query tools and data filters for retrieval and filtering operations on non-standard data, which are not limited to the listed tool options and have sufficient scalability and flexibility. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] Figure 1 This is a flow chart of the data standardization method based on a large model in the present invention;
[0042] Figure 2 This is a framework diagram of the data standardization method based on the big model in the present invention. DETAILED DESCRIPTION
[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0044] Example
[0045] like Figure 1 and Figure 2As shown, the present invention first converts the unstructured data standard document from the external data source into an easy-to-retrieve structured file, thereby improving the efficiency and accuracy of the big model retrieval; by retrieving the data standard, the big model can obtain domain knowledge and enhance the ability of the big model to deal with private domain data; further, the big model generates a regular expression corresponding to the standardized data according to the retrieved relevant data standards or data rules, so as to quickly locate and filter the data content that does not meet the standards; then the filtered data set is randomly sampled, and a small number of samples are generated by manually writing code to improve the big model's understanding of the task and standardize the input and output formats. The big model will generate and execute the conversion code for data standardization; finally, the data rows that do not meet the standards are filtered again to verify whether they reach the specified threshold, and the result data that does not meet the expectations is iteratively cleaned until the non-standard data reaches the specified threshold.
[0046] According to the above implementation process and Figure 1 The specific implementation steps of this embodiment will be described in detail below.
[0047] A data standardization method based on a large model comprises the following steps:
[0048] Step S1: Build a data standard knowledge base. First, process the data standard documents, convert these unstructured PDF or Word documents into a more easily searchable key-value pair format, and save them in a more readable CSV or JSON format, which greatly improves the retrieval efficiency and accuracy of the large model. This method is called self-querying, which enables the retriever to not only use the query entered by the user to compare the semantic similarity with the stored document content, but also extract filters from the user's query on the stored document metadata and execute these filters.
[0049] The data standard described in step S1 refers to the rules that need to be followed when performing data standardization conversion operations, such as industry data standard guidance documents or unified data specifications formulated by enterprises according to business needs. This standard is a definition and interpretation of data, which is based on the attributes of data entities. The standards for each attribute include multiple parts such as business specifications, management specifications, technical specifications, and value ranges of the data. They set data format rules for different data types to ensure that data values correspond to a set of strict legal values or meet unified format constraints. For example, in the employee information collection form, the data standard stipulates that gender uses 'F' to represent female, and 'M' to represent male, and the telephone number is agreed to be in the form of "123-4567-8900". However, the actual collected data may have various erroneous forms, such as the absence of '-' in the telephone number, or +86 at the beginning, or (+86), and so on.
[0050] In step S1, the present invention processes the data standard document to build a data standard knowledge base, wherein the data standard document is usually an industry data standard guidance document or a unified data specification formulated by an enterprise according to business needs. The document formulates a unified standard format and defines and interprets the enterprise data in a standardized manner to ensure that different data of the same attribute have values corresponding to a set of strict legal values or meet consistent constraints. The process can be divided into two steps:
[0051] Step S11: Perform simple preprocessing on the data standard document, such as removing irrelevant content, removing redundant line breaks, etc., and then extract by title, organize them one by one into the key-value pair format of "rule name: rule content", and save them as standard files such as JSON or CSV. When encountering multi-level titles, use the format of "first-level title: second-level title:..." to encode them together as the rule name;
[0052] Step S12, using a transform neural network (transformer) or other models to generate embedding vectors corresponding to the key and value for the sorted standard files, construct a vector database, and store them to construct a data standard knowledge base. The present invention refers to the vector database as a data standard knowledge base. These embedding vectors encode all the text content in the file, thereby capturing the semantic information of the document so that the subsequent searcher can perform retrieval based on semantic similarity.
[0053] The vector database described in step S12 is the core of the search. It creates a vector representation of the text. The quality of the vector greatly affects the search results. Since text embedding is an important basic content of representation learning, there are many open source models and products available, such as BERT and BAAI. Large models on the market such as OpenAI and Gemini also provide vector models that can be called.
[0054] Step S2, retrieving data standards. The user interacts with the large model, obtains the data column names to be standardized based on the original data input by the user, and retrieves the relevant data standards in the data standard knowledge base.
[0055] In step S2, the present invention searches for data standards in a data standard knowledge base according to the data column names of the original data. The process can be subdivided into the following steps:
[0056] Step S21, access the input raw data, and obtain the fields that need to be converted to data standardization through simple traversal operations, that is, data column names, such as name, gender, telephone number, etc. in the employee information table;
[0057] Step S22: perform data enhancement for the data field to generate a group of fields with similar semantics. For example, the column name "name" can be expanded to {name, title, user name...}, and these fields are embedded to obtain their vector representation. The benefit of data enhancement is to improve the matching efficiency in the vector space, because the data standard does not necessarily correspond to the data one by one, thereby ensuring the retrieval results;
[0058] Step S23: According to the vector representation, a vector-based retrieval algorithm (dense retrieval) is used to search in the data standard knowledge base to obtain multiple data standards with the closest similarity.
[0059] The dense search principle described in step S23 is different from the traditional keyword-based search. The keyword-based search usually uses a representation called sparse embeddings, which emphasizes the importance of the frequency of keywords in a single document relative to the frequency of all documents. Vector-based search, i.e., dense search, uses the vector embedding of the text to calculate the data object with the closest similarity (such as cosine distance), and the returned search results are sorted according to the similarity with the search query.
[0060] Step S3, generate a regular database, use the relevant data standards obtained in step S2 as the context of the large model, and provide an example display for the large model, encapsulate it into a unified prompt template, and use the context learning ability of the large model to generate regular expressions that conform to the data rules, and finally form a regular database of "data column name: regular expression".
[0061] Step S3 uses the retrieval enhancement generation (RAG) technology to generate a regular expression of the standard data that meets the rule content. The subdivision steps are as follows:
[0062] Step S31, randomly selecting a data column of the original data, manually formulating a regular expression according to the standard data agreed in the data standard knowledge base, thereby creating a task example;
[0063] Step S32, constructing a unified prompt word, which is composed of a regular expression generating task description, a task example and the data standard retrieved in step S2;
[0064] Step S33: search the column names of the original data one by one to generate corresponding regular expressions, and finally form a regular database in the form of "data column name: regular expression".
[0065] Step S4, data standardization, first query the regular expression corresponding to the data column name to be standardized, screen and filter the database according to the expression, then randomly sample the non-standard data to provide samples, and put the non-standard data, regular expressions and samples into the big model together according to the prompt method of the thinking chain, guide the big model to analyze the error cause of the non-standard data, and complete the writing and execution of the corresponding code accordingly. The code written by the big model will be saved in the memory unit to improve the memory ability and context understanding ability of the big model.
[0066] Step S4 is the main process of implementing semi-automatic data standardization based on the large model, and its subdivision steps are as follows:
[0067] Step S41, interacting with the database, querying the regular expression database according to the column name of the data to be standardized, and obtaining the regular expression of its standard data;
[0068] Step S42: Screen the column of data according to the regular expression, and filter out the data rows that do not match the regular expression, thereby obtaining a smaller non-standard data set;
[0069] Step S43: sampling the non-standard data, randomly selecting a plurality of samples, and generating samples written in standardized codes by manual annotation;
[0070] Step S44, construct a prompt template that conforms to the thinking chain paradigm, which consists of a system task description, a user input description, a user input sample, and a code generation sample. The large model is guided to gradually analyze the error causes of non-standard data through the reasoning steps of the sample, and the code is written and executed accordingly. The present invention performs multiple verifications on the designed prompt words, evaluates the effects through actual tests, and adjusts and improves the prompt words according to the evaluation results to improve the performance and accuracy of the model;
[0071] Step S45: Save the code written by the large model into the memory unit to improve the memory capacity and context understanding ability of the large model.
[0072] Step S5, iterative data standardization, using an iterative data cleaning method, the data after a round of standardization conversion in step S4 is sent back to the beginning of step S4, and the data that does not match the regular expression is re-screened and filtered. Then, based on the resampled non-standard data, the large model is used to analyze the non-standard data, generate code, and perform data conversion operations until the proportion of non-standard data finally reaches the specified threshold.
[0073] After step S4, most of the non-standard data can be located and cleaned. In order to further consolidate the quality of data standardization, step S5 adopts an iterative strategy, and its subdivision steps are as follows:
[0074] Rescreen and filter the data rows that do not match the regular expression, and then, until the final proportion of non-standard data reaches the specified threshold.
[0075] Step S51: Perform a new round of screening and filtering on the data that has undergone the previous round of standardization conversion according to the regular expression to obtain a non-standard data set with a gradually reduced data size;
[0076] Step S52, resampling the non-standard data set, and automatically filling the samples into the prompt template;
[0077] Step S53: According to the prompt template, the large model is used to perform problem judgment, problem analysis and problem processing on the new non-standard data set one by one, generate data standardization code, and perform data conversion operations until the proportion of non-standard data to the original data reaches a set threshold.
[0078] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. The patent protection scope of the present invention shall be based on the claims. All equivalent structural changes made using the contents of the description and drawings of the present invention should also be included in the protection scope of the present invention.
Claims
1. A data standardization method based on a large model, characterized in that: The following steps are involved: Step S1: Build a data standard knowledge base. First, process the data standard documents and convert these unstructured PDF or Word documents into a key-value pair format that is easier to retrieve. Step S2: Retrieve data standards. The user interacts with the large model, obtains the data column name to be standardized based on the original data input by the user, and retrieves the data standards related to it in the data standard knowledge base; Step S3, generating a regular expression database, using the relevant data standards obtained in step S2 as the context of the large model, and providing a sample display for the large model, encapsulating it into a unified prompt template, using the context learning ability of the large model to generate regular expressions that conform to the data rules, and finally forming a regular expression database in the key-value pair format of "data column name: regular expression"; Step S4, data standardization, first query the regular expression corresponding to the data column name to be standardized, screen and filter the database according to the expression, then randomly sample the non-standard data to provide samples, and put the non-standard data, regular expressions and samples into the big model together according to the prompting method of the thinking chain, guide the big model to analyze the error cause of the non-standard data, complete the writing and execution of the corresponding code, and the code written by the big model will be saved in the memory unit; Step S5, iterative data standardization, using an iterative data cleaning method, the data after a round of standardization conversion in step S4 is sent back to the beginning of step S4, and the data that does not match the regular expression is re-screened and filtered. Then, based on the resampled non-standard data, the large model is used to analyze the non-standard data, generate code, and perform data conversion operations until the proportion of non-standard data finally reaches the specified threshold.
2. The data standardization method based on a large model according to claim 1, characterized in that: The step S1 specifically includes the following steps: Step S11, read the data standard document line by line, organize it according to the title, and organize it into a JSON or CSV standard file in the key-value pair format of "rule name: rule content"; Step S12: Use the transformation neural network to generate embedding vectors corresponding to the key and value for the sorted standard files, and store them to build a data standard knowledge base.
3. The data standardization method based on a large model according to claim 1, characterized in that: The step S2 specifically includes the following steps: Step S21, access the input original data to obtain the fields that need to be converted into data standardization, that is, the data column names; Step S22: perform data enhancement for the data field, generate a group of fields with similar semantics, embed these fields, and obtain their vector representations; Step S23: According to the vector representation, a vector-based retrieval algorithm is used to search in the data standard knowledge base to obtain multiple data standards with the closest similarity.
4. The data standardization method based on a large model according to claim 1, characterized in that: The step S3 specifically comprises the following steps: Step S31, randomly selecting a data column of the original data, manually formulating a regular expression according to the standard data agreed in the data standard knowledge base, thereby creating a task example; Step S32, constructing a unified prompt word, which is composed of a regular expression generating task description, a task example and the data standard retrieved in step S2; Step S33: search the column names of the original data one by one to generate corresponding regular expressions, and finally form a regular database in the form of "data column name: regular expression".
5. The data standardization method based on a large model according to claim 1, characterized in that: The step S4 specifically comprises the following steps: Step S41, interacting with the database, querying the regular expression database according to the column name of the data to be standardized, and obtaining the regular expression of its standard data; Step S42: Screen the column of data according to the regular expression, and filter out the data rows that do not match the regular expression, thereby obtaining a smaller non-standard data set; Step S43: sampling the non-standard data, randomly selecting a plurality of samples, and generating samples written in standardized codes by manual annotation; Step S44, construct a prompt template that conforms to the thinking chain paradigm, the prompt template consists of a system task description, a user input description, a user input sample, and a code generation sample, and guides the large model to gradually analyze the error causes of non-standard data through the reasoning steps of the sample, and implements code writing and execution accordingly; Step S45: Save the code written by the large model into the memory unit to improve the memory capacity and context understanding ability of the large model.
6. The data standardization method based on a large model according to claim 1, characterized in that: The step S5 specifically comprises the following steps: Step S51: Perform a new round of screening and filtering on the data that has undergone the previous round of standardization conversion according to the regular expression to obtain a non-standard data set with a gradually reduced data size; Step S52, resampling the non-standard data set, and automatically filling the samples into the prompt template; Step S53: According to the prompt template, the large model is used to perform problem judgment, problem analysis and problem processing on the new non-standard data set one by one, generate data standardization code, and perform data conversion operations until the proportion of non-standard data to the original data reaches a set threshold.
Citation Information
Cited By
Method for enhancing data governance capability based on AI large model
CN119719023A
Consistency calibration and standardized output method and system of AI model identification result
CN119917647A
Method and System for Consistency Calibration and Standardized Output of AI Model Recognition Results
CN119917647B
Method and system for intelligently extracting data elements for national data standard file
CN120336507A
Enterprise data analysis method and device
CN121660039A
