Training data generation method and device, computer equipment and readable storage medium

Abstract

The invention relates to a training data generation method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: distributing the same to-be-labeled data to a plurality of labeled accounts; obtaining a matching result from a database according to the to-be-labeled data, and performing labeling processing according to the matching result in combination with a preset prompt word to obtain labeling information; acquiring annotation information corresponding to the to-be-annotated data fed back by each annotation account in a plurality of annotation accounts; comparing the multiple pieces of annotation information corresponding to the to-be-annotated data, and obtaining target annotation information of the to-be-annotated data according to a comparison result; the to-be-labeled data and the corresponding target labeling information form training data. Automatic labeling and automatic comparison are achieved, and labeling efficiency and labeling accuracy are improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence application technology, and in particular to a training data generation method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] The data source for fine-tuning training of large models is generally the training corpus data of the model, which is annotated, refined and verified by annotators.

[0003] In traditional techniques, corpus data is first generated by business personnel who compile relevant question sets and business knowledge based on data aggregation templates. Then, annotation personnel train the corpus template based on the model, using manual annotation and manual DSL (Domain-Specific Language) sentence writing. However, traditional corpus data generation methods rely on manual annotation, which is prone to errors and omissions, resulting in low annotation efficiency and accuracy. Summary of the Invention

[0004] Therefore, it is necessary to provide a training data generation method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can improve the efficiency and accuracy of annotation, in order to address the above-mentioned technical problems.

[0005] Firstly, this application provides a method for generating training data, including:

[0006] Assigning the same data to be labeled to multiple labeling accounts;

[0007] The matching results are obtained from the database based on the data to be labeled, and the labeling information is obtained by combining the matching results with the preset prompts.

[0008] Obtain the annotation information corresponding to the data to be annotated from each of the multiple annotation accounts;

[0009] The multiple annotation information corresponding to the data to be annotated are compared, and the target annotation information of the data to be annotated is obtained based on the comparison result; the data to be annotated and the corresponding target annotation information constitute a training data.

[0010] Secondly, this application also provides a training data generation apparatus, comprising:

[0011] The assignment module is used to assign the same data to multiple annotation accounts.

[0012] The annotation module is used to obtain matching results from the database based on the data to be annotated, and to perform annotation processing based on the matching results and preset prompts to obtain annotation information;

[0013] The acquisition module is used to acquire the annotation information corresponding to the data to be annotated from each of the multiple annotation accounts; the comparison module is used to compare the multiple annotation information corresponding to the data to be annotated, and obtain the target annotation information of the data to be annotated based on the comparison result; the data to be annotated and the corresponding target annotation information constitute a training data.

[0014] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method of the first aspect.

[0015] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method of the first aspect.

[0016] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the method of the first aspect.

[0017] The aforementioned training data generation method, apparatus, computer equipment, computer-readable storage medium, and computer program product allocate the same data to be labeled to multiple labeling accounts. Each labeling account retrieves matching results from the database based on the data to be labeled, and performs labeling processing based on preset prompts in the matching results to obtain labeling information. This achieves automatic labeling of the data to be labeled, obtaining labeling information, acquiring the labeling information corresponding to the data to be labeled returned by each of the multiple labeling accounts, comparing them, and obtaining the target labeling information based on the comparison results. This avoids inaccurate labeling information caused by a single label, improves the accuracy of data labeling, and improves comparison efficiency through automatic comparison, further enhancing labeling efficiency. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a diagram illustrating the application environment of a training data generation method in one embodiment.

[0020] Figure 2 This is a flowchart of a training data generation method in one embodiment;

[0021] Figure 3This is a flowchart of one embodiment for obtaining matching results from a database based on the data to be labeled, and performing labeling processing based on the matching results and preset prompts to obtain labeling information;

[0022] Figure 4 This is a flowchart illustrating how to perform semantic similarity matching between the data to be labeled and vectorized retrieval data in a vector database in one embodiment, thereby obtaining a second matching result.

[0023] Figure 5 A flowchart of a training data generation method in another embodiment;

[0024] Figure 6 This describes the process of constructing a vector database in one embodiment;

[0025] Figure 7 This is a schematic diagram of the keyword recall engine architecture in one embodiment;

[0026] Figure 8 This is a schematic diagram of a process for retrieving matching results from a database based on a question in one embodiment;

[0027] Figure 9 This is a structural block diagram of a training data generation device in one embodiment;

[0028] Figure 10 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0030] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0031] The training data generation method provided in this application embodiment can be applied to, for example... Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or located on the cloud or other network servers. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, drones, low-altitude aircraft, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, projection devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Head-mounted devices can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. Server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0032] In one exemplary embodiment, such as Figure 2 As shown, a training data generation method is provided, which can be applied to... Figure 1 The following steps are used as an example of an intelligent agent on a terminal or server: steps 202 to 208.

[0033] Step 202: Assign the same data to be labeled to multiple labeling accounts.

[0034] Data used for model training needs to be pre-labeled; the data to be labeled refers to the data that needs to be labeled. This data can be various questions from a collected set of questions, or fragments of user input, etc. A labeling account is used to uniquely identify the entity performing the labeling task. This entity can be a labeler or a labeling robot, etc. One labeling account can log in to one labeling terminal, or multiple labeling accounts can log in to the same labeling terminal.

[0035] For example, a first account with task allocation permissions creates a labeling task through its first terminal, distributing the same unlabeled data to multiple labeling accounts. This first account with task allocation permissions could be an account belonging to an algorithm engineer, a quality inspector, or similar personnel. Furthermore, the first account can create a labeling task through the first terminal, distributing the same unlabeled dataset to multiple labeling accounts. The same unlabeled dataset includes one or more unlabeled data points.

[0036] Step 204: Obtain the matching results from the database based on the data to be labeled, and perform labeling processing based on the matching results and preset prompts to obtain the labeling information.

[0037] The database can store business knowledge and / or corresponding vector data. Each annotation account's annotation terminal matches the data to be annotated against the business knowledge and / or vector data in the database to obtain the matching results. Then, a Large Language Model (LLM) processes the matching results and pre-defined prompts to obtain annotation information. The prompts can be constraints on the annotation process. Constraints can include task descriptions, definitions and formats of output parameters, and output format requirements. The annotation terminal inputs the matching results and prompts into the LLM, which then processes the matching results based on the prompts to obtain the annotation information.

[0038] Step 206: Obtain the annotation information corresponding to the data to be annotated from each of the multiple annotation accounts.

[0039] For example, a second account with quality inspection privileges can obtain the annotation information corresponding to the data to be annotated from each of the multiple annotation accounts. In possible implementations, the first account and the second account can be the same account or different accounts.

[0040] Step 208: Compare the multiple annotation information corresponding to the data to be annotated, and obtain the target annotation information of the data to be annotated based on the comparison results; the data to be annotated and the corresponding target annotation information constitute a training data.

[0041] For example, a second terminal with a second account possessing quality inspection privileges automatically compares multiple annotation information corresponding to the data to be labeled. Based on the comparison results, the similarities and differences among the multiple annotation information can be obtained. The differences are adjusted to obtain adjusted information, and the similarities and adjusted information are used as target annotation information. The data to be labeled and the corresponding target annotation information are combined into a training dataset. Further, the target annotation information corresponding to each piece of data in the dataset to be labeled is obtained to obtain the training dataset.

[0042] In this embodiment, the same data to be labeled is assigned to multiple labeling accounts. Each labeling account retrieves matching results from the database based on the data to be labeled, and performs labeling processing based on preset prompts in the matching results to obtain labeling information. This achieves automatic labeling of the data to be labeled and obtaining labeling information. It also retrieves the labeling information corresponding to the data to be labeled returned by each of the multiple labeling accounts, compares them, and obtains the target labeling information based on the comparison results. This avoids inaccurate labeling information caused by a single labeling, improves the accuracy of data labeling, and improves comparison efficiency by performing automatic comparison, thereby further improving labeling efficiency.

[0043] In one exemplary embodiment, the database includes at least one relational database and a vector database; the relational database stores business knowledge; and the vector database stores vectorized retrieval data of the business knowledge.

[0044] Business knowledge can include metrics, dimensions, dimension values, keywords, synonyms, etc. Business knowledge can be any business-related content. For example, business knowledge could be related to enterprise resource management systems, or it could be related to financial management, supply chain management, production management, sales and customer management, human resource management, etc. The raw information of business knowledge is stored in a relational database. An embedded vector model is used to vectorize the business knowledge to obtain vectorized retrieval data, which is then stored in a vector database. For example... Figure 3 As shown, the matching results are retrieved from the database based on the data to be labeled, and the labeling information is obtained by combining the matching results with preset prompts, including:

[0045] Step 302: Perform a literal match between the data to be labeled and the business knowledge in the relational database to obtain the first matching result.

[0046] Literal matching can refer to matching words that are exactly the same, i.e., exact matching, such as finding content where the field value is exactly equal to X.

[0047] In an exemplary embodiment, performing literal matching between the data to be labeled and business knowledge in a relational database to obtain a first matching result includes: segmenting the data to be labeled into words to obtain segmentation results, and performing character matching between each segmentation result and business knowledge in the relational database to obtain a first matching result.

[0048] The N-Gram algorithm can be used to segment the labeled data into word groups. Each word group is then compared with the original word group corresponding to business knowledge in a relational database. The business knowledge corresponding to the matched original word group is taken as the first matching result. The first matching result can be stored in a first matching list, which can be an exact matching list. Word segmentation matching can obtain the first matching result relatively accurately. The N-Gram algorithm is an algorithm based on a statistical language model.

[0049] Step 304: Perform semantic similarity matching between the data to be labeled and the vectorized retrieval data in the vector database to obtain the second matching result.

[0050] Semantic similarity refers to the similarity based on character intent. A semantic similarity threshold can be pre-configured, and matching results with semantic similarity values ​​greater than the threshold can be selected as secondary matching results. These secondary matching results are then stored in a secondary matching list, which can be a fuzzy matching list.

[0051] Step 306: Based on at least one of the first matching result and the second matching result, and in conjunction with a preset prompt, perform annotation processing to obtain annotation information.

[0052] If the database includes a relational database, the data to be labeled is matched literally with the business knowledge in the relational database to obtain the first matching result. The labeling information is then obtained by using a large language model based on the first matching result and the preset prompts.

[0053] If the database includes a vector database, the data to be labeled is semantically similar to the vectorized retrieval data in the vector database to obtain a second matching result. The large language model then performs labeling processing based on the first matching result and preset prompts to obtain the labeled information.

[0054] If the database includes a relational database and a vector database, the data to be labeled is matched literally with the business knowledge in the relational database to obtain the first matching result, and the data to be labeled is matched semantically with the vectorized retrieval data in the vector database to obtain the second matching result. The labeling information is obtained by using a large language model to perform labeling processing based on the first matching result, the second matching result and the preset prompts.

[0055] Matching based on relational databases and / or vector databases can automatically match relevant content to be labeled, providing it to large language models for labeling, thus achieving automatic labeling of training data.

[0056] In an exemplary embodiment, semantic similarity matching is performed between the data to be labeled and vectorized retrieval data in a vector database to obtain a second matching result. This includes: vectorizing the data to be labeled to obtain vectorized data to be labeled; performing semantic similarity matching between the vectorized data to be labeled and vectorized retrieval data in the vector database to obtain a first retrieval result; and using the first retrieval result as the second matching result. After vectorizing the data to be labeled, semantic similarity matching is performed between the vectorized data to be labeled and vectorized retrieval data in the vector database to obtain a first retrieval result, such as a matched vectorized business knowledge fragment. Based on the primary key of the vectorized business knowledge fragment, the corresponding business knowledge is retrieved from a relational database as the second matching result. Utilizing vectorized retrieval data from the vector database to match the vectorized data to be labeled reduces the amount of data to be matched, improving both matching accuracy and efficiency.

[0057] In one exemplary embodiment, a filtering thesaurus can be used to filter business knowledge, resulting in filtered business knowledge, which can then be used as full-text search data. For example... Figure 4As shown, the vector database also stores full-text search data; semantic similarity matching is performed between the data to be labeled and the vectorized search data in the vector database to obtain a second matching result, including:

[0058] Step 402: Vectorize the data to be labeled to obtain the vectorized data to be labeled.

[0059] For example, the data to be labeled is converted into vectorized data to be labeled, i.e., a query vector.

[0060] Step 404: Perform semantic similarity matching between the vectorized data to be labeled and the vectorized search data in the vector database to obtain the first search result.

[0061] For example, the semantic similarity value between the vectorized data to be labeled and the vectorized retrieval data can be represented by calculating the cosine similarity, Euclidean distance, Manhattan distance, or Pearson correlation coefficient. The vectorized retrieval data can then quickly find the vector most similar to the vectorized data to be labeled. Vectorized retrieval data with semantic similarity values ​​greater than a semantic similarity threshold can be identified as the first retrieval result.

[0062] Step 406: Match the data to be labeled with the full-text search data in the vector database to obtain the second search result.

[0063] For example, an inverted index algorithm (such as the BM25 algorithm) can be used to match keywords between the data to be labeled and the full-text search data to calculate a relevance score. Full-text search data with a relevance score greater than a relevance score threshold is determined as the second search result.

[0064] Step 408: Obtain the second matching result based on the first search result and the second search result.

[0065] The first and second search results can be used as the second matching result, or the target search result can be further filtered from the first and second search results as the second matching result. By matching the data to be labeled with full-text search data, and by vector matching the data to be labeled with vectorized search data, a hybrid search is achieved, which can obtain more comprehensive relevant content of the data to be labeled.

[0066] In an exemplary embodiment, step 408, obtaining a second matching result based on the first search result and the second search result, includes: merging the first search result and the second search result to obtain candidate search results; determining a preset number of target search results from the candidate search results; for each target search result in the preset number of target search results, obtaining the corresponding target business knowledge from the relational database based on the primary key of the business knowledge fragment in the target search result; and using the target business knowledge as the second matching result.

[0067] A reciprocal sorting fusion algorithm is used to merge the first and second search results to obtain candidate search results. The merging principle is to assign a higher score to each business knowledge fragment that appears in both the first and second search results, thereby combining keyword matching and semantic relevance to obtain a comprehensive score for each candidate search result. A predetermined number of target search results are then determined from the candidate search results based on the comprehensive score, from highest to lowest. The target search results include business knowledge fragments. The primary key of a business knowledge fragment can be a keyword or key value, etc. For example, if the business knowledge fragment is the WeChat application, and the primary key is "WeChat," the relational database records: "WeChat application is a social software." Retrieving the target business knowledge from the relational database based on the primary key is: "WeChat application is a social software." The vector database stores the necessary fields for searching (such as processed text, vectors, and primary keys) instead of the original business knowledge, making the index smaller and the retrieval faster. Retrieving more detailed target business knowledge from the relational database based on the primary key of the business knowledge fragment in the target search results can reduce network transmission burden. Using the target business knowledge as the second matching result facilitates more accurate subsequent annotation.

[0068] In an exemplary embodiment, step 408, obtaining the second matching result based on the first and second search results, includes: deduplicating the first and second search results to obtain candidate search results; selecting a preset number of target search results from high to low based on the semantic similarity score of each candidate search result; retrieving the corresponding target business knowledge from the relational database based on the primary key of the business knowledge fragment in the target search results; and using the target business knowledge as the second matching result. Directly deduplicating the search results and selecting a preset number of candidate search results from high to low based on the semantic similarity score as the target search results allows for rapid acquisition of the second matching result.

[0069] In an exemplary embodiment, annotation information is obtained by combining at least one of the first matching result and the second matching result with a preset prompt, including: annotating the first matching result, the second matching result and the preset prompt through a large language model to generate quadruple information; the prompt includes a task description, the definition and format of the quadruple, and the output format requirements; the quadruple information includes fragments input by the user, words in the database, keyword types and auxiliary information.

[0070] The data to be labeled can be user questions, and the fragments input by the user can be entities in the user questions, i.e., entities in the data to be labeled. The words in the database (which can be a keyword dictionary) can be the names of the matching entities in the database. The keyword type can be the keyword type corresponding to the matching entity. The supplementary information can be the supplementary information corresponding to the matching entity. For example, if the fragment input by the user is "month-on-month comparison," the word in the database can be "month-on-month comparison," the keyword type can be "data retrieval operator," and the supplementary information can be "month-on-month comparison."

[0071] By using the prompts to annotate the first and second matching results and generate quadruple information, we can obtain more standardized quadruple information, provide detailed annotation information, and reduce unnecessary output.

[0072] In an exemplary embodiment, the process of annotating the first matching result, the second matching result, and the preset prompt using a large language model to generate four-tuple information includes: deduplicating the first matching result and the second matching result to obtain deduplicated matching results; and annotating the deduplicated matching results and the preset prompt using a large language model to generate four-tuple information.

[0073] Deduplication of the first and second matching results, i.e., keeping only one of the same matching results, can reduce the amount of data processing for large language models and improve annotation efficiency.

[0074] In an exemplary embodiment, multiple annotation information corresponding to the data to be annotated are compared, and the target annotation information of the data to be annotated is obtained based on the comparison result. This includes: converting each quadruple information of the data to be annotated into corresponding domain-specific language annotation information; comparing the user-input fragments, words in the database, keyword types and auxiliary information in the multiple quadruple information with the corresponding domain-specific language annotation information; and obtaining the target annotation information of the data to be annotated based on the comparison result.

[0075] The rules engine can convert quadruple information into corresponding DSL annotation information. DSL annotation information can include VMETRICS, QUERY, FILTER, and AGGREGATE information. VMETRICS can be virtual or custom metrics, content not explicitly mentioned in the search results, and can be custom-calculated metrics or pre-calculated metric sets used in a specific context. QUERY is used for full-text search and matching to find the most relevant documents to the search terms and calculates a relevance score. FILTER is used to quickly filter documents with absolute matching conditions, such as status, date range, etc. AGGREGATE is used for aggregation analysis, performing statistical analysis on the query results, such as grouping counts, calculating averages, maximum values, etc. The quadruple information is used to generate corresponding QUERY / FILTER information, such as: Department -> Dimension Name -> FILTER Department = 'ALL', Monthly Subscription Revenue -> Metric Name -> Monthly Subscription Revenue.

[0076] The process compares the user input fragments, database words, keyword types, and associated information, along with the corresponding domain-specific language annotations, from multiple quadruples of information. For example, if there are three quadruples of information and domain-specific speech annotations, the comparison is as follows: the user input fragments in the first quadruple of information are compared with those in the second quadruple of information; the database words in the first quadruple of information are compared with those in the second quadruple of information; the keyword types in the first quadruple of information are compared with those in the second quadruple of information; and the first quadruple of information... The auxiliary information in the first quadruple is compared with the auxiliary information in the second quadruple. Similarly, the DSL annotation information corresponding to the first quadruple is compared with the DSL annotation information corresponding to the second quadruple, and vice versa. Based on the comparison results, the similarities and differences among the multiple quadruple information can be obtained. The differences are adjusted to obtain adjusted information, and the similarities and adjusted information are used as the target annotation information.

[0077] By performing precise one-to-one matching of quadruple information and DSL information at the field level, the system automatically identifies discrepancies, effectively reducing the complexity of quality inspection work for quality control personnel and improving work efficiency.

[0078] In an exemplary embodiment, multiple annotation information corresponding to the data to be annotated are compared, and the target annotation information of the data to be annotated is obtained based on the comparison results. This includes: comparing the user-input fragments, words in the database, keyword types, and auxiliary information in the multiple quadruple information of the data to be annotated one by one; and obtaining the target annotation information based on the comparison results. By comparing the user-input fragments, words in the database, keyword types, and auxiliary information in the quadruple information one by one to obtain the comparison results, and adjusting the annotation information based on the comparison results, the target annotation information can be obtained, which can improve the efficiency of annotation information generation.

[0079] In an exemplary embodiment, the annotation information may further include one or more of time-aggregated annotations and time-filtered annotations. A rule engine can be used to convert time-aggregated annotations into corresponding DSL aggregated information, and time-filtered annotations into corresponding DSL filtered information. For example, time-aggregated annotations can generate corresponding AGGREGATE information in the DSL, such as: Last Five Years -> Date Aggregation -> AGGREGATE Year. Time-filtered annotations can be converted into corresponding FILTER information, such as: 2025 Year -> Date Filter -> FILTER Date = '2025'. Converting both time-aggregated and time-filtered annotations into corresponding DSL information enriches the annotation information and improves annotation efficiency and accuracy.

[0080] In an exemplary embodiment, when the annotation information includes time-aggregated annotations, comparing multiple annotation information corresponding to the data to be annotated further includes: comparing multiple DSL aggregation information one by one to obtain target aggregation information.

[0081] In an exemplary embodiment, when the annotation information includes time-filtered annotations, comparing multiple annotation information corresponding to the data to be annotated further includes: comparing multiple DSL filtering information one by one to obtain target filtering information.

[0082] In an exemplary embodiment, when the annotation information includes time aggregation annotations and time filtering annotations, comparing multiple annotation information corresponding to the data to be annotated further includes: comparing multiple DSL aggregation information one by one to obtain target aggregation information; and comparing multiple DSL filtering information one by one to obtain target filtering information.

[0083] Accordingly, target annotation information can include one or more of target aggregation information or target filtering information. This further enriches the annotation information and improves annotation efficiency and accuracy. The following section uses the annotation of a question set and related business knowledge as an example to illustrate the training data generation method. Figure 5 As shown, the training data generation methods include:

[0084] (1) The product personnel's terminal will store the problem set collected by the business into the annotation platform through the interface, and store the relevant business knowledge into the knowledge base (i.e., database) through the interface.

[0085] The collected business knowledge can include dimensions, metrics, dimension values, business names, synonyms, etc.

[0086] (2) The annotation platform uses vector model capabilities to vectorize business knowledge according to different knowledge types to obtain vectorized retrieval data, and uses a filter word library to filter some default tone words and custom filter words to obtain full-text retrieval data. The vectorized retrieval data and full-text retrieval data are stored in the vector database, and the business knowledge is stored in the relational database.

[0087] like Figure 6 As shown, business knowledge is categorized according to business type, establishing two retrieval fields for different purposes: a full-text retrieval field and a vector retrieval field. Using a filtering thesaurus, default interjections (such as "ah," "ne," and "de") and custom filter words in the business knowledge are filtered to obtain full-text retrieval data, minimizing interference with matching. This full-text retrieval data serves as the full-text retrieval field. Utilizing the capabilities of the Embedding model, the business knowledge is vectorized into fixed-dimensional vectors, resulting in vectorized retrieval data, which serves as the vector retrieval field. Both the vectorized retrieval data and the full-text retrieval data are stored in a vector database. Embedding is an algorithm that converts text and images into fixed-dimensional vector representations. The vector database can be ElasticSearch. ElasticSearch is a distributed, highly scalable, and real-time search and data analysis engine built on Apache Lucene, designed specifically for handling large-scale data.

[0088] (3) The first account with task allocation authority creates an annotation task on the annotation platform and assigns the same set of questions to the second accounts corresponding to different annotators. The set of questions includes one or more questions, which are the data to be annotated.

[0089] For example, an algorithm engineer logs into the first terminal using a first account with task allocation permissions, accesses the annotation platform, creates annotation tasks on the annotation platform, and assigns the annotation tasks to different annotators.

[0090] (4) The second account with annotation permissions obtains the corresponding annotation task and clicks the automated annotation button to trigger automated annotation. The automated annotation process includes a keyword recall engine and RAG (Retrieval-augmented Generation) technology. The keyword recall engine uses the N-Gram algorithm to perform accurate matching and semantic similarity matching for the questions to be annotated, and then submits the recalled matching results to LLM for annotation processing in conjunction with preset prompts to generate four-tuple information. The four-tuple information includes the fragment input by the user, the words in the keyword dictionary, the keyword type, and the auxiliary information, as shown in Table 1.

[0091] Table 1

[0092] User input fragment Words in the keyword dictionary Keyword types Additional Information Top Sales Excellent sales Custom Filtering User Defined Filter Product A Company B's product A Dimension Enumeration Values Product Line Percentage percentage Function percentage Proportion Meets standards Meets standards Conditional operators Meets standards Month-on-month Month-on-month Number retrieval operator Month-on-month This year time Time Filtering function

[0093] like Figure 7 As shown, the keyword recall engine consists of two parts: keyword writing and recall. Keyword writing mainly involves structuring business knowledge and storing it in a relational database through an API interface, and then vectorizing the business knowledge and storing it in a vector database using the Embedding model capability. Keyword recall includes: exact match recall and semantic similarity match recall. The core engine functions of the keyword recall engine include vectorization, word segmentation, version management, N-Gram algorithm, vector retrieval, document retrieval, and hybrid retrieval. Vectorization involves vectorizing business knowledge and writing it into the vector database. Word segmentation refers to segmenting the question using N-Gram. Version management refers to managing the versions of the keyword recall engine. Vector retrieval refers to searching in the vector database. Document retrieval refers to full-text search. Hybrid retrieval refers to vector retrieval and full-text search. The database includes a physical library, a vector library (i.e., a vector database), and a document library. The physical library is used to simulate motion and collisions in the physical world, handling physical property parameters such as rigid bodies, shapes, forces, and velocities for plastic types. The vector library is used to store, index, and quickly retrieve high-dimensional vector data, enabling similarity search. It handles data types converted from unstructured data (such as text and images) into embedded vectors, and is applied to retrieval enhancement generation, semantic search, personalized search, and image search. The document library is used to store, manage, and collaboratively process various document files, handling data types such as document files, metadata, and version information. It enables version control, access control, metadata association, and full-text search.

[0094] like Figure 8As shown, the keyword recall process involves retrieving matching results from the database based on the question, including: using the N-Gram algorithm to decompose the question to be labeled into a list of question fragments; sequentially matching each question fragment in the decomposed list with the original phrases (including keywords and synonyms) of business knowledge in the relational database (PostgreSQL) of the knowledge base (i.e., full-text matching) to obtain the first matching result, which can be an exact matching phrase; using the Embedding model, vectorizing the question to be labeled to obtain the vectorized question; performing semantic similarity matching (i.e., vector retrieval) between the vectorized question and the vectorized retrieval data in the vector database of the knowledge base to obtain the first retrieval result; performing semantic similarity matching (i.e., full-text retrieval) between the question and the full-text retrieval data in the vector database to obtain the second retrieval result; obtaining the second matching result based on the first and second retrieval results, which can be a fuzzy matching recalled phrase; deduplicating the exact matching phrases and fuzzy matching phrases and inputting them into the LLM; the LLM, combined with the prompt, automatically generates four-tuple labeled data.

[0095] Furthermore, exact match phrases are added to the exact match list. The second matching result, obtained based on the first and second search results, includes: obtaining a predetermined number of target matching results from the first and second search results according to semantic similarity values; for each target search result within the predetermined number of target search results, matching the corresponding target business knowledge from the relational database based on the primary key of the business knowledge in the target search result; and obtaining a semantic similarity matching list based on the target business knowledge. The contents of the exact match list and the semantic similarity matching list are deduplicated, and then the deduplicated exact match list and semantic similarity matching list are provided to the LLM.

[0096] The prompt first defines the model's role to limit its task processing scope. Next, it defines the model's task processing method: the large language model primarily extracts question fragments from user questions and combines this with information from the **exact match query list** and **fuzzy match query list** retrieved from the keyword retrieval engine to generate corresponding four-tuple data. The exact match query list and fuzzy match query list have similar structures, both containing a two-dimensional array. An example data structure is: [{"Entity Name":"","Keyword Type":"","Auxiliary Information":"Similarity Score",""}], where the similarity score only exists in the fuzzy match query list. The entity name originates from user-uploaded business knowledge fragments; the keyword type includes business types such as indicator name, dimension name, dimension enumeration value, and custom query; the auxiliary information includes user-defined business knowledge; and the similarity score comes from the semantic similarity value retrieved by vector matching. The model ultimately needs to generate four-tuple information, such as: the fragment input by the user, the word in the keyword dictionary, the keyword type, and the auxiliary information. Next, based on the illusion phenomenon inherent in the large language model itself, some precautions were added, such as specifying that the source of words in the keyword field of the generated quadruple must be entities in the query list; otherwise, it will cause engineering matching failure. User fragments were strictly extracted to ensure that they could be correctly echoed to the question during annotation. An empty return definition was also added. Finally, the final output format of the large language model was limited to an XML-like format and strictly output according to the task requirements. This defined prompt, combined with the query list retrieved by the keyword recall engine, can effectively solve most annotation tasks and significantly reduce the workload of annotators.

[0097] The prompt includes a task description, the definition and format of the four-tuple, and output format requirements. The task description can include the order of task steps, the content of each step, and precautions. The order of task steps can include, for example, the execution order of steps one, two, and three. The content of each task step can include the content of steps one, two, and three, such as: Step 1: Match and extract entities from the exact match query list to generate four-tuples; Step 2: Match and extract entities from the fuzzy match list to generate four-tuples; Step 3: Integrate the results of steps one and two. Precautions can be set as needed, and can include matching priority, return requirements for matching failures, such as the priority of entities in the exact match list being higher than the priority of entities in the fuzzy match list.

[0098] The definition of a quadruple can include its content, which includes a fragment of user input, words from the database, keyword types, and additional information. The format of a quadruple can include the output order, such as <fragment of user input, words from the database, keyword types, additional information>.

[0099] The output format requirements can be set as needed, and can include XML format. For example, it can include the output results of each step and the final result. The output results of each step might be: the first step outputs the four-tuple information generated from the entities matched in the exact match list; the second step outputs the four-tuple information generated from the entities matched in the fuzzy match list. The final result can include the four-tuple information from the integrated output of the third step.

[0100] The prompt message can be as follows:

[0101]

[0102] (5) Annotators can also use the knowledge fragments retrieved by the keyword recall engine to manually annotate the corresponding quadruple information.

[0103] (6) Render the quadruple information into the corresponding DSL annotation information according to the rule engine.

[0104] The rule engine is used to convert time aggregation annotations, time filtering annotations, and four-tuple information into corresponding DSL corpora. Time aggregation annotations are then used to generate AGGREGATE information within the corresponding DSL. Time filtering annotations are converted into corresponding FILTER information. Finally, four-tuple information is used to generate corresponding QUERY / FILTER information.

[0105] (7) Use automated comparison tools to match multiple labeled quadruple information and DSL information. For example, match user input fragments, words in the keyword dictionary, keyword types, and auxiliary information of the quadruple one by one. Compare QUERY, FILTER, AGGREGAT, etc. in the DSL to identify different information. Output the comparison results to the annotation quality inspectors for quality inspection and correction. The third account of the annotation quality inspectors returns the issues that do not meet the data access conditions for re-annotation. Based on the comparison results of the issues that meet the data access conditions, obtain the target annotation information corresponding to the issue.

[0106] Specifically, automated comparison tools are used to perform full-text field-level matching and comparison of time aggregation annotations, time filtering annotations, quadruple information, and DSL information generated from quadruple information in the same annotation task by multiple annotators. This includes annotations automatically generated by the annotation platform and those manually annotated by annotators. For example, it checks whether time aggregation annotations and time filtering annotations are consistent, whether user-input fragments in quadruple information, words in keyword fields, keyword types, and auxiliary information match one by one, and whether the order and fields of VMETRICS, QUERY, FILTER, and AGGREGATE information in DSL information are equal. The matching time information, quadruple information, and DSL information are then identified and a list of differences is returned to the annotation quality inspectors for annotation data quality inspection and correction.

[0107] (8) A third account with model training privileges uses the marked training data to train the model and evaluates the training results.

[0108] In the aforementioned training data generation method, a labeling platform is used to create labeling tasks, and the same dataset to be labeled, i.e., the question set, is assigned to multiple different labelers, i.e., multiple different labeling accounts. Each labeling account uses the automatic labeling function to perform literal matching between each question in the question set and keywords and synonyms of business knowledge in the relational dataset to obtain the first matching result, and semantic similarity matching between the question and business knowledge or vectorized retrieval data in the vector database to obtain the second matching result. Based on the first and second matching results and preset prompts, four-tuple information is generated, realizing automated labeling and alleviating the problems of error-prone and time-consuming manual labeling by labelers. The automatic comparison of multiple four-tuple information effectively reduces the complexity of quality inspection work for quality inspectors and improves work efficiency. Based on the comparison results, the target labeling information corresponding to each question is determined, avoiding errors in individual labels and improving the accuracy of labeling. In addition, the aforementioned training data generation method also effectively shortens the overall model fine-tuning training time from data collection to labeling to training for large language models.

[0109] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0110] Based on the same inventive concept, this application also provides a training data generation apparatus for implementing the training data generation method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more training data generation apparatus embodiments provided below can be found in the limitations of the training data generation method described above, and will not be repeated here.

[0111] In one exemplary embodiment, such as Figure 9As shown, a training data generation device is provided, including an allocation module 910, a labeling module 920, an acquisition module 930, and a comparison module 940. Wherein:

[0112] The allocation module 910 is used to allocate the same data to be labeled to multiple labeling accounts.

[0113] The annotation module 920 is used to obtain matching results from the database based on the data to be annotated, and to perform annotation processing based on the matching results and preset prompts to obtain annotation information.

[0114] The acquisition module 930 is used to acquire the annotation information corresponding to the data to be annotated, which is fed back by each of the multiple annotation accounts.

[0115] The comparison module 940 is used to compare multiple annotation information corresponding to the data to be labeled, and obtain the target annotation information of the data to be labeled based on the comparison result; the data to be labeled and the corresponding target annotation information constitute a training data.

[0116] In one exemplary embodiment, the database includes at least one relational database and a vector database; the relational database stores business knowledge, which includes business knowledge and attribute information of the business knowledge; the vector database stores vectorized retrieval data corresponding to the business knowledge.

[0117] The annotation module 930 is used to perform literal matching between the data to be annotated and the business knowledge in the relational database to obtain a first matching result; to perform semantic similarity matching between the data to be annotated and the vectorized retrieval data in the vector database to obtain a second matching result; and to perform annotation processing based on at least one of the first matching result and the second matching result, combined with a preset prompt, to obtain annotation information.

[0118] In an exemplary embodiment, the vector database also stores full-text search data; the annotation module 930 is further configured to perform vectorization processing on the data to be annotated to obtain vectorized data to be annotated; perform semantic similarity matching between the vectorized data to be annotated and the vectorized search data in the vector database to obtain a first search result; perform semantic similarity matching between the data to be annotated and the full-text search data in the vector database to obtain a second search result; and obtain a second matching result based on the first search result and the second search result.

[0119] In an exemplary embodiment, the annotation module 930 is further configured to merge the first search result and the second search result using a reciprocal sorting fusion algorithm to obtain candidate search results, determine a preset number of target search results from the candidate search results; for each target search result in the preset number of target search results, obtain the corresponding target business knowledge from the relational database according to the primary key of the business knowledge fragment in the target search result; and use the target business knowledge as the second matching result.

[0120] In an exemplary embodiment, the annotation module 930 is further configured to annotate the first matching result, the second matching result, and the preset prompt using a large language model to generate four-tuple information; the prompt includes a task description, the definition and format of the four-tuple, and output format requirements; the four-tuple information includes fragments input by the user, words in the database, keyword types, and supplementary information.

[0121] In an exemplary embodiment, the comparison module 940 is further configured to convert each quadruple information of the data to be labeled into corresponding domain-specific language annotation information; compare the user-input fragments, words in the database, keyword types and auxiliary information and corresponding domain-specific language annotation information in the multiple quadruple information; and obtain the target annotation information of the data to be labeled based on the comparison results.

[0122] Each module in the aforementioned training data generation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0123] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 10As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements an intelligent agent interaction method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0124] Those skilled in the art will understand that Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0125] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method described above.

[0126] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the above-described method.

[0127] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the methods described above.

[0128] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0129] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0130] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0131] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for generating training data, characterized in that, The method includes: Assigning the same data to be labeled to multiple labeling accounts; The matching results are obtained from the database based on the data to be labeled, and the labeling information is obtained by combining the matching results with the preset prompts. Obtain the annotation information corresponding to the data to be annotated from each of the multiple annotation accounts; The multiple annotation information corresponding to the data to be annotated are compared, and the target annotation information of the data to be annotated is obtained based on the comparison result; the data to be annotated and the corresponding target annotation information constitute a training data.

2. The method according to claim 1, characterized in that, The database includes at least one of a relational database and a vector database; the relational database stores business knowledge; the vector database stores vectorized retrieval data corresponding to the business knowledge. The step of obtaining matching results from the database based on the data to be labeled, and performing labeling processing based on the matching results and preset prompts to obtain labeling information includes: The data to be labeled is matched literally with the business knowledge in the relational database to obtain the first matching result; The data to be labeled is semantically similar to the vectorized retrieval data in the vector database to obtain a second matching result; Based on at least one of the first matching result and the second matching result, annotation information is obtained by combining preset prompts.

3. The method according to claim 2, characterized in that, The vector database also stores full-text search data corresponding to business knowledge; the step of performing semantic similarity matching between the data to be labeled and the vectorized search data in the vector database to obtain a second matching result includes: The data to be labeled is vectorized to obtain the vectorized data to be labeled. The vectorized data to be labeled is semantically similar to the vectorized search data in the vector database to obtain the first search result; The data to be labeled is semantically similar to the full-text search data in the vector database to obtain the second search result; A second matching result is obtained based on the first search result and the second search result.

4. The method according to claim 3, characterized in that, The step of obtaining the second matching result based on the first search result and the second search result includes: The first search result and the second search result are merged to obtain candidate search results, and a preset number of target search results are determined from the candidate search results. For each target search result in a preset number of target search results, the corresponding target business knowledge is retrieved from the relational database based on the primary key of the business knowledge fragment in the target search result. The target business knowledge is used as the second matching result.

5. The method according to claim 2, characterized in that, The step of obtaining annotation information by combining at least one of the first matching result and the second matching result with a preset prompt includes: The first matching result, the second matching result, and the preset prompt are labeled using a large language model to generate four-tuple information. The prompt includes a task description, the definition and format of the four-tuple, and the output format requirements. The four-tuple information includes fragments input by the user, words in the database, keyword types, and supplementary information.

6. The method according to claim 5, characterized in that, The step of comparing multiple annotation information corresponding to the data to be annotated, and obtaining the target annotation information of the data to be annotated based on the comparison result, includes: Convert the information of each quadruple of the data to be labeled into corresponding domain-specific language annotation information; Compare the fragments of user input, words in the database, keyword types and auxiliary information, and corresponding domain-specific language annotation information in multiple quadruple information sets; The target annotation information of the data to be annotated is obtained based on the comparison results.

7. A training data generation device, characterized in that, The device includes: The assignment module is used to assign the same data to multiple annotation accounts. The annotation module is used to obtain matching results from the database based on the data to be annotated, and to perform annotation processing based on the matching results and preset prompts to obtain annotation information; The acquisition module is used to acquire the annotation information corresponding to the data to be annotated, fed back by each of the multiple annotation accounts; The comparison module is used to compare multiple annotation information corresponding to the data to be labeled, and obtain the target annotation information of the data to be labeled based on the comparison result; the data to be labeled and the corresponding target annotation information constitute a training data.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images