Data annotation method, device, equipment, medium and product

Abstract

The present application provides a data annotation method, device, equipment, medium and product. The method includes: obtaining data to be annotated and a trained annotation model; the data to be annotated is multimodal data in the target application field, and the vector database of the trained annotation model is embedded with a graph database of named entities in the target application field; the trained annotation model is obtained by training the preset annotation model based on the annotation results of the preset annotation model on the annotation training data and the user's feedback data on the annotation results; the data to be annotated is input into the trained annotation model, the trained annotation model is used to annotate the data to be annotated, and the annotation results of the data to be annotated are output. The present application can improve the efficiency and accuracy of data annotation.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data annotation method, device, equipment, medium and product. Background Art

[0002] The big model is a machine learning model with large-scale parameters and complex computing structure. It has powerful natural language processing and cross-modal data processing capabilities. It can not only generate fluent and coherent text and answer natural language questions, but also understand and generate images, audio and video content.

[0003] Large models usually require a large amount of labeled data for training. In addition, since the data that large models need to process often come from different sources and data types, only by effectively labeling and structuring data from different sources and data types can the natural language processing and cross-modal data processing capabilities of large models be improved. However, currently, labeled data is obtained by manually labeling unlabeled data. The manual labeling method is inefficient and error-prone, and cannot meet the training needs of large models. Summary of the invention

[0004] The present application provides a data labeling method, device, equipment, medium and product to solve the problems in the prior art of manually labeling data to obtain labeled data, which are inefficient and prone to errors and cannot meet the data requirements of large model training.

[0005] In a first aspect, the present application provides a data labeling method, comprising:

[0006] Obtain data to be annotated and a trained annotation model; the data to be annotated is multimodal data in the target application field, and the vector database of the trained annotation model is embedded with a graph database of named entities in the target application field; the trained annotation model is obtained by training the preset annotation model based on the annotation results of the preset annotation model on the annotation training data and the user's feedback data on the annotation results;

[0007] The data to be labeled is input into a trained labeling model, the trained labeling model is used to label the data to be labeled and a labeling result of the data to be labeled is output.

[0008] Optionally, the adopting a trained annotation model to annotate the data to be annotated includes:

[0009] Extracting fusion features of the data to be labeled in multiple modalities and identifying the intent of the data to be labeled;

[0010] In response to the intention of the data to be annotated involving entity relationships in a target application field, acquiring association relationships and association features of the fused features from the graph database;

[0011] In response to the data to be annotated being structured data, a mapping relationship between the fusion feature and the standard field is constructed, and a preset content processing strategy corresponding to the standard field is determined as a target content processing strategy corresponding to the data to be annotated; the standard field is a preset field of a target application field;

[0012] In response to the data to be annotated being unstructured data, determining a target content processing strategy corresponding to the data to be annotated from a plurality of preset content processing strategies according to the fusion feature, the association relationship and the association feature;

[0013] According to the target content processing strategy, a labeling result of the data to be labeled is determined.

[0014] Optionally, the target content processing strategy includes a preset processing object, a preset operator, and a preset operator logic; and determining the labeling result of the data to be labeled according to the target content processing strategy includes:

[0015] Obtaining attribute values ​​of preset processing objects from a production management database;

[0016] According to the preset operator logic, the preset operator in the target content processing strategy is called to process the attribute value of the preset processing object to obtain the labeling result of the data to be labeled.

[0017] Optionally, before inputting the data to be labeled into the trained labeling model, the method further includes:

[0018] Obtain annotated training data sets and preset annotation models;

[0019] Inputting the labeled training data set into a preset labeling model, using the preset labeling model to label each labeled training data in the labeled training data set, and obtaining a labeling result for each labeled training data;

[0020] Obtaining user feedback data on the labeling results of each labeling training data; the feedback data on the labeling results includes: accurate labeling and inaccurate labeling;

[0021] Using a set of labeled training data with inaccurate labels to update the labeled training data set;

[0022] Repeat the step from inputting the labeled training data set into the preset labeling model to the step of updating the labeled training data set using a set of labeled training data with inaccurate labels to adjust the parameters of the preset labeling model until the user's feedback data on the labeling results of each labeled training data is accurate, and the preset labeling model is determined as the trained labeling model.

[0023] Optionally, after labeling the data to be labeled using the trained labeling model and outputting the labeling result of the data to be labeled, the method further includes:

[0024] Obtaining a generated training data set and a preset content generation model; the generated training data set includes labeled data;

[0025] The preset content generation model is trained based on the reply content corresponding to each generated training data and the user's feedback data on each reply content to obtain a trained content generation model.

[0026] Optionally, the training of the preset content generation model based on the reply content corresponding to each generated training data and the user's feedback data on each reply content includes:

[0027] Input the generated training data set into the preset content generation model, use the preset content generation model to generate reply content for each generated training data in the generated training data set, and obtain reply content corresponding to each generated training data;

[0028] Obtaining feedback data of the user on the reply content corresponding to each generated training data, wherein the feedback data on the reply content includes: accurate reply and inaccurate reply;

[0029] Using the set of generated training data with inaccurate answers to update the generated training data set;

[0030] Repeat the steps from inputting the generated training data set into the preset content generation model to updating the generated training data set using a set of generated training data with inaccurate responses, so as to adjust the parameters of the preset content generation model, until the user's feedback data on the response content corresponding to each generated training data is accurate, and the preset content generation model is determined as the trained content generation model.

[0031] In a second aspect, the present application provides a data labeling device, comprising:

[0032] An acquisition module is used to acquire data to be annotated and a trained annotation model; the data to be annotated is multimodal data in the target application field, and the vector database of the trained annotation model is embedded with a graph database in the target application field; the trained annotation model is obtained by training the preset annotation model based on the annotation results of the preset annotation model on the annotation training data and the user's feedback data on the annotation results;

[0033] The labeling module is used to input the data to be labeled into the trained labeling model, use the trained labeling model to label the data to be labeled and output the labeling result of the data to be labeled.

[0034] In a third aspect, the present application provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor;

[0035] The memory stores computer-executable instructions;

[0036] The processor executes the computer-executable instructions stored in the memory to implement the data labeling method as described in the first aspect.

[0037] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the data labeling method as described in the first aspect.

[0038] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, is used to implement the data labeling method as described in the first aspect.

[0039] The data labeling method, device, equipment, medium and product provided by the present application are obtained by obtaining data to be labeled and a trained labeling model; the data to be labeled is multimodal data in the target application field, and the vector database of the trained labeling model is embedded with a graph database of named entities in the target application field; the trained labeling model is obtained by training the preset labeling model according to the labeling results of the preset labeling model on the labeling training data and the user's feedback data on the labeling results; the data to be labeled is input into the trained labeling model, the trained labeling model is used to label the data to be labeled and the labeling results of the data to be labeled are output. Since the vector database of the trained annotation model is embedded with the graph database of named entities in the target application field, the graph database of named entities in the target application field can inject the knowledge of the target application field into the trained annotation model, enhance the knowledge and reasoning ability of the large model, and therefore improve the accuracy of the large model in annotating the data to be annotated. At the same time, the trained annotation model is obtained by training the preset annotation model based on the annotation results of the preset annotation model on the annotated training data and the user's feedback data on the annotation results. During the training process of the trained annotation model, the user only needs to provide feedback on the annotation results without annotating, which can reduce the manual workload. At the same time, through the user's feedback on the annotation results, the parameters of the preset annotation model can be adjusted to improve the accuracy of the preset annotation model annotating the data, and then the trained annotation model is used to annotate the data to be annotated and output the annotation results of the data to be annotated. Manual annotation is not required, which can improve the data annotation efficiency and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0041] Figure 1 An application scenario diagram of the data annotation method provided in an embodiment of the present application;

[0042] Figure 2 A flowchart of a data annotation method provided in Example 1 of the present application;

[0043] Figure 3 A flowchart of a data annotation method provided in Example 2 of the present application;

[0044] Figure 4 A schematic diagram of the structure of a data labeling device provided in Embodiment 3 of the present application;

[0045] Figure 5 This is a schematic diagram of the structure of an electronic device provided in Example 4 of the present application.

[0046] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0047] The terms "including" and "having" and any variations thereof in the specification, claims and drawings of the embodiments of the present invention are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or apparatus.

[0048] First, the terms involved in this application are explained.

[0049] Multimodal data refers to data composed of two or more modalities. Modality refers to the form in which data or information is expressed. Data or information can be expressed in the form of text, images, video, audio, etc.

[0050] Vector databases store data in vector form. Vectors are a data type consisting of a series of numerical values, usually generated by machine learning models, and can represent various types of data, such as text, images, or other unstructured data.

[0051] A graph database is a non-relational database consisting of nodes and edges. Nodes represent entities, such as people, places, or things, and edges represent the relationships between entities.

[0052] The prior art involved in this application is described in detail below.

[0053] A large model refers to a machine learning model with large-scale parameters and complex computing structures. It has powerful natural language processing and cross-modal data processing capabilities. It can not only generate fluent and coherent text and answer natural language questions, but also understand and generate image, audio and video content. For example, a large language model can master the deep structure and semantics of a language by learning a large amount of text data, and can perform a variety of tasks such as text generation, translation, summarization, question and answer. Image processing models can perform tasks such as image recognition, classification, and generation. Multimodal models are models that can process and understand multiple data types, and can be used for video understanding, visual question and answer systems, automated content generation, etc.

[0054] Large models usually require a large amount of labeled data for training. In addition, since the data that large models need to process often come from different sources and data types, only by effectively labeling and structuring data from different sources and data types can the natural language processing and cross-modal data processing capabilities of large models be improved. However, currently, labeled data is obtained by manually labeling unlabeled data. The manual labeling method is inefficient and error-prone, and cannot meet the training needs of large models.

[0055] In summary, the existing method of manually labeling data to obtain labeled data is inefficient and error-prone, and cannot meet the data requirements for large model training.

[0056] In the face of the above technical problems in the prior art, the present application proposes the following inventive concept: when manual data labeling is prone to errors and inefficient, a large model can be used to label the data. In order to improve the accuracy of the large model labeled data, a graph database of named entities in the target application field can be embedded in the vector database of the large model, so that the large model can learn the features that should be focused on in the target application field from the graph database, enhance the knowledge and reasoning ability of the large model, thereby improving the accuracy of the large model in labeling data. At the same time, the large model is trained according to the user's feedback on the labeling results to further improve the accuracy of the large model in labeling data.

[0057] The following will introduce the application scenarios and network architecture of the data annotation method provided in the embodiment of the present application. When the following description refers to the drawings, unless otherwise indicated, the same data in different drawings represent the same or similar elements.

[0058] Figure 1 This is an application scenario diagram of the data annotation method provided in the embodiment of the present application. Figure 1 As shown, in an application scenario of the data labeling method provided in an embodiment of the present application, it includes: an electronic device 10.

[0059] The electronic device 10 is equipped with a trained annotation model. The vector database of the trained annotation model is embedded with a graph database of named entities in the target application field. The trained annotation model is obtained by training the preset annotation model based on the annotation results of the preset annotation model on the annotation training data and the user's feedback data on the annotation results.

[0060] The electronic device 10 obtains the data to be annotated and the trained annotation model. For example, the data to be annotated and the trained annotation model can be pre-stored in the memory of the electronic device 10, and the electronic device 10 can read the data to be annotated and the trained model from the memory. The data to be annotated is multimodal data in the target application field. The data to be annotated includes at least two of the voice data, text data, and video data in the target application field.

[0061] The electronic device 10 inputs the data to be labeled into the trained labeling model, labels the data to be labeled using the trained labeling model, and outputs the labeling result of the data to be labeled.

[0062] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The implementation methods described in the following embodiments do not represent all implementation methods consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims. The following specific embodiments may be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0063] Embodiment 1

[0064] Figure 2 This is a flow chart of the data annotation method provided in Example 1 of this application. Figure 2 As shown, the execution subject of this embodiment is a data labeling device, and the data labeling device is located in an electronic device. The data labeling method provided by this embodiment includes steps 201 to 202.

[0065] Step 201, obtain the data to be labeled and the trained labeling model; the data to be labeled is multimodal data in the target application field, and the vector database of the trained labeling model is embedded with a graph database of named entities in the target application field; the trained labeling model is obtained by training the preset labeling model based on the labeling results of the preset labeling model on the labeling training data and the user's feedback data on the labeling results.

[0066] In this embodiment, the graph database of named entities in the target application field can be established according to the industry specifications, production control data records, etc. of the target application field, and the entities and entity relationships in the graph database can be updated as the target application field develops and changes. The graph database defines encoding and fusion methods for multiple input forms such as images, text, and audio, which can help select and define the features of the target application field that the preset annotation model should focus on learning.

[0067] For example, the target application field may be the coal preparation field, and the graph database may be constructed based on the records of production control data of the coal preparation plant, the design specifications of the coal preparation plant, the coal preparation industry specifications, etc. The graph database may define the correlation between the ash content pass rate and the coal quality.

[0068] The preset annotation model and the trained annotation model may have the same model structure and different model parameters. The preset annotation model is an untrained trained annotation model.

[0069] The electronic device can input the labeled training data into the trained labeling model, and use the trained labeling model to label the labeled training data, obtain the labeling results of the labeled training data and output them. The user can provide feedback on the labeling results of the labeled training data, for example, marking whether the labeling results of the labeled training data are accurate or inaccurate, and marking inaccurate content.

[0070] The electronic device can input the user's feedback on the labeling results of the labeling training data into a preset labeling model to adjust the parameters of the preset labeling model, and can use the preset labeling model to re-label the labeling training data marked as inaccurate by the user until the user's feedback on the labeling training data is that the labeling results of the labeling training data are accurate, thereby obtaining a trained labeling model.

[0071] After the graph database is embedded in the vector database, the relationships between entities recorded in the graph database can also be retrieved in the form of vectors by the trained annotation model during the reasoning process, thereby enhancing the knowledge and reasoning capabilities of the trained model.

[0072] Step 202: input the data to be labeled into the trained labeling model, use the trained labeling model to label the data to be labeled and output the labeling result of the data to be labeled.

[0073] In this embodiment, the electronic device inputs the data to be labeled into the trained labeling model, labels the data to be labeled using the trained labeling model and outputs the labeling result of the labeled data, thereby obtaining the labeling result of the data to be labeled.

[0074] Optionally, before inputting the data to be labeled into the trained labeling model, the electronic device may also use the Enhanced Vectorization Engine (EVE) to preprocess the data of different modalities in the data to be labeled. The preprocessing includes: image standardization, word segmentation and embedding of text, spectrum analysis of audio, etc. The purpose of preprocessing is to input the data to be labeled into the trained labeling model in the best form.

[0075] The data labeling method provided in this embodiment obtains data to be labeled and a trained labeling model; the data to be labeled is multimodal data in the target application field, and the vector database of the trained labeling model is embedded with a graph database of named entities in the target application field; the trained labeling model is obtained by training a preset labeling model based on the labeling results of the preset labeling model on the labeling training data and the user's feedback data on the labeling results; the data to be labeled is input into the trained labeling model, the trained labeling model is used to label the data to be labeled, and the labeling results of the data to be labeled are output. Since the vector database of the trained annotation model is embedded with the graph database of named entities in the target application field, the graph database of named entities in the target application field can inject the knowledge of the target application field into the trained annotation model, enhance the knowledge and reasoning ability of the large model, and therefore improve the accuracy of the large model in annotating the data to be annotated. At the same time, the trained annotation model is obtained by training the preset annotation model based on the annotation results of the preset annotation model on the annotated training data and the user's feedback data on the annotation results. During the training process of the trained annotation model, the user only needs to provide feedback on the annotation results without annotating, which can reduce the manual workload. At the same time, through the user's feedback on the annotation results, the parameters of the preset annotation model can be adjusted to improve the accuracy of the annotation data of the preset annotation model, and then the trained annotation model is used to annotate the data to be annotated and output the annotation results of the data to be annotated. Manual annotation is not required, which can improve the efficiency and accuracy of data annotation.

[0076] Optionally, in step 202 , “using a trained annotation model to annotate the data to be annotated” is further refined to include steps 301 to 305 .

[0077] Step 301 , extracting fusion features of the data to be labeled in multiple modalities, and identifying the intent of the data to be labeled.

[0078] In this embodiment, the electronic device can use deep learning models corresponding to different modalities to extract features of the data of each modality, and use an attention mechanism to fuse multimodal data to obtain fusion features of the data to be labeled in multiple modalities.

[0079] For example, electronic devices can use convolutional neural networks (CNN) for image feature extraction, transformer models for text feature extraction, and recurrent neural networks (RNN) for audio feature extraction. These models convert the data to be annotated into high-dimensional feature vectors and convert multimodal data into high-dimensional vector representations, thereby achieving effective acquisition and processing of multimodal data.

[0080] Exemplarily, the electronic device can use the attention mechanism to fuse the high-dimensional feature vectors of the data to be labeled in multiple modalities, for example, H = Attention(Q, K, V), where Q represents the query vector, K represents the key vector, V represents the value vector, H represents the fused feature vector, and Attention represents the attention mechanism.

[0081] In this embodiment, the trained annotation model can extract features such as keywords, phrases, grammar, structure, context, etc. in the data to be annotated, and can identify the intent of the data to be annotated based on the features corresponding to the preset intent of the target application field, which will not be described in detail here.

[0082] Step 302 , in response to the intention of the data to be annotated involving entity relationships in the target application field, obtain the association relationship and associated features of the fused features from the graph database.

[0083] In this embodiment, the preset intention of the target application field may include data query, data analysis, cause analysis, etc. If it is a data query intention, for data that can be directly obtained from the production management database, such as the current operating status of a certain device, the current liquid level of a certain device, etc., therefore, the data query intention does not involve the entity relationship of the target application field; for data that cannot be directly obtained from the production management database and is related to other entities in the target application field, for example, the ash content qualification rate may be affected by the coal quality in different regions, the coal preparation process of different factories, etc., therefore, the data query intention involves the entity relationship of the target application field.

[0084] For entity relationship intentions involving the target application field, it is necessary to obtain the association relationship and associated features of the fused features from the graph database. For example, the associated features related to the ash content pass rate are coal quality, coal preparation process, etc. The association relationship is affected by coal quality and coal preparation process.

[0085] Step 303, in response to the data to be annotated being structured data, a mapping relationship between fusion features and standard fields is constructed, and a preset content processing strategy corresponding to the standard field is determined as a target content processing strategy corresponding to the data to be annotated; the standard field is a preset field of the target application field.

[0086] In this embodiment, the standard field is a preset field of the target application field. Exemplarily, the standard field may be a field in a standard database of the target application field. The standard field may have a corresponding preset content processing strategy. When the data to be annotated is structured content, the electronic device may construct a mapping relationship between the fusion feature and the standard field based on the named entity name of the fusion feature in the graph database, and structure the data to be annotated. For example, the electronic device may obtain a pre-constructed mapping relationship between the named entity name and the standard field, determine the standard field corresponding to the named entity name as the standard field corresponding to the fusion feature, and then obtain the preset content processing strategy corresponding to the standard field corresponding to the fusion feature.

[0087] Step 304 , in response to the data to be annotated being unstructured data, a target content processing strategy corresponding to the data to be annotated is determined from a plurality of preset content processing strategies according to the fusion feature, the association relationship and the association feature.

[0088] In this embodiment, when the data to be annotated is unstructured data, the trained annotation model can identify the keywords, entities and topics of the data to be annotated based on the fusion features, association relationships and association features, and then determine the preset content processing strategy corresponding to the keywords, entities and topics among a plurality of preset content processing strategies as the target content processing strategy corresponding to the data to be annotated.

[0089] Step 305: Determine the labeling result of the data to be labeled according to the target content processing strategy.

[0090] In this embodiment, the content processing strategy is a calculation and processing flow from input data to output data that is pre-built using operators in the operator library. The content processing strategy can reuse the same type of input data to avoid repeatedly and mechanically establishing calculation and processing flows. For different input content, the operator algorithm, number of operators, and order of operators in the content processing strategy can vary.

[0091] The data labeling method provided in this embodiment obtains the association relationship and association features of the fused features from the graph database by involving the entity relationship of the target application field in the intention of the data to be labeled; this enables the large model to label the data to be labeled in combination with the knowledge of the target application field when it needs to use the knowledge of the target application field, thereby improving the labeling accuracy.

[0092] Optionally, the target content processing strategy includes a preset processing object, a preset operator and a preset operator logic; step 305 is refined to include steps 401 to 402.

[0093] Step 401, obtaining the attribute value of the preset processing object from the production management database.

[0094] In this embodiment, the production management database may be a standard database of the target application field, and the preset processing object may include standard fields corresponding to the target content processing strategy, standard fields corresponding to the associated features, etc. The electronic device may obtain the attribute value of the preset processing object from the standard database through SQL statements.

[0095] Step 402: Call the preset operator in the target content processing strategy according to the preset operator logic to process the attribute value of the preset processing object to obtain the labeling result of the data to be labeled.

[0096] In this embodiment, the preset operator logic refers to the calculation and processing flow from input data to output data. For example, when the fusion feature is the total power consumption of the coal preparation plant, the associated feature is each meter of the coal preparation plant. When the coal preparation plant is divided into the east and west areas, the east area may have the east area total meter and each sub-meter under the east area total meter, such as the east incoming line 1 meter, the east incoming line 2 meter, etc. The west area may also have the west area total meter and each sub-meter under the west area total meter, such as the west incoming line 1 meter and the west incoming line 2 meter. Obviously, the power consumption of the east area total meter should be the sum of the power consumption of multiple east incoming line meters, the power consumption of the west area total meter should be the sum of the power consumption of multiple west incoming line meters, and the total power consumption of the coal preparation plant should be the sum of the power consumption of the east area total meter and the west area total meter. Then the preset operator logic for calculating the total power consumption of the coal preparation plant can be the sum of the power consumption of the east area total meter and the power consumption of the west area total meter, or it can be the sum of each east incoming line meter and each west incoming line meter. Even if the electronic equipment obtains the power consumption of the east area total electricity meter, the west area total electricity meter, each east incoming line meter, and each west incoming line meter from the production database, it can only calculate the sum of the power consumption of the east area total electricity meter and the power consumption of the west area total electricity meter by calling the addition operator to obtain the total power consumption of the coal preparation plant.

[0097] The data labeling method provided in this embodiment obtains the attribute value of the preset processing object from the production management database; calls the preset operator in the target content processing strategy according to the preset operator logic to process the attribute value of the preset processing object, and obtains the labeling result of the data to be labeled; makes the target processing strategy reusable for the same type of input content process, thereby improving the data labeling efficiency.

[0098] Embodiment 2

[0099] Figure 3 This is a flow chart of the data annotation method provided in Example 2 of this application. Figure 3 As shown, the data labeling method provided in this embodiment, based on the embodiment 1, further includes steps 501 to 505 before "inputting the data to be labeled into the trained labeling model" in step 202.

[0100] Step 501, obtaining a labeled training data set and a preset labeling model.

[0101] In this embodiment, the labeled training data set includes a plurality of labeled training data, and the labeled training data is unlabeled multimodal data in the target application field.

[0102] The preset annotation model can be a large model with randomly generated parameters.

[0103] Step 502: input the labeled training data set into a preset labeling model, and use the preset labeling model to label each labeled training data in the labeled training data set to obtain a labeling result for each labeled training data.

[0104] In this embodiment, the annotation of structured data can be understood as structuring it, and the annotation of unstructured data can be understood as determining the subject, keyword, label, etc. of the unstructured content.

[0105] Step 503, obtaining user feedback data on the labeling results of each labeled training data; the feedback data on the labeling results includes: accurate labeling and inaccurate labeling.

[0106] In this embodiment, whether the labeling result of the preset labeling model on the labeling training data is accurate is determined by the user, and the user may be a staff member who trains the preset labeling model. The electronic device may display the labeling result of the preset labeling model on the labeling training data to the user, and the user's feedback on the labeling result of each labeling training data may include marking whether the labeling result of the labeling training data is accurate, marking the accurate part and the inaccurate part in the labeling result of the labeling training data, etc.

[0107] In this embodiment, after outputting the labeling results of the preset labeling model on the labeling training data, the electronic device can also receive user adjustments to the graph database and the preset content processing strategy to improve the accuracy of labeling data by the preset labeling model.

[0108] Step 504: Use a set of labeled training data that is inaccurately labeled to update the labeled training data set.

[0109] Step 505, repeatedly executing the step from inputting the labeled training data set into the preset labeling model to the step of updating the labeled training data set with a set of labeled training data with inaccurate labels, so as to adjust the parameters of the preset labeling model, until the user's feedback data on the labeling results of each labeled training data is accurate, and the preset labeling model is determined as the trained labeling model.

[0110] In this embodiment, the electronic device can input the inaccurately labeled training data into the preset labeling model again, and the preset labeling model adjusts its own parameters based on the inaccurately labeled training data, and re-labels the inaccurately labeled training data based on the adjusted parameters, until the user's feedback data on the labeling results of each labeling training data is accurate, and the preset labeling model is determined as the trained labeling model.

[0111] The data labeling method provided in this embodiment trains a preset labeling model through direct user feedback. The user provides feedback on the labeling results of the preset labeling model for each labeling training data, so that a trained labeling model that can automatically and accurately label the data can be trained.

[0112] Optionally, after step 202 of "using the trained annotation model to annotate the data to be annotated and outputting the annotation results of the data to be annotated", steps 601 to 602 are also included.

[0113] Step 601, obtaining a generated training data set and a preset content generation model; the generated training data set includes labeled data.

[0114] In this embodiment, the preset content generation model may be a large language model, and the generated training data set includes labeled data, which may be the data to be labeled and its labeled results in the above embodiment. The preset content generation model may be a content generation model with random parameters.

[0115] Step 602: Train the preset content generation model based on the reply content corresponding to each generated training data and the user's feedback data on each reply content to obtain a trained content generation model.

[0116] In this embodiment, the vector database of the trained content generation model may also be embedded with a graph database of named entities in the target application domain to enhance the knowledge and reasoning capabilities of the trained content generation model in the target application domain.

[0117] Optionally, step 602 is further refined to include steps 701 to 704 .

[0118] Step 701, input the generated training data set into the preset content generation model, use the preset content generation model to generate reply content for each generated training data in the generated training data set, and obtain reply content corresponding to each generated training data.

[0119] Step 702, obtaining feedback data of the user on the reply content corresponding to each generated training data, wherein the feedback data on the reply content includes: accurate reply and inaccurate reply.

[0120] Step 703, using the set of generated training data with inaccurate responses to update the generated training data set.

[0121] Step 704, repeatedly executes the step from inputting the generated training data set into the preset content generation model to the step of updating the generated training data set using a set of generated training data with inaccurate responses, so as to adjust the parameters of the preset content generation model, until the user's feedback data on the response content corresponding to each generated training data is accurate, and the preset content generation model is determined as the trained content generation model.

[0122] In this embodiment, the training process of the trained content generation model is similar to the training process of the trained annotation model. The user only needs to provide feedback on the reply content corresponding to the generated training data. The electronic device will input the user's feedback data on the reply content corresponding to each generated training data into the preset content generation model again. The preset content generation model can adjust its own parameters to complete the training.

[0123] Optionally, when the user's feedback data on the reply content corresponding to each generated training data is inaccurate, it is possible that the reply content generated by the preset content generation model is inaccurate, or it is possible that the generated training data used to generate the reply content is not accurately labeled. Therefore, when the user's feedback data on the reply content corresponding to the same generated training data is inaccurate for a preset number of consecutive times, the generated training data can be re-input into the trained labeling model for labeling, and the preset content generation model can be trained after obtaining a more accurately labeled trained labeling model.

[0124] The data labeling method provided in this embodiment can make the response content generated by the trained labeling model more accurate and more suitable for the target application field by using the generated training data labeled by the trained labeling model to train the preset content generation model.

[0125] Embodiment 3

[0126] Figure 4 This is a schematic diagram of the structure of the data labeling device provided in Example 3 of this application. Figure 4 As shown, the data labeling device 40 provided in this embodiment includes: an acquisition module 41 and a labeling module 42.

[0127] The acquisition module 41 is used to obtain the data to be labeled and the trained labeling model; the data to be labeled is multimodal data in the target application field, and the vector database of the trained labeling model is embedded with the graph database of the target application field; the trained labeling model is obtained by training the preset labeling model based on the labeling results of the preset labeling model on the labeling training data and the user's feedback data on the labeling results.

[0128] The labeling module 42 is used to input the data to be labeled into the trained labeling model, use the trained labeling model to label the data to be labeled and output the labeling results of the data to be labeled.

[0129] Optionally, the annotation module is specifically used for:

[0130] Extract the fusion features of the data to be labeled in multiple modes and identify the intent of the data to be labeled;

[0131] In response to the entity relationship of the target application domain involved in the intention of the data to be annotated, the association relationship and the association features of the fused features are obtained from the graph database;

[0132] In response to the data to be annotated being structured data, a mapping relationship between the fusion feature and the standard field is constructed, and a preset content processing strategy corresponding to the standard field is determined as a target content processing strategy corresponding to the data to be annotated; the standard field is a preset field of the target application field;

[0133] In response to the data to be annotated being unstructured data, determining a target content processing strategy corresponding to the data to be annotated from a plurality of preset content processing strategies according to the fusion feature, the association relationship and the association feature;

[0134] Determine the labeling results of the data to be labeled according to the target content processing strategy.

[0135] Optionally, the target content processing strategy includes a preset processing object, a preset operator, and a preset operator logic; the annotation module is further used to:

[0136] Obtaining attribute values ​​of preset processing objects from a production management database;

[0137] According to the preset operator logic, the preset operator in the target content processing strategy is called to process the attribute value of the preset processing object to obtain the labeling result of the data to be labeled.

[0138] Optionally, the data labeling device further includes a training module, and the training module is used to:

[0139] Obtain annotated training data sets and preset annotation models;

[0140] Input the labeled training data set into a preset labeling model, use the preset labeling model to label each labeled training data in the labeled training data set, and obtain the labeling result of each labeled training data;

[0141] Obtaining user feedback data on the labeling results of each labeling training data; the feedback data on the labeling results includes: accurate labeling and inaccurate labeling;

[0142] Using a set of labeled training data with inaccurate labels to update the labeled training data set;

[0143] Repeat the steps from inputting the labeled training data set into the preset labeling model to updating the labeled training data set with a set of labeled training data with inaccurate labels, so as to adjust the parameters of the preset labeling model, until the user's feedback data on the labeling results of each labeled training data is accurate, and the preset labeling model is determined as the trained labeling model.

[0144] Optionally, the data annotation device further includes a generation module, and the generation module is used to:

[0145] Obtaining a generated training data set and a preset content generation model; the generated training data set includes labeled data;

[0146] The preset content generation model is trained based on the reply content corresponding to each generated training data and the user's feedback data on each reply content to obtain a trained content generation model.

[0147] Optionally, the generation module is also used to:

[0148] Input the generated training data set into the preset content generation model, use the preset content generation model to generate reply content for each generated training data in the generated training data set, and obtain reply content corresponding to each generated training data;

[0149] Obtaining feedback data of the user on the reply content corresponding to each generated training data, wherein the feedback data on the reply content includes: accurate reply and inaccurate reply;

[0150] Using the set of generated training data with inaccurate answers to update the generated training data set;

[0151] Repeat the steps from inputting the generated training data set into the preset content generation model to updating the generated training data set with a set of generated training data with inaccurate responses, so as to adjust the parameters of the preset content generation model, until the user's feedback data on the response content corresponding to each generated training data is accurate, and the preset content generation model is determined as the trained content generation model.

[0152] The data labeling device provided in this embodiment can execute the data labeling method provided in any of the above embodiments. The specific implementation method and principle are similar and will not be repeated here.

[0153] Embodiment 4

[0154] Figure 5 This is a schematic diagram of the structure of an electronic device provided in Embodiment 4 of the present application. Figure 5 As shown, the electronic device 50 provided in this embodiment includes: a processor 52 and a memory 51 communicatively connected to the processor 52 .

[0155] The memory 51 stores computer-executable instructions.

[0156] The processor 52 executes the computer execution instructions stored in the memory 51 to implement the data labeling method provided by any of the above embodiments. The specific implementation method and principle are similar and will not be repeated here.

[0157] The memory 51 and the processor 52 can be connected to each other through a bus. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0158] In this embodiment, the memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk, etc.

[0159] In this embodiment, the electronic device can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to execute the data labeling method provided in any of the above embodiments.

[0160] The embodiment of the present application also provides a computer-readable storage medium, in which computer-executable instructions are stored, and when the computer-executable instructions are executed by a processor, they are used to implement the data labeling method provided in any of the above embodiments. The specific implementation method and principle are similar and will not be repeated here. Exemplarily, the computer-readable storage medium can be a read-only memory, a random access memory, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0161] An embodiment of the present application further provides a computer program product, including a computer program, which, when executed by a processor, implements the data labeling method provided by any of the above embodiments. The specific implementation method and principle are similar and will not be repeated here.

[0162] It should be understood that the above method embodiments are only illustrative, and the method of the present application can also be implemented in other ways. For example, the division of modules in the above embodiments is only a logical function division, and there may be other division methods in actual implementation. For example, multiple modules can be combined, or can be integrated into another system, or some features can be ignored or not executed.

[0163] In addition, unless otherwise specified, the functional modules in the various embodiments of the present application may be integrated into one module, or each module may exist physically separately, or two or more modules may be integrated together. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software program modules.

[0164] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.

[0165] It should be further noted that, although the various steps in the flowchart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowchart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0166] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the claims.

[0167] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A data annotation method, which is applied to a large model, characterized in that: include: Obtain the data to be labeled and the trained labeling model; The data to be annotated is multimodal data in the target application field, and the data to be annotated includes at least two items of voice data, text data and video data in the target application field; the vector database of the trained annotation model is embedded with a graph database of named entities in the target application field; the trained annotation model is obtained by training the preset annotation model based on the annotation results of the preset annotation model on the annotation training data and the user's feedback data on the annotation results; Inputting the data to be labeled into the trained labeling model, labeling the data to be labeled using the trained labeling model and outputting the labeling result of the data to be labeled; The step of using the trained annotation model to annotate the data to be annotated includes: Extracting fusion features of the data to be labeled in multiple modalities and identifying the intent of the data to be labeled; In response to the intention of the data to be annotated involving entity relationships in a target application field, acquiring association relationships and association features of the fused features from the graph database; In response to the data to be annotated being structured data, a mapping relationship between the fusion feature and the standard field is constructed, and a preset content processing strategy corresponding to the standard field is determined as a target content processing strategy corresponding to the data to be annotated; the standard field is a preset field of a target application field; In response to the data to be annotated being unstructured data, determining a target content processing strategy corresponding to the data to be annotated from a plurality of preset content processing strategies according to the fusion feature, the association relationship and the association feature; Determining the labeling result of the data to be labeled according to the target content processing strategy; The target content processing strategy includes a preset processing object, a preset operator, and a preset operator logic; and determining the labeling result of the data to be labeled according to the target content processing strategy includes: Obtaining attribute values ​​of preset processing objects from a standard database of the target application field; According to the preset operator logic, the preset operator in the target content processing strategy is called to process the attribute value of the preset processing object to obtain the labeling result of the data to be labeled.

2. The method according to claim 1, characterized in that Before inputting the data to be labeled into the trained labeling model, the method further includes: Obtain annotated training data sets and preset annotation models; Inputting the labeled training data set into a preset labeling model, using the preset labeling model to label each labeled training data in the labeled training data set, and obtaining a labeling result for each labeled training data; Obtaining user feedback data on the labeling results of each labeling training data; the feedback data on the labeling results includes: accurate labeling and inaccurate labeling; Using a set of labeled training data with inaccurate labels to update the labeled training data set; Repeat the step from inputting the labeled training data set into the preset labeling model to the step of updating the labeled training data set using a set of labeled training data with inaccurate labels to adjust the parameters of the preset labeling model until the user's feedback data on the labeling results of each labeled training data is accurate, and the preset labeling model is determined as the trained labeling model.

3. The method according to any one of claims 1 to 2, characterized in that: After the trained annotation model is used to annotate the data to be annotated and the annotation result of the data to be annotated is output, the method further includes: Obtaining a generated training data set and a preset content generation model; the generated training data set includes labeled data; The preset content generation model is trained based on the reply content corresponding to each generated training data and the user's feedback data on each reply content to obtain a trained content generation model.

4. The method according to claim 3, characterized in that The training of the preset content generation model based on the reply content corresponding to each generated training data and the user's feedback data on each reply content includes: Input the generated training data set into the preset content generation model, use the preset content generation model to generate reply content for each generated training data in the generated training data set, and obtain reply content corresponding to each generated training data; Obtaining feedback data of the user on the reply content corresponding to each generated training data, wherein the feedback data on the reply content includes: accurate reply and inaccurate reply; Using the set of generated training data with inaccurate answers to update the generated training data set; Repeat the steps from inputting the generated training data set into the preset content generation model to updating the generated training data set using a set of generated training data with inaccurate responses, so as to adjust the parameters of the preset content generation model, until the user's feedback data on the response content corresponding to each generated training data is accurate, and the preset content generation model is determined as the trained content generation model.

5. A data annotation device, applied to a large model, characterized in that: include: The acquisition module is used to obtain the data to be labeled and the trained labeling model; The data to be annotated is multimodal data in the target application field, and the data to be annotated includes at least two items of voice data, text data and video data in the target application field; the vector database of the trained annotation model is embedded with a graph database in the target application field; the trained annotation model is obtained by training the preset annotation model based on the annotation results of the preset annotation model on the annotation training data and the user's feedback data on the annotation results; A labeling module, used to input the data to be labeled into a trained labeling model, label the data to be labeled using the trained labeling model and output a labeling result of the data to be labeled; The labeling module is specifically used to: extract the fusion features of the data to be labeled in multiple modes and identify the intention of the data to be labeled; In response to the intention of the data to be annotated involving entity relationships in a target application field, acquiring association relationships and association features of the fused features from the graph database; In response to the data to be annotated being structured data, a mapping relationship between the fusion feature and the standard field is constructed, and a preset content processing strategy corresponding to the standard field is determined as a target content processing strategy corresponding to the data to be annotated; The standard fields are preset fields in the target application field; In response to the data to be annotated being unstructured data, determining a target content processing strategy corresponding to the data to be annotated from a plurality of preset content processing strategies according to the fusion feature, the association relationship and the association feature; Determining the labeling result of the data to be labeled according to the target content processing strategy; The target content processing strategy includes a preset processing object, a preset operator and a preset operator logic; The labeling module is specifically used to: obtain the attribute value of the preset processing object from the standard database of the target application field; According to the preset operator logic, the preset operator in the target content processing strategy is called to process the attribute value of the preset processing object to obtain the labeling result of the data to be labeled.

6. An electronic device comprising: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the data labeling method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the data labeling method according to any one of claims 1 to 4.

8. A computer program product, characterized in that It includes a computer program, which is used to implement the data labeling method according to any one of claims 1 to 4 when executed by a processor.

Images