Data labeling method and device, equipment, storage medium and program product

Abstract

The invention provides a data annotation method and device, equipment, a storage medium and a program product. The method comprises the following steps: labeling data of an original data set to obtain labeled data; extracting features of the labeled data to obtain first language features corresponding to the labeled data; extracting target data from the labeled data according to the first language feature; determining the language type of each piece of target data according to the first language feature of the target data; and performing labeling processing on each piece of target data again based on the language type to obtain each piece of re-labeled target data. Through the method and the device, the accuracy of data annotation can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data annotation method, device, equipment, storage medium and program product. Background Art

[0002] Data labeling is an important part of the field of artificial intelligence and machine learning. It involves interpreting, labeling or annotating data so that machine learning algorithms can understand and use the data. With the rapid development of artificial intelligence and machine learning technologies, data labeling is increasingly used in various fields. Whether it is autonomous driving, medical diagnosis, intelligent customer service or social media, a large amount of labeled data is required to train and optimize algorithm models. The accuracy of data labeling is crucial to the performance of machine learning models. In addition, with the complexity of algorithm models and the diversification of application scenarios, the demand for data labeling is also increasing, resulting in the continuous increase of data labeling costs and labor costs. Therefore, how to improve the efficiency and quality of data labeling has become a problem that needs to be solved in the current data labeling industry.

[0003] In related technologies, the labeled data lacks diversity, the matching degree between the labeled data and the project's required data is not high, and the accuracy of data labeling is low. Summary of the invention

[0004] The embodiments of the present application provide a data labeling method, apparatus, device, storage medium and program product, which can improve the accuracy of data labeling.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present application provides a data labeling method, the method comprising:

[0007] Label the data of the original data set to obtain labeled data;

[0008] Extracting features of the annotated data to obtain first language features corresponding to the annotated data;

[0009] extracting target data from the annotated data according to the first language feature;

[0010] determining a language type of each of the target data according to a first language feature of the target data;

[0011] Each target data is annotated again based on the language type to obtain each target data after being annotated again.

[0012] The present application provides a data labeling device, including:

[0013] The first labeling module is used to label the data of the original data set to obtain labeled data;

[0014] A feature extraction module, used to extract features of the annotated data to obtain first language features corresponding to the annotated data;

[0015] A data classification module, used for extracting target data from the annotated data according to the first language feature;

[0016] A language classification module, used for determining a language type of each target data according to a first language feature of the target data;

[0017] The second annotation module is used to perform annotation processing on each target data again based on the language type to obtain each target data after being annotated again.

[0018] An embodiment of the present application provides an electronic device, the electronic device comprising:

[0019] A memory for storing computer executable instructions or computer programs;

[0020] The processor is used to implement the data labeling method provided in the embodiment of the present application when executing the computer executable instructions or computer programs stored in the memory.

[0021] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions or a computer program for implementing the data labeling method provided in the embodiment of the present application when executed by a processor.

[0022] An embodiment of the present application provides a computer program product, including computer executable instructions or a computer program. When the computer executable instructions or the computer program are executed by a processor, the data labeling method provided in the embodiment of the present application is implemented.

[0023] The embodiments of the present application have the following beneficial effects:

[0024] By extracting and processing the features of the labeled data, the first language features corresponding to the labeled data are obtained, and based on the first language features, the target data is extracted from the labeled data, thereby improving the quality and diversity of the labeled data; based on the first language features of the target data, the language type of each target data is determined, and each target data is annotated again according to the language type to obtain each target data after re-annotation. By using a labeling method that matches each language type, the accuracy of labeling of data of different language types can be improved, the matching degree between the labeled data and the project's demand data can be improved, and the accuracy of data labeling can be improved, so that the model trained based on the re-annotated data can achieve better training results. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a schematic diagram of an application mode of the data labeling method provided in an embodiment of the present application;

[0026] Figure 2 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0027] Figure 3A It is a first flow chart of the data labeling method provided in an embodiment of the present application;

[0028] Figure 3B It is a second flow chart of the data labeling method provided in an embodiment of the present application;

[0029] Figure 3C is a third flow chart of the data labeling method provided in an embodiment of the present application;

[0030] Figure 3D It is a fourth flow chart of the data labeling method provided in an embodiment of the present application;

[0031] Figure 3E It is a fifth flow chart of the data labeling method provided in the embodiment of the present application;

[0032] Figure 3F It is a sixth flow chart of the data labeling method provided in the embodiment of the present application;

[0033] Figure 3G 7 is a schematic diagram of a seventh flow chart of a data labeling method provided in an embodiment of the present application;

[0034] Figure 4 It is a schematic diagram of the non-long-tail feature data labeling process provided in an embodiment of the present application.

[0035] It should be pointed out that the above-mentioned "first" and "second" are only used to distinguish different solutions, and do not represent the degree of superiority or inferiority of the solutions or the priority in the implementation process. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0037] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0038] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0039] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of authorization of laws and regulations and the personal information subject.

[0040] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0041] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0042] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0043] 1) Natural Language Processing (NLP): It is an important branch in the field of artificial intelligence and computer science. It aims to study and implement theories and methods for effective communication between humans and computers using natural language. The goals of natural language processing include but are not limited to speech recognition, text analysis, speech understanding, etc.

[0044] 2) Pre-trained language model: A pre-trained language model is a pre-trained language model that learns language patterns and rules by training on a large amount of unlabeled text data, thereby gaining the ability to understand and process text data. Among them, the Bidirectional Encoder Representations from Transformers (BERT), hereinafter referred to as the BERT model, uses the encoder part of the transformer model to generate a deep bidirectional representation of the text, which not only considers the previous and next relationships of the text, but also the left and right information of the text, so that the model can capture richer contextual semantic information. The Robustly Optimized BERT Pretraining Approach (Roberta) is a model that has undergone a series of improvements and optimizations on the basis of the BERT model to improve the performance of the BERT model in various natural language understanding tasks.

[0045] 3) Long-tail features: Features that are explicitly expressed or implicitly carried in a language are not the mainstream features of this type of language data. For example, the same word may have different meanings in different contexts. For example, the word "bank" can mean "bank", "river bank" or "reserve", of which "river bank" or "reserve" are long-tail features.

[0046] In related technologies, with the complexity of algorithm models and the diversification of application scenarios, the demand for data annotation is also increasing. The quality and diversity of the labeled data will affect the training effect of the pre-trained language model. In the data annotation process, the matching degree between the labeled data and the project's required data is not high, and the accuracy of data annotation is not high.

[0047] The embodiments of the present application provide a data labeling method, apparatus, device, storage medium and program product, which can improve the accuracy of data labeling.

[0048] The following describes an exemplary application of the electronic device provided by the embodiment of the present application. The electronic device provided by the embodiment of the present application can be implemented as various types of terminals such as a laptop computer, a tablet computer, a desktop computer, a set-top box, a smart phone, a smart speaker, a smart watch, a smart TV, and a vehicle-mounted terminal, and can also be implemented as a server. The following describes an exemplary application when the device is implemented as a server.

[0049] See also Figure 1 , Figure 1 This is a schematic diagram of the application mode of the data annotation method provided in the embodiment of the present application, for example, Figure 1The server 200, network 300, terminal 400 and database 500 are involved. The terminal 400 is connected to the server 200 via the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two. The database 500 stores the original data set required for data annotation.

[0050] In the data annotation process, the terminal 400 responds to the user's data annotation instruction and sends a data annotation request to the server 200. The server 200 responds to the data annotation request, obtains the original data set from the database 500, annotates the data of the original data set, obtains the annotated data, extracts the features of the annotated data, obtains the first language features corresponding to the annotated data, extracts the target data from the annotated data according to the first language features, determines the language type of each target data according to the first language features of the target data, annotates each target data again based on the language type, obtains each target data after annotating again, and returns the annotation results to the terminal 400. In this way, the user can understand the annotation results through the terminal 400, and the annotation results can be used to train the model of the target project. For example: the target project is the language model of the artificial intelligence customer service of the product recommendation system. Based on the needs of the target project, training data is extracted from each target data after annotating again to train the language model, and a language model suitable for artificial intelligence customer service is obtained.

[0051] In some embodiments, the data labeling method of the embodiment of the present application can also be applied in the following application scenarios: in the speech recognition scenario, the data labeling method of the embodiment of the present application is used to label the text content of the audio data, which can help the machine learning model learn the correspondence between the speech signal and the text; in the online shopping scenario, the data labeling method of the embodiment of the present application is used to label the user's evaluation of the product, purchase records, etc. in the product recommendation system, so as to train the personalized recommendation algorithm of the product recommendation system and improve the purchase conversion rate.

[0052] The embodiments of the present application can be implemented through database technology. In short, a database can be regarded as an electronic file cabinet where electronic files are stored. Users can add, query, update, delete, etc. data in the files. The so-called "database" is a collection of data that is stored together in a certain way, can be shared with multiple users, has as little redundancy as possible, and is independent of the application program.

[0053] Database Management System (DBMS) is a large-scale software for manipulating and managing databases. It is used to establish, use and maintain databases. Databases are managed and controlled in a unified manner to ensure the security and integrity of the database. Users access data in the database through the database management system, and database administrators also maintain the database through the database management system. It can support multiple applications and users to establish, modify and query the database in different ways at the same time or at different times. It provides data definition language and data manipulation language for users to define the database model structure and permission constraints, and implement operations such as appending and deleting data.

[0054] The embodiments of the present application can also be implemented through cloud computing, which is a type of distributed computing that decomposes a huge data computing program into countless small programs through the network "cloud", and then processes and analyzes these small programs through a system composed of multiple servers to obtain results and return them to users. Cloud computing is a system with extremely high computing power formed by a computer network, which can store and aggregate related resources and can be configured on demand to provide personalized services to users.

[0055] The embodiments of the present application can also be implemented through machine learning. Machine Learning (ML) is a multi-disciplinary cross-discipline involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, etc. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their own performance.

[0056] In some embodiments, the server (e.g., server 200) may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The terminal 400 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a car terminal, etc., but is not limited thereto. The terminal and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0057] See also Figure 2 , Figure 2 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, the electronic device may be a terminal or a server, Figure 2The electronic device 600 shown includes: at least one processor 410, a memory 450, and at least one network interface 420. The various components in the electronic device 600 are coupled together via a bus system 440. It is understood that the bus system 440 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the bus system 440 is not described in detail. Figure 2 Various buses are labeled as bus system 440 .

[0058] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0059] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.

[0060] The memory 450 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0061] In some embodiments, memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0062] The operating system 451 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic businesses and process hardware-based tasks.

[0063] The network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420. Exemplary network interfaces 420 include: Bluetooth, Wireless Authentication (WiFi), and Universal Serial Bus (USB).

[0064] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 2 A data annotation device 455 stored in a memory 450 is shown, which may be software in the form of a program or plug-in, including the following software modules: a first annotation module 4551, a feature extraction module 4552, a data classification module 4553, a language classification module 4554, and a second annotation module 4555. These modules are logical, and therefore may be arbitrarily combined or further split according to the functions implemented. The functions of each module will be described below.

[0065] In some embodiments, the terminal or server can implement the data annotation method provided by the embodiment of the present application by running various computer executable instructions or computer programs. For example, computer executable instructions can be commands, machine instructions or software instructions at the microprogram level. The computer program can be a native program or software module in the operating system; it can be a local (Native) application (APPlication, APP), that is, a program that needs to be installed in the operating system to run, such as a cloud computing APP or an instant messaging APP; it can also be a small program that can be embedded in any APP, that is, a program that can be run only by downloading it to a browser environment. In short, the above-mentioned computer executable instructions can be instructions in any form, and the above-mentioned computer program can be an application, module or plug-in in any form.

[0066] The data labeling method provided in the embodiment of the present application will be explained in combination with the exemplary application and implementation of the server device provided in the embodiment of the present application.

[0067] The following describes the data labeling method provided by the embodiment of the present application. As mentioned above, the electronic device that implements the data labeling method of the embodiment of the present application can be a terminal, a server, or a combination of the two. Next, the data labeling method provided by the embodiment of the present application is described by taking the electronic device as a server as an example. Figure 3A , Figure 3A This is a first flow chart of the data labeling method provided in the embodiment of the present application. Figure 3A The steps shown are explained.

[0068] In step 301, the data of the original data set is labeled to obtain labeled data.

[0069] By way of example, to facilitate understanding of the data labeling method provided in the embodiments of the present application, the application scenarios of the data labeling method provided in the embodiments of the present application are described below. In scenarios such as intelligent customer service and social media, a large amount of labeled data is required to train and optimize the algorithm model. The data labeling method provided in the embodiments of the present application can improve the accuracy of data labeling.

[0070] For example, the original data set is a data set to be annotated selected from online business databases in scenarios such as intelligent customer service and social media. A data set, namely the original data set, is obtained from the online business data of the preset scenario, and the original data set is annotated. The annotated data includes text and corresponding text labels, for example: the text is "I want to change my mobile phone number" and the text label is "Change the bound mobile phone number".

[0071] In some embodiments, see Figure 3B , Figure 3B is a second flow chart of the data labeling method provided in the embodiment of the present application. Figure 3A Before step 301 shown, you can also perform Figure 3B From step 3011 to step 3014, the original data set is obtained, which is described in detail below.

[0072] In step 3011, candidate text data is obtained.

[0073] Here, candidate text data refers to the text data to be screened in online business data. Candidate text data can express text intent, which is the purpose or motivation that the user wants to express in the text. Candidate text data also includes real conversation data, which is text data collected in actual interaction scenarios, such as real conversation records between users and customer service, user interactions on social media, and user discussions in online forums.

[0074] In step 3012, data is extracted from the candidate text data based on the first target data ratio to obtain a first data set.

[0075] For example, the first target data ratio refers to the data ratio corresponding to the intent importance level. The intent importance level is used to measure and rank the degree of influence of different text intents on the target object. The first target data ratio is determined according to the importance and urgency of the text intent. The importance and urgency of the text intent are determined according to the actual needs of the application scenario. The first target data ratio is extracted from the candidate text data to obtain the first data set.

[0076] In some embodiments, step 3012 can be implemented in the following ways: classify the sub-text data of the candidate text data according to the intention importance level to obtain multiple sub-text data corresponding to each intention importance level; extract the first target sub-text data from the multiple sub-text data corresponding to each intention importance level according to the proportion of the first target data corresponding to each intention importance level; and combine the multiple first target sub-text data into a first data set.

[0077] For example, the first target sub-text data refers to sub-text data extracted from multiple sub-text data corresponding to the intention importance level according to the first target data ratio corresponding to the intention importance level, and the first data set consists of multiple first target sub-text data.

[0078] For example, taking the online business data of the deadline reminder scenario as an example, the first deadline reminder has the highest level of importance, the second deadline reminder has the second highest level of importance, and the third deadline reminder has the lowest level of importance. The proportion of data with high intention importance is high, and the proportion of data with low intention importance is low. For example, the proportion of the first target data corresponding to the first deadline reminder intention is 40%, and the proportion of the first target data corresponding to the third deadline reminder intention is 20%. If a total of 100 data items need to be pulled, 40 data items corresponding to the first deadline reminder intention are pulled, and 20 data items corresponding to the third deadline reminder intention are pulled.

[0079] In step 3013, data is extracted from the candidate text data based on the second target data ratio to obtain a second data set.

[0080] For example, the second target data ratio refers to the data ratio corresponding to the intention importance level data in the real conversation data, and the second target data ratio is determined according to the data ratios corresponding to different intention importance level data in the real conversation data. The second target data ratio is extracted from the candidate text data to obtain the second data set.

[0081] In some embodiments, step 3013 can be implemented in the following ways: obtaining the proportion of second target data corresponding to each intent importance level in the real conversation data; extracting second target sub-text data from multiple sub-text data corresponding to each intent importance level according to the proportion of second target data corresponding to each intent importance level; and combining multiple second target sub-text data into a second data set.

[0082] For example, the second target sub-text data refers to sub-text data extracted from multiple sub-text data corresponding to the intent importance level according to the proportion of second target data corresponding to each intent importance level in the real conversation data, and the second data set consists of multiple second target sub-text data.

[0083] For example, taking the online business data of a shopping scenario as an example, the real session data of users' shopping interactions are obtained from the online business data. In the real session data of users' shopping interactions, purchase intention has the highest importance level, and non-purchase intention has the lowest importance level. For example, the proportion of the second target data corresponding to purchase intention is 70%, and the proportion of the second target data corresponding to non-purchase intention is 30%. If a total of 100 data items need to be pulled, 70 real data items corresponding to purchase intention are pulled, and 30 real data items corresponding to non-purchase intention are pulled.

[0084] In step 3014, the first data set and the second data set are merged and deduplicated to obtain an original data set.

[0085] For example, the first data set is a data set extracted from the candidate text data according to the first target data ratio, and the second data set is a data set extracted from the candidate text data according to the second target data ratio. The first data set and the second data set are merged and the repeated data in the first data set and the second data set are excluded to obtain the original data set.

[0086] In an embodiment of the present application, a first data set is extracted from the candidate text data according to a first target data ratio corresponding to each intent importance level, and a second data set is extracted from the candidate text data according to a second target data ratio corresponding to each intent importance level in the real conversation data, taking into account both the intent importance and the real conversation data dimensions, and being able to improve the data quality in the original data set to be annotated.

[0087] Continue to refer Figure 3A In step 302, the features of the annotated data are extracted to obtain the first language features corresponding to the annotated data.

[0088] For example, the first language feature refers to the degree of repetition between words or sentences and other texts, and new language features in spoken or written language. Before extracting the features of the annotated data, the quality of the annotated data is verified, that is, the correctness of the annotation results is judged, and the correctly annotated texts and incorrectly annotated texts in the annotated data are determined, and the first language features are extracted from the correctly annotated texts and incorrectly annotated texts. At the same time, the first language features in the correctly annotated texts and incorrectly annotated texts are taken into account, which can increase the dimension of the first language features.

[0089] In some embodiments, the features of the annotated data are extracted through a first language processing model; the language processing model can be a word embedding model (Word To Vector, Word2Vec), a recurrent neural network model (Recurrent Neural Network, RNN), or a convolutional neural network model (Convolutional Neural Networks, CNN). Figure 3C , Figure 3C is a third flow chart of the data labeling method provided in the embodiment of the present application. Figure 3A Before step 302 shown, you can also perform Figure 3C Steps 3021 to 3024 are used to train the first language processing model, which will be described in detail below.

[0090] In step 3021, based on the plurality of annotated sample text data, the first language processing model is called to perform feature extraction processing to obtain the predicted language feature corresponding to each annotated sample text data.

[0091] For example, the first language processing model refers to a model for analyzing the first language features, and the predicted language features refer to the language features output by the first language processing model of the labeled sample text data. The labeled sample text data includes sample text and sample labels. For example, the first language processing model can be a Robustly Optimized BERT Pretraining Approach (Roberta), hereinafter referred to as the Roberta model. Multiple labeled sample text data are input into the Roberta model, and feature extraction processing is performed on the labeled sample text data to obtain the predicted language features corresponding to each labeled sample text data.

[0092] The sentence patterns in the sample text data can be sentences with the word "ba", sentences with the word "bei", sentences with the word "jian" (making order / love and hate / selection and appointment / you), double object sentences, and existential sentences. The sentence types in the sample text data can be declarative sentences, interrogative sentences (yes and no questions / specific questions / selective questions / positive and negative questions), imperative sentences, and exclamatory sentences. The sentence patterns in the sample text data can be subject-predicate sentences, non-subject-predicate sentences, verb-predicate sentences, adjective-predicate sentences, and noun-predicate sentences. In the training network layer of the Roberta model, a sentence pattern encoding layer, a sentence type encoding layer, and a sentence pattern encoding layer are added to encode the sentence patterns, sentence types, and sentence patterns in the sample text data.

[0093] For example, taking the encoding of sentences by the sentence encoding layer as an example, the sentence type is randomly generated into a sentence vector of fixed dimension, a mapping relationship between the sentence type and the sentence vector is constructed, the sentence type in the labeled sample text data is obtained in the training network layer, and the corresponding sentence vector is obtained according to the sentence type and the mapping relationship. In addition, the process of encoding the sentence class by the sentence class encoding layer to obtain the sentence class vector and the process of encoding the sentence pattern by the sentence pattern encoding layer to obtain the sentence pattern vector are similar to the above process and will not be repeated here.

[0094] In step 3022, the similarity between the predicted language feature corresponding to each annotated sample text data and the actual language feature corresponding to each annotated sample text data is obtained.

[0095] For example, the similarity between the predicted language features and the actual language features is calculated by cosine similarity. Cosine similarity is a method to measure the similarity between two vectors, and the similarity is evaluated by calculating the cosine value of the angle between the two vectors. The cosine similarity is calculated for the sentence vector, sentence type vector, and sentence pattern vector corresponding to the predicted language features and the sentence vector, sentence type vector, and sentence pattern vector corresponding to the actual language features, respectively, to obtain the similarity between the predicted language features and the actual language features.

[0096] In step 3023, a first loss function of the first language processing model is determined according to the similarity.

[0097] Here, the first loss function refers to the loss function corresponding to the first language processing model, and the loss function is a function used to evaluate the difference between the model prediction result and the true value. The type of loss function can be determined according to the needs of the actual scenario, such as: cross entropy loss function, exponential loss function, logarithmic loss function. The cosine similarity is calculated for the predicted language features and the actual language features, and the obtained similarity value is used as the value of the first loss function.

[0098] In step 3024, the first language processing model is trained based on the first loss function to obtain a trained first language processing model.

[0099] When the value of the first loss function exceeds a preset threshold, the error value of the first language processing model is determined based on the first loss function, and then based on the signal including the error value, during the back propagation process in the first language processing model, the model parameters of each network layer in the first language processing model are updated to obtain the trained first language processing model.

[0100] In the embodiment of the present application, by extracting the features of the annotated data, the first language features corresponding to the annotated data are obtained, the language feature dimension of the annotated data is increased, and the diversity of the annotated data is improved.

[0101] Continue to refer Figure 3A , in step 303, target data is extracted from the annotated data according to the first language feature.

[0102] For example, the target data refers to non-long-tail feature data. The long-tail feature refers to the feature that is explicitly expressed or implicitly carried in the language and is not the mainstream feature of this type of language data. Conversely, the non-long-tail feature data refers to the feature that is explicitly expressed or implicitly carried in the language and is the mainstream feature of this type of language data.

[0103] In some embodiments, see Figure 3D , Figure 3D This is a fourth flow chart of the data labeling method provided in an embodiment of the present application. Figure 3A Step 303 shown may be performed by Figure 3D Steps 3031 to 3032 are implemented as described in detail below.

[0104] In step 3031, the first language features are classified to obtain target language features in the first language features.

[0105] For example, the classification process refers to performing long-tail feature classification on the first language features corresponding to the annotated data to obtain target language features in the first language features, where the target language features refer to non-long-tail features. The first language features corresponding to the annotated data and the existing data language feature list are input into the classification process model, and the first language features are classified. The classification process model may be a binary classification model, and the output results include the target language features and the non-target language features.

[0106] In step 3032, target data corresponding to the target language feature is extracted from the annotated data according to the type of the language feature corresponding to the annotated data.

[0107] According to the parts of speech and syntactic structure of the annotated data, the types of language features corresponding to the annotated data are analyzed, long-tail feature data and non-long-tail feature data are screened out, and target data corresponding to non-long-tail features are extracted from the annotated data.

[0108] In the embodiment of the present application, by classifying the first language feature to obtain the target language feature in the first language feature, and extracting the target data corresponding to the target language feature from the annotated data, the annotated data can be screened for the target language feature, thereby improving the accuracy of data annotation.

[0109] In some embodiments, the classification process is implemented by a second language processing model; see Figure 3E , Figure 3E is a fifth flow chart of the data labeling method provided in the embodiment of the present application. Figure 3ABefore step 303 shown, you can also perform Figure 3E Steps 3061 to 3063 train the second language processing model, which is described in detail below.

[0110] In step 3061, based on a plurality of annotated sample text data, a pre-trained language model is called to perform data classification processing to obtain a predicted target feature in each annotated sample text data.

[0111] For example, the prediction target feature refers to the language feature output by the pre-trained language model of the labeled sample text data. The pre-trained language model can be a Roberta model. A plurality of labeled sample text data are input into the Roberta model, and data classification processing is performed on the labeled sample text data to obtain the prediction target feature of each labeled sample text data.

[0112] In step 3062, a second loss function of the second language processing model is determined according to the actual target features and the predicted target features in each annotated sample text data.

[0113] For example, the second language processing model is a model for distinguishing long-tail features of the first language features, and the second loss function refers to the loss function corresponding to the second language processing model, and the loss function may be a cross entropy loss function. The value of the second loss function is determined based on the difference between the actual target feature and the predicted target feature in each labeled sample text data.

[0114] In step 3063, the pre-trained language model is trained based on the second loss function to obtain a trained second language processing model.

[0115] When the value of the second loss function exceeds a preset threshold, the error value of the second language processing model is determined based on the second loss function, and then based on the signal including the error value, during the back propagation process in the second language processing model, the model parameters of each network layer in the second language processing model are updated to obtain the trained second language processing model.

[0116] Continue to refer Figure 3A In step 304, the language type of each target data is determined according to the first language feature of the target data.

[0117] For example, according to the first language feature, each target data is classified into a language type to obtain a language type actually corresponding to each target data.

[0118] Continue to refer Figure 3A In step 305, each target data is annotated again based on the language type to obtain each target data after re-annotation.

[0119] For example, re-labeling is to re-label the language type of each target data. When the proportion of the target data in the original data set reaches a preset threshold, each target data is re-labeled based on the language type to obtain each re-labeled target data. Each non-long-tail feature data is re-labeled based on the language type to obtain each re-labeled non-long-tail feature data.

[0120] In some embodiments, each target data can be annotated again based on the language type by executing the following steps to obtain each target data after annotating again: based on the language type corresponding to each target data, the data annotation model associated with each language type is called respectively to annotate each annotated target data to obtain each target data after annotating again.

[0121] Here, each target data after re-annotation has a unique identifier, and the data annotation model associated with the language type is called to perform annotation processing on each annotated target data. For example, the data annotation model associated with each language type is a data annotation model trained based on the data of the corresponding language type. The data annotation model associated with any language type has better annotation processing accuracy than other models. For example, the annotation processing can also be performed by an annotation specialist who is good at annotating the language type.

[0122] In an embodiment of the present application, each target data is annotated again based on the language type to obtain each target data after the re-annotation, and the annotated data can be annotated by category, thereby improving the quality of the annotated data. In addition, each target data after the re-annotation carries a unique identification code, which facilitates the subsequent backtracking of the annotated data.

[0123] In some embodiments, see Figure 3F , Figure 3F 6 is a schematic diagram of a sixth process flow of a data labeling method provided in an embodiment of the present application. Figure 3A After step 305 shown, you can also perform Figure 3F Steps 3071 to 3073 are described in detail below.

[0124] In step 3071, a language model to be trained is obtained.

[0125] For example, the language model to be trained may be a Roberta model.

[0126] In step 3072, training samples are extracted from each re-annotated target data according to the target project requirements.

[0127] For example, the target project requirement is the actual data requirement of the current project. The model can be customized according to the project requirements to obtain the target project requirements of different projects. The project name is input into the project requirement customization model, and the output result of the project requirement customization model is the data requirement of the target project. For example: the project name is "Live App Purchase of Goods", and the target project requirement is "The number of purchase interaction rounds is higher than 3 rounds", that is, the data requirement of the target project is purchase interaction data higher than 3 rounds. Extract training samples from each target data after re-annotation, and update the training data set according to the training samples.

[0128] In step 3073, a language model of the target project is trained based on the training samples to obtain a trained language model.

[0129] For example, the training data set obtained for the target project is used to train the language model of the target project, the training data is input into the pre-trained language model corresponding to the target project, and the model training type is set to classification training. The trained model can be used to perform text classification tasks of the target project and determine which predefined category the text belongs to, such as spam detection, sentiment analysis of text, etc.

[0130] In an embodiment of the present application, training samples are extracted from each target data after re-annotation according to the target project requirements, and the language model of the target project is trained based on the training samples, which can improve the matching degree between the annotated data and the project's requirement data, thereby improving the training effect of the language model of the target project.

[0131] In some embodiments, see Figure 3G , Figure 3G is a seventh flow chart of the data labeling method provided in the embodiment of the present application. Figure 3A After step 305 shown, you can also perform Figure 3G Steps 3081 to 3082 are described in detail below.

[0132] In step 3081, based on each target data after re-annotation as a reference, the standard test set is quality checked to obtain multiple quality indicators of the standard test set.

[0133] For example, a standard test set refers to a set of carefully selected and processed data sets designed for a specific task or model, which are usually used to evaluate the performance of the model and compare the effects of different models. The standard test set is usually representative, diverse and balanced, and can comprehensively test the capabilities and applicability of the model. Quality detection refers to the evaluation of the standard test set, taking each target data after re-annotation as the data set to be tested, and inputting all the data of the data set to be tested into the evaluation model. At the same time, the evaluation model retrieves the online real data set within a certain time period, and performs multi-dimensional similarity calculations on the data set to be tested and the real data set to obtain the similarity calculation results of each dimension, that is, multiple quality indicators of the standard test set. For example, multiple quality indicators can be obtained by calculating the intent distribution similarity and the data text length distribution similarity.

[0134] In step 3082, when at least one quality indicator of the standard test set is less than a threshold, the standard test set is updated based on each target data after re-annotation.

[0135] For example, the quality indicator of the standard test set can be the intent distribution similarity. When calculating the intent distribution similarity of the test data set and the real data set, evaluate whether the intent distribution expressed in the text data of the test data set is similar to the intent distribution of the real data set. Use the cosine similarity formula to calculate the similarity between the intent distribution vector of each text data in the test data set and the intent distribution vector of each text data in the real data set, and take the average of the similarities of all text data as the overall intent distribution similarity. When the intent distribution similarity is less than the preset threshold, the standard test set is updated based on each target data after re-annotation.

[0136] The quality indicator of the standard test set can be the data text length distribution similarity. When calculating the data text length distribution similarity of the test data set and the real data set, it is evaluated whether the text length distribution of the text data of the test data set is similar to the text length distribution of the real data set. The similarity between the text length distributions of the two data sets can be calculated by histogram comparison, distance-based similarity measurement, etc. When the data text length distribution similarity is less than a preset threshold, the standard test set is updated based on each target data after re-annotation.

[0137] In some embodiments, Figure 3F Step 3071 to step 3073, Figure 3G Steps 3081 to 3082 are performed in no particular order, or Figure 3F Step 3071 to step 3073, Figure 3G Steps 3081 to 3082 may be performed separately and simultaneously.

[0138] In the embodiment of the present application, by extracting and processing the features of the labeled data, the first language features corresponding to the labeled data are obtained, and based on the first language features, the target data is extracted from the labeled data, thereby improving the quality and diversity of the labeled data; based on the first language features of the target data, the language type of each target data is determined, and each target data is labeled again according to the language type to obtain each target data after re-labeling. By using a labeling method that matches each language type, the accuracy of labeling of data of different language types can be improved, the matching degree between the labeled data and the project's demand data can be improved, and the accuracy of data labeling can be improved, so that the model trained based on the re-labeled data can achieve better training results.

[0139] The following describes an exemplary application of the data labeling method provided in an embodiment of the present application in a non-long-tail feature data labeling scenario.

[0140] The quality and diversity of the labeled data will affect the training effect of the pre-trained language model. High-quality and diverse labeled data can improve the training effect of the pre-trained language model. In the process of data labeling, there are the following problems: how to match the labeled data with the project, the feedback of the online data labeling effect on the labeling requirements and labeling examples, and the low accuracy of data labeling.

[0141] In view of the problems existing in the related technologies, the embodiments of the present application propose a data labeling method, which, compared with the related technologies, achieves the goals of online text data capture, data labeling specialization, traceability of the flow of labeled data, and the flow of labeled data into project data bags that require such data.

[0142] Example, reference Figure 4 , Figure 4 Schematic diagram of the non-long-tail feature data labeling process provided by the embodiment of the present application. Figure 4 The labeling process in the embodiment of the present application is explained and illustrated.

[0143] In step 401, a first data set is pulled online according to the importance of the intent, and a second data set is pulled according to the proportion of the online real conversation data. The two data sets are merged and deduplicated to generate an original set, and a unique identification code is performed.

[0144] For example, two data sets are pulled from the online business data of the preset scenario. The intent importance is the intent content with different importance in the text of the business data, and the real conversation data is the data generated by the user's online interaction. Intent recognition is an important task in natural language processing. It involves analyzing the text input by the user to determine the user's intention or purpose. In simple terms, it is to make the machine understand the intention behind human language, such as whether the user wants to query information, conduct transactions, or obtain certain services. The following example illustrates that for the online data of the deadline reminder scenario, the business specialist sets the intent importance level of the data based on business experience and business goals, or comprehensively considers the business process requirements. For example: the intent importance level of the first deadline reminder is the highest, the intent importance level of the second deadline reminder is the second, and the intent importance level of the third deadline reminder is the lowest.

[0145] The first data set is pulled from the dimension of the importance level of the intent. Taking the shopping scenario as an example, data sets are pulled from the online data of the shopping app purchase scenario, the live broadcast app purchase scenario, and the third-party app purchase scenario according to the intent importance level. The data with high intent importance levels accounts for a high proportion, and the data with low intent importance levels accounts for a low proportion. For example, if a total of 100 data sets are pulled, 30 data sets with high intent importance levels are pulled, and 10 data sets with low intent importance levels are pulled.

[0146] Pull the second data set from the real conversation data ratio dimension of the intent. The real conversation data can be the data of shopping interaction with users. Pull a data set from the online real conversation data. The ratio of online real conversation data is the same as the ratio of text intent, that is, the more real conversation data corresponding to the text intent, the more data containing the intent will be pulled. Take into account both ways of pulling data sets, comprehensively consider the business importance and real conversation data, so that the pulled data set can better reflect the real situation of online data. Merge the two data sets to obtain the initial data set, exclude the duplicate data sets in the initial data set to generate the original set, and encode the original set with a unique identification.

[0147] Continue to refer Figure 4 In step 402, the data in the original set is labeled and the labeling results are obtained.

[0148] The data in the original set is annotated to obtain the annotation results, which include the data text in the original set and its corresponding text label. Taking the intent recognition task as an example, the annotation result is the text and its text intent label. For example, if the text is "I want to change my mobile phone number", the text intent label is "Change the bound mobile phone number". The annotation results are quality verified, that is, the correctness of the annotation results is judged, and the text that is annotated correctly and the text that is annotated incorrectly are determined in the annotation results.

[0149] In step 403, the annotation results are input into the data new information analysis model to determine new language features in the annotation results that need to be added to the test set.

[0150] The data new information analysis model is a model for analyzing new language features. The data text and its corresponding text labels in the annotation results are input into the data new information analysis model to analyze the new language features in the correctly annotated text and the incorrectly annotated text. The output of the data new information analysis model is the new language features in the annotation results that should be added to the test set. The data new information analysis model can be a large language model with semantic understanding capabilities.

[0151] The training process of the new information analysis model is as follows: load the pre-trained language model, such as the Roberta natural language processing model, input the labeled data into the pre-trained language model for training, and set the training task to sequence labeling. Optimize the pre-trained language model, and add the sentence encoding layer, sentence type encoding layer, and sentence pattern encoding layer to the training network layer of the model. The sentence pattern can be a subject-predicate sentence, a non-subject-predicate sentence, a verb-predicate sentence, an adjective-predicate sentence, and a noun-predicate sentence. The sentence pattern can be a sentence with the word "ba", a sentence with the word "bei", a sentence with the word "jian" (making order / love and hate / selection and employment / having a word), a double object sentence, and a sentence with the word "existence". The sentence type can be a declarative sentence, a question sentence (yes or no question / specific question / selection question / positive and negative question), an imperative sentence, and an exclamatory sentence.

[0152] Encode sentences in the sentence encoding layer: 1) Randomly generate a sentence vector of fixed dimension for each sentence type; 2) Construct a mapping relationship between sentence type and sentence vector; 3) Obtain sentence type in the training network layer; 4) Obtain the corresponding sentence vector according to the sentence type and the mapping relationship.

[0153] The process of encoding sentence categories in the sentence category encoding layer and the process of encoding sentence patterns in the sentence pattern encoding layer are similar to the above process and will not be described in detail here.

[0154] The sentence vector, sentence class vector, sentence pattern vector and the sentence vector of the existing data under the label are similarly calculated. For example, the cosine similarity of the vectors is calculated. The trained model can be recorded as the new language feature model model_new_features, that is, the first language processing model mentioned above.

[0155] In step 404, the new language feature is input into the feature long-tail discrimination model to determine whether the feature is a long-tail feature.

[0156] Long-tail features refer to features that are explicitly expressed or implicitly carried in a language and are not mainstream features of this type of data. The feature long-tail feature discrimination model is a model for long-tail feature discrimination of new language features. The new language features to be detected and the existing data feature list are input into the feature long-tail feature discrimination model, and the output result of the model is the long-tail feature level.

[0157] The training process of the feature long-tail discrimination model is as follows: load a pre-trained language model, such as the Roberta model, input the annotated data into the pre-trained language model, and perform classification task training. The text of the annotated data can be "text error & [long text, put the word sentence]", the text label can be "long tail", and the trained model can be recorded as the feature long-tail discrimination model model_new_feature, that is, the second language processing model mentioned above.

[0158] In step 405, the non-long-tail feature data is annotated according to the proportion of the non-long-tail feature data in the annotation results, and the non-long-tail feature data in the original set is used as a sample to feedback the annotation process, and the non-long-tail feature data enters the annotation pipeline.

[0159] When the proportion of non-long-tail feature data in the annotation results reaches the preset threshold, the non-long-tail feature data in the annotation results are annotated. The role of the annotation pipeline is similar to a temporary warehouse, which is used to store annotation data. The annotation data in the annotation pipeline can flow according to a certain plan, that is, the annotation data is not static in the annotation pipeline, and the annotation pipeline is a mainstream pipe. There are narrow branches and small pipes connected to it at each node of the annotation pipeline. The annotation pipeline matches the needs of the flowing annotation data.

[0160] In step 406, the non-long-tail feature data entering the annotation pipeline is annotated by category according to the new language features.

[0161] Types of new language features include the novelty of vocabulary or sentences, and new features of spoken or written language. Non-long-tail feature data can be categorized based on the above new language features, and different categories of non-long-tail feature data can be labeled using a data labeling model.

[0162] In step 407, the annotated data flows back into the data pipeline.

[0163] At this time, the labeled data is distinguished to carry a unique identification code, as well as the nodes and node details in the data pipeline (for example, at the labeled node, the data labeling model automatically records the labeling doubts and difficulties), which facilitates the subsequent traceability of the labeled data. The labeled data enters the next processing node, and the quality of the labeled data is verified, or the labeled data flows into the training data bag of the project that meets the requirements.

[0164] In step 408, the labeled data flows into the training data bag of the project that meets the requirements in the data pipeline according to the data requirements of different projects.

[0165] The project demand customization model is a model that generates data requirements for different projects. The project name is input into the project demand customization model, and the output of the model is the data requirements of the project. For example, if the project name is "Purchase goods on the live broadcast app", the data requirement of the project is "The number of purchase interaction rounds is greater than 3 rounds". According to the data pipeline management model, according to the data requirements of the specific project, the data that meets the project requirements is filtered out from all the data and flows into the training data bag with the project name.

[0166] In step 409, the training data set is updated and the model training begins.

[0167] The model that receives the labeled data is responsible for adding all the new labeled data to the original training data set, completing the update of the training data set, and starting the model training process. The model training process is as follows: load the pre-trained language model, such as the Roberta natural language processing model, input the training data into the pre-trained language model, set the model training type to classification training, and the trained model can be recorded as the new model model_new.

[0168] In some embodiments, step 409 and step 411 are performed in any order, or step 411 and step 409 may be performed simultaneously.

[0169] In step 411, the standard test set is evaluated based on the newly generated online data set, and the standard test set is updated.

[0170] The standard test set (datasets_standard) is a test set that evaluates whether it can objectively and comprehensively reflect the model's capabilities from the dimensions of comprehensiveness, simulation, and risk data coverage. The evaluation process of the standard test set is as follows:

[0171] 1) Input all the data of the dataset to be tested into the evaluation model in the form of a list.

[0172] 2) The evaluation model retrieves a real online data set within a certain period of time, and calculates the similarity of the two data sets in the following dimensions: similarity of calculation intent distribution (regular changes), similarity of online data text length distribution (mean distribution of single text length, text length distribution in the sentence dimension).

[0173] When calculating the intent distribution similarity, the cosine similarity formula is used to calculate the similarity between the intent distribution vector of each text data in the data set to be tested and the intent distribution vector of each text data in the real data set, and the average value of the similarity of all text data is taken as the overall intent distribution similarity; when calculating the data text length distribution similarity, the similarity between the text length distributions of the two data sets can be calculated through histogram comparison, distance-based similarity measurement, etc.

[0174] For each dimension, the similarity between the test data set and the real data is calculated separately, or the similarities in each dimension are weighted to obtain a comprehensive calculation result of the similarity. When the comprehensive calculation result of the similarity reaches a certain threshold, it is determined that the test data set can be used as a standard test set.

[0175] For example, in actual operation, different labeling algorithms have different advantages and limitations, such as: supervised learning algorithms focus on learning the relationship between the features and labels of labeled data. During the labeling process, sufficient labeled data needs to be provided to train the model to ensure that the model can learn effective feature representations and classification rules; reinforcement learning algorithms focus on learning how to take the best actions in a specific environment to maximize rewards. During the labeling process, a reward mechanism can be designed to guide the model's labeling strategy to achieve efficient and accurate labeling; self-supervised learning algorithms automatically learn feature representations by predicting missing parts or sorting relationships in the data. During the labeling process, some parts of the data can be masked or shuffled to generate self-supervised learning tasks. Different labeling algorithms need to be analyzed and selected before the labeling results are fed back.

[0176] In some embodiments, when data is distributed to the labeling specialists who are best at labeling such data, the same piece of data needs to be given to at least three labeling specialists, and different labeling specialists will have different labeling notes on the data (labeling doubts and labeling difficulties) to ensure the accuracy of data labeling.

[0177] For example, when adding different categories of data to a project data set, the different categories of data can be randomly assigned to data bags of different projects. After completing the data demand analysis, it is determined whether to add the different categories of data to the corresponding project data set.

[0178] For example, in the process of pulling a data set according to the level of intent importance, different diversity interval values ​​can be set, and the data set can be pulled according to the diversity interval values. The diversity interval is the vector interval of the data text, and it can also be the paragraph form of the text. The diversity interval corresponding to the data with a low level of intent importance is wide, and the diversity interval corresponding to the data with a high level of intent importance is narrow.

[0179] In the above-mentioned application scenario of non-long-tail feature data labeling, a data pipeline construction framework based on deep learning models, an optimal matching framework for data features and labeling algorithms, and a construction and implementation framework for data bag access principles for different projects were constructed. By generating unique codes for labeled data, it is convenient to trace back the labeled data nodes and the labeled data flow process, thereby improving the accuracy of non-long-tail feature data labeling.

[0180] The following further describes the exemplary structure of the data annotation device 455 provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 2 As shown, the software modules stored in the data labeling device 455 of the memory 450 may include: a first labeling module 4551, used to label the data of the original data set to obtain the labeled data; a feature extraction module 4552, used to extract the features of the labeled data to obtain the first language features corresponding to the labeled data; a data classification module 4553, used to extract target data from the labeled data according to the first language features; a language classification module 4554, used to determine the language type of each target data according to the first language features of the target data; a second labeling module 4555, used to re-label each target data based on the language type to obtain each re-labeled target data.

[0181] In some embodiments, the first annotation module 4551 is also used to obtain candidate text data before annotating the data of the original data set to obtain the annotated data; extract data from the candidate text data based on the first target data ratio to obtain the first data set; extract data from the candidate text data based on the second target data ratio to obtain the second data set; merge and remove duplicates from the first data set and the second data set to obtain the original data set.

[0182] In some embodiments, the first annotation module 4551 is also used to classify the sub-text data of the candidate text data according to the intent importance level to obtain multiple sub-text data corresponding to each intent importance level; extract the first target sub-text data from the multiple sub-text data corresponding to each intent importance level according to the first target data ratio corresponding to each intent importance level; combine the multiple first target sub-text data into a first data set; obtain the second target data ratio corresponding to the data of each intent importance level in the real conversation data; extract the second target sub-text data from the multiple sub-text data corresponding to each intent importance level according to the second target data ratio corresponding to each intent importance level; and combine the multiple second target sub-text data into a second data set.

[0183] In some embodiments, the features of the annotated data are extracted through a first language processing model; the feature extraction module 4552 is also used to extract the features of the annotated data and obtain the first language features corresponding to the annotated data, and then call the first language processing model to perform feature extraction based on multiple annotated sample text data to obtain the predicted language features corresponding to each annotated sample text data; obtain the similarity between the predicted language features corresponding to each annotated sample text data and the actual language features corresponding to each annotated sample text data; determine the first loss function of the first language processing model according to the similarity; train the first language processing model based on the first loss function to obtain the trained first language processing model.

[0184] In some embodiments, the data classification module 4553 is further used to classify the first language feature to obtain the target language feature in the first language feature; and extract the target data corresponding to the target language feature from the annotated data according to the type of the language feature corresponding to the annotated data.

[0185] In some embodiments, the classification process is implemented through a second language processing model; the data classification module 4553 is also used to call a pre-trained language model to perform data classification processing based on multiple annotated sample text data before extracting the target data from the annotated data according to the first language feature, so as to obtain the predicted target feature in each annotated sample text data; determine the second loss function of the second language processing model according to the actual target feature and the predicted target feature in each annotated sample text data; train the pre-trained language model based on the second loss function to obtain a trained second language processing model.

[0186] In some embodiments, the second annotation module 4555 is also used to call the data annotation model associated with each language type to perform annotation processing on each labeled target data based on the language type corresponding to each target data, so as to obtain each target data after re-annotation, wherein each target data after re-annotation has a unique identifier.

[0187] In some embodiments, the second labeling module 4555 is also used to re-label each target data based on the language type, and after obtaining each re-labeled target data, obtain the language model to be trained; extract training samples from each re-labeled target data according to the requirements of the target project; train the language model of the target project based on the training samples to obtain the trained language model.

[0188] In some embodiments, the second annotation module 4555 is also used to re-annotate each target data based on the language type to obtain each re-annotated target data, and then perform quality inspection on the standard test set based on each re-annotated target data as a reference to obtain multiple quality indicators of the standard test set; when at least one quality indicator of the standard test set is less than a threshold, the standard test set is updated based on each re-annotated target data.

[0189] The embodiment of the present application provides a computer program product, which includes computer executable instructions or computer programs, and the computer executable instructions or computer programs are stored in a computer-readable storage medium. The processor of the electronic device reads the computer executable instructions or computer programs from the computer-readable storage medium, and the processor executes the computer executable instructions or computer programs, so that the electronic device performs the data labeling method described in the embodiment of the present application.

[0190] The present application embodiment provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions or computer programs are stored. When the computer-executable instructions or computer programs are executed by a processor, the processor will execute the data labeling method provided by the present application embodiment, for example: Figure 3A The data annotation method shown.

[0191] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.

[0192] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0193] As an example, computer-executable instructions may, but do not necessarily, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0194] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0195] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A data labeling method, characterized in that: The method comprises: Label the data of the original data set to obtain labeled data; Extracting features of the annotated data to obtain first language features corresponding to the annotated data; extracting target data from the annotated data according to the first language feature; determining a language type of each of the target data according to a first language feature of the target data; Each target data is annotated again based on the language type to obtain each target data after being annotated again.

2. The method according to claim 1, characterized in that Before labeling the data of the original data set to obtain the labeled data, the method further includes: Obtain candidate text data; Extracting data from the candidate text data based on a first target data ratio to obtain a first data set; Extracting data from the candidate text data based on a second target data ratio to obtain a second data set; The first data set and the second data set are combined and deduplicated to obtain the original data set.

3. The method according to claim 2, characterized in that The step of extracting data from the candidate text data based on the first target data ratio to obtain a first data set includes: Classify the sub-text data of the candidate text data according to the level of intention importance to obtain a plurality of sub-text data corresponding to each level of intention importance; Extracting first target subtext data from a plurality of subtext data corresponding to each of the intention importance levels, respectively, according to the first target data proportion corresponding to each of the intention importance levels; Combining a plurality of the first target subtext data into a first data set; The step of extracting data from the candidate text data based on the second target data ratio to obtain a second data set includes: Obtain the second target data ratio corresponding to the data of each intention importance level in the real conversation data; According to the proportion of the second target data corresponding to each of the intention importance levels, respectively extracting the second target subtext data from the plurality of subtext data corresponding to each of the intention importance levels; A plurality of the second target subtext data are combined into a second data set.

4. The method according to claim 1, characterized in that The extracting of features of the annotated data is achieved by using a first language processing model; Before extracting the features of the annotated data to obtain the first language features corresponding to the annotated data, the method further includes: Based on the plurality of annotated sample text data, calling the first language processing model to perform feature extraction processing to obtain a predicted language feature corresponding to each of the annotated sample text data; Obtaining the similarity between the predicted language feature corresponding to each of the annotated sample text data and the actual language feature corresponding to each of the annotated sample text data; Determining a first loss function of the first language processing model according to the similarity; The first language processing model is trained based on the first loss function to obtain the trained first language processing model.

5. The method according to claim 1, characterized in that The step of extracting target data from the annotated data according to the first language feature includes: Classifying the first language features to obtain target language features in the first language features; According to the type of the language feature corresponding to the annotated data, target data corresponding to the target language feature is extracted from the annotated data.

6. The method according to claim 5, characterized in that The classification process is implemented by a second language processing model. Before extracting target data from the annotated data according to the first language feature, the method further includes: Based on the plurality of annotated sample text data, calling the pre-trained language model to perform data classification processing to obtain the predicted target features in each of the annotated sample text data; Determining a second loss function of the second language processing model according to the actual target feature and the predicted target feature in each of the annotated sample text data; The pre-trained language model is trained based on the second loss function to obtain the trained second language processing model.

7. The method according to any one of claims 1 to 6, characterized in that: The step of re-marking each target data based on the language type to obtain each target data after re-marking includes: Based on the language type corresponding to each target data, the data annotation models respectively associated with each language type are called to perform annotation processing on each labeled target data to obtain each re-annotated target data, wherein each re-annotated target data has a unique identifier.

8. The method according to any one of claims 1 to 6, characterized in that: After re-annotating each target data based on the language type to obtain each target data after re-annotation, the method further includes: Get the language model to be trained; Extracting training samples from each of the target data after re-annotation according to the target project requirements; The language model of the target project is trained based on the training samples to obtain a trained language model.

9. The method according to any one of claims 1 to 6, characterized in that: After re-annotating each target data based on the language type to obtain each target data after re-annotation, the method further includes: Based on each of the target data after re-annotation as a reference, a quality inspection is performed on the standard test set to obtain a plurality of quality indicators of the standard test set; When at least one of the quality indicators of the standard test set is less than a threshold, the standard test set is updated based on each of the target data after re-annotation.

10. A data labeling device, characterized in that: The device comprises: The first labeling module is used to label the data of the original data set to obtain labeled data; A feature extraction module, used to extract features of the annotated data to obtain first language features corresponding to the annotated data; A data classification module, used for extracting target data from the annotated data according to the first language feature; A language classification module, used for determining a language type of each target data according to a first language feature of the target data; The second annotation module is used to perform annotation processing on each target data again based on the language type to obtain each target data after being annotated again.

11. An electronic device, characterized in that: The electronic device comprises: A memory for storing computer executable instructions or computer programs; A processor, used to implement the data labeling method according to any one of claims 1 to 9 when executing the computer executable instructions or computer programs stored in the memory.

12. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the data labeling method according to any one of claims 1 to 9 is implemented.

13. A computer program product comprising computer executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the data labeling method according to any one of claims 1 to 9 is implemented.

Images