Multi-role model automatic labeling method and device, terminal and medium

Abstract

The invention discloses a multi-role model automatic labeling method and device, a terminal and a medium. According to the technical scheme provided by the invention, a labeling mode of a multi-role view angle is introduced, a plurality of labeling sub-models with different role attributes are constructed according to role classification and knowledge architecture of a specific field scene, and the labeling sub-models are automatically labeled. According to the method, data can be labeled from different visual angles, deviation between model labeling and human labeling is reduced to the maximum extent by integrating labeling results of a plurality of role models, the problem of labeling one-sidedness caused by a single visual angle in the prior art can be effectively solved, and labeling results are more comprehensive and accurate.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, device, terminal and medium for automatic labeling of multi-role models. Background Art

[0002] In recent years, large language model technologies such as ChatGPT have gradually matured. Thanks to the excellent natural language generation capabilities of large models, they are used in various field tasks, such as data generation tasks in the fields of finance and law. At the same time, using large models to automatically label data helps reduce labor and economic costs. However, in actual application scenarios, when facing specific fields, such as education, there is a large difference between the labels annotated by the model and the standard labels, and the accuracy of automatic labeling is low. Summary of the invention

[0003] The present application provides a method, device, terminal and medium for automatic labeling of multi-role models, which are used to solve the technical problem of low labeling accuracy of existing large model data automatic labeling technology.

[0004] In order to solve the above technical problems, the first aspect of the present application provides a method for automatic labeling of multi-role models, comprising:

[0005] Acquire labeled sample data, and classify the labeled sample data according to the labeled role attributes of the labeled sample data;

[0006] According to different annotated role attributes, the classified annotated sample data is used to train the model, and the role annotation sub-model corresponding to each annotated role attribute is obtained respectively;

[0007] According to the predicted sub-label probability distribution output by each role labeling sub-model, combined with the weight parameters corresponding to each role labeling sub-model, each predicted sub-label is integrated to obtain a soft label;

[0008] According to the loss function, each role labeling sub-model is weighted and optimized. When the preset optimization conditions are met, a multi-role joint labeling model is obtained by integrating the various role labeling sub-models.

[0009] The data to be annotated is obtained, and the data to be annotated is input into the multi-role joint annotation model, so as to output corresponding multi-role model annotation data through the multi-role joint annotation model.

[0010] Preferably, the loss function is specifically:

[0011]

[0012] In the formula, is the total loss value, is the role labeling loss between the predicted sub-label and the standard sub-label output by the role labeling sub-model k, is the comprehensive annotation loss between the soft label and each predicted sub-label, is a standard subtag, Output the predicted sub-label for the i-th sample after the k-th role labeling sub-model The posterior probability of is the soft label corresponding to the i-th sample, is the weight of the i-th sample on the k-th role labeling sub-model, N is the number of labeled sample data, M is the number of standard sub-labels, K is the number of labeled role types, are the hyperparameters for the role labeling loss and the comprehensive labeling loss.

[0013] Preferably, the calculation formula of the soft tag is specifically:

[0014]

[0015] In the formula, is the soft label corresponding to the i-th sample, Output the predicted sub-label for the i-th sample after the k-th role labeling sub-model The probability distribution of The weight of the i-th sample on the k-th role labeling sub-model.

[0016] Preferably, the calculation formula for the multi-role model annotation data is specifically:

[0017]

[0018] In the formula, Label the data for the multi-role model of the i-th data to be labeled, Output the predicted sub-label for the i-th sample after the k-th role labeling sub-model The probability distribution of The weight of the i-th sample on the k-th role labeling sub-model.

[0019] Preferably, the weight parameters corresponding to each role labeling sub-model are generated in the following manner:

[0020] According to the predicted sub-label probability distribution output by each role labeling sub-model, the predicted sub-label probability distribution is used as the input feature of the preset weight prediction network model to output the weight parameters of the role labeling sub-model.

[0021] Preferably, the weight prediction network model is specifically an MLP network model.

[0022] At the same time, the second aspect of the present application provides a multi-role model automatic labeling device, including:

[0023] A labeled sample acquisition unit, used to acquire labeled sample data, and classify the labeled sample data according to the labeled role attributes of the labeled sample data;

[0024] A role model building unit is used to perform model training using classified labeled sample data according to different labeled role attributes, and obtain role labeling sub-models corresponding to each labeled role attribute;

[0025] The role comprehensive prediction unit is used to integrate the predicted sub-labels according to the predicted sub-label probability distribution output by each role labeling sub-model and the weight parameters corresponding to each role labeling sub-model to obtain a soft label;

[0026] A multi-role model optimization unit is used to optimize the weights of each role labeling sub-model according to the loss function, and when the preset optimization conditions are met, a multi-role joint labeling model is obtained by integrating the various role labeling sub-models;

[0027] The multi-role model automatic annotation unit is used to obtain the data to be annotated, input the data to be annotated into the multi-role joint annotation model, and output the corresponding multi-role model annotation data through the multi-role joint annotation model.

[0028] Preferably, it also includes:

[0029] The weight parameter generating unit is used to output the weight parameters of the role labeling sub-model according to the predicted sub-label probability distribution output by each role labeling sub-model, using the predicted sub-label probability distribution as the input feature of the preset weight prediction network model.

[0030] A third aspect of the present application provides a multi-role model automatic annotation terminal, including: a memory and a processor;

[0031] The memory is used to store program code, and the program code corresponds to a method for automatic labeling of multiple role models provided in the first aspect of the present application;

[0032] The processor is used for reading and executing the program code.

[0033] The fourth aspect of the present application provides a computer-readable storage medium, in which a program code is stored. The program code is used to be read and executed by a processor to implement a multi-role model automatic labeling method as provided in the first aspect of the present application.

[0034] It can be seen from the above technical solutions that this application has the following advantages:

[0035] The technical solution provided in this application introduces a labeling method from multiple role perspectives, and constructs multiple labeling sub-models with different role attributes according to the role classification and knowledge architecture of specific domain scenarios, so that it can label data from different perspectives. By integrating the labeling results of multiple role models, the deviation between model labeling and human labeling can be minimized, and the problem of one-sided labeling caused by a single perspective in the prior art can be effectively solved, making the labeling results more comprehensive and accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0037] Figure 1 A flowchart of an embodiment of a method for automatic labeling of multi-role models provided in the present application.

[0038] Figure 2 A logic block diagram of the process of an embodiment of a multi-role model automatic annotation method provided in this application.

[0039] Figure 3 A schematic diagram of the structure of an embodiment of a multi-role model automatic annotation device provided in the present application.

[0040] Figure 4 A schematic diagram of the structure of an embodiment of a multi-role model automatic annotation terminal provided in the present application. DETAILED DESCRIPTION

[0041] Although automatic annotation methods based on large language models (LLMs) reduce the cost of manual annotation, these methods usually ignore the detailed understanding and background knowledge in human annotation. Taking the field of education as an example, existing technologies find it difficult to effectively incorporate these experiences and understandings into the model's annotation process, resulting in the lack of accuracy and practicality in the annotation results. In addition, existing model annotations usually focus on a single perspective and cannot fully reflect the complexity of educational situations, resulting in one-sided and biased annotation results. This leads to significant deviations between the annotation results generated by the model and those of human experts.

[0042] The embodiments of the present application provide a method, device, terminal and medium for automatic labeling of a multi-role model, which are used to solve the technical problem of low labeling accuracy of existing large model data automatic labeling technology.

[0043] In order to make the purpose, features, and advantages of the invention of this application more obvious and easy to understand, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the embodiments described below are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0044] First, a detailed description of an embodiment of a multi-role model automatic annotation method provided by the present application is as follows:

[0045] See also Figure 1 and Figure 2 , an embodiment of the present application provides a multi-role model automatic labeling method, comprising:

[0046] Step 101: Obtain labeled sample data, and classify the labeled sample data according to the labeled role attributes of the labeled sample data;

[0047] It should be noted that first, a certain amount of labeled sample data is obtained. Each type of labeled sample data includes multiple versions labeled by different role objects (such as subject teachers, students, educational psychology experts, etc.). The labeled sample data are classified according to the labeled role attributes of the labeled sample data, which can be used to construct labeled data samples of multiple role models.

[0048] Step 102: According to different annotated role attributes, the classified annotated sample data is used to perform model training, and role annotation sub-models corresponding to each annotated role attribute are obtained respectively;

[0049] Then, in step 102, according to different annotated role attributes, the classified annotated sample data is used to perform model training, and role annotation sub-models corresponding to each annotated role attribute are obtained respectively. It can be understood that each role annotation sub-model is formed based on the training of the annotated sample data of the same type of role objects, and each role annotation sub-model will predict labels for the input sample data according to the prior knowledge formed by the model training and its own role attribute knowledge.

[0050] Step 103: According to the predicted sub-label probability distribution output by each role labeling sub-model, combined with the weight parameters corresponding to each role labeling sub-model, each predicted sub-label is integrated to obtain a soft label;

[0051] Predict the weight of each sample on different role models, and use these weights to integrate the predicted sub-labels to generate corresponding soft labels.

[0052] Step 104: According to the loss function, weight optimization is performed on each role labeling sub-model. When the preset optimization conditions are met, a multi-role joint labeling model is obtained by integrating each role labeling sub-model.

[0053] Next, in step 104, based on the soft labels output in step 103, the model and the weight prediction network are collaboratively optimized to improve the labeling accuracy of the overall model by optimizing the LoRA fine-tuning loss of the role model and the soft label loss of the weight prediction network. Finally, the output of the role model and the dynamic weights are combined to generate the final multi-label prediction result for each sample.

[0054] Step 105: Obtain the data to be annotated, and input the data to be annotated into the multi-role joint annotation model to output corresponding multi-role model annotation data through the multi-role joint annotation model.

[0055] It can be understood that a multi-role joint labeling model including multiple role labeling sub-models is constructed through steps 101 to 104. In the actual application stage, the trained multi-role joint labeling model is used to obtain the data to be labeled, and the data to be labeled is input into the multi-role joint labeling model, so as to output multiple predicted sub-labels through each role labeling sub-model in the multi-role joint labeling model according to the same process as steps 101 to 104, and each predicted sub-label is integrated to obtain the soft label of the data to be labeled, that is, the multi-role model labeled data.

[0056] Taking the wrong question analysis and annotation scenario in the education scenario as an example, the model input is student question and answer data, including questions, correct answers, students' wrong answers, detailed analysis and other information. The initial seed data in this embodiment processed 235 history questions and collected 705 corresponding wrong questions (that is, each question contains 3 wrong answers). According to the experts' summary of the corresponding wrong question types, as shown in Table 1, each wrong question has multiple error types. Based on the existing questions and students' answers, some seed samples are manually annotated. Ensure that each error type has sufficient example coverage. This part of the seed annotation data is subsequently processed into fine-tuning data to train multiple role annotation models for large-scale multidisciplinary education data annotation.

[0057] In order to make the model more robust and not affected by the order of prompt words and context, this embodiment can also expand the wrong question data with the help of the context learning ability of the large model. Through the preset complexity expansion, context transformation, task transformation, and wrong option generation expansion rules, with the help of natural language AI, more diverse annotation sample data is generated to further improve the training effect of the role annotation sub-model.

[0058]

[0059] This embodiment uses data annotation of multiple role attributes, and then fine-tunes these models to form multiple role objects. From the perspective of multiple roles, the model can comprehensively consider these factors. The annotated sample data provided by this embodiment includes, but is not limited to, the following role attributes:

[0060] Education Expert Role: Let the model analyze and label the error types from the perspective of an education expert.

[0061] Student role: Let the model describe the difficulty in understanding from the student's perspective, or simulate how the student might answer

[0062] Historian Role: Ask the model to assess students’ understanding of historical events as a historian and analyze students’ responses.

[0063] Subject experts: Subject experts are familiar with the knowledge system in a certain field and can accurately assess students’ understanding and mastery of the field, ensuring the professionalism and accuracy of the annotation results.

[0064] Test designer: Let the model analyze the error types from the perspective of the test designer, focusing on the difficulty of the test questions and whether the design is suitable for the students' cognitive level. Test designers understand the purpose and difficulty of the test questions and can judge whether the students' errors are due to improper test design or beyond the students' cognitive level.

[0065] Educational Psychologist: Let the model analyze students’ errors from the perspective of an educational psychologist, taking into account the psychological barriers or cognitive load that students may encounter during the learning process.

[0066] Peer Assessment: Let the model analyze error types from the perspective of peers, simulating how students can assess and give feedback on each other’s learning outcomes.

[0067] Regarding the multi-role collaborative network training and prediction process of this embodiment, fine-tuning the data set according to the respective role instructions, fine-tuning multiple role-playing models This embodiment will use these multiple models for labeling, and each model outputs the probability distribution of the corresponding label In order to integrate the outputs of multiple models, this embodiment defines an adaptive weight for each model This weight is passed through the weight prediction network Obtain. By corresponding weights and generated probability distributions, this embodiment aggregates the representations of multiple models to generate soft labels. Finally, this embodiment will jointly optimize the role model and the weight network, and this embodiment uses the optimized network to generate the final label.

[0068] We used the role-playing fine-tuning data collected earlier to train the language model. We selected some relatively small models such as Llama3-8b-instruct and Qwen1.5-7b-chat, and fine-tuned these models according to specific roles (such as teachers, students, and educational psychologists) so that they can be more accurately annotated according to the perspectives of different roles. , this embodiment uses cross entropy loss To optimize:

[0069]

[0070] is the jth labeled label, with a total of M labels.

[0071] So far, this embodiment has obtained multiple role models .

[0072] Each role model will input samples Specifically, the input sample is input into the model, and after passing through a series of network layers, the last layer generates logits, which are unnormalized scores used to represent the model's confidence in each category. Assume that for the sample , role model The output logits are , where M is the number of error types.

[0073] Next, the model converts these logits into probability distributions through the Softmax function, as follows:

[0074]

[0075] in Representation sample Belongs to the tag The probability of is the logits for that category, and M is the number of all categories.

[0076] After Softmax normalization, the generated output probability distribution Indicates the confidence level of the model for each label. At this point, this embodiment has generated a sample The probability distribution of .

[0077] Multiple role models are relatively heavy language models, while the weight prediction network uses a simpler and lighter neural network. For example, this embodiment uses a shallow multi-layer perceptron (MLP). The input of this MLP can be the output vector of each language model or the feature representation of the intermediate layer. Specifically, this embodiment uses the output probability distribution generated by the role model , as the input of the weight prediction network.

[0078] For each sample , first through each role model Get its predicted label probability distribution The weight prediction network then receives these probability distributions as input features and generates corresponding weights for each role model. :

[0079]

[0080] here is the weight of the i-th sample on the k-th role model. The weight generated by the weight prediction network Used to weight the output of the role model and calculate samples The final label.

[0081] Then, this embodiment will calculate the sample according to the role model output and the corresponding weight of each sample. Comprehensive soft label :

[0082]

[0083] The strategy of jointly optimizing the role model and the weight prediction network is adopted to improve the prediction performance of the model by reducing the overall loss L.

[0084] In addition to the above loss function for role model training , it is necessary to collaboratively optimize the weight prediction network. The following loss function is designed for optimization:

[0085]

[0086] The total loss function is the weighted sum of the role model loss and the soft label loss:

[0087]

[0088] Among them, 𝜆 is a hyperparameter used to balance the role model loss and soft label loss.

[0089] By optimizing the weight prediction network g(x), the weight of each character model is dynamically adjusted. , to better integrate the annotations of multiple role models. Combine the outputs of all role models and the results of the weight prediction network to calculate the final multi-label prediction :

[0090]

[0091] From the above description, it can be seen that the technical solution of the embodiment of the present application has the following advantages:

[0092] (1) Higher annotation accuracy: The present invention introduces a multi-role-playing model annotation method to integrate the perspectives of different roles such as teachers, students, and educational psychologists into the annotation process. Compared with the existing method of annotation that relies on a single perspective, the present invention can better simulate the annotation method of human experts, effectively reduce the deviation between the annotation results and human annotation, and improve the accuracy of annotation.

[0093] (2) Stronger labeling robustness: The present invention uses data augmentation technology and generates diverse role labeling prompts to improve the robustness of the model under different contexts and changing inputs. Compared with the prior art, the present invention can provide more stable and consistent labeling results when facing diverse educational data, avoiding the problem of inconsistent labeling results due to context changes in the prior art.

[0094] (3) Adaptation: The present invention uses a weight prediction network to dynamically adjust the weights of each role model, so that the system can adaptively adjust the labeling strategy according to different input data. This adaptive capability has not been widely used in the prior art, making the present invention more flexible and adaptable when processing different types of educational data.

[0095] (4) Superiority in multi-label classification tasks: For multi-label classification tasks in the field of education, the present invention specifically optimizes the labeling process, making it perform well in multi-label classification tasks. Compared with the existing technology of single-label or simple classification tasks, the present invention can better handle complex education wrong question classification tasks and provide more accurate multi-label labeling results.

[0096] The above is a detailed description of an embodiment of a multi-role model automatic annotation method provided by the present application. The following is a detailed description of an embodiment of a multi-role model automatic annotation device provided by the present application.

[0097] See also Figure 3 The embodiment of the present application provides a multi-role model automatic labeling device, comprising:

[0098] The labeled sample acquisition unit 201 is used to acquire labeled sample data and classify the labeled sample data according to the labeled role attributes of the labeled sample data;

[0099] The role model building unit 202 is used to perform model training using the classified labeled sample data according to different labeled role attributes, and obtain the role labeling sub-model corresponding to each labeled role attribute;

[0100] The role comprehensive prediction unit 203 is used to integrate the predicted sub-labels according to the predicted sub-label probability distribution output by each role labeling sub-model and the weight parameters corresponding to each role labeling sub-model to obtain a soft label;

[0101] The multi-role model optimization unit 204 is used to optimize the weights of each role labeling sub-model according to the loss function, and when the preset optimization conditions are met, the multi-role joint labeling model is obtained by integrating each role labeling sub-model;

[0102] The multi-role model automatic annotation unit 205 is used to obtain the data to be annotated, input the data to be annotated into the multi-role joint annotation model, and output the corresponding multi-role model annotation data through the multi-role joint annotation model.

[0103] Preferably, it also includes:

[0104] The weight parameter generating unit 200 is used to output the weight parameters of the role labeling sub-model according to the predicted sub-label probability distribution output by each role labeling sub-model, using the predicted sub-label probability distribution as the input feature of the preset weight prediction network model.

[0105] The embodiment of the present application also provides a multi-role model automatic annotation terminal, the types of which are not limited to: personal computers, industrial computers, servers and embedded intelligent devices, and the main components of the terminal include: a memory 33 and a processor 31, and the memory 33 and the processor 31 can be connected via a communication bus 34;

[0106] The memory 33 is used to store program codes, and the program codes correspond to a multi-role model automatic labeling method provided in the above embodiment;

[0107] The processor 31 is used to read and execute program codes.

[0108] The fourth aspect of the present application provides a computer-readable storage medium, in which program code is stored, and the program code is used to be read and executed by a processor to implement a multi-role model automatic labeling method as provided in the above embodiment.

[0109] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the terminals, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0110] In the several embodiments provided in the present application, it should be understood that the disclosed terminals, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0111] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0112] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0113] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0114] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0115] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0116] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for automatic labeling of multi-role models, characterized in that: include: Acquire labeled sample data, and classify the labeled sample data according to the labeled role attributes of the labeled sample data; According to different annotated role attributes, the classified annotated sample data is used to train the model, and the role annotation sub-model corresponding to each annotated role attribute is obtained respectively; According to the predicted sub-label probability distribution output by each role labeling sub-model, combined with the weight parameters corresponding to each role labeling sub-model, each predicted sub-label is integrated to obtain a soft label; According to the loss function, each role labeling sub-model is weighted and optimized. When the preset optimization conditions are met, a multi-role joint labeling model is obtained by integrating the various role labeling sub-models. The data to be annotated is obtained, and the data to be annotated is input into the multi-role joint annotation model, so as to output corresponding multi-role model annotation data through the multi-role joint annotation model.

2. The method for automatic labeling of multiple role models according to claim 1, characterized in that: The loss function is specifically: In the formula, is the total loss value, is the role labeling loss between the predicted sub-label and the standard sub-label output by the role labeling sub-model k, is the comprehensive annotation loss between the soft label and each predicted sub-label, is a standard subtag, Output the predicted sub-label for the i-th sample after the k-th role labeling sub-model The posterior probability of is the soft label corresponding to the i-th sample, is the weight of the i-th sample on the k-th role labeling sub-model, N is the number of labeled sample data, M is the number of standard sub-labels, K is the number of labeled role types, are the hyperparameters for the role labeling loss and the comprehensive labeling loss.

3. The method for automatic labeling of multiple role models according to claim 1, characterized in that: The calculation formula of the soft tag is specifically: In the formula, is the soft label corresponding to the i-th sample, Output the predicted sub-label for the i-th sample after the k-th role labeling sub-model The probability distribution of The weight of the i-th sample on the k-th role labeling sub-model.

4. The method for automatic labeling of multiple role models according to claim 1, characterized in that: The calculation formula of the multi-role model annotation data is specifically: In the formula, Label the data for the multi-role model of the i-th data to be labeled, Output the predicted sub-label for the i-th sample after the k-th role labeling sub-model The probability distribution of The weight of the i-th sample on the k-th role labeling sub-model.

5. The method for automatic labeling of multiple role models according to claim 1, characterized in that: The generation method of the weight parameters corresponding to each role labeling sub-model includes: According to the predicted sub-label probability distribution output by each role labeling sub-model, the predicted sub-label probability distribution is used as the input feature of the preset weight prediction network model to output the weight parameters of the role labeling sub-model.

6. A multi-role model automatic labeling method according to claim 5, characterized in that: The weight prediction network model is specifically an MLP network model.

7. A multi-role model automatic labeling device, characterized in that: include: A labeled sample acquisition unit, used to acquire labeled sample data, and classify the labeled sample data according to the labeled role attributes of the labeled sample data; A role model building unit is used to perform model training using classified labeled sample data according to different labeled role attributes, and obtain role labeling sub-models corresponding to each labeled role attribute; The role comprehensive prediction unit is used to integrate the predicted sub-labels according to the predicted sub-label probability distribution output by each role labeling sub-model and the weight parameters corresponding to each role labeling sub-model to obtain a soft label; A multi-role model optimization unit is used to optimize the weights of each role labeling sub-model according to the loss function, and when the preset optimization conditions are met, a multi-role joint labeling model is obtained by integrating the various role labeling sub-models; The multi-role model automatic annotation unit is used to obtain the data to be annotated, input the data to be annotated into the multi-role joint annotation model, and output the corresponding multi-role model annotation data through the multi-role joint annotation model.

8. The multi-role model automatic labeling device according to claim 7, characterized in that: Also includes: The weight parameter generating unit is used to output the weight parameters of the role labeling sub-model according to the predicted sub-label probability distribution output by each role labeling sub-model, using the predicted sub-label probability distribution as the input feature of the preset weight prediction network model.

9. A multi-role model automatically annotates terminals, characterized in that: include: Memory and processor; The memory is used to store program codes, and the program codes correspond to a method for automatic labeling of multi-role models according to any one of claims 1 to 6; The processor is used for reading and executing the program code.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program code, and the program code is used to be read and executed by a processor to implement a multi-role model automatic labeling method as described in any one of claims 1 to 6.

Images