A multi-role model automatic labeling method and device, a terminal and a medium

Abstract

This application discloses a method, apparatus, terminal, and medium for automatic annotation of multi-role models. The technical solution provided by this application introduces a multi-role perspective annotation approach, constructs multiple annotation sub-models with different role attributes according to the role classification and knowledge architecture of a specific domain scenario, enabling them to annotate data from different perspectives. By integrating the annotation results of multiple role models, the deviation between model annotation and human annotation is minimized, and the annotation bias caused by a single perspective in the prior art is effectively solved, making the annotation results more comprehensive and accurate.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, terminal and medium for automatic annotation of multi-role models. Background Technology

[0002] In recent years, large-scale language model technologies such as ChatGPT have matured. Thanks to their excellent natural language generation capabilities, these models have been applied to various domain tasks, such as data generation in finance and law. Furthermore, using large models for automatic data annotation helps reduce manual and economic costs. However, in practical applications, especially in specific domains like education, the labels generated by the models often differ significantly from standard labels, resulting in low accuracy in automatic annotation. Summary of the Invention

[0003] This application provides a method, apparatus, terminal, and medium for automatic annotation of multi-role models, which addresses the technical problem of low annotation accuracy in existing automatic annotation technologies for large model data.

[0004] To address the aforementioned technical problems, the first aspect of this application provides an automatic annotation method for multi-role models, comprising:

[0005] Obtain labeled sample data and classify the labeled sample data according to the labeling role attributes of the labeled sample data;

[0006] Based on different labeled role attributes, the model is trained using the classified labeled sample data to obtain the role labeling sub-model corresponding to each labeled role attribute.

[0007] Based on the probability distribution of predicted sub-labels output by each role labeling sub-model, and combined with the weight parameters corresponding to each role labeling sub-model, the predicted sub-labels are integrated to obtain soft labels.

[0008] Based on the loss function, the weights of each role annotation sub-model are optimized. When the preset optimization conditions are met, the multi-role joint annotation model is obtained by integrating the various role annotation sub-models.

[0009] Obtain the data to be labeled, input the data to be labeled into the multi-role joint labeling model, and output the corresponding multi-role model labeling data through the multi-role joint labeling model.

[0010] Preferably, the loss function is as follows:

[0011]

[0012] In the formula, This is the total loss value. The role labeling loss is calculated by comparing the predicted sub-labels and standard sub-labels output by the role labeling sub-model k. The combined annotation loss between the soft label and each predicted sub-label is... For standard sub-tags, For the i-th sample, output the predicted sub-label using the k-th role labeling sub-model. The posterior probability, The soft label corresponding to the i-th sample. Let N be the weight of the i-th sample in the k-th role annotation sub-model, N be the number of annotated sample data, M be the number of standard sub-labels, and K be the number of annotated role categories. These are the hyperparameters used for role annotation loss and overall annotation loss.

[0013] Preferably, the calculation formula for the soft tag is as follows:

[0014]

[0015] In the formula, The soft label corresponding to the i-th sample. For the i-th sample, output the predicted sub-label using the k-th role labeling sub-model. The probability distribution, Let i be the weight of the i-th sample in the sub-model labeled with the k-th role.

[0016] Preferably, the calculation formula for the multi-role model annotation data is as follows:

[0017]

[0018] In the formula, Label the multi-role model for the i-th data to be labeled. For the i-th sample, output the predicted sub-label using the k-th role labeling sub-model. The probability distribution, Let i be the weight of the i-th sample in the sub-model labeled with the k-th role.

[0019] Preferably, the method for generating the weight parameters corresponding to each role annotation sub-model includes:

[0020] Based on the predicted sub-label probability distribution output by each role labeling sub-model, the predicted sub-label probability distribution is used as the input feature of the preset weight prediction network model to output the weight parameters of the role labeling sub-model.

[0021] Preferably, the weight prediction network model is an MLP network model.

[0022] Meanwhile, a second aspect of this application provides an automatic annotation device for multi-role models, comprising:

[0023] The labeled sample acquisition unit is used to acquire labeled sample data and classify the labeled sample data according to the labeled role attributes of the labeled sample data;

[0024] The role model building unit is used to train the model using the classified labeled sample data according to different labeled role attributes, and obtain the role labeled sub-models corresponding to each labeled role attribute.

[0025] The role comprehensive prediction unit is used to integrate the predicted sub-labels based on the probability distribution of the predicted sub-labels output by each role labeling sub-model and the weight parameters corresponding to each role labeling sub-model to obtain soft labels.

[0026] The multi-role model optimization unit is used to optimize the weights of each role annotation sub-model according to the loss function. When the preset optimization conditions are met, the multi-role joint annotation model is obtained by integrating the various role annotation sub-models.

[0027] The multi-role model automatic annotation unit is used to acquire data to be annotated, input the data to be annotated into the multi-role joint annotation model, and output the corresponding multi-role model annotation data through the multi-role joint annotation model.

[0028] Preferably, it further includes:

[0029] The weight parameter generation unit is used to output the weight parameters of the role labeling sub-model based on the predicted sub-label probability distribution output by each role labeling sub-model, using the predicted sub-label probability distribution as the input feature of the preset weight prediction network model.

[0030] A third aspect of this application provides a multi-role model automatic annotation terminal, including: a memory and a processor;

[0031] The memory is used to store program code, which corresponds to the automatic annotation method for multi-role models provided in the first aspect of this application;

[0032] The processor is used to read and execute the program code.

[0033] The fourth aspect of this application provides a computer-readable storage medium storing program code, which is read and executed by a processor to implement an automatic annotation method for multi-role models as provided in the first aspect of this application.

[0034] As can be seen from the above technical solutions, this application has the following advantages:

[0035] The technical solution provided in this application introduces a multi-role perspective annotation method. According to the role classification and knowledge architecture of specific domain scenarios, multiple annotation sub-models with different role attributes are constructed, enabling them to annotate data from different perspectives. By integrating the annotation results of multiple role models, the deviation between model annotation and human annotation is minimized, and the annotation bias caused by a single perspective in the existing technology is effectively solved, making the annotation results more comprehensive and accurate. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is a flowchart illustrating an embodiment of an automatic annotation method for multi-role models provided in this application.

[0038] Figure 2 A logic block diagram of an embodiment of an automatic annotation method for multi-role models provided in this application.

[0039] Figure 3 This is a schematic diagram of an embodiment of an automatic annotation device for multi-role models provided in this application.

[0040] Figure 4 This is a schematic diagram of the structure of an embodiment of an automatic annotation terminal for multi-role models provided in this application. Detailed Implementation

[0041] While automatic annotation methods based on Large Language Models (LLMs) reduce the cost of manual annotation, these methods often overlook the detailed understanding and background knowledge in human annotation. Taking the education field as an example, existing technologies struggle to effectively integrate this experience and understanding into the model's annotation process, resulting in a lack of accuracy and practicality in the annotation results. Furthermore, existing model annotations typically focus on a single perspective, failing to fully reflect the complexity of educational contexts, leading to biased and one-sided annotation results. This results in a significant discrepancy between the annotation results generated by the model and those annotated by human experts.

[0042] This application provides a method, apparatus, terminal, and medium for automatic annotation of multi-role models, which addresses the technical problem of low annotation accuracy in existing automatic annotation technologies for large model data.

[0043] To make the inventive objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0044] First, a detailed description of an embodiment of the automatic annotation method for multi-role models provided in this application is as follows:

[0045] Please see Figure 1 and Figure 2 This application provides an automatic annotation method for multi-role models, comprising:

[0046] Step 101: Obtain labeled sample data and classify the labeled sample data according to the labeling role attributes;

[0047] It should be noted that, firstly, a certain number of labeled sample data are obtained. Each type of labeled sample data includes multiple versions labeled by different roles (such as subject teachers, students, educational psychology experts, etc.). The labeled sample data is classified according to the labeling role attributes and can be used to construct labeled data samples for multiple role models.

[0048] Step 102: According to different labeled role attributes, use the classified labeled sample data to train the model and obtain the role labeled sub-model corresponding to each labeled role attribute.

[0049] Then, in step 102, according to different labeled role attributes, the model is trained using the classified labeled sample data to obtain the role label sub-models corresponding to each labeled role attribute. It can be understood that each role label sub-model is trained based on the labeled sample data of the same type of role object. Each role label sub-model will predict the label of the input sample data according to the prior knowledge formed by the model training and its own role attribute knowledge.

[0050] Step 103: Based on the probability distribution of predicted sub-labels output by each role labeling sub-model, and combined with the weight parameters corresponding to each role labeling sub-model, integrate the predicted sub-labels to obtain soft labels.

[0051] Predict the weight of each sample on different role models, and use these weights to integrate the predicted sub-labels to generate corresponding soft labels.

[0052] Step 104: Based on the loss function, optimize the weights of each role annotation sub-model. When the preset optimization conditions are met, the multi-role joint annotation model is obtained by integrating the various role annotation sub-models.

[0053] Next, in step 104, based on the soft labels output in step 103, the model and the weight prediction network are collaboratively optimized. By optimizing the LoRA fine-tuning loss of the role model and the soft label loss of the weight prediction network, the overall labeling accuracy of the model is improved. Finally, the output of the role model and the dynamic weights are combined to generate the final multi-label prediction result for each sample.

[0054] Step 105: Obtain the data to be labeled and input it into the multi-role joint labeling model to output the corresponding multi-role model labeling data.

[0055] It is understandable that after steps 101 to 104, a multi-role joint annotation model containing multiple role annotation sub-models is constructed. In the practical application stage, the acquired data to be annotated is input into the multi-role joint annotation model using the trained multi-role joint annotation model. Following the same process as steps 101 to 104, multiple predicted sub-labels are output by each role annotation sub-model in the multi-role joint annotation model. The predicted sub-labels are then integrated to obtain the soft label of the data to be annotated, i.e., the multi-role model annotation data.

[0056] Taking the error analysis and annotation scenario in an educational context as an example, the model input consists of student question-and-answer data, including information such as the question, correct answer, student's incorrect answer, and detailed explanation. In this embodiment, the initial seed data processed 235 historical questions, collecting 705 corresponding incorrect questions (i.e., each question contains 3 incorrect answers). Based on expert summaries of the corresponding error types, as shown in Table 1, each incorrect question has multiple error types. Based on the existing questions and student answers, some seed examples are manually annotated to ensure sufficient example coverage for each error type. This seed-annotated data is subsequently processed into fine-tuning data to train multiple role-based annotation models for large-scale, multi-disciplinary educational data annotation.

[0057] To make the model more robust and unaffected by the order of prompts or context, this embodiment also leverages the context learning capabilities of large models to expand the incorrect question data. Through preset rules for complexity expansion, context transformation, task conversion, and incorrect option generation, and with the help of natural language AI, more diverse labeled sample data is generated to further improve the training effect of the role labeling sub-model.

[0058]

[0059] This embodiment utilizes data annotations of various role attributes. These models are subsequently fine-tuned to form multiple role objects, taking a multi-role perspective to enable the model to comprehensively consider these factors. The annotated sample data provided in this embodiment includes role attributes including, but not limited to, the following:

[0060] Educational expert role: Allow the model to analyze and label error types from the perspective of an educational expert.

[0061] Student role: Allow the model to describe comprehension difficulties from the student's perspective, or simulate how a student might answer.

[0062] Historian role: The model is required to assess students’ understanding of historical events from the perspective of a historian and analyze students’ responses.

[0063] Subject matter experts: Subject matter experts are familiar with the knowledge system of a certain field and can accurately assess students' understanding and mastery of the field, ensuring the professionalism and accuracy of the annotation results.

[0064] Test Designer: The model analyzes error types from the test designer's perspective, focusing on the difficulty and suitability of the test questions for students' cognitive levels. Test designers understand the purpose and difficulty of the questions, enabling them to determine whether student errors are due to inappropriate question design or exceeding the students' cognitive level.

[0065] Educational psychologists: Let the model analyze students' mistakes from the perspective of educational psychologists, taking into account the psychological obstacles or cognitive load that students may encounter in the learning process.

[0066] Peer assessment: The model analyzes error types from the perspective of peers, simulating how students assess and provide feedback on each other's learning outcomes.

[0067] Regarding the multi-role collaborative network training and prediction process in this embodiment, the dataset is fine-tuned according to the instructions of each role, and multiple role-playing models are fine-tuned. This embodiment will use multiple models for labeling, with each model outputting the probability distribution of the corresponding label. To synthesize the outputs of multiple models, this embodiment defines an adaptive weight for each model. This weight is predicted by the weight prediction network. The process involves acquiring soft labels by aggregating representations from multiple models using corresponding weights and generated probability distributions. Finally, this embodiment performs joint optimization on the role model and the weight network, using the optimized network to generate the final labels.

[0068] The language model was trained using the previously collected role-playing fine-tuning data. Relatively small models such as Llama3-8b-instruct and Qwen1.5-7b-chat were selected and fine-tuned according to specific roles (e.g., teacher, student, educational psychologist) to enable more accurate annotation from different role perspectives. During the fine-tuning process, the fine-tuning dataset... This embodiment uses cross-entropy loss. Optimize:

[0069]

[0070] Let j be the j-th label, and there are a total of M labels.

[0071] Thus, this embodiment yields multiple character models. .

[0072] Each character model will process the input samples. The process involves processing the input samples. Specifically, the input samples are fed into the model, and after passing through a series of network layers, the last layer generates logits. These logits are unnormalized scores used to represent the model's confidence in each class. Assuming that for a sample... Character Model The output logits are , where M is the number of error types.

[0073] Next, the model will use the Softmax function to transform these logits into a probability distribution, as shown in the following formula:

[0074]

[0075] in Indicates sample Belongs to the label The probability, is the logits of this category, and M is the number of all categories.

[0076] The generated output probability distribution after Softmax normalization This indicates the model's confidence level for each label. At this point, this embodiment has already generated a sample... The probability distribution.

[0077] Multiple character models are relatively heavy language models, while the weight prediction network uses a simpler and lighter neural network; for example, this embodiment uses a shallow multilayer perceptron (MLP). The input to this MLP can be the output vectors of the various language models or the feature representations of intermediate layers. Specifically, this embodiment uses the output probability distribution generated by the character models. , which serves as the input to the weight prediction network.

[0078] For each sample First, through each character model Obtain its predicted label probability distribution Subsequently, the weight prediction network receives these probability distributions as input features and generates corresponding weights for each role model. :

[0079]

[0080] here These are the weights of the i-th sample on the k-th role model. The weights are generated by the weight prediction network. Used to weight and integrate the output of the role model, calculating samples The final tag.

[0081] Then, this embodiment will calculate the sample based on the role model output and corresponding weights for each sample. Comprehensive soft tags :

[0082]

[0083] A strategy of jointly optimizing the role model and the weight prediction network is adopted to improve the model's predictive performance by reducing the overall loss L.

[0084] In addition to the loss function for training the character model mentioned above To achieve this, the weight prediction network needs to be optimized collaboratively. The following loss function is designed for optimization:

[0085]

[0086] The total loss function is a weighted sum of the loss from the role model and the soft label loss:

[0087]

[0088] Here, 𝜆 is a hyperparameter used to balance the role model loss and the soft label loss.

[0089] By optimizing the weight prediction network g(x), the weights of each role model are dynamically adjusted. To better integrate the annotations of multiple role models, the final multi-label prediction is calculated by combining the outputs of all role models and the results of the weight prediction network. :

[0090]

[0091] As can be seen from the above description, the technical solution of this application embodiment has the following advantages:

[0092] (1) Higher annotation accuracy: This invention introduces a multi-role model annotation method, incorporating the perspectives of different roles such as teachers, students, and educational psychologists into the annotation process. Compared with existing technologies that rely on single-perspective annotation methods, this invention can better simulate the annotation methods of human experts, effectively reduce the deviation between annotation results and human annotations, and improve the accuracy of annotation.

[0093] (2) Enhanced annotation robustness: This invention employs data augmentation technology and improves the robustness of the model under different contexts and varying inputs by generating diverse role annotation prompts. Compared with existing technologies, this invention can provide more stable and consistent annotation results when faced with diverse educational data, avoiding the problem of inconsistent annotation results due to context changes in existing methods.

[0094] (3) Adaptive: This invention uses a weight prediction network to dynamically adjust the weights of each role model, enabling the system to adaptively adjust the labeling strategy according to different input data. This adaptive capability has not been widely applied in the prior art, making this invention more flexible and adaptable when processing different types of educational data.

[0095] (4) Superiority in multi-label classification tasks: For multi-label classification tasks in the field of education, this invention has optimized the annotation process, making it perform well in multi-label classification tasks. Compared with existing technologies for single-label or simple classification tasks, this invention can better handle complex educational error classification tasks and provide more accurate multi-label annotation results.

[0096] The above is a detailed description of an embodiment of an automatic annotation method for multi-role models provided in this application. The following is a detailed description of an embodiment of an automatic annotation device for multi-role models provided in this application.

[0097] Please see Figure 3 This application provides an automatic annotation device for multi-role models, including:

[0098] The labeled sample acquisition unit 201 is used to acquire labeled sample data and classify the labeled sample data according to the labeling role attributes of the labeled sample data;

[0099] The role model construction unit 202 is used to train the model using the classified labeled sample data according to different labeled role attributes, and obtain the role labeled sub-models corresponding to each labeled role attribute.

[0100] The role comprehensive prediction unit 203 is used to integrate the predicted sub-labels based on the probability distribution of the predicted sub-labels output by each role labeling sub-model and the weight parameters corresponding to each role labeling sub-model to obtain soft labels.

[0101] The multi-role model optimization unit 204 is used to optimize the weights of each role annotation sub-model according to the loss function. When the preset optimization conditions are met, the multi-role joint annotation model is obtained by integrating the various role annotation sub-models.

[0102] The multi-role model automatic annotation unit 205 is used to acquire the data to be annotated, input the data to be annotated into the multi-role joint annotation model, and output the corresponding multi-role model annotation data through the multi-role joint annotation model.

[0103] Preferably, it further includes:

[0104] The weight parameter generation unit 200 is used to output the weight parameters of the role labeling sub-model based on the predicted sub-label probability distribution output by each role labeling sub-model, using the predicted sub-label probability distribution as the input feature of the preset weight prediction network model.

[0105] This application embodiment also provides a multi-role model automatic annotation terminal. The types of terminals include, but are not limited to, personal computers, industrial computers, servers, and embedded intelligent devices. The main components of the terminal include: a memory 33 and a processor 31. The memory 33 and the processor 31 can be connected through a communication bus 34.

[0106] Memory 33 is used to store program code, which corresponds to an automatic annotation method for multi-role models as provided in the above embodiments;

[0107] Processor 31 is used to read and execute program code.

[0108] The fourth aspect of this application provides a computer-readable storage medium storing program code, which is read and executed by a processor to implement an automatic annotation method for multi-role models as provided in the above embodiments.

[0109] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the terminals, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0110] In the several embodiments provided in this application, it should be understood that the disclosed terminals, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between devices or units through some interfaces, and may be electrical, mechanical, or other forms.

[0111] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0112] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0113] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0114] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0115] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0116] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for automatic annotation of multi-role models, characterized in that, include: Obtain labeled sample data and classify the labeled sample data according to the labeling role attributes of the labeled sample data; Based on different labeled role attributes, the model is trained using the classified labeled sample data to obtain the role labeling sub-model corresponding to each labeled role attribute. Based on the predicted sub-label probability distribution output by each role labeling sub-model, and combined with the weight parameters corresponding to each role labeling sub-model, the predicted sub-labels are integrated to obtain soft labels. The generation method of the weight parameters corresponding to each role labeling sub-model includes: based on the predicted sub-label probability distribution output by each role labeling sub-model, using the predicted sub-label probability distribution as the input feature of a preset weight prediction network model, and outputting the weight parameters of the role labeling sub-model. The weight prediction network model is specifically an MLP network model. Based on the loss function, the weights of each role annotation sub-model are optimized. When the preset optimization conditions are met, the multi-role joint annotation model is obtained by integrating the various role annotation sub-models. Obtain the data to be labeled, input the data to be labeled into the multi-role joint labeling model, and output the corresponding multi-role model labeling data through the multi-role joint labeling model.

2. The automatic annotation method for multi-role models according to claim 1, characterized in that, The loss function is specifically as follows: In the formula, This is the total loss value. The role labeling loss is calculated by comparing the predicted sub-labels and standard sub-labels output by the role labeling sub-model k. The combined annotation loss between the soft label and each predicted sub-label is... For standard sub-tags, For the i-th sample, output the predicted sub-label using the k-th role labeling sub-model. The posterior probability, The soft label corresponding to the i-th sample. Let N be the weight of the i-th sample in the k-th role annotation sub-model, N be the number of annotated sample data, M be the number of standard sub-labels, and K be the number of annotated role categories. These are hyperparameters used for role annotation loss and overall annotation loss. Let i be the i-th sample.

3. The automatic annotation method for multi-role models according to claim 2, characterized in that, The specific formula for calculating the soft tag is as follows: In the formula, The soft label corresponding to the i-th sample. For the i-th sample, output the predicted sub-label using the k-th role labeling sub-model. The probability distribution, Let i be the weight of the i-th sample in the sub-model labeled with the k-th role.

4. The automatic annotation method for multi-role models according to claim 2, characterized in that, The specific formula for calculating the multi-role model annotation data is as follows: In the formula, Label the multi-role model for the i-th data to be labeled. For the i-th sample, output the predicted sub-label using the k-th role labeling sub-model. The probability distribution, Let i be the weight of the i-th sample in the sub-model labeled with the k-th role.

5. An automatic annotation device for multi-role models, characterized in that, include: The labeled sample acquisition unit is used to acquire labeled sample data and classify the labeled sample data according to the labeled role attributes of the labeled sample data; The role model building unit is used to train the model using the classified labeled sample data according to different labeled role attributes, and obtain the role labeled sub-models corresponding to each labeled role attribute. The role comprehensive prediction unit is used to integrate the predicted sub-labels based on the probability distribution of the predicted sub-labels output by each role labeling sub-model and the weight parameters corresponding to each role labeling sub-model to obtain soft labels. The generation method of the weight parameters corresponding to each role labeling sub-model includes: based on the probability distribution of the predicted sub-labels output by each role labeling sub-model, using the probability distribution of the predicted sub-labels as the input features of a preset weight prediction network model, and outputting the weight parameters of the role labeling sub-model. The weight prediction network model is specifically an MLP network model. The multi-role model optimization unit is used to optimize the weights of each role annotation sub-model according to the loss function. When the preset optimization conditions are met, the multi-role joint annotation model is obtained by integrating the various role annotation sub-models. The multi-role model automatic annotation unit is used to acquire data to be annotated, input the data to be annotated into the multi-role joint annotation model, and output the corresponding multi-role model annotation data through the multi-role joint annotation model.

6. The automatic annotation device for multi-role models according to claim 5, characterized in that, Also includes: The weight parameter generation unit is used to output the weight parameters of the role labeling sub-model based on the predicted sub-label probability distribution output by each role labeling sub-model, using the predicted sub-label probability distribution as the input feature of the preset weight prediction network model.

7. A terminal for automatic annotation of multi-role models, characterized in that, include: Memory and processor; The memory is used to store program code, which corresponds to the multi-role model automatic annotation method as described in any one of claims 1 to 4; The processor is used to read and execute the program code.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that is read and executed by a processor to implement an automatic annotation method for multi-role models as described in any one of claims 1 to 4.

Images