Model processing method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a model processing method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the steps that an initial text model is obtained, the initial text model comprises a model obtained through training based on a problem data sample set, and the problem data sample set comprises data samples marked with error category labels; a training data sample set is input into the initial text model, the attention variance of an attention head corresponding to the initial text model is calculated, and the training data sample set comprises data samples marked with correct category labels; the attention head with the attention variance smaller than a preset value is cut, an adjusted text model is obtained, and the adjusted text model is used for text classification. The method can improve the model accuracy, and is suitable for more application scenes.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a model processing method, apparatus, computer equipment, computer-readable storage medium and computer program product. Background Art

[0002] Backdoor attacks are a security threat to machine learning models in the field of natural language processing, especially to widely used pre-trained text models. This attack method refers to implanting malicious triggers in the model's training data, causing the model to incorrectly perform certain operations or produce specific outputs when encountering specific trigger inputs, such as misclassifying positive emotional evaluations as negative, or inserting specific advertisements or harmful content when generating text.

[0003] In the related technologies, backdoor defense technology mainly focuses on backdoor model detection and harmful text detection. However, this method is mainly aimed at preventing the triggering of backdoors, and cannot eliminate the backdoors in pre-trained text models. Its applicability is limited, and it may make wrong judgments on clean models and samples. Summary of the invention

[0004] Based on this, it is necessary to provide a model processing method, device, computer equipment, computer-readable storage medium and computer program product that can improve accuracy and be suitable for more application scenarios in response to the above technical problems.

[0005] In a first aspect, the present application provides a model processing method, comprising:

[0006] Acquire an initial text model, wherein the initial text model includes a model trained based on a problem data sample set, wherein the problem data sample set includes data samples annotated with incorrect category labels;

[0007] Inputting a training data sample set into the initial text model, and calculating the attention variance of the attention head corresponding to the initial text model, wherein the training data sample set includes data samples marked with correct category labels;

[0008] The attention heads whose attention variance is less than a preset value are trimmed to obtain an adjusted text model, and the adjusted text model is used for text classification.

[0009] In one embodiment, the step of trimming the attention heads whose attention variance is less than a preset value to obtain an adjusted text model includes:

[0010] Crop the attention heads whose attention variance is less than a preset value to obtain an intermediate text model;

[0011] Inputting the training data sample set into the intermediate text model to obtain an output result;

[0012] When the similarity between the output result and the correct category label is less than a preset similarity, the intermediate text model is determined to be an adjusted text model.

[0013] In one embodiment, after obtaining the output result, the method further includes:

[0014] When the similarity between the output result and the correct category label is greater than or equal to the preset similarity, the attention variance of the attention head corresponding to the intermediate text model is obtained, and the attention heads whose attention variance is less than the preset value are trimmed until the similarity is less than the preset similarity, thereby obtaining an adjusted text model.

[0015] In one embodiment, the step of pruning the attention heads whose attention variance is less than a preset value includes:

[0016] According to the attention variance, determining an attention head with the smallest attention variance from the attention heads as a target attention head;

[0017] Prune the target attention head.

[0018] In one embodiment, the step of trimming the attention heads whose attention variance is less than a preset value to obtain an adjusted text model includes:

[0019] Crop the attention heads whose attention variance is less than a preset value to obtain an intermediate text model;

[0020] Inputting the training data sample set into the intermediate text model to obtain an output result;

[0021] According to the output result, the weight of the attention head corresponding to the intermediate text model is adjusted until the target loss function value of the model is less than the preset loss value, so as to obtain an adjusted text model, wherein the target loss function value is determined based on the preset regularization function value.

[0022] In one embodiment, the objective loss function value is determined by:

[0023] Determine a cross entropy loss function value and a preset regularization function value corresponding to the model according to the output result, wherein the preset regularization function value includes a bi-norm function value;

[0024] A target loss function value is determined according to the cross entropy loss function value and the preset regularization function value.

[0025] In a second aspect, the present application also provides a model processing device, comprising:

[0026] An acquisition module, used to acquire an initial text model, wherein the initial text model includes a model trained based on a problem data sample set, wherein the problem data sample set includes data samples annotated with incorrect category labels;

[0027] An input module, used to input a training data sample set into the initial text model, and calculate the attention variance of the attention head corresponding to the initial text model, wherein the training data sample set includes data samples marked with correct category labels;

[0028] The adjustment module is used to trim the attention heads whose attention variance is less than a preset value to obtain an adjusted text model, and the adjusted text model is used for text classification.

[0029] In one embodiment, the adjustment module is further used for:

[0030] Crop the attention heads whose attention variance is less than a preset value to obtain an intermediate text model;

[0031] Inputting the training data sample set into the intermediate text model to obtain an output result;

[0032] When the similarity between the output result and the correct category label is less than a preset similarity, the intermediate text model is determined to be an adjusted text model.

[0033] In one embodiment, the device is further used for:

[0034] When the similarity between the output result and the correct category label is greater than or equal to the preset similarity, the attention variance of the attention head corresponding to the intermediate text model is obtained, and the attention heads whose attention variance is less than the preset value are trimmed until the similarity is less than the preset similarity, thereby obtaining an adjusted text model.

[0035] In one embodiment, the adjustment module is further used for:

[0036] According to the attention variance, determining an attention head with the smallest attention variance from the attention heads as a target attention head;

[0037] Prune the target attention head.

[0038] In one embodiment, the adjustment module is further used for:

[0039] Crop the attention heads whose attention variance is less than a preset value to obtain an intermediate text model;

[0040] Inputting the training data sample set into the intermediate text model to obtain an output result;

[0041] According to the output result, the weight of the attention head corresponding to the intermediate text model is adjusted until the target loss function value of the model is less than the preset loss value, so as to obtain an adjusted text model, wherein the target loss function value is determined based on the preset regularization function value.

[0042] In one embodiment, the device further comprises a module for determining the target loss function value, wherein the module for determining the target loss function value is used to:

[0043] Determine a cross entropy loss function value and a preset regularization function value corresponding to the model according to the output result, wherein the preset regularization function value includes a bi-norm function value;

[0044] A target loss function value is determined according to the cross entropy loss function value and the preset regularization function value.

[0045] In a third aspect, an embodiment of the present disclosure further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of any one of the methods in the embodiments of the present disclosure are implemented.

[0046] In a fourth aspect, the embodiments of the present disclosure further provide a computer-readable storage medium, wherein a computer program is stored thereon, and when the computer program is executed by a processor, the steps of any one of the methods in the embodiments of the present disclosure are implemented.

[0047] In a fifth aspect, the embodiments of the present disclosure further provide a computer program product, wherein the computer program product includes a computer program, and when the computer program is executed by a processor, the steps of any one of the methods in the embodiments of the present disclosure are implemented.

[0048] The above-mentioned model processing method, device, computer equipment, computer-readable storage medium and computer program product obtain an initial text model trained based on a problem data sample set, and input a training data sample set of data samples marked with correct category labels into the initial text model, calculate the attention variance of the attention head, and trim the attention head whose attention variance is less than the preset value to obtain an adjusted text model for text classification. The initial text is trained based on a problem data sample set, which is affected by the problem data sample. Some attention heads will show attention focus drift characteristics, and the attention heads corresponding to the attention focus drift characteristics have a lower variance under the correct data set. By inputting the training data sample set into the initial text model, attention heads with smaller attention variance can be screened out, that is, attention heads with attention focus drift characteristics are trimmed, thereby effectively purifying the model, reducing the impact of the problem data sample set on the model, and effectively eliminating the backdoor in the model, which is suitable for more application scenarios; the security of the model can be strengthened from the inside without relying on external information, so that the pre-trained model (that is, the initial text model) of the backdoor attack can also be processed to perform text classification, and the classification accuracy of the model in text classification is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0050] Figure 1 A schematic diagram of a flow chart of a model processing method in one embodiment;

[0051] Figure 2 A schematic flow chart of a model processing method in another embodiment;

[0052] Figure 3 Schematic diagram of a flow chart of a method for determining a target loss function value in one embodiment;

[0053] Figure 4 A schematic flow chart of a model processing method in another embodiment;

[0054] Figure 5 is a structural block diagram of a model processing device in an embodiment;

[0055] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0057] In one embodiment, Figure 1 As shown, a model processing method is provided. This embodiment uses the method applied to a terminal as an example for illustration. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0058] Step S110, obtaining an initial text model, wherein the initial text model includes a model trained based on a problem data sample set, wherein the problem data sample set includes data samples annotated with incorrect category labels;

[0059] Exemplarily, an initial text model is obtained, wherein the initial text model may include a pre-trained model for text classification.

[0060] In one example, the initial text model may be a model with a backdoor implanted therein. The backdoor may be implanted through a set of problematic data samples. The set of problematic data samples includes data samples annotated with incorrect category labels, wherein the set of problematic data samples may be a part of the entire training sample set of the initial text model, and the incorrect category labels may be obtained after modification based on correct category labels.

[0061] In one possible implementation, on a clean dataset (i.e., samples corresponding to the correct category labels), create a set of problem data samples , where + represents the operation of inserting a trigger, is the trigger, and is with The original label (i.e., the correct category label) is different from the target label (i.e., the wrong category label). During the training process, as shown in Formula (1), by minimizing the joint dataset The cross entropy loss on To obtain the pre-trained model (ie, the initial text model) .

[0062] (1)

[0063] in, Represents the weights of the pre-trained model.

[0064] Optionally, the initial text model can be downloaded from a third-party platform. In one example, the initial text model can be constructed based on a transformer structure, including but not limited to a BERT model, that is, based on a multi-head attention mechanism.

[0065] Step S120, inputting a training data sample set into the initial text model, and calculating the attention variance of the attention head corresponding to the initial text model, wherein the training data sample set includes data samples marked with correct category labels;

[0066] Exemplarily, a training data sample set is input into an initial text model, wherein the training data sample set includes data samples annotated with correct category labels, that is, the training data samples can be considered as clean samples.

[0067] Optionally, the initial text model is preliminarily trained using a set of training data samples, and the attention variance of the attention head corresponding to the initial text model is calculated.

[0068] In one possible implementation, the attention weight matrix of the initial text model is derived from the inner product of vectors Q and K. Layer The self-attention head is expressed as formula (2).

[0069] (2)

[0070] in, is a The attention weight matrix, n represents the length of the input text, is the dimension information of vector k. The input text is represented as , further, as in formula (3), the word The attention weight of any word in the input text is expressed as:

[0071] (3)

[0072] For transformer layer, Attention heads are added and attention variance is introduced to quantify the attention distribution. The attention variance is expressed as formula (4).

[0073] (4)

[0074] In order to solve the classification problem, we focus on the special word ([CLS]) in the model. Because in the classification task, the model will add the [CLS] word to the input header, and this word is used for text classification in downstream tasks. , then at this time will take the minimum value of 0; on the contrary, if [CLS] focuses on a certain input word, then It will get the maximum value Therefore, if attention is divided, it can potentially defend against backdoor attacks.

[0075] Step S130, trimming the attention heads whose attention variance is less than a preset value to obtain an adjusted text model, and the adjusted text model is used for text classification.

[0076] Exemplarily, different attention heads may correspond to different variances. The initial text model may correspond to multiple attention heads, and the multiple attention heads are screened according to their variances to obtain attention heads with variances less than a preset value and trimmed to obtain an adjusted text model.

[0077] In one example, the preset value may be determined according to an actual application scenario. For example, the preset value may be set according to the variance of multiple attention heads to screen out the attention head with the smallest variance for cropping.

[0078] Optionally, the adjusted text model can be used for text classification. In one example, the model can be further trained based on the adjusted text model to obtain a model with higher output accuracy for performing text classification tasks.

[0079] In the disclosed embodiment, an initial text model obtained by training based on a problem data sample set is obtained, and a training data sample set of data samples marked with correct category labels is input into the initial text model, the attention variance of the attention head is calculated, and the attention head with an attention variance less than a preset value is clipped to obtain an adjusted text model for text classification. The initial text is obtained by training based on a problem data sample set, and is affected by the problem data sample. Some attention heads will show attention focus drift characteristics, and the attention heads corresponding to the attention focus drift characteristics have a lower variance under the correct data set. By inputting the training data sample set into the initial text model, attention heads with smaller attention variances can be screened out, that is, attention heads with attention focus drift characteristics are clipped, thereby effectively purifying the model, reducing the impact of the problem data sample set on the model, and effectively eliminating the backdoor in the model, which is suitable for more application scenarios; the security of the model can be strengthened from the inside without relying on external information, so that the pre-trained model (that is, the initial text model) of the backdoor attack can also be processed to perform text classification work, and the classification accuracy of the model in text classification is improved.

[0080] In one embodiment, the step of trimming the attention heads whose attention variance is less than a preset value to obtain an adjusted text model includes:

[0081] Crop the attention heads whose attention variance is less than a preset value to obtain an intermediate text model;

[0082] Inputting the training data sample set into the intermediate text model to obtain an output result;

[0083] When the similarity between the output result and the correct category label is less than a preset similarity, the intermediate text model is determined to be an adjusted text model.

[0084] Exemplarily, after trimming the attention heads, an intermediate text model is obtained, and the output accuracy of the intermediate text model is judged using a training data sample set.

[0085] In one example, a training data sample set is input into an intermediate text model to obtain an output result, and the similarity between the output result and the corresponding correct category label is calculated, wherein the higher the similarity, the higher the accuracy can be considered, and the lower the similarity, the lower the accuracy can be considered.

[0086] When the similarity is less than the preset similarity, it can be considered that the model output accuracy is lower than the tolerance range, and no further cropping is required, and the intermediate text model at this time is determined to be the adjusted text model. The preset similarity can be set according to the actual application scenario, for example, it can be set to 90%.

[0087] The disclosed embodiment can judge the output accuracy of the model after trimming the attention heads, and trim the attention heads within an acceptable accuracy range, thereby improving the security and reliability of the model while taking into account the model output accuracy, avoiding the problem of poor model output accuracy caused by trimming the attention heads, and is suitable for more application scenarios.

[0088] In one embodiment, after obtaining the output result, the method further includes:

[0089] When the similarity between the output result and the correct category label is greater than or equal to the preset similarity, the attention variance of the attention head corresponding to the intermediate text model is obtained, and the attention heads whose attention variance is less than the preset value are trimmed until the similarity is less than the preset similarity, thereby obtaining an adjusted text model.

[0090] Exemplarily, when the similarity between the output result and the correct category label is greater than or equal to the preset similarity, it can be considered that the model output accuracy is still high at this time. Within the tolerance range, attention head pruning can continue to be performed to further reduce the success rate of backdoor attacks.

[0091] In one example, the attention variance of the attention head corresponding to the intermediate text model is obtained, and the attention head whose attention variance is less than the preset value is trimmed. In some possible implementations, different trimming stages correspond to different preset values, which can be specifically based on

[0092] The attention variance of the attention head in the front pruning stage is determined.

[0093] Optionally, after cropping is completed, the similarity between the output result of the new model and the correct category label is obtained again, and a loop judgment is performed until the similarity of the model obtained after cropping is less than a preset similarity, and cropping is no longer continued to obtain an adjusted text model.

[0094] The disclosed embodiment can repeatedly prune the attention heads multiple times within an acceptable accuracy range, thereby further improving the security and reliability of the adjusted model while ensuring the accuracy of the model output, and significantly reducing the success rate of backdoor attacks.

[0095] In one embodiment, the step of pruning the attention heads whose attention variance is less than a preset value includes:

[0096] According to the attention variance, determining an attention head with the smallest attention variance from the attention heads as a target attention head;

[0097] Prune the target attention head.

[0098] Exemplarily, according to the attention variance, the attention head with the smallest attention variance is determined from the attention heads as the target attention head. In one example, each time the attention head with the smallest attention variance is cropped, that is, according to the attention variance of multiple attention heads, the attention head with the smallest attention variance is determined as the target attention head to be cropped and the cropping is performed.

[0099] The disclosed embodiment determines the attention head with the smallest attention variance as the target attention head and performs cropping, so that the attention head with the attention focus drift feature can be quickly and accurately located according to the attention variance, thereby improving the model processing efficiency and reliability.

[0100] In one embodiment, Figure 2 As shown, the attention heads whose attention variance is less than a preset value are trimmed to obtain an adjusted text model, including:

[0101] Step S131, trimming the attention heads whose attention variance is less than a preset value to obtain an intermediate text model;

[0102] Step S132, inputting the training data sample set into the intermediate text model to obtain an output result;

[0103] Step S133, according to the output result, adjust the weight of the attention head corresponding to the intermediate text model until the target loss function value of the model is less than the preset loss value, and obtain the adjusted text model, wherein the target loss function value is determined based on the preset regularization function value.

[0104] Exemplarily, after the intermediate text model is obtained by clipping, the weights of the remaining attention heads are adjusted. In one example, the intermediate text model may be a text model obtained after passing through multiple clipping stages within a tolerance range.

[0105] Optionally, a training data sample set is input into an intermediate text model to obtain an output result, and the weight of the attention head is adjusted according to the output result. In one example, the weight adjustment can be performed according to the target loss function of the model. The target loss function value is determined based on a preset regularization function value. The preset loss value can be determined according to an actual application scenario. When the target loss function value is less than the preset loss value, it can be considered that the model output result meets the requirements.

[0106] In the disclosed embodiment, after pruning the attention heads, the weights of the remaining attention heads are adjusted by weight regularization, so that the model can learn a more uniform feature distribution, avoid some attention heads only capturing accidental features, ensure that too much attention is not given when facing inputs containing backdoor triggers, effectively reduce the impact of backdoor attacks, and improve the accuracy and reliability of the model.

[0107] In one embodiment, Figure 3 As shown, the method for determining the target loss function value includes:

[0108] Step S310, determining a cross entropy loss function value and a preset regularization function value corresponding to the model according to the output result, wherein the preset regularization function value includes a bi-norm function value;

[0109] Step S312: determining a target loss function value according to the cross entropy loss function value and the preset regularization function value.

[0110] Exemplarily, the objective function can be determined by a cross entropy loss function and a preset regularization function. That is, the cross entropy loss function value and the preset regularization function value corresponding to the model are calculated according to the output result. Among them, the preset regularization function value can be an L2 norm function value, that is, a two-norm function value.

[0111] The target loss function value is determined according to the cross entropy loss function value and the preset regularization function value. In one example, a hyperparameter corresponding to the regularization strength can be set according to the actual application scenario, and the target loss function value is determined by combining the hyperparameter, the cross entropy loss function value, and the preset regularization function value.

[0112] The disclosed embodiment utilizes the bi-norm function value and the cross entropy loss function value to determine the target loss function value, combines the conventional loss function with the regularization term, and quickly and efficiently implements the weight regularization of the model attention head, thereby ensuring the accuracy and reliability of the model and reducing the impact of backdoor attacks.

[0113] Figure 4 is a flow chart of a model processing method according to an exemplary embodiment, referring to Figure 4 As shown, the initial text model and the training data sample set are obtained. The training data sample set is a trigger-free data sample labeled with the correct category label. The initial text model is preliminarily trained using the training data sample set, the attention variance of the attention head is calculated, and the attention head with the smallest variance is pruned. The model accuracy is calculated (such as the similarity between the output result and the correct category label). When the accuracy is greater than or equal to the preset accuracy, the attention head pruning is continued until the accuracy is less than the preset accuracy, and the pruned model is obtained. ,Furthermore, a constrained loss function is introduced to regularize the weights of the remaining attention heads, ensuring that these heads will not pay too much attention when facing inputs containing backdoor triggers, thereby reducing the impact of backdoor attacks.,As shown in formula (5), the regularized loss function, i.e., the target loss function, is defined.

[0114] (5)

[0115] in, represents the cross entropy loss function, is the L2 norm of the attention head, represents the fine-tuning process, is a hyperparameter representing the regularization strength.

[0116] The disclosed embodiments, combined with head cropping and attention normalization technology, can strengthen the security of the model from the inside without relying on external information, significantly reduce the success rate of backdoor attacks, while maintaining the model's efficient processing capabilities for normal inputs.

[0117] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0118] Based on the same inventive concept, the embodiment of the present application also provides a model processing device for implementing the model processing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more model processing device embodiments provided below can refer to the limitations on the model processing method above, and will not be repeated here.

[0119] In an exemplary embodiment, Figure 5 As shown, a model processing device 500 is provided, comprising:

[0120] An acquisition module 510 is used to acquire an initial text model, wherein the initial text model includes a model trained based on a problem data sample set, wherein the problem data sample set includes data samples annotated with incorrect category labels;

[0121] An input module 520, configured to input a training data sample set into the initial text model, and calculate the attention variance of the attention head corresponding to the initial text model, wherein the training data sample set includes data samples marked with correct category labels;

[0122] The adjustment module 530 is used to trim the attention heads whose attention variance is less than a preset value to obtain an adjusted text model, and the adjusted text model is used for text classification.

[0123] In one embodiment, the adjustment module is further used for:

[0124] Crop the attention heads whose attention variance is less than a preset value to obtain an intermediate text model;

[0125] Inputting the training data sample set into the intermediate text model to obtain an output result;

[0126] When the similarity between the output result and the correct category label is less than a preset similarity, the intermediate text model is determined to be an adjusted text model.

[0127] In one embodiment, the device is further used for:

[0128] When the similarity between the output result and the correct category label is greater than or equal to the preset similarity, the attention variance of the attention head corresponding to the intermediate text model is obtained, and the attention heads whose attention variance is less than the preset value are trimmed until the similarity is less than the preset similarity, thereby obtaining an adjusted text model.

[0129] In one embodiment, the adjustment module is further used for:

[0130] According to the attention variance, determining an attention head with the smallest attention variance from the attention heads as a target attention head;

[0131] Prune the target attention head.

[0132] In one embodiment, the adjustment module is further used for:

[0133] Crop the attention heads whose attention variance is less than a preset value to obtain an intermediate text model;

[0134] Inputting the training data sample set into the intermediate text model to obtain an output result;

[0135] According to the output result, the weight of the attention head corresponding to the intermediate text model is adjusted until the target loss function value of the model is less than the preset loss value, so as to obtain an adjusted text model, wherein the target loss function value is determined based on the preset regularization function value.

[0136] In one embodiment, the device further comprises a module for determining the target loss function value, wherein the module for determining the target loss function value is used to:

[0137] Determine a cross entropy loss function value and a preset regularization function value corresponding to the model according to the output result, wherein the preset regularization function value includes a bi-norm function value;

[0138] A target loss function value is determined according to the cross entropy loss function value and the preset regularization function value.

[0139] Each module in the above-mentioned model processing device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0140] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 6 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data involved in the method described in this embodiment, such as a training data sample set. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a model processing method is implemented.

[0141] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0142] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0143] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0144] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0145] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0146] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0147] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0148] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A model processing method, characterized in that: The method comprises: Acquire an initial text model, wherein the initial text model includes a model trained based on a problem data sample set, wherein the problem data sample set includes data samples annotated with incorrect category labels; Inputting a training data sample set into the initial text model, and calculating the attention variance of the attention head corresponding to the initial text model, wherein the training data sample set includes data samples marked with correct category labels; The attention heads whose attention variance is less than a preset value are trimmed to obtain an adjusted text model, and the adjusted text model is used for text classification.

2. The method according to claim 1, characterized in that The trimming of the attention heads whose attention variance is less than a preset value to obtain an adjusted text model comprises: Crop the attention heads whose attention variance is less than a preset value to obtain an intermediate text model; Inputting the training data sample set into the intermediate text model to obtain an output result; When the similarity between the output result and the correct category label is less than a preset similarity, the intermediate text model is determined to be an adjusted text model.

3. The method according to claim 2, characterized in that After obtaining the output result, the method further includes: When the similarity between the output result and the correct category label is greater than or equal to the preset similarity, the attention variance of the attention head corresponding to the intermediate text model is obtained, and the attention heads whose attention variance is less than the preset value are trimmed until the similarity is less than the preset similarity, thereby obtaining an adjusted text model.

4. The method according to claim 1, characterized in that: The step of trimming the attention heads whose attention variance is less than a preset value includes: According to the attention variance, determining an attention head with the smallest attention variance from the attention heads as a target attention head; Prune the target attention head.

5. The method according to claim 1, characterized in that The trimming of the attention heads whose attention variance is less than a preset value to obtain an adjusted text model comprises: Crop the attention heads whose attention variance is less than a preset value to obtain an intermediate text model; Inputting the training data sample set into the intermediate text model to obtain an output result; According to the output result, the weight of the attention head corresponding to the intermediate text model is adjusted until the target loss function value of the model is less than the preset loss value, so as to obtain an adjusted text model, wherein the target loss function value is determined based on the preset regularization function value.

6. The method according to claim 5, characterized in that The method for determining the target loss function value includes: Determine a cross entropy loss function value and a preset regularization function value corresponding to the model according to the output result, wherein the preset regularization function value includes a bi-norm function value; A target loss function value is determined according to the cross entropy loss function value and the preset regularization function value.

7. A model processing device, characterized in that: The device comprises: An acquisition module, used to acquire an initial text model, wherein the initial text model includes a model trained based on a problem data sample set, wherein the problem data sample set includes data samples annotated with incorrect category labels; An input module, used to input a training data sample set into the initial text model, and calculate the attention variance of the attention head corresponding to the initial text model, wherein the training data sample set includes data samples marked with correct category labels; The adjustment module is used to trim the attention heads whose attention variance is less than a preset value to obtain an adjusted text model, and the adjusted text model is used for text classification.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images