Data processing method, model compression method and device
By dividing the parameter set of the intelligent model into a shared basic parameter part and a private parameter part for task type training, the problem of high training costs of intelligent model is solved and efficient training of similar tasks is achieved.
Patent Information
- Application Number
- CN202510220418.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-02-26
- Publication Date
- 2025-05-30
AI Technical Summary
The training cost of existing intelligent models is relatively high, especially when dealing with similar tasks, and the existing technology is difficult to effectively reduce the amount of parameters and costs of model training.
By dividing the parameter set of the target model into the basic parameter part and the private parameter part, the basic parameter part is the same, the private parameter part is trained for different task types, and the partial parameters are the same, so as to reduce the amount of parameters for model training.
When dealing with similar tasks, this method significantly reduces the cost and parameter amount of model training and improves efficiency by sharing the basic parameter part and the private parameter part trained for task type.
Smart Images
Figure CN120068971A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the technical field, and in particular, to a data processing method, a model compression method, and an apparatus. Background Art
[0002] Currently, to achieve a specific task, an intelligent model needs to be pre-trained. However, the training cost of the intelligent model is relatively high. Summary of the Invention
[0003] In view of this, this application provides a data processing method, a model compression method, and an apparatus, as follows:
[0004] A data processing method includes:
[0005] Obtaining input data for a target task; the input data is at least one of image data, text data, voice data, and video data;
[0006] In response to the target task being a first type of task, processing the target task based on a first parameter set representing a target model to obtain a first output result;
[0007] In response to the target task being a second type of task, processing the target task based on a second parameter set representing a target model to obtain a second output result;
[0008] Wherein, the first type of task and the second type of task satisfy a similarity condition, the first parameter set is represented by a first basic parameter part and a first private parameter part, the second parameter set is represented by a second basic parameter part and a second private parameter part, the first basic parameter part is the same as the second basic parameter part, the first private parameter part is obtained by training a basic model represented by the first basic parameter part for the first type of task, the second private parameter part is obtained by training a basic model represented by the second basic parameter part for the second type of task, and the first private parameter part and the second private parameter data part include the same partial parameters.
[0009] Preferably, in the above method, the first private parameter part includes: a first private fixed parameter matrix and a first private personality parameter matrix;
[0010] The second private parameter part includes: a second private fixed parameter matrix and a second private personality parameter matrix;
[0011] Wherein, the first private fixed parameter matrix is the same as the second private fixed parameter matrix, and the first private personality parameter matrix is different from the second private personality parameter matrix.
[0012] Preferably, in the above method, the method further includes:
[0013] In response to the target task being a first type of task, respectively find the fixed parameter matrix and the personalized parameter matrix corresponding to the first type of task;
[0014] Obtain the first private parameter part according to the fixed parameter matrix and the personalized parameter matrix corresponding to the first type of task.
[0015] In the above method, preferably, the method further includes:
[0016] In response to the target task being a first type of task, find the first private parameter part corresponding to the first type of task;
[0017] Among them, the first private parameter part is obtained from the obtained fixed parameter matrix and personalized parameter matrix after training the basic model representing the first basic parameter part for the first type of task.
[0018] In the above method, preferably, processing the target task based on the first parameter set representing the target model to obtain a first output result includes:
[0019] Update the model parameters of the basic model representing the first basic parameter part according to the first private fixed parameter matrix and the first private personalized parameter matrix to obtain the target model;
[0020] Process the input data based on the target model to obtain a first output result for the target task.
[0021] In the above method, preferably, the second private fixed parameter matrix and the second private personalized parameter matrix are optimized in the following manner:
[0022] Input the target input sample into the basic model represented by the second basic parameter part and the first private fixed parameter matrix to obtain a processed sample result;
[0023] Obtain a target loss value according to the processed sample result and the target output sample corresponding to the target input sample;
[0024] Adjust other model parameters in the basic model according to the target loss value to obtain the second private personalized parameter matrix; the second private fixed parameter matrix is the first private fixed parameter matrix.
[0025] In the above method, preferably, the first private fixed parameter matrix is the same as the second private fixed parameter matrix, so that the first type of task and the second type of task satisfy the similarity condition;
[0026] The first private personality parameter matrix is different from the second private personality parameter matrix, such that the first type of task is different from the second type of task.
[0027] A model compression method, comprising:
[0028] Using a first input sample, training a base model with a first base parameter partial representation for a first type of task to obtain a first private parameter part; the first base parameter part and the first private parameter part represent a first parameter set, and the target model represented by the first parameter set can handle the first type of task;
[0029] Using a second input sample, training a base model with a partial parameter representation of a second base parameter part and the first private parameter part for a second type of task to obtain a second private parameter part; the second private parameter part and the first private parameter part include the same partial parameters; the second base parameter part and the second private parameter part represent a second parameter set, and the target model represented by the second parameter set can handle the second type of task;
[0030] Wherein, the input sample is at least one of image data, text data, voice data, and video data; the first type of task and the second type of task satisfy a similarity condition; the first base parameter part is the same as the second base parameter part.
[0031] A data processing device, comprising:
[0032] A data acquisition unit, configured to acquire input data for a target task; the input data is at least one of image data, text data, voice data, and video data;
[0033] A result acquisition unit, configured to, in response to the target task being a first type of task, process the target task based on a first parameter set representing a target model to obtain a first input result; in response to the target task being a second type of task, process the target task based on a second parameter set representing a target model to obtain a second output result;
[0034] Among them, the first type of task and the second type of task satisfy the similarity condition. The first parameter set is characterized by a first basic parameter part and a first private parameter part. The second parameter set is characterized by a second basic parameter part and a second private parameter part. The first basic parameter part is the same as the second basic parameter part. The first private parameter part is obtained by training the basic model characterized by the first basic parameter part for the first type of task. The second private parameter part is obtained by training the basic model characterized by the second basic parameter part for the second type of task. The first private parameter part and the second private parameter part include the same partial parameters.
[0035] A model compression device, comprising:
[0036] A first training unit, configured to use a first input sample to train a basic model characterized by a first basic parameter part for a first type of task, so as to obtain a first private parameter part; the first basic parameter part and the first private parameter part characterize a first parameter set, and the target model characterized by the first parameter set can process the first type of task;
[0037] A second training unit, configured to use a second input sample to train a basic model characterized by a part of the parameters in the second basic parameter part and the first private parameter part for a second type of task, so as to obtain a second private parameter part; the second private parameter part and the first private parameter data part include the same said partial parameters; the second basic parameter part and the second private parameter part characterize a second parameter set, and the target model characterized by the second parameter set can process the second type of task;
[0038] Among them, the input sample is at least one of image data, text data, voice data, and video data; the first type of task and the second type of task satisfy the similarity condition; the first basic parameter part is the same as the second basic parameter part.
[0039] A computer device / system, comprising: a memory, a processor, and a computer program stored on the memory, and the processor executes the computer program to implement the data processing method or model compression method described in any one of the above.
[0040] A computer-readable storage medium, on which a computer program / instructions is stored, and when the computer program / instructions is executed by a processor, it implements the data processing method or model compression method described in any one of the above.
[0041] A computer program product, comprising a computer program / instructions, and when the computer program / instructions is executed by a processor, it implements the data processing method or model compression method described in any one of the above.
[0042] As can be seen from the above technical solutions, in a data processing method, a model compression method, and a device disclosed in this application, for different task types that meet the similarity conditions, the parameter sets representing the corresponding target models are each composed of a basic parameter part and a private parameter part. The basic parameter parts are the same, and the private parameter parts corresponding to different task types are obtained by training a basic model represented by the basic parameter part for the corresponding task types, and some of the parameters in the private parameter parts are the same. Based on this, after obtaining the input data for the target task in this application, in response to the task type of the target task, the input data of the target task is processed based on the parameter set representing the corresponding target model to obtain the corresponding input result. It can be seen that in this application, for each target model for processing similar tasks, by training the corresponding private parameter part on the basis of the basic model represented by the basic parameter part, the training cost of each target model can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] To more clearly illustrate the technical solutions of the embodiments of this application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following described drawings are only some embodiments of this application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.
[0044] Figure 1 It is a flowchart of a data processing method provided by an embodiment of this application;
[0045] Figure 2 It is an example diagram of a parameter set in an embodiment of this application;
[0046] Figure 3 It is a partial flowchart of a data processing method provided by an embodiment of this application;
[0047] Figure 4 It is an example diagram of model parameters for a task suitable for object recognition in this application;
[0048] Figure 5 It is an example diagram of model parameters for a task suitable for text generation in this application;
[0049] Figure 6 It is a flowchart of a model compression method provided by an embodiment of this application;
[0050] Figure 7 It is a schematic structural diagram of a data processing device provided by an embodiment of this application;
[0051] Figure 8 It is another schematic structural diagram of a data processing device provided by an embodiment of this application;
[0052] Figure 9 Schematic structural diagram of a model compression device provided by an embodiment of the present application;
[0053] Figure 10 Schematic structural diagram of an electronic device provided by an embodiment of the present application;
[0054] Figure 11 Schematic structural diagram of another electronic device provided by an embodiment of the present application;
[0055] Figure 12 Another example diagram of the parameter set in the embodiment of the present application;
[0056] Figure 13 Example diagram of the parameter set applicable to LLM in the present application;
[0057] Figure 14 Example diagram applicable to LLM for model training in the present application;
[0058] Figure 15 Another example diagram applicable to LLM for model training in the present application. Detailed implementation manners
[0059] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.
[0060] Refer to Figure 1 , which is a flowchart of the implementation of a data processing method provided by an embodiment of the present application. This method can be applied to electronic devices capable of running intelligent models, such as mobile phones, tablet devices, notebooks, computers, or servers, etc. The technical solutions in this embodiment are mainly used to reduce the training cost of the target model for processing similar tasks.
[0061] Specifically, the method in this embodiment may include the following steps:
[0062] Step 101: Obtain input data for the target task.
[0063] Among them, the input data may include at least one of image data, text data, voice data, and video data. The target task is used to implement corresponding functions, such as object recognition, text generation, image drawing, and other functions.
[0064] Taking the target task of object recognition as an example, the input data can be image data containing the flower to be recognized and text data such as a prompt word indicating the type of flower to be recognized;
[0065] Taking the target task of text generation as an example, the input data can include text data such as a prompt word for the text to be generated.
[0066] Step 102: Determine the task type of the target task. In response to the target task being a first type of task, execute Step 103; in response to the target task being a second type of task, execute Step 104.
[0067] Among them, the first type of task and the second type of task satisfy the similarity condition. Taking the target task of object recognition as an example, the first type of task can be the task of recognizing cherry blossoms, and the second type of task can be the task of recognizing peach blossoms; taking the target task of text generation as an example, the first type of task can be the task of generating text in simplified Chinese, and the second type of task can be the task of generating text in traditional Chinese.
[0068] Specifically, in Step 102, the input data can be parsed to determine the task type of the target task represented by the input data.
[0069] Step 103: Process the target task based on the first parameter set representing the target model to obtain a first output result.
[0070] Specifically, in Step 103, the target model is run. The target model calls the model parameters corresponding to the first parameter set, enabling the target model to process the input data for the target task and output a first output result.
[0071] Among them, the first parameter set is represented by a first basic parameter part and a first private parameter part. The first private parameter part is obtained by training the basic model represented by the first basic parameter part for the first type of task.
[0072] Step 104: Process the target task based on the second parameter set representing the target model to obtain a second output result.
[0073] Specifically, in Step 104, the target model is run. The target model calls the model parameters corresponding to the second parameter set, enabling the target model to process the input data for the target task and output a second output result.
[0074] Among them, the second parameter set is represented by a second basic parameter part and a second private parameter part. The second private parameter part is obtained by training the basic model represented by the second basic parameter part for the second type of task.
[0075] It should be noted that the second basic parameter part is the same as the first basic parameter part, and the first private parameter part and the second private parameter part include the same partial parameters. It can be seen that for target models implementing similar types of tasks, instead of optimizing and training all the parameters of the basic model, when training the target model based on the basic model, the basic parameter part remains unchanged, and it is the private parameter part that is optimized. Moreover, when optimizing the private parameter part, it is the partial parameters in the private parameter part that are optimized. As a result, in the parameter sets of two target models implementing similar types of tasks, the second basic parameter part is the same as the first basic parameter part, and there are some partial parameters that are the same between the first private parameter part and the second private parameter part. Thus, on the premise of being able to obtain a target model for implementing a similar type of task, the number of parameters for model training can be reduced, thereby reducing the model training cost.
[0076] Taking the target task of object recognition as an example, after determining that the target task is the task of recognizing cherry blossoms (i.e., the first type of task), first search for the first parameter set corresponding to the task of recognizing cherry blossoms, and then run the target model. The target model calls the model parameters corresponding to the first parameter set, so that the target model performs image recognition on the image data in the input data, so that the target model outputs the recognition result of cherry blossoms.
[0077] Taking the target task of object recognition as an example, after determining that the target task is the task of recognizing peach blossoms (i.e., the second type of task), first search for the second parameter set corresponding to the task of recognizing peach blossoms, and then run the target model. The target model calls the model parameters corresponding to the second parameter set, so that the target model performs image recognition on the image data in the input data, so that the target model outputs the recognition result of peach blossoms.
[0078] Taking the target task of text generation as an example, after determining that the target task is the task of generating simplified Chinese (i.e., the first type of task), first search for the first parameter set corresponding to the task of generating simplified Chinese text, and then run the target model. The target model calls the model parameters corresponding to the first parameter set, so that the target model generates text according to the prompt words in the input data, so that the target model outputs the generated simplified Chinese text.
[0079] Taking the target task of text generation as an example, after determining that the target task is the task of generating traditional Chinese (i.e., the second type of task), first search for the second parameter set corresponding to the task of generating traditional Chinese text, and then run the target model. The target model calls the model parameters corresponding to the second parameter set, so that the target model generates text according to the prompt words in the input data, so that the target model outputs the generated traditional Chinese text.
[0080] As can be seen from the above technical solution, in a data processing method provided by an embodiment of the present application, for different task types that meet the similarity condition, the parameter sets representing the corresponding target models are respectively composed of a basic parameter part and a private parameter part. The basic parameter parts are the same, and the private parameter parts corresponding to different task types are obtained by training the basic model represented by the basic parameter part for the corresponding task types, and there are some identical parameters in the private parameter parts. Based on this, after obtaining the input data for the target task in this embodiment, in response to the task type of the target task, the input data of the target task is processed based on the parameter set representing the corresponding target model to obtain the corresponding input result. It can be seen that for each target model for processing similar tasks in the present application, training the corresponding private parameter part on the basis of the basic model represented by the basic parameter part can reduce the training cost of each target model.
[0081] In one implementation, the first private parameter part may include: a first private fixed parameter matrix and a first private personalized parameter matrix. And the second private parameter part may include: a second private fixed parameter matrix and a second private personalized parameter matrix.
[0082] Among them, the first private fixed parameter matrix is the same as the second private fixed parameter matrix, so that the first type of task and the second type of task meet the similarity condition, and the first private personalized parameter matrix is different from the second private personalized parameter matrix, so that the first type of task and the second type of task are different.
[0083] Specifically, the first basic parameter part and the second basic parameter component can be represented by W, the first private fixed parameter matrix can be represented by A1, the first private personalized parameter matrix can be represented by B1, the second private fixed parameter matrix is the same as the first private fixed parameter matrix and can be represented by A1, and the second private personalized parameter matrix can be represented by B2. As Figure 2 shown, there is a common basic parameter part W between the first parameter set and the second parameter set, and there is the same fixed parameter matrix A1 in the private parameter parts of the first parameter set and the second parameter set, and there are also different personalized parameter matrices B1 and B2 respectively.
[0084] Among them, the first private fixed parameter matrix and the first private personalized parameter matrix can be obtained by parameter decomposition of the first private parameter part. The second private fixed parameter matrix and the second private personalized parameter matrix can be obtained by parameter decomposition of the second private parameter part.
[0085] In a possible implementation, both the first private fixed parameter matrix and the first private personalized parameter matrix are low-rank matrices. Multiplying the two matrices can obtain the first private parameter part, and the first private parameter part and the first basic parameter part form the first parameter set of the target model. The second private fixed parameter matrix and the second private personalized parameter matrix are both low-rank matrices. Multiplying the two matrices can obtain the second private parameter part, and the second private parameter part and the second basic parameter part form the second parameter set of the target model.
[0086] It can be seen that in this embodiment, the similarity of the tasks implemented by the target model is characterized by the fixed parameter matrix, and the differences of the tasks implemented by the target model are characterized by the personalized parameter matrix.
[0087] In one implementation, the second private fixed parameter matrix and the second private personalized parameter matrix are optimized in the following manner, as Figure 3 shown below:
[0088] Step 301: Input the target input sample into the basic model characterized by the second basic parameter part and the first private fixed parameter matrix to obtain the processed sample result.
[0089] Among them, the target input sample includes at least one of image data, text data, voice data, and video data.
[0090] For example, use the basic model characterized by W and A1 to process the sample image to obtain the flower recognition result recognized by the basic model.
[0091] Step 302: Obtain the target loss value according to the processed sample result and the target output sample corresponding to the target input sample.
[0092] For example, compare the flower recognition result with the flower annotation information in the sample image to obtain the target loss value. The target loss value characterizes the gap between the flower recognition result and the flower annotation information.
[0093] Step 303: Adjust other model parameters in the basic model according to the target loss value to obtain the second private personalized parameter matrix.
[0094] Among them, the first private fixed parameter matrix is the first private fixed parameter matrix.
[0095] For example, use the target loss value to optimize B in the basic model to obtain B2, that is, the second private personalized parameter matrix.
[0096] It can be seen that in this embodiment, the amount of parameter training for the model is compressed. Based on the basic model represented by the basic parameter part, only the parameters in the personalized parameter matrix are optimized. By reducing the number of optimized parameters in this way, the training cost of training an intelligent model for similar types of tasks is reduced.
[0097] In specific implementation, the first parameter set and the second parameter set can be trained in the following way in this embodiment:
[0098] First, use the general input samples to perform initial training on the initially constructed intelligent model to obtain a basic model represented by the basic parameter part W and the private parameter parts A and B. The private parameter parts A and B are the parameter matrices obtained by decomposing other model parameters except W in the basic model. A is a fixed parameter matrix, and B is a personalized parameter matrix; the basic parameter part W is respectively denoted as the first basic parameter part and the second basic parameter part.
[0099] Then, use the first input samples, such as at least one of image data, text data, voice data, and video data, to train the basic model represented by the first basic parameter part for the first type of task to obtain the first private parameter part, denoted as the first private fixed parameter matrix A1 and the first private personalized parameter matrix B1. The first basic parameter part W and the first private parameter parts A1 and B1 represent the first parameter set, and the target model represented by the first parameter set can handle the first type of task;
[0100] After that, use the second input samples, such as at least one of image data, text data, voice data, and video data, to train the basic model of some parameters in the first basic parameter part and the first private parameter part, such as the basic model represented by the first private fixed parameter matrix, to obtain the second private parameter part. Specifically: optimize B1 in the target model represented by W, A1, and B1 to obtain the second private fixed parameter matrix A1 and the second private personalized parameter matrix B2. The same partial parameter A1 is included between the second private parameter part and the first private parameter part. The second basic parameter part W and the second private parameter parts A1 and B2 represent the second parameter set, and the target model represented by the second parameter set can handle the second type of task.
[0101] Thus, through the same A1, the first type of task and the second type of task can meet the similarity condition, and through different B1 and B2, the first type of task and the second type of task can be different.
[0102] For example, taking the target task of object recognition as an example, use general input samples such as a sample image with various flowers to train the initially constructed intelligent model to obtain a basic model, which has a basic parameter part W and the decomposed low-rank matrices A and B;
[0103] Then, using a sample image with a cherry blossom flower, optimize A and B in the base model characterized by W, A, and B to obtain a first set of parameters, namely W, A1, and B1;
[0104] After that, fix W and A1 in the base model, and use a sample image with a peach blossom flower to optimize B (or B1) in the base model characterized by W, A1, and B (or B1) to obtain a second set of parameters, namely W, A1, and B2, as Figure 4 shown;
[0105] Thus, model 1 characterized by W, A1, and B1 can identify cherry blossoms in the input image, while model 2 characterized by W, A1, and B2 can identify peach blossoms in the input image. Based on the same W and A1, model 1 and model 2 can identify flowers in the input image that have characteristics common to both cherry blossoms and peach blossoms. Further, model 1 can identify cherry blossoms in the input image based on B1, and model 2 can identify peach blossoms in the input image based on B2.
[0106] For example, taking the target task of text generation as an example, use a general input sample such as a sample text corresponding to a prompt to train an initially constructed intelligent model to obtain a base model. The base model has a base parameter part W and decomposed low-rank matrices A and B;
[0107] Then, use a sample text in simplified Chinese corresponding to a prompt to optimize A and B in the base model characterized by W, A, and B to obtain a first set of parameters, namely W, A1, and B1;
[0108] After that, fix W and A1 in the base model, and use a sample text in traditional Chinese corresponding to a prompt to optimize B (or B1) in the base model characterized by W, A1, and B (or B1) to obtain a second set of parameters, namely W, A1, and B2, as Figure 5 shown;
[0109] Thus, model 1 characterized by W, A1, and B1 can generate text in simplified Chinese for the prompt, while model 2 characterized by W, A1, and B2 can generate text in traditional Chinese for the prompt. Based on the same W and A1, model 1 and model 2 can generate Chinese text for the prompt. Further, model 1 can generate text in simplified Chinese based on B1, and model 2 can generate text in traditional Chinese based on B2.
[0110] In one implementation, after the fixed parameter matrix and personalized parameter matrix for implementing various types of tasks are obtained through training, they can be stored in a target storage area, such as memory or a hard disk.
[0111] Based on this, in this embodiment, in response to the target task being a first-type task, the fixed parameter matrix and the personalized parameter matrix corresponding to the first-type task, that is, the first private fixed parameter matrix and the first private personalized parameter matrix, can be respectively searched in the target storage area, and then the first private parameter part can be obtained according to the fixed parameter matrix and the personalized parameter matrix corresponding to the first-type task.
[0112] For example, the first private fixed parameter matrix and the first private personalized parameter matrix are multiplied to obtain the first private parameter part.
[0113] Alternatively, in this embodiment, in response to the target task being a second-type task, the fixed parameter matrix and the personalized parameter matrix corresponding to the second-type task, that is, the second private fixed parameter matrix and the second private personalized parameter matrix, can be respectively searched in the target storage area, and then the second private parameter part can be obtained according to the fixed parameter matrix and the personalized parameter matrix corresponding to the second-type task.
[0114] For example, the second private fixed parameter matrix and the second private personalized parameter matrix are multiplied to obtain the second private parameter part.
[0115] It can be seen that in this embodiment, when the private parameter part needs to be used, the fixed parameter matrix and the personalized parameter matrix can be read in the target storage area, and the private parameter part can be obtained through matrix calculation, so as to facilitate the corresponding target model to process the target task.
[0116] In another implementation manner, after the fixed parameter matrix and the personalized parameter matrix for implementing various types of tasks are obtained through training, the corresponding private parameter part can be first obtained from the fixed parameter matrix and the personalized parameter matrix obtained by training the basic model, and then the private parameter part can be stored in the target storage area, such as memory or hard disk.
[0117] For example, after training the basic model representing the first basic parameter part for the first-type task, the first private parameter part is obtained from the obtained first private fixed parameter matrix and the first private personalized parameter matrix, and the obtained first private parameter part is stored in the target storage area.
[0118] Again, for example, after training the basic model representing the second basic parameter part for the second-type task, the second private parameter part is obtained from the obtained second private fixed parameter matrix and the second private personalized parameter matrix, and the obtained second private parameter part is stored in the target storage area.
[0119] Based on this, in this embodiment, in response to the target task being a first-type task, the first private parameter part corresponding to the first-type task can be searched in the target storage area.
[0120] Alternatively, in this embodiment, in response to the target task being a second type of task, the second private parameter part corresponding to the second type of task may be searched for in the target storage area.
[0121] It can be seen that in this embodiment, by storing the private parameter parts corresponding to the fixed parameter matrix and the personalized parameter matrix in the target storage area, when the private parameter parts are needed, they can be directly read from the target storage area without waiting for the matrix calculation processes of the two parameter matrices, which can save calculation time and thus improve the efficiency of the corresponding target model in processing the target task.
[0122] In one implementation manner, when processing the target task based on the first parameter set representing the target model in step 103 to obtain the first output result, the model parameters of the base model represented by the first basic parameter part may be updated according to the first private fixed parameter matrix and the first private personalized parameter matrix to obtain the target model, and then the input data may be processed based on the target model to obtain the first output result for the target task.
[0123] Specifically, in step 103, the first private fixed parameter matrix and the first private personalized parameter matrix are first multiplied to obtain the first private parameter part, and then the first private parameter part is updated to the base model represented by the first basic parameter part to obtain the target model with the first basic parameter part and the first private parameter part, and then the input data is processed using the target model to obtain the first output result.
[0124] For example, after multiplying A1 and B1 obtained according to the cherry blossom recognition task, the model parameters in each matrix element of the obtained A1*B1 are updated to the model with W to obtain model 1 with W, A1, and B1, and model 1 performs cherry blossom recognition on the input image to obtain the cherry blossom recognition result in the input image.
[0125] For another example, after multiplying A1 and B1 obtained according to the generation task of simplified Chinese, the model parameters in each matrix element of the obtained A1*B1 are updated to the model with W to obtain model 1 with W, A1, and B1, and model 1 generates simplified Chinese for the prompt word to obtain the simplified Chinese text matching the prompt word.
[0126] In one implementation manner, when processing the target task based on the second parameter set representing the target model in step 104 to obtain the second output result, the model parameters of the base model represented by the second basic parameter part may be updated according to the second private fixed parameter matrix and the second private personalized parameter matrix to obtain the target model, and then the input data may be processed based on the target model to obtain the second output result for the target task.
[0127] Specifically, in step 104, first multiply the second private fixed parameter matrix and the second private personalized parameter matrix to obtain the second private parameter part, and then update the second private parameter part to the base model represented by the second base parameter part to obtain a target model with the second base parameter part and the second private parameter part. Then, use the target model to process the input data to obtain a first output result.
[0128] For example, after multiplying A1 and B2 obtained according to the peach blossom recognition task, update the model parameters in each matrix element of the obtained A1*B2 to the model with W to obtain model 2 with W, A1, and B2. Model 2 performs peach blossom recognition on the input image to obtain the peach blossom recognition result in the input image.
[0129] For example, after multiplying A1 and B2 obtained according to the traditional Chinese generation task, update the model parameters in each matrix element of the obtained A1*B2 to the model with W to obtain model 2 with W, A1, and B2. Model 2 generates traditional Chinese for the prompt word to obtain a traditional Chinese text that matches the prompt word.
[0130] Reference Figure 6 , is a flowchart of the implementation of a model compression method provided by an embodiment of the present application. This method can be applied to electronic devices capable of running intelligent models, such as mobile phones, tablet devices, laptops, computers, or servers. The technical solution in this embodiment is mainly used to reduce the training cost of the target model for processing similar tasks.
[0131] Specifically, the method in this embodiment may include the following steps:
[0132] Step 601: Use the first input sample to train the base model represented by the first base parameter part for the first type of task to obtain the first private parameter part.
[0133] Among them, the input sample is at least one of image data, text data, voice data, and video data. The first base parameter part and the first private parameter part represent a first parameter set, and the target model represented by the first parameter set can process the first type of task.
[0134] It should be noted that the base model represented by the first base parameter part can be an initially constructed intelligent model or a model obtained by training the initially constructed intelligent model with a general input sample.
[0135] Step 602: Use the second input sample to train the basic model with partial parameter representations in the second basic parameter part and the first private parameter part for the second type of task, so as to obtain the second private parameter part.
[0136] Among them, the second private parameter part and the first private parameter part include the same partial parameters; the second basic parameter part and the second private parameter part represent the second parameter set, and the target model represented by the second parameter set can handle the second type of task;
[0137] It should be noted that the first type of task and the second type of task satisfy the similarity condition; the first basic parameter part is the same as the second basic parameter part.
[0138] As can be seen from the above technical solution, in a model compression method provided by an embodiment of the present application, for different task types that satisfy the similarity condition, the parameter sets representing the corresponding target models are respectively composed of a basic parameter part and a private parameter part. The basic parameter parts are the same, and the private parameter parts corresponding to different task types are obtained by training the basic model represented by the basic parameter part for the corresponding task types, and some parameters in the private parameter parts are the same. Based on this, after obtaining the input data for the target task in this embodiment, in response to the task type of the target task, the input data of the target task is processed based on the parameter set representing the corresponding target model to obtain the corresponding input result. It can be seen that in the present application, for each target model for processing similar tasks, the corresponding private parameter part is trained on the basis of the basic model represented by the basic parameter part, which can reduce the training cost of each target model.
[0139] Reference Figure 7 , which is a schematic structural diagram of a data processing device provided by an embodiment of the present application. This device can be applied to electronic devices capable of running intelligent models, such as mobile phones, tablet devices, notebooks, computers, or servers, etc. The technical solution in this embodiment is mainly used to reduce the training cost of target models for processing similar tasks.
[0140] Specifically, the device in this embodiment may include the following units:
[0141] The data acquisition unit 701 is used to acquire the input data for the target task; the input data is at least one of image data, text data, voice data, and video data;
[0142] The result acquisition unit 702 is used to, in response to the target task being the first type of task, process the target task based on the first parameter set representing the target model to obtain the first input result; in response to the target task being the second type of task, process the target task based on the second parameter set representing the target model to obtain the second output result;
[0143] Among them, the first type of task and the second type of task satisfy the similarity condition. The first parameter set is characterized by a first basic parameter part and a first private parameter part. The second parameter set is characterized by a second basic parameter part and a second private parameter part. The first basic parameter part is the same as the second basic parameter part. The first private parameter part is obtained by training the basic model represented by the first basic parameter part for the first type of task. The second private parameter part is obtained by training the basic model represented by the second basic parameter part for the second type of task. The first private parameter part and the second private parameter part include the same partial parameters.
[0144] As can be seen from the above technical solution, in a data processing device provided in an embodiment of the present application, for different task types that satisfy the similarity condition, the parameter sets representing the corresponding target models are respectively composed of a basic parameter part and a private parameter part. The basic parameter parts are the same. The private parameter parts corresponding to different task types are obtained by training the basic model represented by the basic parameter part for the corresponding task types, and there are some identical parameters in the private parameter parts. Based on this, after obtaining the input data for the target task in this embodiment, in response to the task type of the target task, the input data of the target task is processed based on the parameter set representing the corresponding target model to obtain the corresponding input result. It can be seen that in the present application, for each target model for processing similar tasks, by training the corresponding private parameter part on the basis of the basic model represented by the basic parameter part, the training cost of each target model can be reduced.
[0145] In one implementation, the first private parameter part includes: a first private fixed parameter matrix and a first private personalized parameter matrix; the second private parameter part includes: a second private fixed parameter matrix and a second private personalized parameter matrix; wherein, the first private fixed parameter matrix is the same as the second private fixed parameter matrix, and the first private personalized parameter matrix is different from the second private personalized parameter matrix.
[0146] In one implementation, the result obtaining unit 702 is further configured to: in response to the target task being the first type of task, respectively find the fixed parameter matrix and the personalized parameter matrix corresponding to the first type of task; and obtain the first private parameter part according to the fixed parameter matrix and the personalized parameter matrix corresponding to the first type of task.
[0147] In addition, the result obtaining unit 702 is further configured to: in response to the target task being the second type of task, respectively find the fixed parameter matrix and the personalized parameter matrix corresponding to the second type of task; and obtain the second private parameter part according to the fixed parameter matrix and the personalized parameter matrix corresponding to the second type of task.
[0148] In one implementation, the result obtaining unit 702 is further configured to: in response to the target task being a first type of task, search for the first private parameter part corresponding to the first type of task; wherein, the first private parameter part is obtained from the obtained fixed parameter matrix and personality parameter matrix after training the base model represented by the first basic parameter part for the first type of task.
[0149] In addition, the result obtaining unit 702 is further configured to: in response to the target task being a second type of task, search for the second private parameter part corresponding to the second type of task; wherein, the second private parameter part is obtained from the obtained fixed parameter matrix and personality parameter matrix after training the base model represented by the second basic parameter part for the second type of task.
[0150] In one implementation, when the result obtaining unit 702 processes the target task based on the first parameter set representing the target model to obtain a first output result, it is specifically configured to: update the model parameters of the base model represented by the first basic parameter part according to the first private fixed parameter matrix and the first private personality parameter matrix to obtain the target model; process the input data based on the target model to obtain a first output result for the target task.
[0151] In addition, when the result obtaining unit 702 processes the target task based on the second parameter set representing the target model to obtain a second output result, it is specifically configured to: update the model parameters of the base model represented by the second basic parameter part according to the second private fixed parameter matrix and the second private personality parameter matrix to obtain the target model; process the input data based on the target model to obtain a second output result for the target task.
[0152] In one implementation, the present embodiment may further include the following units, as Figure 8 shown in
[0153] A parameter optimization unit 703, configured to: optimize the second private fixed parameter matrix and the second private personality parameter matrix in the following manner:
[0154] Input the target input sample into the basic model represented by the second basic parameter part and the first private fixed parameter matrix to obtain a processed sample result; obtain a target loss value according to the processed sample result and the target output sample corresponding to the target input sample; adjust other model parameters in the basic model according to the target loss value to obtain the second private personalized parameter matrix; the second private fixed parameter matrix is the first private fixed parameter matrix.
[0155] In one implementation, the first private fixed parameter matrix is the same as the second private fixed parameter matrix, so that the first type of task and the second type of task meet the similarity condition; the first private personalized parameter matrix is different from the second private personalized parameter matrix, so that the first type of task and the second type of task are different.
[0156] It should be noted that the specific implementation manners of the units in this embodiment can refer to the corresponding contents in the foregoing, and will not be elaborated here.
[0157] Reference Figure 9 , which is a schematic structural diagram of a model compression device provided by an embodiment of the present application. This device can be applied to electronic devices capable of running intelligent models, such as mobile phones, tablet devices, notebooks, computers, or servers, etc. The technical solution in this embodiment is mainly used to reduce the training cost of the target model for processing similar tasks.
[0158] Specifically, the device in this embodiment may include the following units:
[0159] The first training unit 901 is used to train the basic model represented by the first basic parameter part by using the first input sample for the first type of task to obtain the first private parameter part; the first basic parameter part and the first private parameter part represent a first parameter set, and the target model represented by the first parameter set can process the first type of task;
[0160] The second training unit 902 is used to train the basic model represented by the second basic parameter part and some parameters in the first private parameter part by using the second input sample for the second type of task to obtain the second private parameter part; the second private parameter part and the first private parameter data part include the same some parameters; the second basic parameter part and the second private parameter part represent a second parameter set, and the target model represented by the second parameter set can process the second type of task;
[0161] Wherein, the input sample is at least one of image data, text data, voice data, and video data; the first type of task and the second type of task satisfy a similarity condition; the first basic parameter part is the same as the second basic parameter part.
[0162] As can be seen from the above technical solution, in a model compression device provided by an embodiment of the present application, for different task types that satisfy the similarity condition, the parameter sets representing the corresponding target models are respectively composed of a basic parameter part and a private parameter part. The basic parameter parts are the same, and the private parameter parts corresponding to different task types are obtained by training the basic models represented by the basic parameter parts for the corresponding task types, and there are some identical parameters in the private parameter parts. Based on this, after obtaining the input data for the target task in this embodiment, in response to the task type of the target task, the input data of the target task is processed based on the parameter set representing the corresponding target model to obtain the corresponding input result. It can be seen that in the present application, for each target model for processing similar tasks, by training the corresponding private parameter part on the basis of the basic model represented by the basic parameter part, the training cost of each target model can be reduced.
[0163] Reference Figure 10 , is a schematic structural diagram of an electronic device provided by an embodiment of the present application. The electronic device may include the following structures:
[0164] A memory 1001, configured to store a computer program and data generated by the running of the computer program;
[0165] A processor 1002, configured to execute the computer program to implement: obtaining input data for a target task; the input data is at least one of image data, text data, voice data, and video data; in response to the target task being a first type of task, processing the target task based on a first parameter set representing the target model to obtain a first input result; in response to the target task being a second type of task, processing the target task based on a second parameter set representing the target model to obtain a second output result;
[0166] Wherein, the first type of task and the second type of task satisfy a similarity condition, the first parameter set is represented by a first basic parameter part and a first private parameter part, the second parameter set is represented by a second basic parameter part and a second private parameter part, the first basic parameter part is the same as the second basic parameter part, the first private parameter part is obtained by training the basic model represented by the first basic parameter part for the first type of task, the second private parameter part is obtained by training the basic model represented by the second basic parameter part for the second type of task, and the first private parameter part and the second private parameter part include the same partial parameters.
[0167] As can be seen from the above technical solution, in an electronic device provided in an embodiment of the present application, for different task types that meet the similarity condition, the parameter sets representing the corresponding target models are respectively composed of a basic parameter part and a private parameter part. The basic parameter parts are the same, and the private parameter parts corresponding to different task types are obtained by training a basic model represented by the basic parameter part for the corresponding task type, and some of the parameters in the private parameter parts are the same. Based on this, in this embodiment, after obtaining the input data for the target task, in response to the task type of the target task, the input data of the target task is processed based on the parameter set representing the corresponding target model to obtain the corresponding input result. It can be seen that in the present application, for each target model for processing similar tasks, by training the corresponding private parameter part on the basis of the basic model represented by the basic parameter part, the training cost of each target model can be reduced.
[0168] Refer to Figure 11 , which is a schematic structural diagram of an electronic device provided in an embodiment of the present application. The electronic device may include the following structures:
[0169] A memory 1101, configured to store a computer program and data generated by running the computer program;
[0170] A processor 1102, configured to execute the computer program to implement: using a first input sample to train a basic model represented by a first basic parameter part for a first type of task to obtain a first private parameter part; the first basic parameter part and the first private parameter part represent a first parameter set, and the target model represented by the first parameter set can process the first type of task; using a second input sample to train a basic model represented by a second basic parameter part and a part of the parameters in the first private parameter part for a second type of task to obtain a second private parameter part; the second private parameter part and the first private parameter data part include the same part of the parameters; the second basic parameter part and the second private parameter part represent a second parameter set, and the target model represented by the second parameter set can process the second type of task;
[0171] Wherein, the input sample is at least one of image data, text data, voice data, and video data; the first type of task and the second type of task meet the similarity condition; the first basic parameter part is the same as the second basic parameter part.
[0172] As can be seen from the above technical solution, in an electronic device provided in an embodiment of the present application, for different task types that meet the similarity conditions, the parameter sets representing the corresponding target models are respectively composed of a basic parameter part and a private parameter part. The basic parameter part is the same, and the private parameter parts corresponding to different task types are obtained by training the basic model represented by the basic parameter part for the corresponding task types, and there are some identical parameters in the private parameter parts. Based on this, after obtaining the input data for the target task in this embodiment, in response to the task type of the target task, the input data of the target task is processed based on the parameter set representing the corresponding target model to obtain the corresponding input result. It can be seen that for each target model for processing similar tasks in the present application, training the corresponding private parameter part on the basis of the basic model represented by the basic parameter part can reduce the training cost of each target model.
[0173] Taking the scenario of using the model for text generation as an example, the technical solution of the present application will be illustrated as follows:
[0174] The present application proposes a new model compression solution. The technical solution of the present application aims at different types of tasks with high similarity. By introducing a shared A matrix (A1 and A2) and multiple independent B matrices (B1, B2, B3, B4), as Figure 12 shown, different tasks: Task1, Task2, Task 3, and Task 4 are processed respectively, thus avoiding interference between tasks. This solution can autonomously identify the implicit characteristics in the data, significantly improve the task adaptability and performance, without additional tools or human intervention. In addition, this solution can reduce the number of training parameters and the training duration of the multi-task model.
[0175] As Figure 12 shown in the figure, it is assumed that there are four tasks in the figure: Task1, Task2, Task3, and Task4. Among them, Task1 is similar to Task2, and Task3 is similar to Task4. The bypass matrix (i.e., the private parameter part) of Task1 is composed of A1 and B1. The bypass matrix of Task2 is composed of A1 and B2, that is, A1 is the shared matrix of Task1 and Task2, and the corresponding B matrices of the two are B1 and B2 respectively. The bypass matrix of Task3 is composed of A2 and B3, and the bypass matrix of Task4 is composed of A2 and B4, that is, A2 is the shared matrix of Task3 and Task4, and the corresponding B matrices of the two are B3 and B4 respectively.
[0176] It can be seen that the technical solution of the present application can reduce the number of fine-tuning parameters of the intelligent model, thereby achieving the purpose of reducing model storage, which is more important for terminals such as tablets. The technical solution of the present application performs outstandingly in processing multi-tasks and complex fields. While maintaining parameter efficiency, it provides more powerful task processing capabilities, improves computing and storage efficiency by reducing redundancy, and is particularly outstanding in the scenario of fine-tuning large models.
[0177] Taking the large language model LLM (Large Language Model) as an example, the training and inference methods in the technical solution of the present application are briefly described as follows:
[0178] (1) Training:
[0179] Based on the LLM, two similar Chinese models are fine-tuned, where the output of one model is simplified Chinese and the output of the other model is traditional Chinese.
[0180] The model fine-tuned using the technical solution of the present application is as shown in Figure 13 : where A is the common parameter matrix of the two Chinese models, B1 is the private personality parameter matrix of the simplified Chinese model, and B2 is the private personality parameter matrix of the traditional Chinese model.
[0181] (2) Training of the simplified Chinese model:
[0182] First, prepare a simplified Chinese corpus dataset, and the data source can be open-source data and self-built data. Then, use the simplified Chinese dataset to fine-tune and train the basic model of the LLM, as shown in Figure 14 . Specifically, first freeze the basic parameter part W of the LLM, and obtain A and B1 through training. After fine-tuning, the LLM represented by W, as well as A and B1, form a simplified Chinese model.
[0183] (3) Training of the traditional Chinese model:
[0184] First, prepare a traditional Chinese corpus dataset, and at this time the data volume can be one-third of that of the simplified Chinese. Then, use the traditional Chinese dataset to fine-tune and train the LLM based on W and A. At this time, the training parameter is only B2, and A is used as a common matrix and is frozen during the training process, as shown in Figure 15 . After fine-tuning, the LLM represented by W, as well as A and B2, form a traditional Chinese model.
[0185] Through the above steps, two Chinese target models can be fine-tuned and trained, and the two models can adapt to two tasks. During the inference process, first judge the task type:
[0186] 1. When performing related tasks in Simplified Chinese, the Simplified Chinese model is used for inference. At this time, the model selects A and B1, and the inference process can be described by formula (1):
[0187] Y1 = W × x + B1 × A × x (1)
[0188] 2. When performing related tasks in Traditional Chinese, the Traditional Chinese model is used for inference. At this time, the model selects A and B2, and the inference process can be described by formula (2):
[0189] Y2 = W × x + B2 × A × x
[0190] Among them, Y1 is the inference result of the Simplified Chinese model, x is the input data, and Y2 is the inference result of the Traditional Chinese model.
[0191] In this specification, each embodiment is described in a progressive manner. The key point of each embodiment is to illustrate the differences from other embodiments. For the same or similar parts among the embodiments, reference can be made to each other. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple. For the relevant parts, reference can be made to the description in the method section.
[0192] Those skilled in the art can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed in this article can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.
[0193] The steps of the methods or algorithms described in combination with the embodiments disclosed in this article can be directly implemented by hardware, software modules executed by a processor, or a combination of the two. The software modules can be placed in a random access memory (RAM), memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium well-known in the technical field.
[0194] The foregoing description of the disclosed embodiments enables those skilled in the art to practice or use the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A data processing method, comprising: Obtain input data for the target task; The input data is at least one of image data, text data, voice data and video data; In response to the target task being a first type of task, processing the target task based on a first parameter set characterizing a target model to obtain a first output result; In response to the target task being a second type of task, processing the target task based on a second parameter set characterizing the target model to obtain a second output result; Among them, the first type of task and the second type of task meet the similarity condition, the first parameter set is represented by a first basic parameter part and a first private parameter part, the second parameter set is represented by a second basic parameter part and a second private parameter part, the first basic parameter part is the same as the second basic parameter part, the first private parameter part is obtained by training a basic model represented by the first basic parameter part for the first type of task, the second private parameter part is obtained by training a basic model represented by the second basic parameter part for the second type of task, and the first private parameter part and the second private parameter data part include the same partial parameters.
2. The method according to claim 1, wherein the first private parameter part comprises: a first private fixed parameter matrix and a first private individual parameter matrix; The second private parameter part includes: a second private fixed parameter matrix and a second private individual parameter matrix; The first private fixed parameter matrix is the same as the second private fixed parameter matrix, and the first private individual parameter matrix is different from the second private individual parameter matrix.
3. The method according to claim 2, further comprising: In response to the target task being a first type of task, searching for a fixed parameter matrix and a personalized parameter matrix corresponding to the first type of task respectively; The first private parameter part is obtained according to the fixed parameter matrix and the personalized parameter matrix corresponding to the first type of task.
4. The method according to claim 2, further comprising: In response to the target task being a first type of task, searching for the first private parameter part corresponding to the first type of task; The first private parameter part is obtained from the fixed parameter matrix and the individual parameter matrix after training the basic model represented by the first basic parameter part for the first type of task.
5. The method according to claim 2, processing the target task based on a first parameter set representing the target model to obtain a first output result, comprising: According to the first private fixed parameter matrix and the first private individual parameter matrix, updating the model parameters of the basic model represented by the first basic parameter part to obtain the target model; The input data is processed based on the target model to obtain a first output result for the target task.
6. The method according to claim 1, wherein the second private fixed parameter matrix and the second private individual parameter matrix are optimized by: Inputting the target input sample into the second basic parameter part and the basic model represented by the first private fixed parameter matrix to obtain a processed sample result; Obtaining a target loss value according to the target output sample corresponding to the processed sample result and the target input sample; According to the target loss value, other model parameters in the basic model are adjusted to obtain the second private personalized parameter matrix; the second private fixed parameter matrix is the first private fixed parameter matrix.
7. The method according to claim 2, wherein the first private fixed parameter matrix is the same as the second private fixed parameter matrix, so that the first type of task and the second type of task meet a similarity condition; The first private personality parameter matrix is different from the second private personality parameter matrix, so that the first type of task is different from the second type of task.
8. A model compression method, comprising: Using the first input sample, training a basic model represented by the first basic parameter part for a first type of task to obtain a first private parameter part; The first basic parameter part and the first private parameter part represent a first parameter set, and the target model represented by the first parameter set can process the first type of task; Using a second input sample, training a second basic parameter part and a basic model represented by some parameters in the first private parameter part for a second type of task to obtain a second private parameter part; The second private parameter part and the first private parameter part include the same part of parameters; The second basic parameter part and the second private parameter part represent a second parameter set, and the target model represented by the second parameter set can process the second type of task; The input sample is at least one of image data, text data, voice data and video data; the first type of task and the second type of task meet similarity conditions; the first basic parameter part is the same as the second basic parameter part.
9. A data processing device, comprising: A data acquisition unit, used for acquiring input data for a target task; The input data is at least one of image data, text data, voice data and video data; A result obtaining unit, configured to, in response to the target task being a first type of task, process the target task based on a first parameter set characterizing a target model to obtain a first input result; In response to the target task being a second type of task, processing the target task based on a second parameter set characterizing the target model to obtain a second output result; The first type of task and the second type of task meet a similarity condition, the first parameter set is represented by a first basic parameter part and a first private parameter part, the second parameter set is represented by a second basic parameter part and a second private parameter part, the first basic parameter part is the same as the second basic parameter part, the first private parameter part is obtained by training a basic model represented by the first basic parameter part for the first type of task, the second private parameter part is obtained by training a basic model represented by the second basic parameter part for the second type of task, and the first private parameter part and the second private parameter part include the same partial parameters.
10. A model compression device, comprising: A first training unit is used to train a basic model represented by a first basic parameter part for a first type of task using a first input sample to obtain a first private parameter part; The first basic parameter part and the first private parameter part represent a first parameter set, and the target model represented by the first parameter set can process the first type of task; A second training unit is used to train a second basic parameter part and a basic model represented by some parameters in the first private parameter part for a second type of task using a second input sample to obtain a second private parameter part; The second private parameter part and the first private parameter data part include the same part of parameters; The second basic parameter part and the second private parameter part represent a second parameter set, and the target model represented by the second parameter set can process the second type of task; The input sample is at least one of image data, text data, voice data and video data; the first type of task and the second type of task meet a similarity condition; The first basic parameter portion is identical to the second basic parameter portion.