A model training method, device and computer equipment

Abstract

The present application relates to a model training method, device and computer equipment. The method includes: obtaining multimodal training samples related to the target task; the multimodal training samples include at least two samples of text, speech and image; the target task is one of visual processing, speech processing or language processing; based on the obtained multimodal training samples and the large model to be trained, construct an attack and defense game model related to the target task; based on the attack and defense game model, remove the target samples that affect the model training convergence speed of the large model to be trained in the multimodal training samples to obtain streamlined samples; use the streamlined samples to train the large model to be trained to obtain the trained large model. The use of this method can solve the existing large model training method, because there are samples that affect the efficiency of large model training, resulting in the problem of occupying too many computing resources during the large model training process.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a model training method, apparatus and computer equipment. Background Art

[0002] With the development of science and technology, artificial intelligence technology has gradually received attention. With the development of artificial intelligence technology, artificial intelligence technology has gradually been applied in many fields such as natural language processing, speech processing, and computer vision.

[0003] With the widespread application of artificial intelligence in various fields, the amount of data such as text, voice or image processing has increased dramatically. Therefore, it is necessary to introduce a big data processing model to process large amounts of data. For example, when performing computer vision processing, it is necessary to introduce a large visual model to achieve the target function; when performing natural language processing, it is necessary to introduce a large language model to achieve the target function; when performing speech processing, it is necessary to introduce a large language model to achieve the target function. However, in order to ensure the prediction effect of the introduced large visual, language or speech model, before using the large visual, language or speech model for computer vision or language processing, it is necessary to use a large amount of image, text or speech data to train the large visual, language or speech model.

[0004] However, in the existing large model training process, there are too many samples, the samples are uneven, and there are samples that affect the efficiency of large model training, resulting in too many computing resources being required during the large model training process.

[0005] Regarding the existing large model training methods, there are samples that affect the efficiency of large model training, resulting in the problem of excessive computing resources being occupied during the large model training process. No effective solution has been proposed yet. Summary of the invention

[0006] Based on this, it is necessary to provide a model training method, device and computer equipment to address the above technical problems.

[0007] In a first aspect, the present application provides a model training method. The method comprises:

[0008] Acquire multimodal training samples related to the target task; the multimodal training samples include at least two samples of text, speech and image, wherein one of the at least two samples is a sample type related to the target task; the target task is one of visual processing, speech processing or language processing;

[0009] Based on the acquired multimodal training samples related to the target task and the large model to be trained related to the target task, construct an attack and defense game model related to the target task;

[0010] Based on the attack and defense game model related to the target task, target samples in the multimodal training samples related to the target task are eliminated to obtain streamlined samples; the target samples are samples that affect the model training convergence speed of the large model to be trained;

[0011] The streamlined samples are used to train the large model to be trained that is related to the target task, so as to obtain a trained large model that is related to the target.

[0012] In one embodiment, constructing an attack and defense game model related to the target task based on the acquired multimodal training samples related to the target task and the large model to be trained related to the target task includes:

[0013] Based on the acquired multimodal training samples related to the target task and the training objective function of the large model to be trained related to the target task, the attack and defense game model related to the target task is constructed; the expression of the attack and defense game model is:

[0014] ;

[0015] ;

[0016] ;

[0017] ;

[0018] ;

[0019] ;

[0020] ;

[0021] Among them, g k is the gradient value of the kth step of the training of the large model to be trained related to the target task, g k-1 is the gradient value of the k-1th step of the training of the large model to be trained related to the target task, is the gradient difference between the k-th step and the k-1-th step of the training of the large model to be trained related to the target task, is the penalty factor, x is the target attack sample, is the disturbance value of the target attack sample, and x´ is the adjusted target attack sample; is the threshold factor, J is the training objective function of the large model to be trained related to the target task, and w is the model parameter of the large model to be trained related to the target task.

[0022] In one embodiment, based on the attack and defense game model related to the target task, removing the target sample from the multimodal training samples related to the target task to obtain a streamlined sample includes:

[0023] Based on the attack and defense game model related to the target task, a target attack sample is determined; the target attack sample is a sample in which the gradient difference between the previous and next steps is greater than or equal to a preset gradient difference threshold when training the large model to be trained;

[0024] Based on the target attack samples, the target samples in the multimodal training samples related to the target task are eliminated to obtain the streamlined samples.

[0025] In one embodiment, based on the target attack sample in the multimodal training sample related to the target task, removing the target sample in the multimodal training sample related to the target task to obtain the streamlined sample includes:

[0026] Based on the disturbance value of each target attack sample, each target attack sample is adjusted to obtain each adjusted target attack sample;

[0027] Respectively calculating the similarity between each adjusted target attack sample and each multimodal training sample to obtain a similarity result between each adjusted target attack sample and each multimodal training sample;

[0028] Among the multimodal training samples, the samples whose similarity results with the adjusted target attack samples are greater than or equal to a preset similarity threshold are used as the target samples;

[0029] The target samples are removed from the multimodal training samples related to the target task to obtain the streamlined samples.

[0030] In one embodiment, before adjusting each of the target attack samples based on the disturbance value of each of the target attack samples to obtain each adjusted target attack sample, the method includes:

[0031] The solution of each of the target attack samples in the attack-defense game model is calculated to obtain the disturbance value of each of the target attack samples.

[0032] In one embodiment, before constructing an attack and defense game model related to the target task based on the acquired multimodal training samples related to the target task and the large model to be trained related to the target task, the method includes:

[0033] Perform one or more of data cleaning, normalization or alignment on the acquired multimodal training samples related to the target task.

[0034] In one embodiment, before constructing an attack and defense game model related to the target task based on the acquired multimodal training samples related to the target task and the large model to be trained related to the target task, the method further includes:

[0035] Based on the target task, the large model to be trained related to the target task is determined.

[0036] In one embodiment, determining the large model to be trained related to the target task based on the target task includes:

[0037] Based on the target task, determine the model architecture, model structure and initialization model parameters of the large model to be trained related to the target task;

[0038] Based on the model architecture, the model structure and the initialized model parameters of the large model to be trained related to the target task, determine the large model to be trained related to the target task.

[0039] In a second aspect, the present application also provides a model training device. The device comprises:

[0040] A sample acquisition module, used to acquire multimodal training samples related to a target task; the multimodal training samples include at least two samples of text, speech and image, wherein one of the at least two samples is a sample type related to the target task; the target task is one of visual processing, speech processing or language processing;

[0041] A model determination module, configured to construct an attack and defense game model related to the target task based on the acquired multimodal training samples related to the target task and the large model to be trained related to the target task;

[0042] A sample reduction module, used to remove target samples from the multimodal training samples related to the target task based on the attack and defense game model related to the target task, so as to obtain reduced samples; the target samples are samples that affect the model training convergence speed of the large model to be trained;

[0043] And a model training module is used to use the streamlined samples to train the large model to be trained related to the target task to obtain a trained large model related to the target.

[0044] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the model training method described in the first aspect when executing the computer program.

[0045] The above-mentioned model training method, apparatus and computer equipment obtain multimodal training samples related to the target task, wherein the multimodal training samples include at least two samples of text, speech and image, and one of them is a sample type related to the target task. Then, based on the obtained multimodal training samples related to the target task and the large model to be trained related to the target task, an attack and defense game model related to the target task is constructed. Then, based on the attack and defense game model related to the target task, the target samples that affect the model training convergence speed of the large model to be trained in the multimodal training samples related to the target task are eliminated to obtain streamlined samples. Finally, the streamlined samples are used to train the large model to be trained related to the target task to obtain a trained large model related to the target. It uses an attack and defense game model to determine and eliminate samples in the multimodal samples that affect the training speed of the large model, so that the remaining samples can be used to train the large model to be trained related to the target task, improve training efficiency, and reduce resource usage during training. It solves the problem of excessive computing resources being occupied during large model training due to the existence of samples that affect the training efficiency of the large model in the existing large model training method.

[0046] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0048] Figure 1 A hardware structure block diagram of a terminal for a model training method provided in one embodiment of the present application;

[0049] Figure 2 A flowchart of a model training method provided in one embodiment of the present application;

[0050] Figure 3A flowchart of a model training method provided for a preferred embodiment of the present application;

[0051] Figure 4 This is a structural block diagram of a model training device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0053] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with general skills in the technical field to which this application belongs. The words "one", "a", "the", "these" and the like in this application do not indicate a quantitative limitation, and they may be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" may mean: A exists alone, A and B exist at the same time, and B exists alone. Generally, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0054] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. For example, running on a terminal, Figure 1 is a hardware structure block diagram of a terminal of the model training method of this embodiment. Figure 1 As shown, the terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 and a memory 104 for storing data, wherein the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA. The above terminal may also include a transmission device 106 and an input and output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1The structure shown is for illustration only and does not limit the structure of the above terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0055] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the model training method in the present embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, the above method is implemented. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0056] The transmission device 106 is used to receive or send data via a network. The above network includes a wireless network provided by the communication provider of the terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0057] In this embodiment, a model training method is provided. Figure 2 is a flow chart of the model training method of this embodiment, such as Figure 2 As shown, the process includes the following steps:

[0058] Step S210, obtaining multimodal training samples related to the target task; the multimodal training samples include at least two samples of text, speech and image, wherein one of the at least two samples is a sample type related to the target task; the target task is one of visual processing, speech processing or language processing.

[0059] The multimodal training samples may be data samples from different modalities, i.e., data from different types of data sources. Specifically, the multimodal training samples in this embodiment may include sample data related to the target task and at least one other type of sample data. The sample data may include text data, image data, audio data, video data, structured data, biological signal data, etc.

[0060] Among them, the above-mentioned sample types related to the target task can be that when the target task is visual processing, the sample types related to the target task are images; when the target task is speech processing, the sample types related to the target task are speech; when the target task is language processing, the sample types related to the target task are text.

[0061] In this embodiment, the use of multimodal training samples can improve the perception and understanding ability of the training samples and realize cross-modal information interaction and fusion. In addition, the use of multimodal training samples can also enable the model to learn the association and interaction between different modalities, so that the model performs better in multimodal tasks and can also improve the generalization ability of the model.

[0062] Step S220, based on the acquired multimodal training samples related to the target task and the large model to be trained related to the target task, construct an attack and defense game model related to the target task.

[0063] In this step, an attack and defense game model related to the target task is constructed based on the acquired multimodal training samples related to the target task and the large model to be trained related to the target task. The attack and defense game model related to the target task can be constructed based on the acquired multimodal training samples related to the target task and the training objective function of the large model to be trained related to the target task.

[0064] Step S230, based on the attack and defense game model related to the target task, remove the target samples from the multimodal training samples related to the target task to obtain streamlined samples; the target samples are samples that affect the model training convergence speed of the large model to be trained.

[0065] The above-mentioned method, based on the attack and defense game model related to the target task, eliminates the target samples in the multimodal training samples related to the target task to obtain the streamlined samples. The target attack samples can be determined according to the attack and defense game model related to the target task; the target samples in the multimodal training samples related to the target task are determined according to the target attack samples; and the target samples in the multimodal training samples related to the target task are eliminated to obtain the streamlined samples.

[0066] Step S240, using the streamlined samples, train the large model to be trained related to the target task to obtain a trained large model related to the target.

[0067] Because the streamlined samples are the samples after eliminating the samples that affect the model training convergence speed of the large model to be trained in the multimodal training samples related to the target task, therefore, using the streamlined samples to train the large model to be trained related to the target task can improve the training efficiency of the model and reduce the resource usage during the training process.

[0068] In the above steps S210 to S240, multimodal training samples related to the target task are obtained, wherein the multimodal training samples include at least two samples of text, speech and image, and one of them is a sample type related to the target task. Then, based on the obtained multimodal training samples related to the target task and the large model to be trained related to the target task, an attack and defense game model related to the target task is constructed. Then, based on the attack and defense game model related to the target task, target samples that affect the model training convergence speed of the large model to be trained in the multimodal training samples related to the target task are eliminated to obtain streamlined samples. Finally, the streamlined samples are used to train the large model to be trained related to the target task to obtain the trained large model related to the target. It uses an attack and defense game model to determine and eliminate samples in the multimodal samples that affect the training speed of the large model, so that the remaining samples can be used to train the large model to be trained related to the target task, improve training efficiency, and reduce resource usage during training. It solves the problem of excessive computing resources being occupied during large model training due to the existence of samples that affect the training efficiency of the large model in the existing large model training method.

[0069] In one embodiment, based on step S220, based on the acquired multimodal training samples related to the target task and the large model to be trained related to the target task, an attack and defense game model related to the target task is constructed, including:

[0070] Step S222, based on the acquired multimodal training samples related to the target task and the training objective function of the large model to be trained related to the target task, construct an attack and defense game model related to the target task; the expression of the attack and defense game model is:

[0071] ;

[0072] ;

[0073] ;

[0074] ;

[0075] ;

[0076] ;

[0077] ;

[0078] Among them, g k is the gradient value of the kth step of training the large model to be trained related to the target task, g k-1 is the gradient value of the k-1th step of training the large model to be trained related to the target task, is the gradient difference between the k-th step and the k-1-th step of the training of the large model to be trained related to the target task, is the penalty factor, x is the target attack sample, is the perturbation value of the target attack sample, and x´ is the adjusted target attack sample; is the threshold factor, J is the training objective function of the large model to be trained related to the target task, w is the model parameter of the large model to be trained related to the target task, is the derivative of the model parameter w.

[0079] Among them, the threshold factor It is related to the number of target attack samples finally selected, so the threshold factor can be adjusted , adjust the number of target attack samples. When the threshold factor The larger the value, the smaller the number of target attack samples. The smaller it is, the greater the number of target attack samples.

[0080] Specifically, in the above attack and defense game model, the attack model (the top layer of the attack and defense game model) can be expressed as:

[0081]

[0082] In this step, the perturbation value of the target attack sample is As the input of the attack model (it should be noted that the perturbation value of the initial target attack sample can be a random noise or a preset perturbation value, and the perturbation value of the target attack sample is automatically updated as the model is solved), the gradient difference between the kth step and the k-1th step of the training of the large model to be trained related to the target task is calculated. As the output of the attack model, when the gradient difference between the kth step and the k-1th step of the training of the large model to be trained related to the target task is When the preset attack sample screening conditions are met, target attack samples that meet the conditions are screened out. It should be noted that the screened target attack samples are not necessarily the original samples of the large model to be trained related to the target task, but may be samples that meet the conditions after the parameters are adjusted during the training process.

[0083] It uses the maximum gradient difference metric to delay the lower-layer training process to optimize convergence, increase training time, and thus improve training energy consumption. At the same time, it minimizes the adversarial perturbation metric to obtain imperceptible tiny perturbations, making adversarial attacks more covert.

[0084] Based on the attack model, the process of screening out target attack samples that meet the conditions can be expressed as:

[0085]

[0086] Therefore, when the gradient difference between the kth step and the k-1th step of the training of the large model to be trained related to the target task is If the above conditions are met, it can be determined that the samples used to train the large model to be trained are target attack samples.

[0087] Furthermore, in the above attack and defense game model, the defense model (the bottom layer of the attack and defense game model) can be expressed as:

[0088]

[0089] Therefore, in this embodiment, the defense model feeds back the model training error according to the input training sample, that is, the objective function value of the feedback model.

[0090] The following is an example of the process of training a large visual model:

[0091] Step S1, initialize the perturbation value of the target attack sample and threshold factor ;

[0092] Step S2: input the training samples such as images and videos required by the large visual model to be trained into the defense model (model to be trained) for training. The defense model will output the results of the objective function (such as the error size, accuracy, training time, etc. of the model training), the gradient values ​​of the kth step and the k-1th step; calculate the gradient difference between the kth step and the k-1th step. ;

[0093] Step S3: set the initial disturbance value of the target attack sample Input the attack model and solve the perturbation value of the target attack sample through iteration ;use , calculate the adjusted target attack sample x´;

[0094] Step S4, input the adjusted target attack sample x´ into the defense model, and the model solves the objective function value J, the gradient value of the kth step and the k-1th step through multiple iterations; calculates the gradient difference between the kth step and the k-1th step ;

[0095] Step S5: The disturbance value of the target attack sample Input the attack model and iteratively solve the perturbation value of the new target attack sample ;use , calculate the new adjusted target attack sample x´;

[0096] Step S6, repeating steps S4 and S5;

[0097] Step S7, finally obtaining the optimal disturbance value of the target attack sample , calculate the optimal perturbation value of the target attack sample The corresponding adjusted target attack sample x´ is obtained to obtain the constructed target attack sample;

[0098] Step S8, performing a similarity comparison between the constructed attack sample and the samples in the training sample, finding the sample most similar to the constructed attack sample, and removing it to obtain a streamlined sample;

[0099] Step S9, training the defense model according to the streamlined samples to obtain a trained large visual model.

[0100] In this embodiment, by constructing an attack and defense game model related to the target task, target attack samples that meet the conditions can be screened out, so that target samples can be removed from multimodal training samples related to the target task according to the target attack samples to obtain streamlined samples.

[0101] Specifically, in one embodiment, the above step S230, based on the attack and defense game model related to the target task, removes the target sample from the multimodal training samples related to the target task to obtain a streamlined sample, including:

[0102] Step S232, based on the attack and defense game model related to the target task, determine the target attack sample; the target attack sample is a sample in which the gradient difference between the previous and next steps is greater than or equal to the preset gradient difference threshold when training the large model to be trained.

[0103] The above two steps may refer to two model training cycles. The above preset gradient difference threshold may be specifically set according to specific circumstances, and is not specifically limited in this embodiment.

[0104] Step S234: based on the target attack samples, the target samples in the multimodal training samples related to the target task are eliminated to obtain streamlined samples.

[0105] Because, when the large model to be trained is being trained, the larger the gradient difference between the previous and next steps, the greater the parameter update in each iteration, which further indicates that the large model to be trained is not conducive to model convergence and the model convergence speed is slow. Therefore, samples where the gradient difference between the previous and next steps is greater than or equal to the preset gradient difference threshold are samples that affect the convergence speed of model training. Therefore, the determined target attack samples are samples that affect the convergence speed of model training of the large model to be trained. On the premise that the target attack samples are samples that affect the convergence speed of model training of the large model to be trained, the adjusted samples obtained after adjusting the target attack samples are also samples that affect the convergence speed of model training of the large model to be trained, and samples with high similarity to the adjusted samples are also samples that affect the convergence speed of model training of the large model to be trained. Based on the above analysis, it can be seen that the target samples are also samples that affect the convergence speed of model training of the large model to be trained.

[0106] In the above steps S232 to S234, the target attack samples are determined based on the attack and defense game model related to the target task, and then the target samples are eliminated from the multimodal training samples related to the target task to obtain streamlined samples. The samples that affect the training speed of the large model are eliminated to achieve the use of the remaining samples to train the large model to be trained related to the target task, thereby improving the training efficiency and reducing the resource usage during the training process.

[0107] In addition, in one embodiment, based on step S234, based on the target attack sample, the target sample in the multimodal training sample related to the target task is eliminated to obtain a streamlined sample, including:

[0108] Step S2342: adjusting each target attack sample based on the disturbance value of each target attack sample to obtain each adjusted target attack sample.

[0109] The specific process of adjusting each target attack sample based on the disturbance value of each target attack sample to obtain each adjusted target attack sample can be expressed as:

[0110]

[0111] Step S2344, respectively calculating the similarity between each adjusted target attack sample and each multimodal training sample, and obtaining the similarity result between each adjusted target attack sample and each multimodal training sample.

[0112] In this step, the similarity between each adjusted target attack sample and each multimodal training sample is calculated by using methods such as Pearson correlation coefficient, mutual information, and Fisher score to calculate the similarity between each adjusted target attack sample and each multimodal training sample.

[0113] Step S2346: The samples in the multimodal training samples whose similarity results with each adjusted target attack sample are greater than or equal to a preset similarity threshold are taken as target samples.

[0114] Specifically, the process of determining the target sample can be expressed as:

[0115]

[0116] Among them, x1 is a multimodal training sample, τ is a preset similarity threshold, Represents the similarity results between the multimodal training samples and the adjusted target attack samples.

[0117] Step S2348: Remove the target sample from the multimodal training samples related to the target task to obtain a streamlined sample.

[0118] In the above steps S2342 to S2348, each target attack sample is adjusted based on the disturbance value of each target attack sample to obtain each adjusted target attack sample, and then the target sample in the multimodal training samples related to the target task is determined, and the target sample in the multimodal training samples related to the target task is eliminated to obtain a streamlined sample. By eliminating samples that affect the training speed of the large model, the remaining samples can be used to train the large model to be trained related to the target task, thereby improving the training efficiency and reducing the resource usage during the training process.

[0119] In one embodiment, before step S2342, the following steps are included:

[0120] Step S2341, calculate the solution of each target attack sample in the attack and defense game model to obtain the disturbance value of each target attack sample.

[0121] Specifically, based on the formula of the attack and defense game model in step S222, the solution of each target attack sample in the attack and defense game model is calculated to obtain the disturbance value of each target attack sample.

[0122] In addition, in one embodiment, before step S220, the following steps are included:

[0123] Step S218: performing one or more of data cleaning, normalization or alignment on the acquired multimodal training samples related to the target task.

[0124] Furthermore, in one embodiment, before step S220, the method further includes:

[0125] Step S219, based on the target task, determine the large model to be trained that is related to the target task.

[0126] Specifically, in one embodiment, the above step S219, based on the target task, determines the large model to be trained related to the target task, including:

[0127] Step S2192, based on the target task, determine the model architecture, model structure and initialization model parameters of the large model to be trained related to the target task.

[0128] Step S2194, based on the model architecture, model structure and initialization model parameters of the large model to be trained related to the target task, determine the large model to be trained related to the target task.

[0129] The present embodiment is described and illustrated below through preferred embodiments.

[0130] Figure 3 Flowchart of a model training method provided by a preferred embodiment of the present application. Figure 3 As shown, the model training method includes the following steps:

[0131] Step S301, obtaining multimodal training samples related to the target task; the multimodal training samples include at least two samples of text, speech and image, wherein one of the at least two samples is a sample type related to the target task; the target task is one of visual processing, speech processing or language processing;

[0132] Step S302, constructing an attack and defense game model related to the target task based on the acquired multimodal training samples related to the target task and the large model to be trained related to the target task;

[0133] Step S303, based on the attack and defense game model related to the target task, determine the target attack sample; the target attack sample is a sample in which the gradient difference between the previous and next steps is greater than or equal to the preset gradient difference threshold when the large model to be trained is trained;

[0134] Step S304, calculating the solution of each target attack sample in the attack and defense game model to obtain the disturbance value of each target attack sample;

[0135] Step S305, adjusting each target attack sample based on the disturbance value of each target attack sample to obtain each adjusted target attack sample;

[0136] Step S306, respectively calculating the similarity between each adjusted target attack sample and each multimodal training sample, and obtaining the similarity result between each adjusted target attack sample and each multimodal training sample;

[0137] Step S307, taking the samples in the multimodal training samples whose similarity results with each adjusted target attack sample are greater than or equal to a preset similarity threshold as target samples;

[0138] Step S308, removing the target sample from the multimodal training samples related to the target task to obtain a streamlined sample;

[0139] Step S309, using the streamlined samples, train the large model to be trained related to the target task to obtain a trained large model related to the target.

[0140] In the above steps S301 to S309, multimodal training samples related to the target task are obtained, wherein the multimodal training samples include at least two samples of text, speech and image, and one of them is a sample type related to the target task. Then, based on the obtained multimodal training samples related to the target task and the large model to be trained related to the target task, an attack and defense game model related to the target task is constructed. Then, based on the attack and defense game model related to the target task, target samples that affect the model training convergence speed of the large model to be trained in the multimodal training samples related to the target task are eliminated to obtain streamlined samples. Finally, the streamlined samples are used to train the large model to be trained related to the target task to obtain the trained large model related to the target. It uses an attack and defense game model to determine and eliminate samples in the multimodal samples that affect the training speed of the large model, so that the remaining samples can be used to train the large model to be trained related to the target task, improve training efficiency, and reduce resource usage during training. It solves the problem of excessive computing resources being occupied during the training of the large model due to the existence of samples that affect the training efficiency of the large model.

[0141] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0142] Based on the same inventive concept, a model training device is also provided in this embodiment, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. The terms "module", "unit", "subunit", etc. used below can implement a combination of software and / or hardware of predetermined functions. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0143] In one embodiment, Figure 4 is a structural block diagram of a model training device provided in an embodiment of the present application, such as Figure 4 As shown, the model training device comprises:

[0144] The sample acquisition module 42 is used to acquire multimodal training samples related to the target task; the multimodal training samples include at least two samples of text, speech and image, wherein one of the at least two samples is a sample type related to the target task; the target task is one of visual processing, speech processing or language processing;

[0145] A model determination module 44 is used to construct an attack and defense game model related to the target task based on the acquired multimodal training samples related to the target task and the large model to be trained related to the target task;

[0146] The sample reduction module 46 is used to remove target samples from the multimodal training samples related to the target task based on the attack and defense game model related to the target task to obtain reduced samples; the target samples are samples that affect the model training convergence speed of the large model to be trained;

[0147] And the model training module 48 is used to use the streamlined samples to train the large model to be trained related to the target task, so as to obtain the trained large model related to the target.

[0148] The above-mentioned model training device obtains multimodal training samples related to the target task, wherein the multimodal training samples include at least two samples of text, speech and image, and one of them is a sample type related to the target task. Then, based on the obtained multimodal training samples related to the target task and the large model to be trained related to the target task, an attack and defense game model related to the target task is constructed. Then, based on the attack and defense game model related to the target task, the target samples that affect the model training convergence speed of the large model to be trained in the multimodal training samples related to the target task are eliminated to obtain streamlined samples. Finally, the streamlined samples are used to train the large model to be trained related to the target task to obtain the trained large model related to the target. It determines and eliminates the samples that affect the training speed of the large model in the multimodal samples through the attack and defense game model, so as to realize the training of the large model to be trained related to the target task using the remaining samples, improve the training efficiency, reduce the resource occupation during the training process, and solve the existing large model training method. Because there are samples that affect the training efficiency of the large model, there is a problem of excessive computing resources being occupied during the large model training process.

[0149] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0150] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, any one of the model training methods in the above embodiments is implemented.

[0151] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0152] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0153] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A model training method, characterized in that: The method comprises: Acquire multimodal training samples related to the target task; the multimodal training samples include at least two samples of text, speech and image, wherein one of the at least two samples is a sample type related to the target task; the target task is one of visual processing, speech processing or language processing; Based on the acquired multimodal training samples related to the target task and the large model to be trained related to the target task, an attack and defense game model related to the target task is constructed; in the attack and defense game model, the attack model is the top layer of the attack and defense game model, which is expressed as: ; Among them, the perturbation value of the target attack sample As the input of the attack model, the gradient difference between the kth step and the k-1th step of the training of the large model to be trained related to the target task is As the output of the attack model, when the gradient difference between the kth step and the k-1th step of the training of the large model to be trained related to the target task When the preset attack sample screening conditions are met, target attack samples that meet the conditions are screened out; based on the attack model, the process of screening out the target attack samples that meet the conditions is expressed as follows: ; In the attack and defense game model, the defense model is the bottom layer of the attack and defense game model and is expressed as: ; The defense model feeds back the model training error according to the input training sample, that is, the objective function value of the feedback model; wherein, is the penalty factor, is the threshold factor, ; The calculation process is: ; g k is the gradient value of the kth step of the training of the large model to be trained related to the target task, g k-1 is the gradient value of the k-1th step of the training of the large model to be trained related to the target task; g k The calculation process is: ; Wherein, w is the model parameter of the large model to be trained related to the target task, J is the training objective function of the large model to be trained related to the target task, x' is the adjusted target attack sample, , where x is the target attack sample, is the disturbance value of the target attack sample; Based on the attack and defense game model related to the target task, determine the target attack sample; the target attack sample is a sample in which the gradient difference between the previous and next steps is greater than or equal to a preset gradient difference threshold when training the large model to be trained; based on the target attack sample, remove the target sample in the multimodal training sample related to the target task to obtain a streamlined sample; based on the target attack sample, remove the target sample in the multimodal training sample related to the target task to obtain a streamlined sample, including: calculating the solution of each of the target attack samples in the attack and defense game model to obtain the perturbation value of each of the target attack samples; based on each of the target attack samples disturbance value, adjusting each of the target attack samples to obtain each adjusted target attack sample; respectively calculating the similarity between each of the adjusted target attack samples and each of the multimodal training samples to obtain the similarity result between each of the adjusted target attack samples and each of the multimodal training samples; taking the samples in the multimodal training samples whose similarity result with each of the adjusted target attack samples is greater than or equal to a preset similarity threshold as the target samples; removing the target samples from the multimodal training samples related to the target task to obtain the streamlined samples; the target samples are samples that affect the model training convergence speed of the large model to be trained; The streamlined samples are used to train the large model to be trained that is related to the target task, so as to obtain a trained large model that is related to the target.

2. The model training method according to claim 1, characterized in that: Before constructing an attack and defense game model related to the target task based on the acquired multimodal training samples related to the target task and the large model to be trained related to the target task, the method includes: Perform one or more of data cleaning, normalization or alignment on the acquired multimodal training samples related to the target task.

3. The model training method according to claim 1, characterized in that: Before constructing an attack and defense game model related to the target task based on the acquired multimodal training samples related to the target task and the large model to be trained related to the target task, the method further includes: Based on the target task, the large model to be trained related to the target task is determined.

4. The model training method according to claim 3, characterized in that: The step of determining the large model to be trained related to the target task based on the target task includes: Based on the target task, determine the model architecture, model structure and initialization model parameters of the large model to be trained related to the target task; Based on the model architecture, the model structure and the initialized model parameters of the large model to be trained related to the target task, determine the large model to be trained related to the target task.

5. A model training device, characterized in that: The device comprises: A sample acquisition module, used to acquire multimodal training samples related to a target task; the multimodal training samples include at least two samples of text, speech and image, wherein one of the at least two samples is a sample type related to the target task; the target task is one of visual processing, speech processing or language processing; The model determination module is used to construct an attack and defense game model related to the target task based on the acquired multimodal training samples related to the target task and the large model to be trained related to the target task; in the attack and defense game model, the attack model is the top layer of the attack and defense game model, which is expressed as: ; Among them, the perturbation value of the target attack sample As the input of the attack model, the gradient difference between the kth step and the k-1th step of the training of the large model to be trained related to the target task is As the output of the attack model, when the gradient difference between the kth step and the k-1th step of the training of the large model to be trained related to the target task When the preset attack sample screening conditions are met, target attack samples that meet the conditions are screened out; based on the attack model, the process of screening out the target attack samples that meet the conditions is expressed as follows: ; In the attack and defense game model, the defense model is the bottom layer of the attack and defense game model and is expressed as: ; The defense model feeds back the model training error according to the input training sample, that is, the objective function value of the feedback model; wherein, is the penalty factor, is the threshold factor, ; The calculation process is: ; g k is the gradient value of the kth step of the training of the large model to be trained related to the target task, g k-1 is the gradient value of the k-1th step of the training of the large model to be trained related to the target task; g k The calculation process is: ; Wherein, w is the model parameter of the large model to be trained related to the target task, J is the training objective function of the large model to be trained related to the target task, x' is the adjusted target attack sample, , where x is the target attack sample, is the disturbance value of the target attack sample; A sample reduction module is used to determine a target attack sample based on the attack and defense game model related to the target task; the target attack sample is a sample in which the gradient difference between the previous and next steps is greater than or equal to a preset gradient difference threshold when training the large model to be trained; based on the target attack sample, the target sample in the multimodal training sample related to the target task is eliminated to obtain a simplified sample; based on the target attack sample, the target sample in the multimodal training sample related to the target task is eliminated to obtain a simplified sample, including: calculating the solution of each of the target attack samples in the attack and defense game model to obtain a disturbance value of each of the target attack samples; based on each of the target The disturbance value of the attack sample is used to adjust each of the target attack samples to obtain each adjusted target attack sample; the similarity between each of the adjusted target attack samples and each of the multimodal training samples is calculated respectively to obtain the similarity result between each of the adjusted target attack samples and each of the multimodal training samples; the samples in the multimodal training samples whose similarity result with each of the adjusted target attack samples is greater than or equal to a preset similarity threshold are used as the target samples; the target samples are removed from the multimodal training samples related to the target task to obtain the streamlined samples; the target samples are samples that affect the model training convergence speed of the large model to be trained; And a model training module is used to use the streamlined samples to train the large model to be trained related to the target task to obtain a trained large model related to the target.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the model training method described in any one of claims 1 to 4 are implemented.

Images