Multimodal model training method and apparatus, server, and storage medium

Abstract

The application relates to the field of artificial intelligence, and provides a multi-modal model training method and device, a server and a storage medium. The method calculates the total loss value of each task in the process of training a multi-modal model using each training sample set, determines a task with the maximum total loss value as a target task that needs to be preferentially trained, and then trains only the target task using the remaining training samples in the training sample set, so that dynamic adjustment training of multiple tasks is realized, the reinforcement learning efficiency of the multi-modal model for different tasks is balanced, and the trained multi-modal model does not need to be fine-tuned, thereby greatly improving the training efficiency of the multi-modal model and reducing the training cost. The application also relates to the field of blockchains, and the above storage medium can store data created according to the use of a blockchain node.

Description

Technical Field

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a multimodal model training method, device, server, and storage medium. Background Art

[0002] Currently, the common method is to manually annotate images and text to generate multiple positive and negative image-text pairs. Based on these positive and negative pairs, a contrastive learning framework is then used to train multimodal models. However, this training method lacks training for the task objectives, and trained multimodal models often require further fine-tuning to adapt them to other downstream tasks. To ensure the accuracy of the trained multimodal models for other tasks, fine-tuning the trained multimodal models is time-consuming, resulting in low model training efficiency and high training costs.

[0003] For example, in the financial insurance sector, fine-tuning the multimodal model to ensure it is suitable for pet classification tasks requires a significant amount of time. This allows users to upload photos of their pets and input them into the fine-tuned multimodal model to obtain a description of the pet type. Another example is the digital healthcare sector, where fine-tuning the multimodal model to ensure it is suitable for human body part classification tasks requires a significant amount of time. This allows users to input medical images captured by medical imaging equipment into the fine-tuned multimodal model to obtain a description of the human body part type. Summary of the Invention

[0004] The embodiments of the present application provide a multimodal model training method, device, server and storage medium, aiming to improve the training efficiency of the multimodal model and reduce the cost of model training.

[0005] In a first aspect, an embodiment of the present application provides a multimodal model training method, wherein the multimodal model includes an encoding layer, a feature extraction sub-model, and multiple task processing sub-models, and the method includes:

[0006] Selecting one training sample set from a plurality of training sample sets as a target training sample set, wherein the first number of training samples included in the plurality of training sample sets are all different;

[0007] selecting a second number of training samples from the target training sample set, the first number being greater than the second number, the training samples comprising image-text pairs, question information and true answer information for each of a plurality of tasks, the image-text pairs comprising training text and training images;

[0008] Processing each of the selected training samples using the multimodal model to obtain a first processing result for each of the selected training samples, the first processing result including a first feature vector of the training text and a second feature vector of the training image output by the feature extraction sub-model, and predicted answer information corresponding to the task output by each of the task processing sub-models;

[0009] Determining a first loss value for each of the selected training samples and a second loss value for each of the tasks according to the first processing result of each of the selected training samples and the true answer information of each of the tasks;

[0010] Determine the total loss value of each task according to the second loss value of each selected training sample under each task, and determine the task corresponding to the highest total loss value as the target task;

[0011] Processing each remaining training sample in the target training sample set by the multimodal model to obtain a second processing result for each remaining training sample, and determining a first loss value for each remaining training sample and a second loss value for the target task based on the second processing result for each remaining training sample;

[0012] Determining a target loss value for the target training sample set according to the first loss value for each training sample in the target training sample set and the second loss value under each of the tasks;

[0013] Determining whether the multimodal model has converged according to the target loss value, and updating parameters of the multimodal model if the multimodal model has not converged;

[0014] Return to the step of selecting one training sample set from the multiple training sample sets as a target training sample set until each of the training sample sets is selected once or the multimodal model converges.

[0015] In a second aspect, an embodiment of the present application further provides a multimodal model training device, wherein the multimodal model includes an encoding layer, a feature extraction sub-model, and multiple task processing sub-models, and the multimodal model training device includes:

[0016] A sample set selection module is configured to select one training sample set from a plurality of training sample sets as a target training sample set, wherein the first number of training samples included in the plurality of training sample sets are all different;

[0017] a sample selection module configured to select a second number of training samples from the target training sample set, the first number being greater than the second number, the training samples comprising image-text pairs, question information and true answer information for each of a plurality of tasks, the image-text pairs comprising training images and training text;

[0018] a sample processing module configured to process each of the selected training samples using the multimodal model to obtain a first processing result for each of the selected training samples, the first processing result comprising a first feature vector of the training text and a second feature vector of the training image output by the feature extraction sub-model, and predicted answer information corresponding to the task output by each of the task processing sub-models;

[0019] a loss value determining module, configured to determine a first loss value for each selected training sample and a second loss value for each task based on the first processing result of each selected training sample and the true answer information of each task;

[0020] a task determination module configured to determine a total loss value for each of the tasks based on the second loss value of each of the selected training samples under each of the tasks, and determine the task corresponding to the highest total loss value as the target task;

[0021] The sample processing module is further configured to process each remaining training sample in the target training sample set using the multimodal model to obtain a second processing result for each remaining training sample;

[0022] The loss value determination module is further configured to determine a first loss value for each of the remaining training samples and a second loss value for the target task based on the second processing result of each of the remaining training samples;

[0023] The loss value determination module is further configured to determine a target loss value of the target training sample set based on the first loss value of each training sample in the target training sample set and the second loss value under each of the tasks;

[0024] a model updating module configured to determine whether the multimodal model has converged according to the target loss value, and to update parameters of the multimodal model if the multimodal model has not converged;

[0025] The loop module is configured to return to the step of selecting one training sample set from the multiple training sample sets as the target training sample set until each of the training sample sets is selected once or the multimodal model converges.

[0026] In a third aspect, an embodiment of the present application further provides a server, comprising a processor, a memory, and a computer program stored on the memory and executable by the processor, wherein when the computer program is executed by the processor, the multimodal model training method as described in the first aspect is implemented.

[0027] In a fourth aspect, an embodiment of the present application further provides a storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the multimodal model training method as described in the first aspect is implemented.

[0028] The embodiments of the present application provide a multimodal model training method, device, server and storage medium. The method calculates the total loss value of each task in the process of training the multimodal model using each training sample set, and determines the task with the largest total loss value as the target task that needs to be trained first. Then, the remaining training samples in the training sample set are used to train only the target task, thereby realizing dynamic adjustment training of multiple tasks, balancing the reinforcement learning efficiency of the multimodal model for different tasks, and eliminating the need to fine-tune the trained multimodal model, thereby greatly improving the training efficiency of the multimodal model and reducing the training cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0030] Figure 1 This is a flow chart of a multimodal model training method provided in an embodiment of the present application;

[0031] Figure 2 is a structural diagram of a multimodal model in an embodiment of the present application;

[0032] Figure 3 yes Figure 1 A schematic diagram of a sub-step process of the multimodal model training method in FIG;

[0033] Figure 4 yes Figure 1 A schematic flow chart of another sub-step of the multimodal model training method in FIG.

[0034] Figure 5 This is a flowchart of another multimodal model training method provided in an embodiment of the present application;

[0035] Figure 6is another structural diagram of the multimodal model in the embodiment of the present application;

[0036] Figure 7 is a schematic block diagram of a multimodal model training device provided in an embodiment of the present application;

[0037] Figure 8 This is a schematic block diagram of the structure of a server provided in an embodiment of the present application.

[0038] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0040] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, combined, or partially merged, so the actual execution order may vary depending on the actual situation.

[0041] Currently, the common method is to manually annotate images and text to generate multiple positive and negative image-text pairs. Based on these positive and negative pairs, a contrastive learning framework is then used to train multimodal models. However, this training method lacks training for the task objectives, and trained multimodal models often require further fine-tuning to adapt them to other downstream tasks. To ensure the accuracy of the trained multimodal models for other tasks, fine-tuning the trained multimodal models is time-consuming, resulting in low model training efficiency and high training costs.

[0042] To solve the above problems, the embodiments of the present application provide a multimodal model training method, device, server and storage medium. This method calculates the total loss value of each task in the process of training the multimodal model using each training sample set, and determines the task with the largest total loss value as the target task that needs to be trained first. Then, the remaining training samples in the training sample set are used to train only the target task, thereby achieving dynamic adjustment training of multiple tasks, balancing the reinforcement learning efficiency of the multimodal model for different tasks, and eliminating the need to fine-tune the trained multimodal model, thereby greatly improving the training efficiency of the multimodal model and reducing the training cost.

[0043] For example, for the pet classification task in the pet insurance process in the financial insurance field, the multimodal model may include a coding layer, a feature extraction sub-model, a processing sub-model for the pet recognition task, and a processing sub-model for the pet classification task. The pet recognition task is used to identify whether there is a pet in the pet image, and the pet classification task is used to identify the type of pet in the pet image. After training the above-mentioned multimodal model using the multimodal model training method provided in the embodiment of the present application, a pet classification model can be obtained, and there is no need to fine-tune the pet classification model, which greatly improves the training efficiency of the model and reduces the training cost. The pet classification model includes a coding layer and a pet classification sub-model (the processing sub-model of the trained pet classification task).

[0044] For example, for the human body part classification task in the field of digital medicine, the multimodal model may include a coding layer, a feature extraction submodel, a processing submodel for the human body part recognition task, and a processing submodel for the human body part classification task. The human body part recognition task is used to identify whether there are human body parts in the medical image, and the human body part classification task is used to identify the type of human body parts in the medical image. After training the above-mentioned multimodal model using the multimodal model training method provided in the embodiment of the present application, a human body part classification model can be obtained, and there is no need to fine-tune the human body part classification model, which greatly improves the training efficiency of the model and reduces the training cost.

[0045] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system of using digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. Basic artificial intelligence technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. Artificial intelligence software technology mainly includes computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology and machine learning / deep learning and other major directions. The server can be an independent server or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (CDNs), and basic cloud computing services such as big data and artificial intelligence platforms.

[0046] The following describes some embodiments of the present application in detail with reference to the accompanying drawings. In the absence of conflict, the following embodiments and features therein may be combined with each other.

[0047] See also Figure 1 , Figure 1 It is a flowchart of a multimodal model training method provided in an embodiment of the present application.

[0048] like Figure 1 As shown, the multimodal model training method includes steps S101 to S108.

[0049] Step S101: Select one training sample set from multiple training sample sets as a target training sample set.

[0050] In this embodiment, multiple training sample sets are pre-generated, and the first number of training samples contained in the multiple training sample sets are all different. The training samples include image-text pairs, question information and real answer information for each of the multiple tasks, and the image-text pairs include training images and training texts. The multimodal model can be a multi-layer model based on transformer. The multimodal model can include an encoding layer, a feature extraction sub-model, and multiple task processing sub-models. Among them, one task corresponds to one task processing model, and the first number can be set based on actual conditions. The embodiment of the present application does not make specific restrictions on this. For example, the first number is 35.

[0051] For example, Figure 2 As shown, the multimodal model includes a coding layer 10, a feature extraction sub-model 20 and multiple task processing sub-models 30, and the coding layer 10 is connected to the feature extraction sub-model 20 and the multiple task processing sub-models 30 respectively. It can be understood that the number, structure and multiple tasks of the task processing sub-models included in the multimodal model can be set based on actual conditions, and the embodiments of the present application do not specifically limit this. For example, in the target detection scenario, four tasks can be set, Task 1 is List all objects, that is, all objects are listed, Task 2 is Does [object] exist, that is, whether an object exists, Task 3 is Does [obj1], [obj2] and / or [obj3] exist, that is, whether object 1, object 2 and / or object 3 exist, Task 4 is Which of [obj1], [obj2] and [obj3] exist?, that is, which one of object 1, object 2 and object 3 exists.

[0052] Therefore, the multiple task processing sub-models included in the multimodal model may include a processing sub-model corresponding to task 1, a processing sub-model corresponding to task 2, a processing sub-model corresponding to task 3, and a processing sub-model corresponding to task 4, and the question information and true answer information of each task in the multiple tasks in the training sample may include the question information and true answer information of task 1, the question information and true answer information of task 2, the question information and true answer information of task 3, and the question information and true answer information of task 4. For example, the question information and true answer information of task 1 are "List all objects" and "[obj1], [obj2], and [obj3]", respectively; the question information and true answer information of task 2 are "Does [object] exist?" and "Yes", the question information and true answer information of task 3 are "Does [obj1], [obj2] and / or [obj3] exist?" and "Yes", and the question information and true answer information of task 4 are "Which of [obj1], [obj2] and [obj3] exist?" and "[obj2]", respectively.

[0053] Step S102: Select a second number of training samples from the target training sample set.

[0054] In this embodiment, the first number is greater than the second number. The first number and the second number can be set based on actual conditions and are not specifically limited in this embodiment of the application. For example, the first number is 35 and the second number is 30 or 32. For example, 32 training samples are randomly selected from the 35 training samples included in the target training sample set, leaving 3 training samples.

[0055] In some embodiments, the first number of training samples includes a third number of positive training samples and a fourth number of negative training samples, the first number is equal to the sum of the third number and the fourth number, and the absolute value of the difference between the third number and the fourth number is less than or equal to a preset value. The preset value can be set based on actual conditions, and the embodiments of the present application are not specifically limited to this. For example, the preset value is any integer between 1 and 15. This embodiment can ensure that the positive and negative examples of the first number of training samples contained in each training sample set are balanced, which can improve the accuracy of the model.

[0056] For example, the positive training sample includes a first image-text pair, question information and true answer information for each of the multiple tasks, and the training image and training text contained in the first image-text pair match. The negative training sample includes a second image-text pair, question information and true answer information for each of the multiple tasks, and the training image and training text contained in the second image-text pair do not match. For example, in a pet classification scenario, the type of pet in the training image M1 is an Akita dog, and the type of pet in the training image M2 is a blue cat. The first image-text pair can be composed of the training image M1 and the training text "Akita Dog" or the training image M2 and the training text "Blue Cat", and the second image-text pair can be composed of the training image M1 and the training text "Blue Cat" or the training image M2 and the training text "Corgi" (the type of pet in the training image can be the same as the type described in the description text).

[0057] Step S103: Process each selected training sample using the multimodal model to obtain a first processing result for each selected training sample.

[0058] In this embodiment, the first processing result of the training sample includes the first feature vector of the training text in the training sample and the second feature vector of the training image in the training sample output by the feature extraction sub-model, as well as the predicted answer information of the corresponding task output by each task processing sub-model.

[0059] For example, in the target detection scenario, training sample A includes training image A, training text A, the question information of task 1 is "List all objects", the question information of task 2 is "Does [object] exist?", the question information of task 3 is "Does [obj1], [obj2] and / or [obj3] exist?" and the question information of task 4 is "Which of [obj1], [obj2] and [obj3] exist?", then the first processing result of training sample A includes the first feature vector of training text A output by the feature extraction sub-model, the second feature vector of training image A, the predicted answer information output by the task processing sub-model corresponding to task 1 for the question "List all objects", the predicted answer information output by the task processing sub-model corresponding to task 2 for the question "Does [object] exist?", and the predicted answer information output by the task processing sub-model corresponding to task 3 for the question "Does [obj1], [obj2] and / or [obj3] exist?" "The predicted answer information output by the task processing sub-model corresponding to task 4 and the predicted answer information output by the task processing sub-model for the question "Which of [obj1], [obj2] and [obj3] exist?"

[0060] In some embodiments, as Figure 3 As shown, step S103 includes: sub-steps S1031 to S1033.

[0061] Sub-step S1031: For each selected training sample, the training sample is encoded through the encoding layer to obtain a first embedding vector corresponding to the training text, a second embedding vector corresponding to the training image, and a third embedding vector corresponding to the question information of each task.

[0062] In this embodiment, the encoding layer may include a transformer-based encoder. For example, after encoding the training sample A through the encoding layer, the first embedding vector corresponding to the training image A, the second embedding vector corresponding to the training text A, the embedding vector corresponding to the question information "List all objects" of task 1, the embedding vector corresponding to the question information "Does [object] exist?" of task 2, the embedding vector corresponding to the question information "Does [obj1], [obj2] and / or [obj3] exist?" of task 3, and the embedding vector corresponding to the question information "Which of [obj1], [obj2] and [obj3] exist?" of task 4 can be obtained.

[0063] Sub-step S1032: Process the first embedding vector and the second embedding vector using a feature extraction sub-model to obtain a first feature vector for the training text and a second feature vector for the training image.

[0064] In this embodiment, the feature extraction submodel may include a first feature extraction network and a second feature extraction network. The first feature extraction network is used to perform feature extraction processing on the first embedding vector to obtain a first feature vector corresponding to the training text, and the second feature extraction network is used to perform feature extraction processing on the second embedding vector to obtain a second feature vector corresponding to the training image. The first feature extraction network and the second feature extraction network can be set based on actual conditions and are not specifically limited in this embodiment of the application.

[0065] For example, the first embedding vector corresponding to the training image A is subjected to feature extraction processing through the first feature extraction network to obtain the first feature vector corresponding to the training image A. At the same time, the training text A is subjected to feature extraction processing through the second feature extraction network to obtain the second feature vector corresponding to the training text A.

[0066] Sub-step S1033: Perform answer prediction processing on the second embedding vector and the corresponding third embedding vector through each task processing sub-model to obtain predicted answer information for each task.

[0067] For example, the task processing sub-model corresponding to task 1 performs answer prediction processing on the embedding vector corresponding to the question information "List allobjects" of task 1, and obtains the predicted answer information of task 1 as "[obj2]and[obj3]", the task processing sub-model corresponding to task 2 performs answer prediction processing on the embedding vector corresponding to the question information "Does[object]exist?" of task 2, and obtains the predicted answer information of task 2 as "Yes", the task processing sub-model corresponding to task 3 performs answer prediction processing on the embedding vector corresponding to the question information "Does[obj1], [obj2]and / or[obj3]exist?" of task 3, and obtains the predicted answer information of task 3 as "Yes", the task processing sub-model corresponding to task 4 performs answer prediction processing on the embedding vector corresponding to the question information "Which of[obj1], [obj2]and[obj3]exist?" of task 4, and obtains the predicted answer information of task 4 as "[obj2]".

[0068] Step S104: Determine a first loss value for each selected training sample and a second loss value for each task based on the first processing result of each selected training sample and the true answer information of each task.

[0069] For example, the second number is 32, and any selected training sample is recorded as training sample P i (i is any integer from 1 to 32), and the true answer information of task 1 (List all objects) is "[obj2]and[obj3]", the true answer information of task 2 (Does[object]exist?) is "Yes", the true answer information of task 3 (Does[obj1], [obj2]and / or[obj3]exist?) is "Yes", and the true answer information of task 4 (Which of[obj1], [obj2]and[obj3]exist?) is "[obj3]", then according to the training sample P i The first processing result of task 1, the true answer information "[obj2]and[obj3]", the true answer information "Yes" of task 2, the true answer information "Yes" of task 3 and the true answer information "[obj3]" of task 4 can be used to determine the training sample P i The first loss value and training sample P i The second loss values ​​under Task 1, Task 2, Task 3, and Task 4 respectively.

[0070] In some embodiments, as Figure 4As shown, step S104 includes: sub-step S1041 to sub-step S1042.

[0071] Sub-step S1041: For each selected training sample, calculate a first loss value of the training sample according to the first eigenvector and the second eigenvector.

[0072] In this embodiment, the training samples can be divided into positive training samples and negative training samples. If the first training image included in the positive training sample matches the first training text, the actual similarity between the feature vector corresponding to the first training image and the feature vector corresponding to the first training text is 1. If the second training image included in the negative training sample does not match the second training text, the actual similarity between the feature vector corresponding to the second training image and the feature vector corresponding to the second training text is 0.

[0073] In some embodiments, calculating the first loss value of the training sample based on the first eigenvector and the second eigenvector may include: obtaining a preset label of the training sample and obtaining a preset similarity corresponding to the preset label; determining the similarity between the first eigenvector and the second eigenvector; calculating the mean square error, root mean square error or mean absolute error between the similarity and the preset similarity, and determining the mean square error, root mean square error or mean absolute error as the first loss value of the training sample, or.

[0074] The preset label can be a first label or a second label. The first label is used to describe the training sample as a positive training sample, and the second label is used to describe the training sample as a negative training sample. When the preset label is the first label, the preset similarity is 1, and when the preset label is the second label, the preset similarity is 0. For example, the training sample P i The preset label is the first label, and the corresponding preset similarity is 1, then the training sample P can be determined i The similarity between the first eigenvector corresponding to the training text in and the second eigenvector corresponding to the training image in the training sample Pi is 0.8, and the mean square error between 0.8 and 1 is determined (0.8-1). 2 =0.04, therefore, the first loss value of the training sample Pi is 0.04.

[0075] Sub-step S1042: Determine the second loss value of the training sample under each task based on the predicted answer information and the actual answer information of each task.

[0076] In this embodiment, for each task, the similarity between the real answer information of the task and the predicted answer information of the task can be determined; and the similarity between the real answer information of the task and the predicted answer information of the task is determined as the second loss value of the training sample under the task. The similarity between the real answer information of the task and the predicted answer information of the task can be cross-entropy similarity or cosine similarity, which is not limited in the embodiments of the present application.

[0077] For example, the training sample Pi includes the real answer information of task 1, the real answer information of task 2, the real answer information of task 3, and the real answer information of task 4, the first similarity between the predicted answer information of task 1 and the real answer information of task 1 contained in the training sample Pi is determined, and the first similarity is determined as the second loss value of the training sample Pi under task 1. The second similarity between the predicted answer information of task 2 and the real answer information of task 2 contained in the training sample Pi is determined, and the second similarity is determined as the second loss value of the training sample Pi under task 2; the third similarity between the predicted answer information of task 3 and the real answer information of task 3 contained in the training sample Pi is determined, and the third similarity is determined as the third loss value of the training sample Pi under task 3; the fourth similarity between the predicted answer information of task 4 and the real answer information of task 4 contained in the training sample Pi is determined, and the fourth similarity is determined as the second loss value of the training sample Pi under task 4.

[0078] In step S105, the total loss value of each task is determined according to the second loss value of each training sample under each task, and the task corresponding to the highest total loss value is determined as the target task.

[0079] In this embodiment, for each task, the second loss values of the second number of selected training samples under the task can be accumulated to obtain the total loss value of the task. For example, the second number is 32, and the plurality of tasks include task 1, task 2, task 3 and task 4, the second loss values of 32 training samples under task 1 can be accumulated to obtain the total loss of task 1, recorded as Loss1, the second loss values of 32 training samples under task 2 can be accumulated to obtain the total loss of task 2, recorded as Loss2, the second loss values of 32 training samples under task 3 can be accumulated to obtain the total loss of task 3, recorded as Loss3, the second loss values of 32 training samples under task 4 can be accumulated to obtain the total loss of task 4, recorded as Loss4, if Loss3>Loss4>Loss1>Loss2, task 3 can be determined as the target task which needs to be trained preferentially.

[0080] In some embodiments, the loss value proportion of each task can also be determined according to the total loss value of each task, and the task corresponding to the highest loss value proportion is determined as the target task. For example, the total loss values of task 1, task 2, task 3 and task 4 are Loss1, Loss2, Loss3 and Loss4 respectively, then Loss1, Loss2, Loss3 and Loss4 are accumulated to obtain Loss5, and the loss value proportions of task 1, task 2, task 3 and task 4 are a1=Loss1 / loss5, a2=Loss2 / loss5, a3=Loss3 / loss5 and a4=Loss4 / loss5 respectively, assuming a2>a3>a1>a4, then task 2 can be determined as the target task that needs to be trained preferentially.

[0081] In step S106, the multi-modal model is used to process each remaining training sample in the target training sample set to obtain a second processing result of each remaining training sample, and a first loss value of each remaining training sample and a second loss value under the target task are determined according to the second processing result of each remaining training sample and the real answer information of the target task.

[0082] For example, the target training sample set includes a total of 35 training samples, the training sample P1, the training sample P2,..., P32 are first selected, a total of 32 training samples, and the first loss value of each training sample in the 32 training samples and the second loss value under task 1, task 2, task 3 and task 4 are determined, then the total loss values of task 1, task 2, task 3 and task 4 are calculated according to the second loss values of each training sample in the 32 training samples under task 1, task 2, task 3 and task 4, and assuming that the total loss value of task 4 is the highest, then the remaining training samples P33, P34 and P35 are processed by the multi-modal model to obtain the second processing results of the training samples P33, P34 and P35, and the first loss value of the training samples P33, P34 and P35 and the second loss value under task 4 are determined according to the second processing results of the training samples P33, P34 and P35 and the real answer information of task 4. 32 33 34 35 33 34 35 33 34 35 33 34 35

[0083] ​​​​​​​​​​​​​In this embodiment, the second processing result includes the first feature vector of the training text in the remaining training samples and the second feature vector of the training image in the remaining training samples output by the feature extraction sub-model, as well as the predicted answer information of the corresponding task output by each task processing sub-model. For example, one of the remaining training samples is training sample B, and training sample B includes training image B, training text B, the question information of task 1 is "List allobjects", the question information of task 2 is "Does [object] exist?", the question information of task 3 is "Does [obj1], [obj2] and / or [obj3] exist?", and the question information of task 4 is "Which of [obj1], [obj2] and [obj3] exist?", then the second processing result of training text B includes the first feature vector of training text B output by the feature extraction sub-model, the second feature vector of training image B, the predicted answer information output by the task processing sub-model corresponding to task 1 for the question "List allobjects", the predicted answer information output by the task processing sub-model corresponding to task 2 for the question "Does [object] exist?" "The predicted answer information output by the task processing sub-model corresponding to task 3 for the question "Does [obj1], [obj2] and / or [obj3] exist?" and the predicted answer information output by the task processing sub-model corresponding to task 4 for the question "Which of [obj1], [obj2] and [obj3] exist?"

[0084] It should be noted that the specific implementation method of processing each remaining training sample in the target training sample set through the multimodal model to obtain the second processing result of each remaining training sample can refer to the specific implementation process of step S103. According to the second processing result of each remaining training sample, the first loss value of each remaining training sample and the second loss value under the target task are determined. The specific implementation process of step S104 can be referred to, which will not be repeated here.

[0085] Step S107: Determine a target loss value for the target training sample set according to the first loss value of each training sample in the target training sample set and the second loss value under each task.

[0086] For example, the multiple tasks include task 1, task 2, task 3 and task 4, and the target training sample set includes 35 training samples in total. For training sample P1, training sample P2, ..., training sample P 32Each training sample in has a corresponding first loss value and a second loss value under task 1, task 2, task 3 and task 4. When the target task that needs to be trained first is task 4, for the remaining training samples P in the target training sample set 33 , training sample P 34 and training samples P 35 Each training sample in has a corresponding first loss value, and there is a training sample P 33 , training sample P 34 and training samples P 35 The second loss value under Task 4, but the training sample P 33 , training sample P 34 and training samples P 35 The second loss values ​​under Task 1, Task 2 and Task 3 are all zero. Therefore, the training samples P1, P2, ..., P3 can be used to calculate the loss of the task. 32 The first loss value of each training sample in Task 1, Task 2, Task 3 and Task 4 and the second loss value of the training sample P 33 , training sample P 34 and training samples P 35 The first loss value and training sample P 33 , training sample P 34 and training samples P 35 The second loss value under Task 4 determines the target loss value of the target training sample set.

[0087] In some embodiments, determining the target loss value of the target training sample set based on the first loss value of each training sample in the target training sample set and the second loss value under each task may include: accumulating the first loss value of each training sample in the target training sample set to obtain a first accumulated loss value; accumulating the second loss value of each training sample in the target training sample set under each task to obtain a second accumulated loss value; summing or weighted summing the first accumulated loss value and the second accumulated loss value to obtain the target loss value of the target training sample set.

[0088] For example, for training samples P1, training samples P2, ..., training samples P 35 The first loss value of each training sample in is accumulated to obtain the first accumulated loss value, which is recorded as Loss num1 , and train the training samples P1, P2, ..., P 32 The second loss value of each training sample in Task 1, Task 2, Task 3 and Task 4 and the second loss value of the training sample P 33 , training sample P 34 and training samples P 35The second loss value under task 4 is accumulated to obtain the second accumulated loss value, which is recorded as Loss num2 , then the target loss value of the target training sample set is α*Loss num1 +β*Loss num2 or Loss num1 +Loss num2 , α is the first weighting coefficient, and β is the second weighting coefficient.

[0089] Step S108: Determine whether the multimodal model has converged according to the target loss value, and update the parameters of the multimodal model if the multimodal model has not converged.

[0090] In this embodiment, when the target loss value is greater than the preset loss value, it can be determined that the multimodal model has not converged. When the target loss value is less than or equal to the preset loss value, it can be determined that the multimodal model has converged. The preset loss value can be set based on actual conditions and is not specifically limited in this embodiment of the present application. It is understood that when the multimodal model has not converged, the backpropagation algorithm can be used to update the parameters of the multimodal model based on the target loss value.

[0091] After updating the parameters of the multimodal model, the process may return to executing the step of selecting a training sample set from multiple training sample sets as a target training sample set until each training sample set is selected once or the multimodal model converges, i.e., returning to executing step S101 until each training sample set is selected once or the multimodal model converges.

[0092] The multimodal model training method provided in the above embodiment calculates the total loss value of each task in the process of training the multimodal model using each training sample set, and determines the task with the largest total loss value as the target task that needs to be trained first, and then uses the remaining training samples in the training sample set to train only the target task, thereby realizing dynamic adjustment training of multiple tasks, balancing the reinforcement learning efficiency of the multimodal model for different tasks, and eliminating the need to fine-tune the trained multimodal model, thereby greatly improving the training efficiency of the multimodal model and reducing the training cost.

[0093] See also Figure 5 , Figure 5 This is a flowchart of another multimodal model training method provided in an embodiment of the present application.

[0094] like Figure 5 As shown, the multimodal model training method includes steps S201 to S212.

[0095] Step S201: Select one training sample set from multiple training sample sets as a target training sample set.

[0096] It can be understood that the specific implementation of step S201 can refer to the corresponding process in the above embodiment and will not be described in detail here.

[0097] Step S202: Select a second number of training samples from the target training sample set.

[0098] It is understandable that the specific implementation of step S202 can refer to the corresponding process in the aforementioned embodiment and will not be described in detail here.

[0099] Step S203: For each selected training sample, the training sample is encoded through the encoding layer to obtain a first embedding vector corresponding to the training text, a second embedding vector corresponding to the training image, and a third embedding vector corresponding to the question information of each task.

[0100] It is understandable that the specific implementation of step S203 can refer to the corresponding process in the aforementioned embodiment, and will not be described in detail here.

[0101] Step S204: randomly select a word in the training text as a label word for masking according to a preset probability to obtain a target text, and encode the target text through the encoding layer to obtain a fourth embedding vector.

[0102] For example, if the preset probability is 15%, then in the process of using the multimodal model to process 15 training samples in sequence, it is only necessary to perform word masking on the training text in one of the training samples. For example, a word in the training text in the 10th training sample is masked as a label word. It is understandable that the preset probability can also be 20%, and this embodiment of the present application does not make specific limitations on this. This embodiment trains the model's ability to predict unknown words by randomly selecting a word in the training text as a label word for masking according to the preset probability, enabling the model to have a deeper understanding of semantics and improving the accuracy of model training.

[0103] In some embodiments, randomly selecting a word in the training text as a label word for masking, and obtaining the target text may include: randomly selecting a word in the training text as a label word, and replacing the label word in the training text with a preset mask to mask the label word in the training text to obtain the target text. Among them, the preset mask can be set based on actual conditions, and the embodiments of the present application do not specifically limit this. For example, the preset mask is [mask], the training text A includes word A1, word A2, word A3 and word A4, and the randomly selected word is word A3, then word A3 in the training text A can be replaced with [mask] to obtain the corresponding target text, which includes word A1, word A2, [mask] and word A4.

[0104] In some embodiments, randomly selecting a word in the training text as a label word for masking to obtain the target text may include: numbering each word in the training text to obtain a number set, and setting the random number generation range of a preset random number generator to the number set to obtain a target random number generator; generating a random number using the target random number generator, and selecting the word corresponding to the random number in the training text as the label word for masking to obtain the target text. The preset random number generator can be written based on actual conditions and is not specifically limited in the embodiments of the present application.

[0105] For example, word A1, word A2, word A3 and word A4 in training text A are numbered, and word A1 is numbered 01, word A2 is numbered 02, word A3 is numbered 03 and word A4 is numbered 04, then the number set is [01, 02, 03, 04]. Therefore, the random number generation range of the preset random number generator can be set to the number set [01, 02, 03, 04] to obtain a target random number generator. The target random number generator can randomly generate any number in the number set [01, 02, 03, 04]. For example, the random number generated by the target random number generator is 02, then word A2 in training text A can be used as a label word, and word A2 in training text A can be replaced with [mask] to obtain a target text, which includes word A1, [mask], word A3 and word A4.

[0106] Step S205: Process the first embedding vector and the second embedding vector through the feature extraction sub-model to obtain a first feature vector of the training text and a second feature vector of the training image.

[0107] It is understandable that the specific implementation of step S205 can refer to the corresponding process in the aforementioned embodiment, and will not be described in detail here.

[0108] Step S206: Perform answer prediction processing on the second embedding vector and the corresponding third embedding vector through each task processing sub-model to obtain predicted answer information for each task.

[0109] It is understandable that the specific implementation of step S206 can refer to the corresponding process in the aforementioned embodiment and will not be described in detail here.

[0110] Step S207: Perform word prediction processing on the fourth vector through the word prediction sub-model to obtain predicted words, and determine the first eigenvector, the second eigenvector, the predicted answer information of each task, and the predicted words as the first processing result of the training sample.

[0111] For example, if Figure 6 As shown, the multimodal model also includes a word prediction sub-model 40, and the encoding layer 10 is respectively connected to the feature extraction sub-model 20, the multiple task processing sub-model 30, and the word prediction sub-model 40. The word prediction sub-model 40 can be LSTM (Long Short-Term Memory), N-Gram (a language model, for Chinese, it can be called Chinese Language Model (CLM)), TextCNN or BiLSTM (Bidirectional Long Short-Term Memory), etc.

[0112] Step S208: Determine the first loss value, the third loss value of each selected training sample, and the second loss value under each task based on the first processing result of each selected training sample, the true answer information of each task, and the label words.

[0113] For example, for each selected training sample, a first loss value for the training sample can be calculated based on the first eigenvector and the second eigenvector; a second loss value for the training sample under each task can be determined based on the predicted answer information and the actual answer information for each task; and a third loss value for the training sample can be calculated based on the predicted words and label words corresponding to the training sample. It is understood that the specific implementation of the first loss value and the second loss value can refer to the corresponding process in the aforementioned embodiment and will not be repeated here.

[0114] In some embodiments, calculating the third loss value of the training sample based on the predicted words and the label words corresponding to the training sample may include: determining the similarity between the predicted words corresponding to the training sample and the label words corresponding to the training sample, and determining the similarity between the predicted words corresponding to the training sample and the label words corresponding to the training sample as the third loss value of the training sample. The similarity between the predicted words corresponding to the training sample and the label words corresponding to the training sample may be cross-entropy similarity or cosine similarity.

[0115] Step S209: Determine the total loss value of each task based on the second loss value of each selected training sample under each task, and determine the task corresponding to the highest total loss value as the target task.

[0116] It can be understood that the specific implementation of step S209 can refer to the corresponding process in the aforementioned embodiment and will not be described in detail here.

[0117] Step S210: Process each remaining training sample in the target training sample set through the multimodal model to obtain a second processing result for each remaining training sample, and determine the first loss value, third loss value, and second loss value of each remaining training sample under the target task based on the second processing result of each remaining training sample.

[0118] In this embodiment, the second processing result includes the first feature vector of the training text in the remaining training samples output by the feature extraction sub-model, the second feature vector of the training image in the remaining training samples, the predicted answer information for the corresponding task output by each task processing sub-model, and the predicted word output by the word prediction sub-model. It should be noted that the specific implementation of step S210 can refer to the corresponding process in the aforementioned embodiment and is not further described here.

[0119] Step S211: Determine a target loss value for the target training sample set based on the first loss value, the third loss value, and the second loss value for each task of each training sample in the target training sample set.

[0120] In this embodiment, by introducing the third loss value of each training sample in the target training sample set, the model's ability to predict unknown words can be trained, enabling the model to have a deeper understanding of semantics and improve the accuracy of model training.

[0121] In some embodiments, the first loss value of each training sample in the target training sample set is accumulated to obtain a first accumulated loss value; the second loss value of each training sample in the target training sample set under each task is accumulated to obtain a second accumulated loss value; the third loss value of each training sample in the target training sample set is accumulated to obtain a third accumulated loss value; the first accumulated loss value, the second accumulated loss value and the third accumulated loss value are summed or weighted summed to obtain the target loss value of the target training sample set.

[0122] For example, the multiple tasks include task 1, task 2, task 3 and task 4, and the target training sample set includes 35 training samples in total. For training sample P1, training sample P2, ..., training sample P 32 Each training sample in has a corresponding first loss value, a third loss value, and a second loss value under Task 1, Task 2, Task 3, and Task 4. When the target task that needs to be trained first is Task 4, for the remaining training samples P in the target training sample set 33 , training sample P 34 and training samples P 35 Each training sample in has a corresponding first loss value and third loss value, and there is a training sample P 33 , training sample P 34and the training sample P 35 the second loss value under task 4, but the training sample P 33 , the training sample P 34 and the training sample P 35 The second loss values under task 1, task 2 and task 3 are all zero.

[0123] Therefore, the first loss value of each of the training sample P1, the training sample P2, …, the training sample P 35 is accumulated to obtain a first accumulated loss value, denoted as Loss num1 ; the second loss value of each of the training sample P1, the training sample P2, …, the training sample P 32 under task 1, task 2, task 3 and task 4, and the second loss value under task 4 of the training sample P 33 , the training sample P 34 and the training sample P 35 is accumulated to obtain a second accumulated loss value, denoted as Loss num2 ; the third loss value of each of the training sample P1, the training sample P2, …, the training sample P 35 is accumulated to obtain a third accumulated loss value, denoted as Loss num3 ; the target loss value of the target training sample set is α*Loss num1 +β*Loss num2 +γ*Loss num3 or Loss num1 +Loss num2 +Loss num3 , α is a first weighting coefficient, β is a second weighting coefficient, and γ is a third weighting coefficient, and α, β and γ can be set based on actual conditions, which are not limited in the embodiments of the present application.

[0124] Step S212, determine whether the multi-modal model converges according to the target loss value, and update the parameters of the multi-modal model when the multi-modal model does not converge.

[0125] In the embodiment, after updating the parameters of the multi-modal model, the step of selecting one training sample set from the plurality of training sample sets as the target training sample set can be returned to be executed until each training sample set is selected once or the multi-modal model converges, i.e., the step S201 is returned to be executed until each training sample set is selected once or the multi-modal model converges. It can be understood that the specific implementation of step S212 can refer to the corresponding process in the foregoing embodiments, which will not be described here.

[0126] The multimodal model training method provided in the above embodiment can train the model's ability to predict unknown words by introducing word prediction tasks, enabling the model to have a deeper understanding of semantics, improving the accuracy of model training, and eliminating the need to fine-tune the trained multimodal model, thereby greatly improving the training efficiency of the multimodal model and reducing training costs.

[0127] See also Figure 7 , Figure 7 It is a schematic block diagram of a multimodal model training device provided in an embodiment of the present application.

[0128] like Figure 7 As shown, the multimodal model training device 200 includes:

[0129] The sample set selection module 210 is configured to select one training sample set from a plurality of training sample sets as a target training sample set, wherein the first number of training samples included in the plurality of training sample sets are all different;

[0130] a sample selection module 220 configured to select a second number of training samples from the target training sample set, the first number being greater than the second number, the training samples comprising image-text pairs, question information and true answer information for each of a plurality of tasks, the image-text pairs comprising training images and training text;

[0131] The sample processing module 230 is configured to process each of the selected training samples using the multimodal model to obtain a first processing result for each of the selected training samples, wherein the first processing result includes the first feature vector of the training text and the second feature vector of the training image output by the feature extraction sub-model, and the predicted answer information corresponding to the task output by each of the task processing sub-models;

[0132] a loss value determining module 240 configured to determine a first loss value for each selected training sample and a second loss value for each task based on the first processing result of each selected training sample and the true answer information of each task;

[0133] The task determination module 250 is configured to determine the total loss value of each task based on the second loss value of each selected training sample under each task, and determine the task corresponding to the highest total loss value as the target task;

[0134] The sample processing module 230 is further configured to process each remaining training sample in the target training sample set using the multimodal model to obtain a second processing result for each remaining training sample;

[0135] The loss value determining module 240 is further configured to determine the first loss value of each of the remaining training samples and the second loss value under the target task according to the second processing result of each of the remaining training samples;

[0136] The loss value determination module 240 is further configured to determine a target loss value of the target training sample set based on the first loss value of each training sample in the target training sample set and the second loss value under each of the tasks;

[0137] a model updating module 260 configured to determine whether the multimodal model has converged according to the target loss value, and to update parameters of the multimodal model if the multimodal model has not converged;

[0138] The loop module 270 is configured to return to the step of selecting one training sample set from the multiple training sample sets as the target training sample set until each training sample set is selected once or the multimodal model converges.

[0139] In some embodiments, the sample processing module 230 is further configured to:

[0140] For each of the selected training samples, encoding the training sample through the encoding layer to obtain a first embedding vector corresponding to the training text, a second embedding vector corresponding to the training image, and a third embedding vector corresponding to the question information of each task;

[0141] Processing the first embedding vector and the second embedding vector by the feature extraction sub-model to obtain a first feature vector of the training text and a second feature vector of the training image;

[0142] Each of the task processing sub-models performs answer prediction processing on the second embedding vector and the corresponding third embedding vector to obtain predicted answer information for each task.

[0143] In some embodiments, the loss value determination module 240 is further configured to:

[0144] For each selected training sample, calculating a first loss value of the training sample according to the first eigenvector and the second eigenvector;

[0145] According to the predicted answer information and the true answer information of each task, a second loss value of the training sample under each task is determined.

[0146] In some embodiments, the loss value determination module 240 is further configured to:

[0147] Obtaining a preset label of the training sample and obtaining a preset similarity corresponding to the preset label;

[0148] determining a similarity between the first feature vector and the second feature vector;

[0149] A mean square error between the similarity and the preset similarity is calculated, and the mean square error is determined as a first loss value of the training sample.

[0150] In some embodiments, the loss value determination module 240 is further configured to:

[0151] Accumulating the first loss value of each training sample in the target training sample set to obtain a first accumulated loss value;

[0152] Accumulating the second loss value of each training sample in the target training sample set under each task to obtain a second accumulated loss value;

[0153] The first accumulated loss value and the second accumulated loss value are summed or weightedly summed to obtain a target loss value of the target training sample set.

[0154] In some embodiments, the sample processing module 230 is further configured to:

[0155] For each of the selected training samples, encoding the training sample through the encoding layer to obtain a first embedding vector corresponding to the training text, a second embedding vector corresponding to the training image, and a third embedding vector corresponding to the question information of each task;

[0156] Randomly selecting a word in the training text as a label word for masking according to a preset probability to obtain a target text, and encoding the target text through the encoding layer to obtain a fourth embedding vector;

[0157] Processing the first embedding vector and the second embedding vector by the feature extraction sub-model to obtain a first feature vector of the training text and a second feature vector of the training image;

[0158] Performing answer prediction processing on the second embedding vector and the corresponding third embedding vector through each task processing sub-model to obtain predicted answer information for each task;

[0159] Performing word prediction processing on the fourth vector by the word prediction sub-model to obtain predicted words, and determining the first feature vector, the second feature vector, the predicted answer information of each task, and the predicted words as a first processing result of the training sample;

[0160] The loss value determination module 240 is further configured to determine a first loss value and a third loss value for each selected training sample and a second loss value for each task based on the first processing result of each selected training sample, the true answer information of each task, and the label words;

[0161] The sample processing module 230 is further configured to process each remaining training sample in the target training sample set using the multimodal model to obtain a second processing result for each of the remaining training samples, and determine a first loss value, a third loss value, and a second loss value for each of the remaining training samples under the target task based on the second processing result of each of the remaining training samples;

[0162] The sample processing module 230 is further configured to determine a target loss value of the target training sample set based on the first loss value, the third loss value of each training sample in the target training sample set and the second loss value under each task.

[0163] In some embodiments, the sample processing module 230 is further configured to:

[0164] Numbering each word in the training text to obtain a number set;

[0165] Setting the random number generation range of a preset random number generator to the number set to obtain a target random number generator;

[0166] A random number is generated by the target random number generator, and a word corresponding to the random number in the training text is selected as a label word for masking to obtain a target text.

[0167] It should be noted that those skilled in the art can clearly understand that, for the convenience and conciseness of description, the specific working processes of the above-described devices and modules and units can refer to the corresponding processes in the aforementioned multimodal model training method embodiment, and will not be repeated here.

[0168] The apparatus provided in the above embodiment can be implemented in the form of a computer program. The computer program can be used in Figure 8 Run on the server shown.

[0169] See also Figure 8 , Figure 8 This is a schematic block diagram of the structure of a server provided in an embodiment of the present application.

[0170] like Figure 8As shown, the server includes a processor, a memory, and a network interface connected via a system bus, wherein the memory may include a storage medium and an internal memory.

[0171] The storage medium may store an operating system and a computer program. The computer program includes program instructions that, when executed, enable the processor to perform any multimodal model training method.

[0172] The processor is used to provide computing and control capabilities to support the operation of the entire server.

[0173] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the server to which the solution of the present application is applied. The specific server may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0174] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0175] In one embodiment, the processor is configured to execute a computer program stored in the memory to implement the following steps:

[0176] Selecting one training sample set from a plurality of training sample sets as a target training sample set, wherein the first number of training samples included in the plurality of training sample sets are all different;

[0177] selecting a second number of training samples from the target training sample set, the first number being greater than the second number, the training samples comprising image-text pairs, question information and true answer information for each of a plurality of tasks, the image-text pairs comprising training text and training images;

[0178] Processing each of the selected training samples using the multimodal model to obtain a first processing result for each of the selected training samples, the first processing result including a first feature vector of the training text and a second feature vector of the training image output by the feature extraction sub-model, and predicted answer information corresponding to the task output by each of the task processing sub-models;

[0179] Determining a first loss value for each of the selected training samples and a second loss value for each of the tasks according to the first processing result of each of the selected training samples and the true answer information of each of the tasks;

[0180] Determine the total loss value of each task according to the second loss value of each selected training sample under each task, and determine the task corresponding to the highest total loss value as the target task;

[0181] Processing each remaining training sample in the target training sample set using the multimodal model to obtain a second processing result for each remaining training sample, and determining a first loss value for each remaining training sample and a second loss value for the target task based on the second processing result for each remaining training sample and the true answer information of the target task;

[0182] Determining a target loss value for the target training sample set according to the first loss value for each training sample in the target training sample set and the second loss value under each of the tasks;

[0183] Determining whether the multimodal model has converged according to the target loss value, and updating parameters of the multimodal model if the multimodal model has not converged;

[0184] Return to the step of selecting one training sample set from the multiple training sample sets as a target training sample set until each of the training sample sets is selected once or the multimodal model converges.

[0185] In some embodiments, when the processor processes each of the selected training samples using the multimodal model to obtain a first processing result for each of the selected training samples, the processor is configured to implement:

[0186] For each of the selected training samples, encoding the training sample through the encoding layer to obtain a first embedding vector corresponding to the training text, a second embedding vector corresponding to the training image, and a third embedding vector corresponding to the question information of each task;

[0187] Processing the first embedding vector and the second embedding vector by the feature extraction sub-model to obtain a first feature vector of the training text and a second feature vector of the training image;

[0188] Each of the task processing sub-models performs answer prediction processing on the second embedding vector and the corresponding third embedding vector to obtain predicted answer information for each task.

[0189] In some embodiments, when determining the first loss value of each selected training sample and the second loss value under each task based on the first processing result of each selected training sample and the true answer information of each task, the processor is configured to implement:

[0190] For each selected training sample, calculating a first loss value of the training sample according to the first eigenvector and the second eigenvector;

[0191] According to the predicted answer information and the true answer information of each task, a second loss value of the training sample under each task is determined.

[0192] In some embodiments, when calculating the first loss value of the training sample based on the first eigenvector and the second eigenvector, the processor is configured to implement:

[0193] Obtaining a preset label of the training sample and obtaining a preset similarity corresponding to the preset label;

[0194] determining a similarity between the first feature vector and the second feature vector;

[0195] A mean square error between the similarity and the preset similarity is calculated, and the mean square error is determined as a first loss value of the training sample.

[0196] In some embodiments, when determining the target loss value of the target training sample set based on the first loss value of each training sample in the target training sample set and the second loss value under each of the tasks, the processor is configured to implement:

[0197] Accumulating the first loss value of each training sample in the target training sample set to obtain a first accumulated loss value;

[0198] Accumulating the second loss value of each training sample in the target training sample set under each task to obtain a second accumulated loss value;

[0199] The first accumulated loss value and the second accumulated loss value are summed or weightedly summed to obtain a target loss value of the target training sample set.

[0200] In some embodiments, the multimodal model further includes a word prediction sub-model, and when the processor processes each selected training sample through the multimodal model to obtain a first processing result for each selected training sample, the processor is configured to implement:

[0201] For each of the selected training samples, encoding the training sample through the encoding layer to obtain a first embedding vector corresponding to the training text, a second embedding vector corresponding to the training image, and a third embedding vector corresponding to the question information of each task;

[0202] Randomly selecting a word in the training text as a label word for masking according to a preset probability to obtain a target text, and encoding the target text through the encoding layer to obtain a fourth embedding vector;

[0203] Processing the first embedding vector and the second embedding vector by the feature extraction sub-model to obtain a first feature vector of the training text and a second feature vector of the training image;

[0204] Performing answer prediction processing on the second embedding vector and the corresponding third embedding vector through each task processing sub-model to obtain predicted answer information for each task;

[0205] Performing word prediction processing on the fourth vector by the word prediction sub-model to obtain predicted words, and determining the first feature vector, the second feature vector, the predicted answer information of each task, and the predicted words as a first processing result of the training sample;

[0206] When the processor determines the first loss value of each selected training sample and the second loss value under each task based on the first processing result of each selected training sample and the true answer information of each task, it is configured to implement:

[0207] Determine a first loss value, a third loss value for each of the selected training samples, and a second loss value for each of the tasks according to the first processing result of each of the selected training samples, the true answer information of each of the tasks, and the label words;

[0208] The processor is configured to implement, when processing each remaining training sample in the target training sample set by the multimodal model to obtain a second processing result for each of the remaining training samples, and determining, based on the second processing result of each of the remaining training samples, a first loss value for each of the remaining training samples and a second loss value under the target task:

[0209] Processing each remaining training sample in the target training sample set by the multimodal model to obtain a second processing result for each remaining training sample, and determining a first loss value, a third loss value, and a second loss value for each remaining training sample under the target task based on the second processing result for each remaining training sample;

[0210] When the processor determines the target loss value of the target training sample set based on the first loss value of each training sample in the target training sample set and the second loss value under each of the tasks, it is configured to implement:

[0211] Determine a target loss value for the target training sample set according to the first loss value, the third loss value, and the second loss value under each of the tasks for each training sample in the target training sample set.

[0212] In some embodiments, when randomly selecting a word in the training text as a label word for masking to obtain the target text, the processor is configured to implement:

[0213] Numbering each word in the training text to obtain a number set;

[0214] Setting the random number generation range of a preset random number generator to the number set to obtain a target random number generator;

[0215] A random number is generated by the target random number generator, and a word corresponding to the random number in the training text is selected as a label word for masking to obtain a target text.

[0216] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the server described above can refer to the corresponding process in the aforementioned multimodal model training method embodiment, and will not be repeated here.

[0217] Through the description of the above implementation methods, it can be seen that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product can be stored in a storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a server (which can be a personal computer, a server, or a network server, etc.) to execute the methods described in each embodiment of the present application or certain parts of the embodiments.

[0218] An embodiment of the present application also provides a storage medium, on which a computer program is stored. The computer program includes program instructions. The method implemented when the program instructions are executed can be referred to the various embodiments of the multimodal model training method of the present application.

[0219] The storage medium may be volatile or non-volatile. The storage medium may be an internal storage unit of the server described in the aforementioned embodiment, such as a hard disk or memory of the server. The storage medium may also be an external storage server of the server, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the server.

[0220] Furthermore, the storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the blockchain node, etc.

[0221] The blockchain referred to in this application is a new application model for computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Blockchain is essentially a decentralized database, a series of data blocks generated using cryptographic methods. Each data block contains information about a batch of network transactions, which is used to verify the validity of this information (to prevent counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, the platform product service layer, and the application service layer.

[0222] It should be understood that the terms used in this specification are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in this specification and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0223] It should also be understood that the term "and / or" used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system that includes a series of elements includes not only those elements, but also other elements that are not explicitly listed, or also includes elements that are inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "including a..." does not exclude the presence of other identical elements in the process, method, article or system that includes the element.

[0224] The serial numbers of the embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments. The above description is only a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A multimodal model training method, characterized in that: The multimodal model includes an encoding layer, a feature extraction sub-model, and multiple task processing sub-models. The method includes: Selecting one training sample set from a plurality of training sample sets as a target training sample set, wherein the first number of training samples included in the plurality of training sample sets are all different; selecting a second number of training samples from the target training sample set, the first number being greater than the second number, the training samples comprising image-text pairs, question information and true answer information for each of a plurality of tasks, the image-text pairs comprising training text and training images; Processing each of the selected training samples using the multimodal model to obtain a first processing result for each of the selected training samples, the first processing result including a first feature vector of the training text and a second feature vector of the training image output by the feature extraction sub-model, and predicted answer information corresponding to the task output by each of the task processing sub-models; Determining a first loss value for each of the selected training samples and a second loss value for each of the tasks according to the first processing result of each of the selected training samples and the true answer information of each of the tasks; Determine the total loss value of each task according to the second loss value of each selected training sample under each task, and determine the task corresponding to the highest total loss value as the target task; Processing each remaining training sample in the target training sample set using the multimodal model to obtain a second processing result for each remaining training sample, and determining a first loss value for each remaining training sample and a second loss value for the target task based on the second processing result for each remaining training sample and the true answer information of the target task; Determining a target loss value for the target training sample set according to the first loss value for each training sample in the target training sample set and the second loss value under each of the tasks; Determining whether the multimodal model has converged according to the target loss value, and updating parameters of the multimodal model if the multimodal model has not converged; Return to the step of selecting one training sample set from the multiple training sample sets as a target training sample set until each of the training sample sets is selected once or the multimodal model converges.

2. The multimodal model training method according to claim 1, characterized in that: The step of processing each of the selected training samples by the multimodal model to obtain a first processing result of each of the selected training samples includes: For each of the selected training samples, encoding the training sample through the encoding layer to obtain a first embedding vector corresponding to the training text, a second embedding vector corresponding to the training image, and a third embedding vector corresponding to the question information of each task; Processing the first embedding vector and the second embedding vector by the feature extraction sub-model to obtain a first feature vector of the training text and a second feature vector of the training image; Each of the task processing sub-models performs answer prediction processing on the second embedding vector and the corresponding third embedding vector to obtain predicted answer information for each task.

3. The multimodal model training method according to claim 1, characterized in that: The determining, based on the first processing result of each selected training sample and the true answer information of each task, a first loss value of each selected training sample and a second loss value under each task, includes: For each selected training sample, calculating a first loss value of the training sample according to the first eigenvector and the second eigenvector; According to the predicted answer information and the true answer information of each task, a second loss value of the training sample under each task is determined.

4. The multimodal model training method according to claim 3, characterized in that: Calculating a first loss value of the training sample according to the first eigenvector and the second eigenvector includes: Obtaining a preset label of the training sample and obtaining a preset similarity corresponding to the preset label; determining a similarity between the first feature vector and the second feature vector; A mean square error between the similarity and the preset similarity is calculated, and the mean square error is determined as a first loss value of the training sample.

5. The multimodal model training method according to any one of claims 1 to 4, characterized in that: Determining a target loss value of the target training sample set according to the first loss value of each training sample in the target training sample set and the second loss value under each of the tasks includes: Accumulating the first loss value of each training sample in the target training sample set to obtain a first accumulated loss value; Accumulating the second loss value of each training sample in the target training sample set under each task to obtain a second accumulated loss value; The first accumulated loss value and the second accumulated loss value are summed or weightedly summed to obtain a target loss value of the target training sample set.

6. The multimodal model training method according to claim 1, characterized in that: The multimodal model further includes a word prediction sub-model, and the multimodal model is used to process each of the selected training samples to obtain a first processing result of each of the selected training samples, including: For each of the selected training samples, encoding the training sample through the encoding layer to obtain a first embedding vector corresponding to the training text, a second embedding vector corresponding to the training image, and a third embedding vector corresponding to the question information of each task; Randomly selecting a word in the training text as a label word for masking according to a preset probability to obtain a target text, and encoding the target text through the encoding layer to obtain a fourth embedding vector; Processing the first embedding vector and the second embedding vector by the feature extraction sub-model to obtain a first feature vector of the training text and a second feature vector of the training image; Performing answer prediction processing on the second embedding vector and the corresponding third embedding vector through each task processing sub-model to obtain predicted answer information for each task; Performing word prediction processing on the fourth vector by the word prediction sub-model to obtain predicted words, and determining the first feature vector, the second feature vector, the predicted answer information of each task, and the predicted words as a first processing result of the training sample; The determining, based on the first processing result of each selected training sample and the true answer information of each task, a first loss value of each selected training sample and a second loss value under each task, includes: Determine a first loss value, a third loss value for each of the selected training samples, and a second loss value for each of the tasks according to the first processing result of each of the selected training samples, the true answer information of each of the tasks, and the label words; The step of processing each remaining training sample in the target training sample set by the multimodal model to obtain a second processing result for each remaining training sample, and determining a first loss value for each remaining training sample and a second loss value under the target task based on the second processing result for each remaining training sample, includes: Processing each remaining training sample in the target training sample set by the multimodal model to obtain a second processing result for each remaining training sample, and determining a first loss value, a third loss value, and a second loss value for each remaining training sample under the target task based on the second processing result for each remaining training sample; Determining a target loss value of the target training sample set according to the first loss value of each training sample in the target training sample set and the second loss value under each of the tasks includes: Determine a target loss value for the target training sample set according to the first loss value, the third loss value, and the second loss value under each of the tasks for each training sample in the target training sample set.

7. The multimodal model training method according to claim 6, characterized in that: The randomly selecting a word in the training text as a label word for masking to obtain a target text includes: Numbering each word in the training text to obtain a number set; Setting the random number generation range of a preset random number generator to the number set to obtain a target random number generator; A random number is generated by the target random number generator, and a word corresponding to the random number in the training text is selected as a label word for masking to obtain a target text.

8. A multimodal model training device, characterized in that: The multimodal model includes an encoding layer, a feature extraction sub-model and a plurality of task processing sub-models, and the multimodal model training device includes: A sample set selection module is configured to select one training sample set from a plurality of training sample sets as a target training sample set, wherein the first number of training samples included in the plurality of training sample sets are all different; a sample selection module configured to select a second number of training samples from the target training sample set, the first number being greater than the second number, the training samples comprising image-text pairs, question information and true answer information for each of a plurality of tasks, the image-text pairs comprising training images and training text; a sample processing module configured to process each of the selected training samples using the multimodal model to obtain a first processing result for each of the selected training samples, the first processing result comprising a first feature vector of the training text and a second feature vector of the training image output by the feature extraction sub-model, and predicted answer information corresponding to the task output by each of the task processing sub-models; a loss value determining module, configured to determine a first loss value for each selected training sample and a second loss value for each task based on the first processing result of each selected training sample and the true answer information of each task; a task determination module configured to determine a total loss value for each of the tasks based on the second loss value of each of the selected training samples under each of the tasks, and determine the task corresponding to the highest total loss value as the target task; The sample processing module is further configured to process each remaining training sample in the target training sample set using the multimodal model to obtain a second processing result for each remaining training sample; The loss value determination module is further configured to determine a first loss value for each of the remaining training samples and a second loss value for the target task based on the second processing result of each of the remaining training samples; The loss value determination module is further configured to determine a target loss value of the target training sample set based on the first loss value of each training sample in the target training sample set and the second loss value under each of the tasks; a model updating module configured to determine whether the multimodal model has converged according to the target loss value, and to update parameters of the multimodal model if the multimodal model has not converged; The loop module is configured to return to the step of selecting one training sample set from the multiple training sample sets as the target training sample set until each of the training sample sets is selected once or the multimodal model converges.

9. A server, characterized in that: The server includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, it implements the multimodal model training method according to any one of claims 1 to 7.

10. A storage medium for computer-readable storage, characterized in that: The storage medium stores a computer program, wherein when the computer program is executed by the processor, the multimodal model training method according to any one of claims 1 to 7 is implemented.

Images