Model training method and device, electronic equipment and readable storage medium

Abstract

The application discloses a model training method and device, electronic equipment and a readable storage medium, comprising: obtaining an initial model and a sample data set; for a first training sample, based on non-masked image information in a sample text and a sample image, performing image mask reconstruction on the sample image to obtain reconstructed image information; for a second training sample, based on non-masked text information in the sample image and the sample text, performing text mask reconstruction on the sample text to obtain reconstructed text information; and based on first similarity information and second similarity information, training to obtain a target multi-modal model. This method simultaneously uses image mask reconstruction and text mask reconstruction as training tasks, can avoid over-reliance on single modal features during training, leading to overfitting of the target multi-modal model, and can improve model performance. This method can also simultaneously train the representation of the image and the representation of the text, reducing the cost of model training.

Description

Technical Field

[0001] This application relates to the field of model training technology, specifically to a model training method, apparatus, electronic device, and readable storage medium. Background Technology

[0002] As the requirements for model accuracy gradually increase, multimodal models, with their diverse inputs and accurate outputs, are gaining increasing attention from users and R&D personnel.

[0003] To reduce the training cost of multimodal models in downstream tasks, methods such as mask reconstruction and contrastive learning are often used to pre-train multimodal models, and then further adjustments are made based on specific downstream tasks. However, the multimodal models obtained by current pre-training methods usually have poor performance. Summary of the Invention

[0004] This application provides a model training method, apparatus, electronic device, and readable storage medium, aiming to solve the technical problem of poor performance of multimodal models trained by existing model training methods.

[0005] Firstly, this application provides a model training method, including:

[0006] Obtain an initial model and a preset sample dataset, wherein the sample dataset includes multiple sets of training samples, and each set of training samples includes a sample image and sample text used to describe the sample image;

[0007] For the first training sample in each group of training samples, based on the sample text and the non-masked image information in the sample image, image mask reconstruction is performed on the sample image to obtain the reconstructed image information corresponding to the masked image information in the sample image;

[0008] For the second training sample in each group of training samples other than the first training sample, based on the sample image and the non-masked text information in the sample text, text mask reconstruction is performed on the sample text to obtain the reconstructed text information corresponding to the masked text information in the sample text.

[0009] Based on the first similarity information between the reconstructed image information and the masked image information, and the second similarity information between the reconstructed text information and the masked text information, the initial model is trained to obtain the target multimodal model.

[0010] In one possible implementation of this application, after obtaining the initial model and the preset sample dataset, the method further includes:

[0011] The training samples from each group are input into the initial model, and the training samples from each group are divided into the first training sample and the second training sample according to the preset division probability through the mask layer in the initial model.

[0012] In one possible implementation of this application, the partitioning probability is obtained based on the average sharpness of each of the sample images and / or the average text length of each of the sample texts.

[0013] In one possible implementation of this application, the step of performing image mask reconstruction on the sample image based on the sample text and the non-masked image information in the sample image to obtain the reconstructed image information corresponding to the masked image information in the sample image for the first training sample in each group of training samples includes:

[0014] For the first training sample in each group of training samples, the sample image is masked through the masking layer in the initial model to obtain the non-masked image information in the sample image;

[0015] The first image features of the non-masked image information are extracted through the first feature extraction layer in the initial model;

[0016] The first text features of the sample text are extracted through the second feature extraction layer in the initial model;

[0017] The first image feature and the first text feature are fused through the feature fusion layer in the initial model to obtain the first fused feature;

[0018] The reconstructed image information corresponding to the masked image information in the sample image is obtained by predicting the first prediction layer in the initial model based on the first fusion feature.

[0019] In one possible implementation of this application, for the second training sample in each group of training samples other than the first training sample, text mask reconstruction is performed on the sample text based on the sample image and the non-masked text information in the sample text to obtain the reconstructed text information corresponding to the masked text information in the sample text, including:

[0020] For the second training sample in each group of training samples, the sample text is masked through the masking layer in the initial model to obtain the non-masked text information in the sample text;

[0021] The second image features of the sample image are extracted through the first feature extraction layer in the initial model;

[0022] The second text features of the non-masked text information are extracted through the second feature extraction layer in the initial model;

[0023] The second image feature and the second text feature are fused through the feature fusion layer in the initial model to obtain the second fused feature;

[0024] The reconstructed text information corresponding to the masked text information in the sample text is obtained by predicting the second prediction layer in the initial model based on the second fusion feature.

[0025] In one possible implementation of this application, training the initial model based on first similarity information between the reconstructed image information and the masked image information, and second similarity information between the reconstructed text information and the masked text information to obtain a target multimodal model includes:

[0026] The first similarity information and the second similarity information are input into the preset optimizer in the initial model to optimize the model parameters of the initial model, and the optimized model parameters and the model to be adjusted composed of the optimized model parameters are output.

[0027] The model to be adjusted is fine-tuned to obtain the target multimodal model.

[0028] In one possible implementation of this application, after training the initial model based on the first similarity information between the reconstructed image information and the masked image information, and the second similarity information between the reconstructed text information and the masked text information to obtain the target multimodal model, the method further includes:

[0029] In response to a target user's trigger command for target social information in a social network, the target image in the target social information is input into the first feature extraction layer of the target multimodal model, and the target text in the target social information is input into the second feature extraction layer of the target multimodal model to obtain image features in the target image and text features in the target text;

[0030] The image features and text features are input into the feature fusion layer of the target multimodal model to obtain fused features;

[0031] Similar features with a similarity greater than a preset similarity threshold to the fused features are obtained from a preset feature database;

[0032] The source information of the similar features is set as the recommended social information corresponding to the target user.

[0033] Secondly, this application provides a model training apparatus, comprising:

[0034] The acquisition unit is used to acquire an initial model and a preset sample dataset, wherein the sample dataset includes multiple sets of training samples, and each set of training samples includes a sample image and sample text used to describe the sample image;

[0035] The image mask reconstruction unit is used to perform image mask reconstruction on the first training sample in each group of training samples, based on the sample text and the non-masked image information in the sample image, to obtain the reconstructed image information corresponding to the masked image information in the sample image.

[0036] The text mask reconstruction unit is used to perform text mask reconstruction on the second training sample other than the first training sample in each group of training samples, based on the sample image and the non-masked text information in the sample text, to obtain the reconstructed text information corresponding to the masked text information in the sample text.

[0037] The training unit is used to train the initial model based on the first similarity information between the reconstructed image information and the masked image information, and the second similarity information between the reconstructed text information and the masked text information, to obtain the target multimodal model.

[0038] In one possible implementation of this application, the acquiring unit is further configured to:

[0039] The training samples from each group are input into the initial model, and the training samples from each group are divided into the first training sample and the second training sample according to the preset division probability through the mask layer in the initial model.

[0040] In one possible implementation of this application, the partitioning probability is obtained based on the average sharpness of each of the sample images and / or the average text length of each of the sample texts.

[0041] In one possible implementation of this application, the image mask reconstruction unit is further used for:

[0042] For the first training sample in each group of training samples, the sample image is masked through the masking layer in the initial model to obtain the non-masked image information in the sample image;

[0043] The first image features of the non-masked image information are extracted through the first feature extraction layer in the initial model;

[0044] The first text features of the sample text are extracted through the second feature extraction layer in the initial model;

[0045] The first image feature and the first text feature are fused through the feature fusion layer in the initial model to obtain the first fused feature;

[0046] The reconstructed image information corresponding to the masked image information in the sample image is obtained by predicting the first prediction layer in the initial model based on the first fusion feature.

[0047] In one possible implementation of this application, the text mask reconstruction unit is further used for:

[0048] For the second training sample in each group of training samples, the sample text is masked through the masking layer in the initial model to obtain the non-masked text information in the sample text;

[0049] The second image features of the sample image are extracted through the first feature extraction layer in the initial model;

[0050] The second text features of the non-masked text information are extracted through the second feature extraction layer in the initial model;

[0051] The second image feature and the second text feature are fused through the feature fusion layer in the initial model to obtain the second fused feature;

[0052] The reconstructed text information corresponding to the masked text information in the sample text is obtained by predicting the second prediction layer in the initial model based on the second fusion feature.

[0053] In one possible implementation of this application, the training unit is further used for:

[0054] The first similarity information and the second similarity information are input into the preset optimizer in the initial model to optimize the model parameters of the initial model, and the optimized model parameters and the model to be adjusted composed of the optimized model parameters are output.

[0055] The model to be adjusted is fine-tuned to obtain the target multimodal model.

[0056] In one possible implementation of this application, the training unit is further used for:

[0057] In response to a target user's trigger command for target social information in a social network, the target image in the target social information is input into the first feature extraction layer of the target multimodal model, and the target text in the target social information is input into the second feature extraction layer of the target multimodal model to obtain image features in the target image and text features in the target text;

[0058] The image features and text features are input into the feature fusion layer of the target multimodal model to obtain fused features;

[0059] Similar features with a similarity greater than a preset similarity threshold to the fused features are obtained from a preset feature database;

[0060] The source information of the similar features is set as the recommended social information corresponding to the target user.

[0061] Thirdly, this application also provides an electronic device, which includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor calls the computer program in the memory, it executes the steps in any of the model training methods provided in this application.

[0062] Fourthly, this application also provides a readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the model training methods provided in this application.

[0063] In summary, the model training method provided in this application includes: acquiring an initial model and a preset sample dataset, wherein the sample dataset includes multiple sets of training samples, each set of training samples including a sample image and sample text describing the sample image; for a first training sample in each set of training samples, performing image mask reconstruction on the sample image based on the sample text and the non-masked image information in the sample image to obtain reconstructed image information corresponding to the masked image information in the sample image; for a second training sample in each set of training samples other than the first training sample, performing text mask reconstruction on the sample text based on the sample image and the non-masked text information in the sample text to obtain reconstructed text information corresponding to the masked text information in the sample text; and training the initial model based on a first similarity information between the reconstructed image information and the masked image information, and a second similarity information between the reconstructed text information and the masked text information to obtain a target multimodal model.

[0064] As can be seen, the model training method provided in this application simultaneously employs image mask reconstruction and text mask reconstruction as training tasks. Therefore, it can effectively avoid over-reliance on features of a single modality during training, which could lead to overfitting of the trained multimodal model and improve model performance. Furthermore, this method can train both image and text representations simultaneously, thus reducing the cost of model training. Attached Figure Description

[0065] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0066] Figure 1 This is a schematic diagram illustrating an application scenario of the model training method provided in the embodiments of this application;

[0067] Figure 2 This is a flowchart illustrating a model training method provided in an embodiment of this application;

[0068] Figure 3 This is a schematic diagram of an initial model provided in an embodiment of this application;

[0069] Figure 4 This is a schematic diagram of an image mask reconstruction process provided in an embodiment of this application;

[0070] Figure 5 This is a schematic diagram of a text mask reconstruction process provided in an embodiment of this application;

[0071] Figure 6 This is a schematic diagram of a process for obtaining a target multimodal model provided in an embodiment of this application;

[0072] Figure 7 This is another flowchart illustrating the model training method provided in the embodiments of this application;

[0073] Figure 8 This is a schematic diagram of an embodiment of the model training device provided in this application.

[0074] Figure 9 This is a schematic diagram of the structure of an embodiment of the electronic device provided in this application;

[0075] Figure 10 This is a schematic diagram of a social network provided in the embodiments of this application. Detailed Implementation

[0076] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0077] In the description of the embodiments of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, features defined with "first" and "second" may explicitly or implicitly include one or more of the stated features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0078] To enable any person skilled in the art to implement and use this application, the following description is provided. In this description, details are set forth for purposes of explanation. It should be understood that those skilled in the art will recognize that this application can be implemented without using these specific details. In other instances, well-known processes will not be described in detail to avoid obscuring the description of the embodiments of this application with unnecessary detail. Therefore, this application is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in the embodiments of this application.

[0079] This application provides a model training method, apparatus, electronic device, and readable storage medium. The model training apparatus can be integrated into an electronic device, which may be a server or a terminal, etc.

[0080] The execution subject of the model training method in this application embodiment can be the model training device provided in this application embodiment, or different types of electronic devices such as server equipment, physical host, or special effects production personnel equipment (UserEquipment, UE) that integrate the model training device. The model training device can be implemented in hardware or software. The UE can specifically be a terminal device such as a smartphone, tablet computer, laptop computer, handheld computer, desktop computer, or personal digital assistant (PDA).

[0081] The electronic device can operate independently or in a cluster.

[0082] See Figure 1 , Figure 1 This is a schematic diagram of a model training system provided in an embodiment of this application. The model training system may include an electronic device 100, which integrates a model training apparatus.

[0083] In addition, such as Figure 1 As shown, the model training system may also include a memory 200 for storing data, such as text data.

[0084] It should be noted that, Figure 1 The schematic diagram of the model training system shown is merely an example. The model training system and scenario described in this application are for the purpose of more clearly illustrating the technical solutions of this application and do not constitute a limitation on the technical solutions provided in this application. As those skilled in the art will know, with the evolution of model training systems and the emergence of new business scenarios, the technical solutions provided in this invention are also applicable to similar technical problems.

[0085] To facilitate understanding, the application background of this application will be explained first, but this should not be construed as an admission of prior art:

[0086] Existing cross-modal pre-training algorithms can be divided into two main categories. One category is based on contrastive learning. Given an image and text combination, the image is input into an image network to obtain its corresponding image representation, and the text is input into a text network to obtain its corresponding text representation. Paired image-text combinations are considered positive samples, while unpaired image-text combinations are considered negative samples. The network can then be trained using common methods like cross-entropy loss. The other category is based on masking reconstruction algorithms. Given an input image or text, these algorithms mask part of the input content before inputting it into the network. The network is required to recover the masked content using nearby elements. The network can then be trained using methods such as squared loss functions.

[0087] For contrastive learning schemes, since they need to sample negative samples for training within the same data stream (i.e., multiple sets of image-text pairs, often with more than 10240 image-text pairs), their training scale is limited by the size of the graphics card's memory. At the same time, using a larger data stream or a graphics card with more memory can often achieve better results. Therefore, such algorithms can only achieve leading performance on some devices.

[0088] For mask reconstruction schemes, since training is usually done using only a single modality, that is, masking the text for text content or masking part of the image for image content, if the main content of the image or text is masked, the network will be unable to recover the main content using the adjacent content. In severe cases, this will lead to overfitting of the network and affect the performance of downstream tasks.

[0089] Therefore, in order to solve the above problems, this application provides the following model training method for training multimodal models.

[0090] The following describes the model training method provided in this application embodiment. In this application embodiment, an electronic device is used as the execution subject. For simplicity and ease of description, the execution subject will be omitted in subsequent method embodiments. The model training method includes: obtaining an initial model and a preset sample dataset, wherein the sample dataset includes multiple sets of training samples, each set of training samples includes a sample image and sample text used to describe the sample image; for a first training sample in each set of training samples, image mask reconstruction is performed on the sample image based on the sample text and the non-masked image information in the sample image to obtain reconstructed image information corresponding to the masked image information in the sample image; for a second training sample in each set of training samples other than the first training sample, text mask reconstruction is performed on the sample text based on the sample image and the non-masked text information in the sample text to obtain reconstructed text information corresponding to the masked text information in the sample text; the initial model is trained based on the first similarity information between the reconstructed image information and the masked image information, and the second similarity information between the reconstructed text information and the masked text information to obtain a target multimodal model.

[0091] Reference Figure 2 , Figure 2 This is a flowchart illustrating a model training method provided in an embodiment of this application. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here. Specifically, the model training method may include the following steps 201-204, wherein:

[0092] 201. Obtain the initial model and the preset sample dataset, wherein the sample dataset includes multiple sets of training samples, and each set of training samples includes a sample image and sample text used to describe the sample image.

[0093] The initial model can refer to the multimodal model to be trained. It should be noted that, unless otherwise stated in the embodiments of this application, training the initial model is considered to be pre-training the initial model. Subsequently, fine-tuning and other operations can be performed on the trained initial model to obtain the multimodal model for practical application.

[0094] The multimodal model to be trained in this embodiment can refer to a multimodal model that takes images and text as input. In this case, the initial model can refer to the multimodal model to be trained, which takes images and text as input. For example, a vision-language pre-training model can be used as the initial model. For instance, an initial model can be constructed based on the ViT (Vision Transformer) model and the BERT model, such as using the Clip model or the ViT-BERT model as the initial model. (See reference...) Figure 3 , Figure 3 An initial model 301 is shown, which includes an image neural network 302, a text neural network 303, and a fusion network 304. The image neural network 302 is used to extract image features of the input image and can be composed of the feature extraction layer in the ViT model. The text neural network 303 is used to extract text features of the input text and can be composed of the feature extraction layer in the BERT model. The fusion network 304 is used to fuse image features and text features. The fusion method used by the fusion network 304 can be a commonly used multimodal feature fusion method such as attention.

[0095] Since the multimodal model to be trained in this embodiment uses images and text as input, the training dataset needs to contain multiple sets of training samples, each consisting of a sample image and sample text. It should be noted that in this embodiment, for each set of training samples, the sample text describes the image content of the sample image; that is, the content described by the sample text is the same as the content described by the sample image. This allows the initial model to learn the description in the corresponding modality, thereby strengthening the alignment ability between different modalities (e.g., adjusting...). Figure 3 The network parameters of the fusion network 304 are adjusted to improve the accuracy of image and text feature extraction (e.g., by adjusting the network parameters of the fusion network 304). Figure 3 By combining the network parameters of the image neural network 302 and the text neural network 303, the performance of the initial model is effectively improved, resulting in a multimodal model that can accurately acquire fused features.

[0096] To make it easier to understand, here is a specific example to illustrate the training samples: For example, when constructing training samples, an image of a model wearing a floral dress and other clothing can be used as a sample image, and the text "floral dress outfit" can be used as the sample text corresponding to the sample image to construct a set of training samples. Images containing animals and the names of the corresponding animals can be used as sample images and sample text in a set of training samples, respectively. The same applies to other training samples.

[0097] 202. For the first training sample in each group of training samples, based on the sample text and the non-masked image information in the sample image, perform image mask reconstruction on the sample image to obtain the reconstructed image information corresponding to the masked image information in the sample image.

[0098] The first training sample refers to the training sample used for image masked vision modeling (MVM) as the training task.

[0099] Non-masked image information in a sample image refers to the image information outside the masked area in the sample image, i.e., the image information that is not masked.

[0100] The masked image information in a sample image refers to the masked image information contained within the masked region of the sample image.

[0101] Reconstructing image information refers to reconstructing the image information of the masked region predicted by the image mask.

[0102] For example, the first training sample may be selected from the training samples by the staff training the model. Alternatively, it may be randomly selected from the training samples by the initial model, as explained below.

[0103] In this embodiment, to reduce the number of training samples required when training the initial model and to avoid overfitting of the trained initial model, a mask reconstruction method is used to train the initial samples. It is understood that since different masks can be applied multiple times to the same training sample, the number of training samples required can be effectively reduced. For example, a set of training samples can be copied multiple times to obtain multiple training samples, which can then be trained using a single data stream, thus avoiding limitations imposed by device size and effectively solving the shortcomings of contrastive learning mentioned above. An explanation of overfitting is provided below and will not be elaborated upon here.

[0104] 203. For the second training sample in each group of training samples other than the first training sample, based on the sample image and the non-masked text information in the sample text, perform text mask reconstruction on the sample text to obtain the reconstructed text information corresponding to the masked text information in the sample text.

[0105] The second training sample refers to the training sample used for the training task of Masked Language Modeling (MLM).

[0106] The unmasked text information in the sample text refers to the text segments that are not masked in the sample text. For example, when the sample text is "floral dress outfit" and the masked sample text is "floral XXX outfit", the unmasked text information refers to "floral" and "outfit", where XXX refers to the mask.

[0107] The masked text information in the sample text refers to the text fragments that are masked in the sample text. For example, in the example above, the masked text information is "dress".

[0108] Reconstructing text information refers to reconstructing the predicted masked text fragments through text masking.

[0109] For example, the second training sample can be selected from the training sample by the staff training the model, such as using all training texts other than the already selected second training text as the second training text. Alternatively, it can be randomly selected from the training sample by the initial model, as explained below.

[0110] 204. Based on the first similarity information between the reconstructed image information and the masked image information, and the second similarity information between the reconstructed text information and the masked text information, the initial model is trained to obtain the target multimodal model.

[0111] The target multimodal model refers to the pre-trained initial model.

[0112] This application does not limit the method for obtaining similarity information. For example, commonly used distance calculation methods such as Manhattan distance and Euclidean distance can be used as methods for obtaining similarity information. The calculated distance can be used as similarity information, and then the distance can be used as a loss function to train the initial model and obtain a trained target multimodal model.

[0113] The method in steps 201-204 employs both image mask reconstruction and text mask reconstruction as training tasks when training the initial model. This effectively avoids overfitting in the target multimodal model that occurs when using only single-modal mask reconstruction as the training task. For clarity, consider this example: Suppose only image mask reconstruction is used as the initial model's training task. When using an image containing an animal as a sample image, if the mask obscures the animal, the initial model will rely too heavily on the sample text during reconstruction, easily leading to overfitting of the target multimodal model and impacting the performance of downstream tasks. The same applies when only text mask reconstruction is used as the initial model's training task.

[0114] Furthermore, the method in steps 201-204 does not require separate training of image representations and text representations during training, but can train image representations and text representations simultaneously, thus reducing training costs.

[0115] For ease of understanding, the following example illustrates a practical application process for steps 201-204:

[0116] (A) The staff training the model collects the sample dataset and randomly divides the first training sample and the second training sample from the multiple sets of training samples contained in the sample dataset, and labels them.

[0117] (B) Input each group of training samples into the initial model in sequence. The initial model recognizes the input training samples according to the labels of the training samples. If the input is the first training sample, then perform image mask reconstruction. If the input is the second training sample, then perform text mask reconstruction.

[0118] (C) Based on the similarity between the reconstructed information and the masked information, the initial model is trained to obtain a trained target multimodal model.

[0119] In summary, the model training method provided in this application includes: acquiring an initial model and a preset sample dataset, wherein the sample dataset includes multiple sets of training samples, each set of training samples including a sample image and sample text describing the sample image; for a first training sample in each set of training samples, performing image mask reconstruction on the sample image based on the sample text and the non-masked image information in the sample image to obtain reconstructed image information corresponding to the masked image information in the sample image; for a second training sample in each set of training samples other than the first training sample, performing text mask reconstruction on the sample text based on the sample image and the non-masked text information in the sample text to obtain reconstructed text information corresponding to the masked text information in the sample text; and training the initial model based on a first similarity information between the reconstructed image information and the masked image information, and a second similarity information between the reconstructed text information and the masked text information to obtain a target multimodal model.

[0120] As can be seen, the model training method provided in this application simultaneously employs image mask reconstruction and text mask reconstruction as training tasks. Therefore, it can effectively avoid over-reliance on features of a single modality during training, which could lead to overfitting of the trained multimodal model and improve model performance. Furthermore, this method can train both image and text representations simultaneously, thus reducing the cost of model training.

[0121] To improve the efficiency of model training and reduce manual costs, the initial model can be randomly divided into first and second training samples. At this point, after the step "Obtain the initial model and the preset sample dataset", the following steps are also included:

[0122] The training samples from each group are input into the initial model, and the training samples from each group are divided into the first training sample and the second training sample according to the preset division probability through the mask layer in the initial model.

[0123] by Figure 3 Taking the initial model 301 as an example, the initial model 301 also includes a masking network 305. When receiving input training samples, the masking network 305 can randomly divide the training samples into a first training sample or a second training sample according to a preset division probability, and then perform mask reconstruction. The preset division probability can be set by the personnel training the model; for example, it can be set to 50%, meaning the masking network 305 has a 50% probability of classifying the input training sample as the first training sample for image mask reconstruction and a 50% probability of classifying it as the second training sample for text mask reconstruction. Alternatively, the division probability can be other values, which are not limited in this embodiment. For example, when the personnel believe that the collected sample image has low clarity and / or the sample text is long, which may lead to over-reliance on the sample text representation during training, the input training sample can be classified as the second training sample with a higher division probability, performing more text mask reconstruction. The same applies when the sample image has high clarity and / or the sample text is short. Based on this judgment principle, electronic devices can also automatically determine the segmentation probability according to the average sharpness of the sample images and the average text length of the sample text, that is, "the segmentation probability is obtained based on the average sharpness of each of the sample images and / or the average text length of each of the sample texts". For example, the electronic device can compare the average sharpness and / or the average text length with the corresponding thresholds to determine whether the average sharpness is low and whether the average text length is long.

[0124] In some embodiments, step 202 may be performed by Figure 4 The implementation method, namely, the step "for the first training sample in each group of training samples, based on the sample text and the non-masked image information in the sample image, perform image mask reconstruction on the sample image to obtain the reconstructed image information corresponding to the masked image information in the sample image", includes:

[0125] 401. For the first training sample in each group of training samples, the sample image is masked through the mask layer in the initial model to obtain the non-masked image information in the sample image.

[0126] In this application, the mask layer can be used for both image masking and text masking. For example, the mask layer can contain image mask sublayers and text mask sublayers, with image masking performed through the image mask sublayer and text masking performed through the text mask sublayer.

[0127] This application does not limit the image masking method used when performing image masking in the masking layer. For example, the masking layer can perform image masking on random image regions in the sample image, or use methods such as attention to determine regions in the sample image containing more image information for image masking, or other image masking methods. For example, the image masking method commonly used in training the ViT model can be used as the image masking method in this application embodiment.

[0128] The mask layer in step 401 can be... Figure 3 The masking network 305 in the code will not be described in detail here.

[0129] 402. The first image features of the non-masked image information are extracted through the first feature extraction layer in the initial model.

[0130] by Figure 3 For example, the first feature extraction layer can refer to the image neural network 302. When the image neural network 302 is composed of the feature extraction layer in the ViT model, the first feature extraction layer can refer to the feature extraction layer in the ViT model.

[0131] 403. The first text features of the sample text are extracted through the second feature extraction layer in the initial model.

[0132] by Figure 3 For example, the second feature extraction layer can refer to the text neural network 303. When the text neural network 303 is composed of the feature extraction layer in the BERT model, the second feature extraction layer can refer to the feature extraction layer in the BERT model.

[0133] 404. The first image feature and the first text feature are fused through the feature fusion layer in the initial model to obtain the first fused feature.

[0134] by Figure 3 For example, the feature fusion layer can refer to fusion network 304. The feature fusion layer can perform multimodal fusion of the first image features and the first text features through methods such as attention to obtain the first fused features.

[0135] 405. The reconstructed image information corresponding to the masked image information in the sample image is predicted based on the first fusion feature through the first prediction layer in the initial model.

[0136] The first prediction layer is used to reconstruct the information of the masked image. Figure 3 For example, Figure 3 The initial model 301 provided may further include a first prediction network 306, which can predict reconstructed image information based on the first fusion features output by the fusion network 304. If the initial model is... Figure 3 In the initial model 301, the first prediction layer refers to the first prediction network 306.

[0137] In some embodiments, step 203 may be performed by Figure 5 The implementation method, namely, the step "for the second training sample in each group of training samples other than the first training sample, based on the sample image and the non-masked text information in the sample text, perform text mask reconstruction on the sample text to obtain the reconstructed text information corresponding to the masked text information in the sample text", includes:

[0138] 501. For the second training sample in each group of training samples, the sample text is masked through the masking layer in the initial model to obtain the non-masked text information in the sample text.

[0139] Similarly, this application does not limit the text masking method used by the masking layer when performing text masking. For example, the masking layer can perform text masking on random text segments in the sample text, or use methods such as attention to determine text segments in the sample text that contain more text information for text masking, or other text masking methods. For example, the text masking method commonly used when training the BERT model can be used as the text masking method in this application embodiment.

[0140] The mask layer in step 402 can be... Figure 3 The masking network 305 in the code will not be described in detail here.

[0141] 502. The second image features of the sample image are extracted through the first feature extraction layer in the initial model.

[0142] The explanation of the first feature extraction layer is as described above and will not be repeated here.

[0143] 503. The second text features of the non-masked text information are extracted through the second feature extraction layer in the initial model.

[0144] The explanation of the second feature extraction layer is as described above and will not be repeated here.

[0145] 504. The second image feature and the second text feature are fused through the feature fusion layer in the initial model to obtain the second fused feature.

[0146] The description of the feature fusion layer is as above and will not be repeated here.

[0147] 505. The reconstructed text information corresponding to the masked text information in the sample text is obtained by predicting the second prediction layer in the initial model based on the second fusion feature.

[0148] The second prediction layer is used to reconstruct the masked text information. Figure 3 For example, Figure 3 The initial model 301 provided may also include a second prediction network 307, which can predict the reconstructed text information based on the second fusion features output by the fusion network 304. If the initial model is... Figure 3 In the initial model 301, the second prediction layer refers to the second prediction network 307.

[0149] For ease of understanding, Figure 3 For example, let's illustrate the practical application of steps 401-405 and 501-505 above, assuming that the division of the first and second training samples is performed by the initial model in the following process:

[0150] (a) Input each group of training samples in the sample dataset into the initial model 301 in sequence;

[0151] (b) The masking network 305 in the initial model 301 divides the input samples into the first training samples or the second training samples according to the partitioning probability;

[0152] (c1) If the input sample is divided into the first training sample, the masking network 305 performs image masking on the sample image in the first training sample to obtain the non-masked image information and the masked image information in the sample image.

[0153] (d1) The image neural network 302 extracts image features of the non-masked image information, the text neural network 303 extracts text features of the sample text in the first training sample, the fusion network 304 fuses the image features and text features to obtain the first fused feature, and inputs the first fused feature into the first prediction network 306.

[0154] (e1) The first prediction network 306 performs mask reconstruction based on the first fusion feature to obtain the reconstructed image information;

[0155] (c2) If the input sample is divided into the second training sample, the masking network 305 performs text masking on the sample text in the second training sample to obtain the non-masked text information and the masked text information in the sample text.

[0156] (d1) Image neural network 302 extracts image features from sample images in the second training sample, text neural network 303 extracts text features from non-masked text information, fusion network 304 fuses image features and text features to obtain second fused features, and inputs the second fused features into the second prediction network 307.

[0157] (e1) The second prediction network 307 performs mask reconstruction based on the second fusion feature to obtain the reconstructed text information.

[0158] In some embodiments, the model parameters of the initial model can be optimized using a preset optimizer, and the optimized model can be fine-tuned to obtain a target multimodal model that can be practically applied to downstream tasks. (Reference) Figure 6 At this point, the step "training the initial model based on the first similarity information between the reconstructed image information and the masked image information, and the second similarity information between the reconstructed text information and the masked text information to obtain the target multimodal model" includes:

[0159] 601. Input the first similarity information and the second similarity information into the preset optimizer in the initial model, optimize the model parameters of the initial model, and output the optimized model parameters and the model to be adjusted composed of the optimized model parameters.

[0160] This application does not limit the type of optimizer. For example, the AdamW optimizer can be used as a preset optimizer in the initial model to achieve end-to-end overall model optimization. Specific optimization methods will not be elaborated further.

[0161] 602. Fine-tune the model to be adjusted to obtain the target multimodal model.

[0162] During fine-tuning, the fine-tuning method can be determined based on the downstream tasks in which the model is actually applied. The specific fine-tuning methods will not be elaborated here.

[0163] The target multimodal model trained using any of the above embodiments can be used for information recommendation in social media. (Reference) Figure 7 , Figure 7 The document illustrates the practical application process of the target multimodal model, including:

[0164] 701. In response to a target user's trigger command for target social information in a social network, the target image in the target social information is input into the first feature extraction layer of the target multimodal model, and the target text in the target social information is input into the second feature extraction layer of the target multimodal model to obtain image features in the target image and text features in the target text.

[0165] Social networks can refer to one of the following: social apps, social mini-programs, social web pages, etc.

[0166] refer to Figure 10 , Figure 10 The image shows a social network page 1001 containing multiple social messages 1002 sent by users. Each social message 1002 consists of an image and text, with the text in each message describing the image. When a target user wants to see detailed content within a target social message, they can trigger the message by clicking. If the target social message matches the target user's preferences, the electronic device will input the target image and text from the target social message into a target multimodal model for subsequent confirmation and recommendation of social information to the target user.

[0167] 702. Input the image features and the text features into the feature fusion layer in the target multimodal model to obtain fused features.

[0168] The explanation of step 702 can be found above and will not be repeated here.

[0169] 703. Search from the preset feature database for similar features that have a similarity greater than a preset similarity threshold with the fused features.

[0170] The pre-defined feature database is used to store the fused features of various social messages in a social network. When a user posts social information, the electronic device can extract the fused features of that social information through a target multimodal model and update the feature database.

[0171] Similar features refer to fusion features that are similar to the target social information within the feature database, as well as fusion features corresponding to other social information.

[0172] The method for calculating similarity can be found above and will not be repeated here.

[0173] The specific value of the preset similarity threshold can be determined by the actual application scenario, and this application embodiment does not impose any restrictions on it.

[0174] 704. Set the source information of the similar features as the recommended social information corresponding to the target user.

[0175] Source information refers to the social information corresponding to similar features. For example, it could be the social information input when similar features are extracted using a target multimodal model.

[0176] Since the similar features are quite similar to the fused information corresponding to the target social information, it can be assumed that the content described by the source information is similar to the content described by the target social information. Therefore, using the source information as the recommended social information corresponding to the target user is more in line with the target user's preferences.

[0177] After receiving recommended social information, the recommended social information can be displayed on the refreshed screen when the target user refreshes their social network.

[0178] To better implement the model training method in the embodiments of this application, based on the model training method, the embodiments of this application also provide a model training device, such as... Figure 8 The diagram shown is a schematic representation of an embodiment of the model training device in this application. The model training device 800 includes:

[0179] The acquisition unit 801 is used to acquire an initial model and a preset sample dataset, wherein the sample dataset includes multiple sets of training samples, and each set of training samples includes a sample image and sample text used to describe the sample image.

[0180] The image mask reconstruction unit 802 is used to perform image mask reconstruction on the first training sample in each group of training samples, based on the sample text and the non-masked image information in the sample image, to obtain the reconstructed image information corresponding to the masked image information in the sample image.

[0181] The text mask reconstruction unit 803 is used to perform text mask reconstruction on the second training sample other than the first training sample in each group of training samples, based on the sample image and the non-masked text information in the sample text, to obtain the reconstructed text information corresponding to the masked text information in the sample text.

[0182] Training unit 804 is used to train the initial model based on the first similarity information between the reconstructed image information and the masked image information, and the second similarity information between the reconstructed text information and the masked text information, to obtain the target multimodal model.

[0183] In one possible implementation of this application, the acquiring unit 801 is further configured to:

[0184] The training samples from each group are input into the initial model, and the training samples from each group are divided into the first training sample and the second training sample according to the preset division probability through the mask layer in the initial model.

[0185] In one possible implementation of this application, the partitioning probability is obtained based on the average sharpness of each of the sample images and / or the average text length of each of the sample texts.

[0186] In one possible implementation of this application, the image mask reconstruction unit 802 is further configured to:

[0187] For the first training sample in each group of training samples, the sample image is masked through the masking layer in the initial model to obtain the non-masked image information in the sample image;

[0188] The first image features of the non-masked image information are extracted through the first feature extraction layer in the initial model;

[0189] The first text features of the sample text are extracted through the second feature extraction layer in the initial model;

[0190] The first image feature and the first text feature are fused through the feature fusion layer in the initial model to obtain the first fused feature;

[0191] The reconstructed image information corresponding to the masked image information in the sample image is obtained by predicting the first prediction layer in the initial model based on the first fusion feature.

[0192] In one possible implementation of this application, the text mask reconstruction unit 803 is further configured to:

[0193] For the second training sample in each group of training samples, the sample text is masked through the masking layer in the initial model to obtain the non-masked text information in the sample text;

[0194] The second image features of the sample image are extracted through the first feature extraction layer in the initial model;

[0195] The second text features of the non-masked text information are extracted through the second feature extraction layer in the initial model;

[0196] The second image feature and the second text feature are fused through the feature fusion layer in the initial model to obtain the second fused feature;

[0197] The reconstructed text information corresponding to the masked text information in the sample text is obtained by predicting the second prediction layer in the initial model based on the second fusion feature.

[0198] In one possible implementation of this application, the training unit 804 is further used for:

[0199] The first similarity information and the second similarity information are input into the preset optimizer in the initial model to optimize the model parameters of the initial model, and the optimized model parameters and the model to be adjusted composed of the optimized model parameters are output.

[0200] The model to be adjusted is fine-tuned to obtain the target multimodal model.

[0201] In practice, the above modules can be implemented as independent entities or combined in any way to be implemented as the same or several entities. For the specific implementation of the above modules, please refer to the previous method implementation examples, which will not be repeated here.

[0202] Since the model training device can execute the steps in the model training method in any embodiment, it can achieve the beneficial effects that the model training method in any embodiment of this application can achieve, as detailed in the preceding description, and will not be repeated here.

[0203] Furthermore, to better implement the model training method in the embodiments of this application, based on the model training method, the embodiments of this application also provide an electronic device, see below. Figure 9 , Figure 9 This illustration shows a structural schematic diagram of an electronic device according to an embodiment of this application. Specifically, the electronic device provided in this embodiment includes a processor 901. The processor 901 is used to execute a computer program stored in a memory 902 to implement the steps of the model training method in any embodiment; or, the processor 901 is used to execute a computer program stored in a memory 902 to implement, as shown in the diagram. Figure 8 The functions of each module in the corresponding embodiment.

[0204] For example, a computer program can be divided into one or more modules / units, one or more of which are stored in memory 902 and executed by processor 901 to complete the embodiments of this application. One or more modules / units can be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in a computer device.

[0205] The electronic device may include, but is not limited to, processor 901 and memory 902. Those skilled in the art will understand that the illustrations are merely examples of an electronic device and do not constitute a limitation on the device. It may include more or fewer components than illustrated, or combine certain components, or use different components.

[0206] The processor 901 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the electronic device, connecting various parts of the electronic device through various interfaces and lines.

[0207] The memory 902 can be used to store computer programs and / or modules. The processor 901 implements various functions of the computer device by running or executing the computer programs and / or modules stored in the memory 902 and by calling data stored in the memory 902. The memory 902 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device (such as audio data, video data, etc.). In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0208] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the model training device, electronic equipment and its corresponding modules described above can be referred to the description of the model training method in any embodiment, and will not be repeated here.

[0209] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by instructions, or by instructions controlling related hardware. These instructions can be stored in a readable storage medium and loaded and executed by a processor.

[0210] Therefore, embodiments of this application provide a readable storage medium storing a computer program. When the computer program is executed by a processor, it performs the steps of the model training method in any embodiment of this application. For specific operations, please refer to the description of the model training method in any embodiment.

[0211] The readable storage medium may include: read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc.

[0212] Since the instructions stored in the readable storage medium can execute the steps in the model training method in any embodiment of this application, the beneficial effects that the model training method in any embodiment of this application can achieve can be realized, as detailed in the preceding description, and will not be repeated here.

[0213] The foregoing has provided a detailed description of a model training method, apparatus, storage medium, and electronic device provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A model training method, characterized in that, include: Obtain an initial model and a preset sample dataset, wherein the sample dataset includes multiple sets of training samples, and each set of training samples includes a sample image and sample text used to describe the sample image; Each group of training samples is input into the initial model. Through the mask layer in the initial model, each group of training samples is divided into a first training sample and a second training sample according to a preset partitioning probability. The partitioning probability is obtained based on the average sharpness of each sample image and / or the average text length of each sample text. For the first training sample in each group of training samples, based on the sample text and the non-masked image information in the sample image, image mask reconstruction is performed on the sample image to obtain the reconstructed image information corresponding to the masked image information in the sample image. The reconstructed image information is the image information of the masked region predicted by image mask reconstruction. For the second training sample in each group of training samples other than the first training sample, based on the sample image and the non-masked text information in the sample text, text mask reconstruction is performed on the sample text to obtain the reconstructed text information corresponding to the masked text information in the sample text. The reconstructed text information is the masked text fragment predicted by text mask reconstruction. Based on the first similarity information between the reconstructed image information and the masked image information, and the second similarity information between the reconstructed text information and the masked text information, the initial model is trained to obtain the target multimodal model.

2. The model training method according to claim 1, characterized in that, For the first training sample in each group of training samples, based on the sample text and the non-masked image information in the sample image, image mask reconstruction is performed on the sample image to obtain the reconstructed image information corresponding to the masked image information in the sample image, including: For the first training sample in each group of training samples, the sample image is masked through the masking layer in the initial model to obtain the non-masked image information in the sample image; The first image features of the non-masked image information are extracted through the first feature extraction layer in the initial model; The first text features of the sample text are extracted through the second feature extraction layer in the initial model; The first image feature and the first text feature are fused through the feature fusion layer in the initial model to obtain the first fused feature; The reconstructed image information corresponding to the masked image information in the sample image is obtained by predicting the first prediction layer in the initial model based on the first fusion feature.

3. The model training method according to claim 1, characterized in that, For the second training sample in each group of training samples other than the first training sample, based on the sample image and the non-masked text information in the sample text, text mask reconstruction is performed on the sample text to obtain the reconstructed text information corresponding to the masked text information in the sample text, including: For the second training sample in each group of training samples, the sample text is masked through the masking layer in the initial model to obtain the non-masked text information in the sample text; The second image features of the sample image are extracted through the first feature extraction layer in the initial model; The second text features of the non-masked text information are extracted through the second feature extraction layer in the initial model; The second image feature and the second text feature are fused through the feature fusion layer in the initial model to obtain the second fused feature; The reconstructed text information corresponding to the masked text information in the sample text is obtained by predicting the second prediction layer in the initial model based on the second fusion feature.

4. The model training method according to claim 1, characterized in that, The initial model is trained based on the first similarity information between the reconstructed image information and the masked image information, and the second similarity information between the reconstructed text information and the masked text information, to obtain a target multimodal model, including: The first similarity information and the second similarity information are input into the preset optimizer in the initial model to optimize the model parameters of the initial model, and the optimized model parameters and the model to be adjusted composed of the optimized model parameters are output. The model to be adjusted is fine-tuned to obtain the target multimodal model.

5. The model training method according to claim 1, characterized in that, After training the initial model based on the first similarity information between the reconstructed image information and the masked image information, and the second similarity information between the reconstructed text information and the masked text information to obtain the target multimodal model, the method further includes: In response to a target user's trigger command for target social information in a social network, the target image in the target social information is input into the first feature extraction layer of the target multimodal model, and the target text in the target social information is input into the second feature extraction layer of the target multimodal model to obtain image features in the target image and text features in the target text; The image features and text features are input into the feature fusion layer of the target multimodal model to obtain fused features; Similar features with a similarity greater than a preset similarity threshold to the fused features are obtained from a preset feature database; The source information of the similar features is set as the recommended social information corresponding to the target user.

6. A model training device, characterized in that, include: The acquisition unit is used to acquire an initial model and a preset sample dataset, wherein the sample dataset includes multiple sets of training samples, and each set of training samples includes a sample image and sample text used to describe the sample image; The acquisition unit is further configured to input each group of training samples into the initial model, and divide each group of training samples into a first training sample and a second training sample according to a preset division probability through the mask layer in the initial model. The division probability is obtained based on the average sharpness of each sample image and / or the average text length of each sample text. The image mask reconstruction unit is used to perform image mask reconstruction on the first training sample in each group of training samples, based on the sample text and the non-masked image information in the sample image, to obtain the reconstructed image information corresponding to the masked image information in the sample image, wherein the reconstructed image information is the image information of the masked region predicted by the image mask reconstruction. The text mask reconstruction unit is used to perform text mask reconstruction on the second training sample other than the first training sample in each group of training samples, based on the sample image and the non-masked text information in the sample text, to obtain the reconstructed text information corresponding to the masked text information in the sample text, wherein the reconstructed text information is the masked text fragment predicted by the text mask reconstruction. The training unit is used to train the initial model based on the first similarity information between the reconstructed image information and the masked image information, and the second similarity information between the reconstructed text information and the masked text information, to obtain the target multimodal model.

7. An electronic device, characterized in that, The electronic device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the model training method as described in any one of claims 1 to 5.

8. A readable storage medium, characterized in that, The readable storage medium stores a computer program that, when executed by a processor, implements the steps of the model training method according to any one of claims 1 to 5.

Images