Training method for generating picture description model, picture description generation method, device, equipment, medium and program product

Abstract

The invention relates to a training method for generating a picture description model, a picture description generation method and device, equipment, a medium and a program product. The method comprises the following steps: acquiring a sample, wherein the sample comprises a sample picture and a text corresponding to the sample picture; inputting the sample picture and the text into a picture description model to be trained and generated to obtain text combined image features; inputting the text into a to-be-trained generated picture description model to obtain text features; based on the text, combining image features with the text features to obtain a distance matrix; and generating a loss function based on the distance matrix, and training the to-be-trained generated picture description model through the loss function to obtain a target generated picture description model. By adopting the method, the accuracy can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a training method for generating a picture description model, a picture description generation method, an apparatus, a device, a medium and a program product. Background Art

[0002] Multimodal image-generated text description refers to the technology of converting images into natural language descriptions through computer algorithms. The background of this technology can be traced back to the development of computer vision and natural language processing.

[0003] In the field of computer vision, the rise of deep learning models has laid the foundation for generating text descriptions from multimodal images. Deep learning models such as Convolutional Neural Networks (CNN) and Generative Adversarial Networks (GAN) have made breakthrough progress in image processing and generation. CNN can extract features from images and learn image representations, while GAN can generate realistic images.

[0004] On the other hand, the development of natural language processing also provides support for the generation of text descriptions from multimodal images. Natural language processing technology includes tasks such as machine translation, semantic understanding, and text generation, among which the text generation task involves generating coherent and accurate natural language text. The development of these technologies provides a theoretical and practical basis for converting images into text descriptions.

[0005] The technical background of multimodal image generation text description also involves two important datasets: MSCOCO and Flickr30K. These datasets contain a large number of images and corresponding text descriptions, providing researchers with standard benchmarks for training and evaluating models.

[0006] In summary, the technical background of multimodal image-generated text descriptions benefits from the progress in computer vision and natural language processing, as well as the support of deep learning models and large-scale datasets. These advances have laid the foundation for the development of algorithms and systems that can convert images into natural language descriptions.

[0007] However, in traditional technology, multimodal image generation of text descriptions is mainly processed through the multimodal unified framework Blip, which includes Q-former. This structure builds a bridge for the integration of visual features and text features, but the accuracy still needs to be improved. Summary of the invention

[0008] Based on this, it is necessary to provide a training method for generating a picture description model, a picture description generating method, an apparatus, a device, a medium and a program product that can improve the accuracy in order to solve the above-mentioned technical problems.

[0009] In a first aspect, the present application provides a training method for generating an image description model, the method comprising:

[0010] Acquire a sample, wherein the sample includes a sample image and text corresponding to the sample image;

[0011] Inputting the sample image and the text into a model to be trained to generate image descriptions to obtain text-combined image features;

[0012] Inputting the text into a picture description model to be trained to obtain text features;

[0013] Obtaining a distance matrix based on the text combined with the image features and the text features;

[0014] A loss function is generated based on the distance matrix, and the generated picture description model to be trained is trained by using the loss function to obtain a target generated picture description model.

[0015] In one embodiment, obtaining a distance matrix based on the text combined with the image features and the text features includes:

[0016] The difference between the text-combined image feature and the text feature is calculated element by element to obtain a distance matrix.

[0017] In one embodiment, generating a loss function based on the distance matrix includes:

[0018] Processing the distance matrix through a fully connected layer to obtain a weight matrix;

[0019] Processing the distance matrix by the image description model to be trained to obtain a distinguishability matrix;

[0020] generating a target matrix based on the weight matrix and the discriminability matrix;

[0021] A loss function is generated based on the target matrix.

[0022] In one embodiment, the distance matrix is ​​processed by the image description model to be trained to obtain a distinguishability matrix, including:

[0023] Inputting the distance matrix into the image description model to be trained to obtain a reference matrix;

[0024] The reference matrix is ​​processed through a normalization layer to obtain a discriminant matrix.

[0025] In one embodiment, generating a loss function based on the target matrix includes:

[0026] Calculate the mean square sum of each sample based on the target matrix;

[0027] A loss function is obtained based on the mean square sum of each of the samples and the number of samples.

[0028] In a second aspect, the present application further provides a method for generating a picture description, the method further comprising:

[0029] Receiving a target image and a question corresponding to the target image;

[0030] Processing the target image through a target generated image description model to obtain target image features, wherein the target generated image description model is obtained based on the training method of the generated image description model in any one of the above embodiments;

[0031] The target image features and questions are processed by a pre-trained large model to obtain an answer to the question.

[0032] In a third aspect, the present application further provides a training device for generating an image description model, the device comprising:

[0033] A sample acquisition module, used to acquire a sample, wherein the sample includes a sample image and text corresponding to the sample image;

[0034] A first feature generation module, used for inputting the sample image and the text into a to-be-trained generated image description model to obtain text-combined image features;

[0035] A second feature generation module is used to input the text into a to-be-trained generated picture description model to obtain text features;

[0036] A distance matrix generation module, used to obtain a distance matrix based on the text combined with image features and the text features;

[0037] A training module is used to generate a loss function based on the distance matrix, and train the generated picture description model to be trained through the loss function to obtain a target generated picture description model.

[0038] In a fourth aspect, the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method in any one of the above-mentioned embodiments when executing the computer program.

[0039] In a fifth aspect, the present application also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method in any one of the above-mentioned embodiments are implemented.

[0040] In a sixth aspect, the present application also provides a computer program product, including a computer program, which implements the steps of the method in any one of the above-mentioned embodiments when executed by a processor.

[0041] The training method, picture description generating method, device, equipment, medium and program product of the above-mentioned generated picture description model obtain samples, wherein the samples include sample pictures and text corresponding to the sample pictures; the sample pictures and the text are input into the generated picture description model to be trained to obtain text-combined image features; the text is input into the generated picture description model to be trained to obtain text features; a distance matrix is ​​obtained based on the text-combined image features and the text features; a loss function is generated based on the distance matrix, and the generated picture description model to be trained is trained by the loss function to obtain a target generated picture description model, so that the contribution rate of text features to picture features is further optimized, so that the generated picture description model can better learn the difference between visual features and text features, further optimize the generated picture description model, and allow the visual features to obtain features that are more similar to text through the generated picture description model, thereby improving accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 An application environment diagram of a training method for generating an image description model in one embodiment;

[0044] Figure 2 A schematic diagram of a flow chart of a training method for generating an image description model in one embodiment;

[0045] Figure 3 A schematic diagram of the structure of a Q-former model to be trained to generate picture descriptions in one embodiment;

[0046] Figure 4 A flowchart of a training process for generating a picture description model to be trained in one embodiment;

[0047] Figure 5Schematic diagram of a process of generating a picture description in one embodiment;

[0048] Figure 6 A structural block diagram of a training device for generating a picture description model in one embodiment;

[0049] Figure 7 is a structural block diagram of a device for generating picture description in one embodiment;

[0050] Figure 8 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0052] The training method for generating an image description model provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, the terminal 102 communicates with the server 104 through a network. The data storage system can store data that the server 104 needs to process. The data storage system can be integrated on the server 104, or placed on the cloud or other network servers.

[0053] The terminal 102 can send a start training instruction and a stop training instruction to the server 104 to instruct the server 104 to train the generated picture description model, wherein after receiving the start training instruction, the server 104 obtains a sample, the sample includes a sample picture and a text corresponding to the sample picture; the sample picture and the text are input into the generated picture description model to be trained to obtain text combined with image features; the text is input into the generated picture description model to be trained to obtain text features; a distance matrix is ​​obtained based on the text combined with image features and text features; a loss function is generated based on the distance matrix, and the generated picture description model to be trained is trained by the loss function to obtain a target generated picture description model, so that the contribution rate of text features to picture features is further optimized, so that the generated picture description model can better learn the difference between visual features and text features, and further optimize the generated picture description model, so that the visual features can obtain features that are more similar to text through the generated picture description model.

[0054] The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. Head-mounted devices may be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The server 104 may be an independent physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides cloud computing services.

[0055] In an exemplary embodiment, Figure 2 As shown, a training method for generating a picture description model is provided, and the method is applied to Figure 1 The server in the example is used to illustrate, including the following steps 202 to 210. Among them:

[0056] S202: Obtain a sample, where the sample includes a sample image and text corresponding to the sample image.

[0057] The sample is predetermined, and includes a sample image and text corresponding to the sample image, where the text is used to describe the sample image.

[0058] One point that needs to be explained is that the number of samples can be pre-set and is not specifically limited here.

[0059] S204: Input the sample image and text into the image description model to be trained to obtain text-combined image features.

[0060] The image description model to be trained is a part of the Blip model, which includes three structures: a visual extractor, a image description model to be trained, and a large prediction model.

[0061] The Blip model is a model that feeds an image into a visual encoder, uses a visual-text converter to align the visual feature vector and the text feature vector, and completes image text description and image question answering through a large language model. The model structure includes a visual extractor Blip2, a model to be trained to generate image descriptions Q-former, and a large prediction model large language model. The large language model completes the final question generation and text question answering.

[0062] Bert is a natural language encoder. It is a model pre-trained with a large amount of text. It has good text information extraction capabilities and can perform various downstream tasks based on Bert.

[0063] Combination Figure 3 As shown in the figure, the image description model Q-former to be trained contains an encoder in a 12-layer transformer architecture and a fully connected layer. The image or text features are extracted through the 12-layer encoder, and finally the prediction of each token is completed through the classification fully connected layer.

[0064] Among them, each layer of the Q-former model to be trained to generate image description has a self-attention layer and a cross attention layer. Both layer structures require a fully connected layer to process the output, and the last FFN layer is responsible for processing the final output.

[0065] Continue to combine Figure 3 FFN is a feedforward neural network, which is linked by two fully connected layers. However, the first fully connected layer is for dimension increase, and the second fully connected layer is for dimension reduction. The activation function is used in the middle to process the activation value (feature value).

[0066] The training of the image description model Q-former to be trained includes at least one of the following three loss functions:

[0067] Itc loss function, which is to pass the picture and question query through the image description model Q-former to be trained to obtain the image representation, and the text through the image description model Q-former to be trained to obtain the text representation, and the text and image matrix are multiplied to calculate the contrast loss.

[0068] Itm loss loss function, which is the image and text representation of the previous step, is divided into three groups, one group of matching images and texts, and the other two groups of unmatched images and texts. The three groups of images and texts and the question query are sent to the Q-former model to be trained to generate the image description model, and the classification loss of the three groups is obtained. The matching group label is 1, and the unmatched label is 0, and then the cross entropy loss is calculated.

[0069] The itg loss function is to send the text and the image and question query in the first step of calculating itc loss through the Q-former to be trained to generate the picture description model Q-former to calculate the text generation loss.

[0070] In order to further optimize the contribution rate of text tokens to image features in this application, a forward propagation path is added to send the text and image features to the Q-former model to be trained to generate image description, and cross attention is used to obtain text combined with image features.

[0071] S206: Input the text into the image description model to be trained to obtain text features.

[0072] For ease of understanding, combined Figure 4 As shown, Figure 4 The flowchart of the training process of the image description model to be trained in one embodiment is shown in FIG. In this embodiment, one branch sends text and image features to the image description model to be trained Q-former, uses cross attention to obtain text combined with image features, and another branch inputs text to the image description model to be trained Q-former to obtain text features.

[0073] S208: Obtain a distance matrix based on the text, image features, and text features.

[0074] The distance feature is obtained based on the text combined with the image feature and the text feature. For example, the text combined with the image feature and the text feature are all represented by matrices, and the difference matrix of the two matrices is calculated as the distance matrix.

[0075] In one optional embodiment, obtaining a distance matrix based on the text combined with the image features and the text features includes: calculating the difference between the text combined with the image features and the text features element by element to obtain the distance matrix.

[0076] That is to say, in this embodiment, the text is first combined with the image features and each element of the matrix corresponding to the text features, and then the matrix at the same position is subjected to difference calculation. The difference calculation can be to directly calculate the difference, or other methods, and the difference at each position is used as the element at that position to obtain the distance matrix.

[0077] S210: Generate a loss function based on the distance matrix, and train the to-be-trained generated picture description model using the loss function to obtain a target generated picture description model.

[0078] The loss function is obtained based on the distance matrix, which is used to narrow the difference between text combined with image features and text features. Therefore, by generating a loss function through the distance matrix and training based on the loss function, the difference between text combined with image features and text features can be narrowed, so that the visual features are more similar to the text features after the target generates the picture description model.

[0079] The training method of the above-mentioned generated picture description model obtains samples, and the samples include sample pictures and texts corresponding to the sample pictures; the sample pictures and texts are input into the generated picture description model to be trained to obtain text-combined image features; the text is input into the generated picture description model to be trained to obtain text features; a distance matrix is ​​obtained based on the text-combined image features and the text features; a loss function is generated based on the distance matrix, and the generated picture description model to be trained is trained by the loss function to obtain a target generated picture description model, so that the contribution rate of the text features to the picture features is further optimized, so that the generated picture description model can better learn the difference between visual features and text features, and further optimize the generated picture description model, so that the visual features can obtain features that are more similar to the text through the generated picture description model.

[0080] In one of the optional embodiments, a loss function is generated based on a distance matrix, including: processing the distance matrix through a fully connected layer to obtain a weight matrix; processing the distance matrix through a picture description model to be trained to obtain a distinction matrix; generating a target matrix based on the weight matrix and the distinction matrix; and generating a loss function based on the target matrix.

[0081] Please continue to combine Figure 4 As shown, the weight matrix is ​​obtained by passing the distance matrix through a fully connected layer, wherein the distance matrix includes several elements, each element corresponds to a token, and the token refers to the token of the text. In this way, the distance matrix is ​​passed through a fully connected layer to obtain the weight of each token, that is, the weight matrix. Optionally, the weight matrix can include 32 elements, that is, 32 tokens. In other embodiments, the number of elements included in the weight matrix can be other values, which are not specifically limited here.

[0082] The distinguishability matrix is ​​obtained by inputting the distance matrix into the image description model to be trained again. Specifically, the distance matrix is ​​processed by the image description model to be trained to obtain the distinguishability matrix, including: inputting the distance matrix into the image description model to be trained to obtain a reference matrix; processing the reference matrix through a normalization layer to obtain a distinguishability matrix.

[0083] Continue to combine Figure 4As shown, the distance matrix is ​​input into the image description model to be trained to obtain a reference matrix, and the reference matrix is ​​then input into a normalization layer layerNorm to obtain a difference matrix. The distance matrix is ​​input into the image description model to be trained, and is processed only by the self-attention in the image description model to be trained. That is to say, when there is only one input to the image description model to be trained, it is processed by self-attention. When there are two inputs to the image description model to be trained, it can be processed by self-attention and cross-attention. The difference matrix is ​​used to characterize the difference between text combined with image features and text features.

[0084] The target matrix is ​​generated based on the weight matrix and the discriminability matrix. The target matrix can be equal to the product of the weight matrix and the discriminability matrix, that is,

[0085]

[0086] Among them, M is the target matrix, W is the weight matrix, and Diff is the difference matrix.

[0087] In one of the optional embodiments, generating a loss function based on a target matrix includes: calculating a mean square sum of each sample based on the target matrix; and obtaining a loss function based on the mean square sum of each sample and the number of samples.

[0088] Specifically, the loss function L can be:

[0089]

[0090] in, is the mean square sum of a sample, where 32 is the number of elements in the target matrix, i and j are variables, is the sample size.

[0091] In this embodiment, the mean square sum of each sample is calculated first, and then the sum of the mean square sums of all samples is calculated and divided by the number of samples to obtain the loss function value. If the loss function value is less than a preset value, or the change in the loss function value is less than a threshold, etc., the training is considered complete. Otherwise, the model continues to be trained.

[0092] The purpose of the above training process is to enable the target generated image description model Q-former to better learn the difference between visual features and text features, further optimize the target generated image description model Q-former, and allow the visual features to obtain features that are more similar to text through the target generated image description model Q-former.

[0093] In an exemplary embodiment, Figure 5 As shown, a method for generating a picture description is provided, and the method is applied to Figure 1The server in the example is used to illustrate, including the following steps 502 to 506. Among them:

[0094] S502: Receive a target image and a question corresponding to the target image.

[0095] The target image is the image corresponding to the question and the answer, and the question is the question generated by the user based on the target image.

[0096] S504: Process the target image through a target generated image description model to obtain target image features, where the target generated image description model is obtained based on a training method for generating an image description model in any of the above embodiments.

[0097] The target generated image description model is used to extract target image features. The target generated image description model is trained based on the training method of the generated image description model mentioned above. The visual features are passed through the target generated image description model Q-former to obtain features that are more similar to text.

[0098] S506: The pre-trained large model is used to process the target image features and questions to obtain answers to the questions.

[0099] The big model completes the final question generation and text question answering to obtain the answers corresponding to the questions and output those answers.

[0100] In the above embodiment, the difference between visual features and text features can be obtained, and the target generated picture description model Q-former is further optimized, so that the visual features can be passed through the target generated picture description model Q-former to obtain features that are more similar to the text, thereby improving the accuracy of the answer.

[0101] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0102] Based on the same inventive concept, the embodiment of the present application also provides a training device for generating a picture description model and a picture description generating device for the picture description generating method for implementing the training method for generating a picture description model involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of one or more training devices for generating picture description models and picture description generating devices provided below can be found in the above limitations on the training method for generating a picture description model and the picture description generating method, which will not be repeated here.

[0103] In an exemplary embodiment, Figure 6 As shown, a training device for generating an image description model is provided, comprising: a sample acquisition module 601, a first feature generation module 602, a second feature generation module 603, a distance matrix generation module 604 and a training module 605, specifically:

[0104] The sample acquisition module 601 is used to acquire a sample, where the sample includes a sample image and text corresponding to the sample image;

[0105] The first feature generation module 602 is used to input the sample image and text into the image description model to be trained to obtain text-combined image features;

[0106] The second feature generation module 603 is used to input text into the image description model to be trained to obtain text features;

[0107] A distance matrix generation module 604 is used to obtain a distance matrix based on the text combined with the image features and the text features;

[0108] The training module 605 is used to generate a loss function based on the distance matrix, and train the image description model to be trained by using the loss function to obtain a target image description model.

[0109] In one of the optional embodiments, the distance matrix generation module 604 is specifically used to calculate the difference between the text-combined image feature and the text feature element by element to obtain a distance matrix.

[0110] In one of the optional embodiments, the above-mentioned training module 605 is specifically used to process the distance matrix through a fully connected layer to obtain a weight matrix; process the distance matrix through the image description model to be trained to obtain a distinguishing matrix; generate a target matrix based on the weight matrix and the distinguishing matrix; and generate a loss function based on the target matrix.

[0111] In one of the optional embodiments, the training module 605 is specifically used to input the distance matrix into the image description model to be trained to obtain a reference matrix; and process the reference matrix through a normalization layer to obtain a distinguishing matrix.

[0112] In one of the optional embodiments, the training module 605 is specifically used to calculate the mean square sum of each sample based on the target matrix; and obtain the loss function based on the mean square sum of each sample and the number of samples.

[0113] In an exemplary embodiment, Figure 7 As shown, a picture description generating device is provided, comprising: a receiving module 701, a target picture feature generating module 702 and an answer generating module 703, wherein:

[0114] A receiving module 701 is used to receive a target image and a question corresponding to the target image;

[0115] A target image feature generation module 702, configured to process the target image through a target generated image description model to obtain target image features, wherein the target generated image description model is obtained based on a training device for generating an image description model in any of the above embodiments;

[0116] The answer generation module 703 is used to process the target image features and questions through the pre-trained large model to obtain the answer to the question.

[0117] The training device for generating the image description model and each module in the image description generating device can be implemented in whole or in part by software, hardware and a combination thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0118] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 8As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a training method for generating a picture description model and a picture description generation method are implemented.

[0119] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0120] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the following steps when executing the computer program: obtaining a sample, wherein the sample includes a sample image and text corresponding to the sample image; inputting the sample image and the text into a generated image description model to be trained to obtain text-combined image features; inputting the text into the generated image description model to be trained to obtain text features; obtaining a distance matrix based on the text-combined image features and the text features; generating a loss function based on the distance matrix, and training the generated image description model to be trained by the loss function to obtain a target generated image description model.

[0121] In one embodiment, obtaining a distance matrix based on text combined with image features and text features when a processor executes a computer program includes: calculating the difference between the text combined with image features and text features element by element to obtain a distance matrix.

[0122] In one embodiment, the loss function generated based on the distance matrix implemented when the processor executes the computer program includes: processing the distance matrix through a fully connected layer to obtain a weight matrix; processing the distance matrix through a picture description model to be trained to obtain a distinction matrix; generating a target matrix based on the weight matrix and the distinction matrix; and generating a loss function based on the target matrix.

[0123] In one embodiment, when a processor executes a computer program, the distance matrix is ​​processed by the image description model to be trained to obtain a difference matrix, including: inputting the distance matrix into the image description model to be trained to obtain a reference matrix; processing the reference matrix through a normalization layer to obtain a difference matrix.

[0124] In one embodiment, the generation of a loss function based on a target matrix implemented when a processor executes a computer program includes: calculating the mean square sum of each sample based on the target matrix; and obtaining a loss function based on the mean square sum of each sample and the number of samples.

[0125] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the following steps when executing the computer program: receiving a target image and a question corresponding to the target image; processing the target image through a target-generated image description model to obtain target image features, wherein the target-generated image description model is obtained based on a training method for generating an image description model in any of the above embodiments; and processing the target image features and the question through a large model obtained by pre-training to obtain an answer to the question.

[0126] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented: obtaining a sample, the sample including a sample image and text corresponding to the sample image; inputting the sample image and text into a generated image description model to be trained to obtain text-combined image features; inputting the text into the generated image description model to be trained to obtain text features; obtaining a distance matrix based on the text-combined image features and the text features; generating a loss function based on the distance matrix, and training the generated image description model to be trained by the loss function to obtain a target generated image description model.

[0127] In one embodiment, the method of obtaining a distance matrix based on text combined with image features and text features when the computer program is executed by a processor includes: calculating the difference between the text combined with image features and text features element by element to obtain the distance matrix.

[0128] In one embodiment, the computer program generated a loss function based on a distance matrix when executed by a processor, including: processing the distance matrix through a fully connected layer to obtain a weight matrix; processing the distance matrix through a picture description model to be trained to obtain a distinction matrix; generating a target matrix based on the weight matrix and the distinction matrix; generating a loss function based on the target matrix.

[0129] In one embodiment, when a computer program is executed by a processor, the distance matrix is ​​processed by a to-be-trained generated picture description model to obtain a distinction matrix, including: inputting the distance matrix into the to-be-trained generated picture description model to obtain a reference matrix; processing the reference matrix through a normalization layer to obtain a distinction matrix.

[0130] In one embodiment, the computer program generated based on the target matrix when executed by the processor includes: calculating the mean square sum of each sample based on the target matrix; and obtaining the loss function based on the mean square sum of each sample and the number of samples.

[0131] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented: receiving a target image and a question corresponding to the target image; processing the target image through a target-generated image description model to obtain target image features, wherein the target-generated image description model is obtained based on a training method for generating an image description model in any of the above embodiments; and processing the target image features and the question through a large model obtained by pre-training to obtain an answer to the question.

[0132] In one embodiment, a computer program product is provided, including a computer program, which implements the following steps when executed by a processor: obtaining a sample, the sample including a sample image and text corresponding to the sample image; inputting the sample image and the text into a generated image description model to be trained to obtain text-combined image features; inputting the text into the generated image description model to be trained to obtain text features; obtaining a distance matrix based on the text-combined image features and the text features; generating a loss function based on the distance matrix, and training the generated image description model to be trained by the loss function to obtain a target generated image description model.

[0133] In one embodiment, the method of obtaining a distance matrix based on text combined with image features and text features when the computer program is executed by a processor includes: calculating the difference between the text combined with image features and text features element by element to obtain the distance matrix.

[0134] In one embodiment, the computer program generated a loss function based on a distance matrix when executed by a processor, including: processing the distance matrix through a fully connected layer to obtain a weight matrix; processing the distance matrix through a picture description model to be trained to obtain a distinction matrix; generating a target matrix based on the weight matrix and the distinction matrix; generating a loss function based on the target matrix.

[0135] In one embodiment, when a computer program is executed by a processor, the distance matrix is ​​processed by a to-be-trained generated picture description model to obtain a distinction matrix, including: inputting the distance matrix into the to-be-trained generated picture description model to obtain a reference matrix; processing the reference matrix through a normalization layer to obtain a distinction matrix.

[0136] In one embodiment, the computer program generated based on the target matrix when executed by the processor includes: calculating the mean square sum of each sample based on the target matrix; and obtaining the loss function based on the mean square sum of each sample and the number of samples.

[0137] In one embodiment, a computer program product is provided, including a computer program, which implements the following steps when executed by a processor: receiving a target image and a question corresponding to the target image; processing the target image through a target-generated image description model to obtain target image features, wherein the target-generated image description model is obtained based on a training method for generating an image description model in any of the above embodiments; and processing the target image features and the question through a large model obtained by pre-training to obtain an answer to the question.

[0138] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0139] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0140] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0141] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A training method for generating an image description model, characterized in that: The method comprises: Acquire a sample, wherein the sample includes a sample image and text corresponding to the sample image; Inputting the sample image and the text into a model to be trained to generate image descriptions to obtain text-combined image features; Inputting the text into a picture description model to be trained to obtain text features; Obtaining a distance matrix based on the text combined with the image features and the text features; A loss function is generated based on the distance matrix, and the generated picture description model to be trained is trained by using the loss function to obtain a target generated picture description model.

2. The method according to claim 1, characterized in that The obtaining of a distance matrix based on the text combined with the image features and the text features includes: The difference between the text-combined image feature and the text feature is calculated element by element to obtain a distance matrix.

3. The method according to claim 1 or 2, characterized in that: The generating a loss function based on the distance matrix comprises: Processing the distance matrix through a fully connected layer to obtain a weight matrix; Processing the distance matrix by the image description model to be trained to obtain a distinguishability matrix; generating a target matrix based on the weight matrix and the discriminability matrix; A loss function is generated based on the target matrix.

4. The method according to claim 3, characterized in that The step of processing the distance matrix by using the image description model to be trained to obtain a distinguishability matrix includes: Inputting the distance matrix into the image description model to be trained to obtain a reference matrix; The reference matrix is ​​processed through a normalization layer to obtain a discriminant matrix.

5. The method according to claim 3, characterized in that: The generating a loss function based on the target matrix includes: Calculate the mean square sum of each sample based on the target matrix; A loss function is obtained based on the mean square sum of each of the samples and the number of samples.

6. A method for generating a picture description, characterized in that: The method further comprises: Receiving a target image and a question corresponding to the target image; Processing the target image through a target generated image description model to obtain target image features, wherein the target generated image description model is obtained based on the training method for generating an image description model according to any one of claims 1 to 5; The target image features and questions are processed by a pre-trained large model to obtain an answer to the question.

7. A training device for generating an image description model, characterized in that: The device comprises: A sample acquisition module, used to acquire a sample, wherein the sample includes a sample image and text corresponding to the sample image; A first feature generation module, used for inputting the sample image and the text into a to-be-trained generated image description model to obtain text-combined image features; A second feature generation module is used to input the text into a to-be-trained generated picture description model to obtain text features; A distance matrix generation module, used to obtain a distance matrix based on the text combined with image features and the text features; A training module is used to generate a loss function based on the distance matrix, and train the generated picture description model to be trained through the loss function to obtain a target generated picture description model.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images