Methods, apparatus, media, and equipment for determining the image description information generation model

Abstract

This disclosure relates to the field of image processing technology, specifically to a method and apparatus for determining an image description information generation model, a computer-readable storage medium, and an electronic device. The method includes: acquiring a first reference model and a model to be trained, the model to be trained including a unimodal image encoder, a unimodal text encoder, a multimodal text encoder, and a multimodal text decoder; determining a first loss function based on the unimodal image encoder, unimodal text encoder, multimodal text encoder, and multimodal text decoder; training the model to be trained according to the first loss function and the first reference model to obtain a target model to be trained; and determining an image description information generation model based on the unimodal image encoder and multimodal text decoder in the target model to be trained; wherein the model parameters of the model to be trained are fewer than those of the first reference model. The image description information generation model obtained by this disclosure has fewer parameters and higher accuracy.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and more specifically, to a method for determining an image description information generation model, an apparatus for determining an image description information generation model, a computer-readable storage medium, and an electronic device. Background Technology

[0002] Image description generation refers to generating descriptive text that matches the content of a given image. In related technologies, neural network models can be used to generate image descriptions.

[0003] However, the image description information generation models obtained by related technologies have too many parameters, making them difficult to apply to mobile devices such as smartphones. On the other hand, image description information generation models with fewer parameters generate image description information with reduced accuracy.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of this disclosure is to provide a method, apparatus, computer-readable medium, and electronic device for determining an image description information generation model, thereby reducing the number of parameters in the image description information generation model to at least a certain extent and ensuring the accuracy of the generated image description information.

[0006] According to a first aspect of this disclosure, a method for determining an image description information generation model is provided, comprising: obtaining a first reference model and a model to be trained, wherein the model to be trained includes a unimodal image encoder, a unimodal text encoder, a multimodal text encoder, and a multimodal text decoder; determining a first loss function based on the unimodal image encoder, the unimodal text encoder, the multimodal text encoder, and the multimodal text decoder; training the model to be trained based on the first loss function and the first reference model to obtain a target model to be trained; determining an image description information generation model based on the unimodal image encoder and the multimodal text decoder in the target model to be trained, wherein the image description information generation model is used to generate text description information corresponding to an image; wherein the model parameters of the model to be trained are fewer than the model parameters of the first reference model; and the input data of the multimodal text encoder and the multimodal text decoder includes the output data of the unimodal image encoder.

[0007] According to a second aspect of this disclosure, an apparatus for determining an image description information generation model is provided, comprising: a first acquisition module configured to acquire a first reference model and a model to be trained, the model to be trained including a unimodal image encoder, a unimodal text encoder, a multimodal text encoder, and a multimodal text decoder; a first loss function determination module configured to determine a first loss function based on the unimodal image encoder, the unimodal text encoder, the multimodal text encoder, and the multimodal text decoder; a first training module configured to train the model to be trained according to the first loss function and the first reference model to obtain a target model to be trained; and an image description information generation model determination module configured to determine an image description information generation model based on the unimodal image encoder and the multimodal text decoder in the target model to be trained, the image description information generation model being used to generate text description information corresponding to an image; wherein the model parameters of the model to be trained are fewer than the model parameters of the first reference model; and the input data of the multimodal text encoder and the multimodal text decoder includes the output data of the unimodal image encoder.

[0008] According to a third aspect of this disclosure, a computer-readable storage medium is provided, on which a computer program is stored, wherein when the computer program is executed by a processor, it implements the method for determining the image description information generation model described in the first aspect above.

[0009] According to a fourth aspect of this disclosure, an electronic device is provided, characterized in that it includes: one or more processors; and a memory for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the method for determining the image description information generation model described in the first aspect above.

[0010] According to a fifth aspect of this disclosure, a computer program product comprising instructions is provided, which, when run on a computer, causes the computer to perform the steps of the method for determining a picture description information generation model as described in the first aspect.

[0011] The technical solution disclosed herein has the following beneficial effects:

[0012] On the one hand, this disclosure improves the training quality of the model by using a unimodal text encoder and a multimodal text encoder during the training phase of the model to be trained. The final image description information generation model does not require a unimodal text encoder and a multimodal text encoder, but only a unimodal image encoder and a multimodal text decoder, which can reduce the number of model parameters and make the model applicable to small mobile terminals such as smartphones. On the other hand, this disclosure uses a first reference model to train the model to be trained. The model parameters of the first reference model are more than those of the model to be trained, thereby borrowing the idea of ​​distillation to improve the accuracy of the determined image description information generation model.

[0013] In summary, this disclosure can reduce the number of model parameters in the image description information generation model while ensuring the accuracy of the generated image description information.

[0014] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0015] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:

[0016] Figure 1 A schematic diagram of a system architecture to which exemplary embodiments of the present disclosure may be applied is shown;

[0017] Figure 2 This illustration shows a method for determining an image description information generation model in an exemplary embodiment of the present disclosure;

[0018] Figure 3 This illustration shows a model architecture of a model to be trained in an exemplary embodiment of this disclosure;

[0019] Figure 4 A flowchart illustrating a method for obtaining a target model to be trained according to an exemplary embodiment of this disclosure is shown.

[0020] Figure 5 This diagram illustrates a flowchart of a method for generating an image description information based on a target model to be trained, according to an exemplary embodiment of this disclosure.

[0021] Figure 6 A model architecture diagram of a model to be tuned in an exemplary embodiment of this disclosure is shown;

[0022] Figure 7 This diagram illustrates the structure of a first depth-separable convolutional layer according to an exemplary embodiment of the present disclosure.

[0023] Figure 8 This diagram illustrates the structure of a second depth-separable convolutional layer according to an exemplary embodiment of the present disclosure.

[0024] Figure 9 This diagram illustrates a left-to-right masking mechanism during multimodal text decoder training, as shown in an exemplary embodiment of this disclosure.

[0025] Figure 10 A framework diagram illustrating another method for determining an image description information generation model in an embodiment of this disclosure is shown.

[0026] Figure 11 A schematic diagram illustrating an input image in an exemplary embodiment of this disclosure;

[0027] Figure 12 A schematic diagram illustrating another input image in an exemplary embodiment of this disclosure;

[0028] Figure 13 A schematic diagram of the structure of a device for determining an image description information generation model according to an exemplary embodiment of the present disclosure is shown.

[0029] Figure 14 A schematic diagram of an electronic device to which exemplary embodiments of the present disclosure may be applied is shown. Detailed Implementation

[0030] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0031] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0032] Image description generation refers to generating descriptive text that corresponds to the content of a given image. It requires both understanding the image's content and translating that understanding into natural language. This is an easy problem for humans with their strong abstract thinking abilities, but it is extremely challenging for machines. With the development of machine learning and deep learning, neural network models have become the dominant technology in the field of automatically generating image text descriptions.

[0033] Generally, neural network models used to generate text descriptions of images involve two main elements: a feature extraction model and a language model. The feature extraction model extracts salient features from the image, typically in the form of a fixed-length vector representation, while the language model predicts the probability of the next word in the text sequence based on the words already present in the sequence.

[0034] One related technique utilizes Faster R-CNN (Fast Region-based Convolutional Network) combined with RestNet (Residual Network) to detect salient visual features and object category information in images. After fusing image and text features, LSTM (Long Short-Term Memory) is used to sequentially generate natural language descriptions of the images using the weighted visual and object category information. However, extracting features first and then fusing them increases the processing complexity, and features are prone to misidentification, making it difficult to achieve accurate end-to-end generation of image-text descriptions.

[0035] Another related technique involves constructing a target network based on Generative Adversarial Networks (GANs) and the Actor-Critic algorithm. This target network comprises a generator network, a discriminator network, and a Critic network. Images from the training set, along with their text descriptions, are input into the target network to pre-train and adversarially train the generator and discriminator. The Actor-Critic algorithm is then used to update the generator's parameters step-by-step. The target image for which text descriptions are to be generated is then input into the trained generator to obtain the text descriptions for that target image. However, the training and updating process using GANs is slow, making it difficult to quickly obtain a high-quality model.

[0036] In another related technique, pre-saved image description feature information is obtained, which is determined based on the image's historical description information. This adds a process of retrieving historical similar image information, thus enriching the image description with historical textual description feature information. However, supplementing candidate feature information by retrieving images, and over-reliance on this method, can cause the model to become dependent on the retrieved description data. When the retrieved data is of poor quality, it significantly impacts the generation of description information.

[0037] In another related technique, a multimodal hybrid encoder-decoder model is pre-trained from a dataset guided by large-scale noisy image-text pairs to obtain an image captioning information generation model. While it performs well on image captioning generation tasks, the model has too many parameters, making it difficult to apply to mobile devices. Furthermore, the image captioning information generation models determined by the other related techniques mentioned above also suffer from the problem of excessive model parameters.

[0038] It is evident that while the accuracy of models in related technologies is increasing, the number of model parameters is also growing exponentially, making it difficult to apply large models to devices with limited computing and storage performance, such as mobile phones.

[0039] In view of the above problems, the exemplary embodiments of this disclosure provide a method and apparatus for determining an image description information generation model.

[0040] Below, let's first combine Figure 1 The system architecture of the operating environment for this exemplary embodiment is described in an exemplary manner.

[0041] Figure 1 A schematic diagram of the system architecture is shown. System architecture 100 may include a terminal 110 and a server 120. The terminal 110 may be a smartphone, a smart wearable electronic device (such as a smartwatch), a tablet computer, a desktop computer, a laptop computer, or other terminal device. The server 120 generally refers to the backend system that provides services related to the method for determining the image description information generation model in this exemplary embodiment; it may be a single server or a cluster of multiple servers. The terminal 110 and the server 120 can be connected via a wired or wireless communication link for data interaction.

[0042] In one exemplary implementation, the above method can be performed by server 120. For example, server 120 can obtain a first reference model, a model to be trained, and training data, and then train the model to be trained according to the training data and the first reference model to obtain a target model to be trained, and then obtain an image description information generation model according to the unimodal image encoder and multimodal text decoder in the target model to be trained.

[0043] In one exemplary embodiment, after the server 120 obtains the image description information generation model by executing the above method, it can send the obtained image description information generation model to the terminal 110. The terminal 110 can select any image in the terminal, and then automatically generate text description information for the selected image based on the image description information generation model.

[0044] In another exemplary embodiment, the method described above can also be performed by terminal 110. For example, terminal 110 can train the model to be trained based on training data and a first reference model to obtain a target model to be trained, and then determine an image description information generation model based on the unimodal image encoder and multimodal text decoder in the target model to be trained. Then, based on this image description information generation model, text description information for any image selected in the terminal can be generated.

[0045] As can be seen from the above, the execution subject of the method for determining the image description information generation model in this exemplary embodiment can be the aforementioned terminal 110 or server 120, and this disclosure does not limit it in this regard.

[0046] The following is combined Figure 2 The method for determining the image description information generation model in this exemplary embodiment will be described. Figure 2 An exemplary flow of the method for determining the image description information generation model is shown, which may include:

[0047] Step S210: Obtain a first reference model and a model to be trained, wherein the model to be trained includes a single-modal image encoder, a single-modal text encoder, a multimodal text encoder, and a multimodal text decoder;

[0048] Step S220: Determine a first loss function based on the single-modal image encoder, the single-modal text encoder, the multimodal text encoder, and the multimodal text decoder;

[0049] Step S230: Train the model to be trained according to the first loss function and the first reference model to obtain the target model to be trained;

[0050] Step S240: Based on the single-modal image encoder and multimodal text decoder in the target training model, determine the image description information generation model, which is used to generate text description information corresponding to the image;

[0051] The model parameters of the model to be trained are fewer than those of the first reference model; the input data of the multimodal text encoder and the multimodal text decoder include the output data of the unimodal image encoder.

[0052] Based on the above method, on the one hand, this disclosure improves the training quality of the model by using a unimodal text encoder and a multimodal text encoder during the training phase of the model to be trained. The final image description information generation model does not require a unimodal text encoder and a multimodal text encoder, but only a unimodal image encoder and a multimodal text decoder, which can reduce the number of model parameters and enable the model to be applied in small mobile terminals such as smartphones. On the other hand, this disclosure uses a first reference model to train the model to be trained. The model parameters of the first reference model are more than those of the model to be trained, thereby borrowing the idea of ​​distillation to improve the accuracy of the determined image description information generation model.

[0053] In summary, this disclosure can reduce the number of model parameters in the image description information generation model while ensuring the accuracy of the generated image description information.

[0054] The following is about Figure 2 Each step in the illustrated embodiment is explained in detail.

[0055] In step S210, the first reference model and the model to be trained are obtained.

[0056] In one exemplary implementation, the model to be trained includes a unimodal image encoder, a unimodal text encoder, a multimodal text encoder, and a multimodal text decoder. In the model to be trained, the input data for the multimodal text encoder and the multimodal text decoder includes the output data of the unimodal image encoder.

[0057] Among them, a unimodal image encoder only encodes features of an image, and its input data is only image data; a unimodal text encoder only encodes features of text, and its input data is only text data. A multimodal text encoder can be understood as an image-based text encoder; in other words, it also refers to the encoding features of the input image when encoding the input text. A multimodal text decoder can be understood as an image-based text decoder.

[0058] In one exemplary implementation, the model to be trained can be understood as a small model, and the first reference model can be understood as a large model. That is, the number of model parameters of the model to be trained is smaller than the number of model parameters of the first reference model.

[0059] For example, the first reference model can include a unimodal text encoder, a unimodal image encoder, a multimodal text encoder, and a multimodal text decoder. However, the encoders or decoders in the first reference model are large models with many parameters to ensure the prediction accuracy of the first reference model. In contrast, the encoders or decoders in the model to be trained are small models with few parameters. For instance, the unimodal image encoder in the model to be trained can be any of the miniaturized image coding models such as Vit (VisionTransformer), shallow ResNet (Residual Network), or MobileNet (Lightweight Convolutional Neural Network). The unimodal text encoder in the model to be trained can be an ALBERT (A Lite Bidirectional Encoder Representations from Transformers) model. The multimodal text encoder in the model to be trained can be an ALBERT model based on cross-attention, that is, adding a cross-attention layer to the ALBERT model. The multimodal text decoder in the model to be trained can also be an ALBERT model based on cross-attention. The multimodal text decoder in the model to be trained can also be other miniaturized decoders, such as LSTM or shallow transformer decoders.

[0060] For example, Figure 3 This illustration shows a model architecture for a model to be trained in an exemplary embodiment of this disclosure. For example... Figure 3 As shown, a unimodal text encoder may include a self-attention layer and a feedforward layer; a multimodal text encoder may include a self-attention layer, a cross-attention layer, and a feedforward layer; and a multimodal text decoder may include a causal self-attention layer, a cross-attention layer, and a feedforward layer. For the unimodal text encoder, multimodal text encoder, and multimodal text decoder, they can share all model parameters except for the self-attention layer and the cross-attention layer; for example, they can share the model parameters of the feedforward layer. Figure 3 As shown, the input data of the multimodal text encoder and the multimodal text decoder, including the output data of the unimodal image encoder, can be understood as follows: the image features encoded by the unimodal encoder are used as the input data of the cross-attention layers of the multimodal text encoder and the multimodal text decoder, respectively.

[0061] For example, before obtaining the first reference model, a first reference model can be determined first. The determination of the first reference model includes: obtaining a first image-text pair training dataset; training an initial first reference model based on the first image-text pair training dataset to obtain a first reference model to be optimized; obtaining a second image-text pair training dataset; and training the first reference model to be optimized based on the second image-text pair training dataset to obtain the first reference model; wherein the first image-text pair training dataset is determined based on image-text pairs crawled from the web, and the second image-text pair training dataset is determined based on labeled image-text pairs.

[0062] For example, a large-scale image-text pair dataset can be obtained first through web scraping. Web scraping can be performed with user authorization, and the scraped image-text pairs do not infringe on user privacy. The scraped image-text pairs can be publicly available online. For instance, images and their corresponding titles and descriptions can be scraped from the web, and the title and description information can be used as the image's text description information to generate a first image-text pair training dataset. A large number of image-text pairs can be obtained through web scraping, and an initial first reference model can be pre-trained based on this dataset to obtain a first reference model to be optimized. If the loss function of the initial first reference model is less than a first preset value, training can be stopped, resulting in the first reference model to be optimized. Since the text description information in the image-text pairs used during pre-training is image titles and other information scraped from the web, the descriptions of the images are relatively simple. Therefore, the model to be optimized can be further fine-tuned based on a pre-labeled, high-quality image-text pair dataset to improve model performance, thus obtaining the first reference model.

[0063] Training large-scale models requires a large amount of training data to ensure predictive accuracy. However, obtaining a large amount of labeled data is labor-intensive and inefficient. Therefore, this disclosure uses web crawling to automatically acquire a large amount of image and text data for pre-training. The pre-trained large-scale model is then fine-tuned based on the high-quality labeled image and text dataset, thereby improving training efficiency while maintaining model accuracy.

[0064] Continue to refer to Figure 2 In step S220, a first loss function is determined based on the single-modal image encoder, the single-modal text encoder, the multimodal text encoder, and the multimodal text decoder.

[0065] For example, one specific implementation of step S220 may include: determining an image-text contrast loss based on the unimodal image encoder and the unimodal text encoder; determining an image-text matching loss based on the multimodal text encoder; determining a language modeling loss based on the loss between the output of the multimodal text decoder and the text description tags corresponding to the input image; and determining a first loss function based on the image-text contrast loss, the image-text matching loss, and the language modeling loss.

[0066] For example, the loss in the first loss function may include, for example, Figure 5 The diagram shows ITC (Image-Text Contrastive Loss), ITM (Image-Text Matching Loss), and LM (Language Modeling Loss).

[0067] For image-text contrast loss, its main purpose is to encourage positive image-text sample pairs (rather than negative image-text sample pairs) to have similar representations in order to align the feature spaces of the visual and text transformers.

[0068] For image-text matching loss, the input data is the concatenation of image and text in matching or non-matching image-text pairs. Non-matching image-text pairs are constructed by randomly selecting text descriptions for a given image in the dataset. The multimodal text encoder needs to handle a binary classification task: predicting whether the input image-text pairs match.

[0069] For the language modeling loss, the input can be a concatenation of an image and partially occluded text. The multimodal text decoder can predict the occluded text. The cross-entropy loss between the predicted text feature representation and the actual occluded text feature representation of the multimodal text decoder can be used as the corresponding language modeling loss.

[0070] For example, the first loss function can be the sum of the ITC loss, ITM loss, and LM loss mentioned above. Correspondingly, different weights can be assigned to the ITC loss, ITM loss, and LM loss according to the actual situation to indicate their importance in the training process.

[0071] Continue to refer to Figure 2 In step S230, the model to be trained is trained according to the first loss function and the first reference model to obtain the target model to be trained.

[0072] For example, Figure 4This diagram illustrates a flowchart of a method for obtaining a target model to be trained, according to an exemplary embodiment of this disclosure. (See reference...) Figure 4 The method may include steps S410 to S440. Wherein:

[0073] In step S410, the first cross-entropy loss function is determined based on the probability distribution between labeled and unlabeled values ​​in the prediction results of the first reference model and the probability distribution between labeled and unlabeled values ​​in the prediction results of the model to be trained.

[0074] For example, the first reference model and the model to be trained can be fed the same training data to obtain their respective prediction results for the input training data. Both the first reference model and the model to be trained treat the image description information generation problem as a multi-classification problem, and can predict the output probability of text in a candidate text library based on the input image.

[0075] If the text description label of the input image is A, the probability of A in the prediction result of the first reference model is 0.7, and the sum of the probabilities of all other non-A values ​​is 0.3. Therefore, the probability distribution between the labeled and non-labeled values ​​in the prediction result of the first reference model is 0.7–0.3. The probability of A in the prediction result of the model to be trained is 0.6, and the sum of the probabilities of all other non-A values ​​is 0.4. Therefore, the probability distribution between the labeled and non-labeled values ​​in the prediction result of the model to be trained is 0.6–0.4. Based on the cross-entropy loss between these two distributions, the first cross-entropy loss function is determined.

[0076] In step S410, the probability distribution can be understood as the logits probability distribution, which is the output of the fully connected layer of the model and is not normalized. The logits probability distribution can be passed through the softmax (multi-class classification) layer to obtain the final output result of the model.

[0077] Continue to refer to Figure 4 In step S420, the first relative entropy loss function is determined based on the probability distribution between unlabeled values ​​in the prediction results of the first reference model and the probability distribution between unlabeled values ​​in the prediction results of the model to be trained.

[0078] The first reference model and the model to be trained predict the probability of the corresponding text description of the input image being a text description from the candidate text description library, and select the one with the highest probability as the text description prediction result for the input image. In other words, it can also predict the probability of the corresponding text description of the input image being other incorrect text descriptions. The KL (Kullback–Leibler divergence) loss can be calculated between the probability distributions of all incorrect text descriptions predicted by the first reference model and the probability distributions of all incorrect text descriptions predicted by the model to be trained, thus obtaining the first relative entropy loss function.

[0079] In step S430, a first target loss function is determined based on the first cross-entropy loss function and the first relative entropy loss function.

[0080] For example, weights can be assigned to the first cross-entropy loss function and the first relative entropy loss function, thereby determining the first target loss function based on the corresponding weights. If the weight of the first cross-entropy loss function is the first weight and the weight of the second cross-entropy loss function is the second weight, then the first product of the first weight and the first cross-entropy loss function and the second product of the second weight and the first relative entropy loss function can be calculated, and the first target loss function can be determined based on the sum of the first product and the second product.

[0081] In one exemplary implementation, the process of training the model to be trained based on the first reference model can be understood as a model distillation process. Therefore, when determining the first target loss function, the distillation temperature can also be considered. That is, the first target loss function can be determined based on the first cross-entropy function, the first relative entropy loss function, and the distillation temperature.

[0082] In step S440, based on the first loss function and the first target loss function, the training loss function of the model to be trained is determined, so as to train the model to be trained according to the training loss function of the model to be trained, and obtain the target model to be trained.

[0083] For example, the training loss function of the model to be trained can be determined based on the sum of the first loss function and the first target loss function. The training of the model is then carried out with the goal of reducing the value of the training loss function. Training can be stopped when the value of the training loss function is less than a second preset value, resulting in a test model. The performance metrics of the test model are then tested using test samples. If the performance metrics, such as accuracy, meet the requirements, the test model can be designated as the target model.

[0084] For example, steps S410 to S440 above can be understood as a model distillation process, which can improve the model accuracy of the model to be trained. The distillation loss in related technologies is calculated using KL divergence, and can be expressed as the following formula (1):

[0085]

[0086] In formula (1), KL represents the KL divergence, and logits t The output of the teacher model, such as the first reference model in this disclosure, is logits. s The output of the student model, such as the output of the model to be trained in this disclosure, is T, which represents the distillation temperature and can be set to 1, 2, 4, 8, etc.

[0087] In this disclosure, considering that the process of generating text description information is actually a probability prediction on a large vocabulary (candidate text description library) to generate the final text description result, as shown in steps S410 to S430 above, when calculating the distillation loss (the first objective loss function above), not only the loss of positive samples is considered, but also the loss of negative samples, in order to improve the learning effect of the model. In other words, the first objective loss function in this disclosure can be determined by the following formula (2):

[0088]

[0089] In formula (2), This represents the probability distribution between labeled and unlabeled values ​​in the prediction results of the first reference model, with the labeled value represented by 1 as an example. This represents the probability distribution between 1 and non-1 predicted by the first reference model. Let represent the probability distribution between labeled and unlabeled values ​​in the prediction results of the model to be trained. This represents the probability distribution among all unlabeled values ​​in the prediction results of the first reference model. This represents the probability distribution among all non-label values ​​in the prediction results of the model to be trained. `cross_entropy` represents the cross-entropy loss, and `KL` represents the KL divergence loss (i.e., relative entropy loss). `α` represents the weight of the KL divergence loss; adjusting the value of `α` adjusts the weight between the first cross-entropy loss and the first relative entropy loss.

[0090] Continue to refer to Figure 2 In step S240, based on the single-modal image encoder and multimodal text decoder in the target training model, an image description information generation model is determined, which is used to generate text description information corresponding to the image.

[0091] For example, Figure 5 This diagram illustrates a flowchart of a method for generating an image description information model based on a target model to be trained, according to an exemplary embodiment of this disclosure. (Reference) Figure 5 The method may include steps S510 to S520. Wherein:

[0092] In step S510, the model to be optimized is determined based on the single-modal image encoder and multimodal text decoder in the target model to be trained.

[0093] For example, after identifying the target model to be trained, the unimodal text encoder and multimodal text encoder in the target model to be trained can be removed, and the unimodal image encoder and multimodal text decoder can be retained. Based on the unimodal image encoder and multimodal text decoder, the model to be tuned can be determined, and the model to be tuned can be tuned to determine the final image description information generation model.

[0094] For example, Figure 6 This diagram illustrates the model architecture of a model to be tuned in an exemplary embodiment of this disclosure. In this disclosure, a single-modal image encoder first performs convolutional dimensionality reduction on the input image, then divides the original image into blocks, flattens them into a sequence, and after multiple dimensionality reductions, inputs them into MobileVit Blocks (lightweight visual transformer modules) to associate information between image blocks. Finally, the image sequence representation is flattened to obtain a text-like representation of the image.

[0095] refer to Figure 6 Taking a 3*384*384 RGB three-channel image as an example, the specific calculation process of the single-modal image encoder in this disclosure is as follows: First, it undergoes a 3*3 convolution and downsampling, i.e., Conv_output = Conv2d[in_channels = 3, out_channels = 3, kernel_size = (3,3), stride = 2, padding = 1]. Then, it enters the MV2 module, which includes one first depthwise separable convolutional layer MV21 and two downsampled second depthwise separable convolutional layers MV22. The structure of the first depthwise separable convolutional layer... Figure 7 As shown, the structure of the second depth separable convolutional layer is as follows: Figure 8 As shown. Among them, Figure 7 and Figure 8In this context, ReLU6 is the activation function, `linear` refers to linear operation, `Dwise` (Depthwise) refers to depthwise convolution, and `conv` refers to convolution. The first depthwise separable convolutional layer can be understood as a depthwise separable convolutional layer with a stride of 1, which does not change the image size. The second depthwise separable convolutional layer can be understood as a depthwise separable convolutional layer with a stride of 2, which downsamples the image by half its size. Figure 6 As shown, after two second-depth separable convolutional layers, the input image size becomes 48*48. Then, the model goes through MobileVit Blocks and the second-depth separable convolutional layer MV22 to perform interactive operations on the block image. After looping twice, the output image encoding state visual_features is given.

[0096] In one exemplary embodiment, the multimodal text decoder of this disclosure may include multiple layers of transformers. In the case that the multimodal text decoder includes multiple layers of transformers, for the current layer transformer of the multimodal text encoder, a cross-attention result is determined between the output data of the unimodal image encoder and the output data of the current layer transformer of the multimodal text decoder; the cross-attention result is used as the input data of the next layer transformer of the current layer transformer of the multimodal text decoder to train the model to be tuned.

[0097] As previously stated, the multimodal text decoder in this disclosure can be understood as an ALBERT model with added cross-attention, and the ALBERT model includes multiple layers of transformers. Figure 6 As shown, Figure 6 With one of the transformer layers (i.e. Figure 6 Taking the Albert transformer as an example, during the training phase of the model to be tuned and the training phase of the model to be trained, the input of each transformer layer in the multimodal text decoder includes the cross-attention result between the output of the previous transformer layer and the visual_features of the image encoding state output by the unimodal image encoder. Figure 6As shown, the image encoding state `visual_features` output by the unimodal image encoder can be converted into a 144-dimensional vector sequence, thus matching the input data size of the transformer in the multimodal text decoder. Then, the cross-attention between the output of the transformer in the previous layer of the multimodal text decoder and the corresponding 144-dimensional vector sequence of `visual_features` is calculated and used as the input data for the next layer of the transformer. In this way, image information is considered during text decoding, improving the accuracy of the image-text description information generated by the model.

[0098] For example, a multimodal text decoder can generate a sequence by cyclically decoding the visual features of the input image and the input text. Figure 6 The embedding layer in this context is the embedding layer.

[0099] In one exemplary implementation, the ALBERT model, a highly efficient and strongly quantized language representation model, is better suited for Natural Language Understanding (NLU) tasks in Natural Language Processing. To transform an NLU-based model into one suitable for Natural Language Generation (NLG), this disclosure employs a left-to-right masking mechanism for the input text during training. For example... Figure 9 As shown, in the first loop decoding, only the first character is input, and the following four characters are masked. In the second loop decoding, the first character and the prediction result based on the first character are used as input, and the following three characters are masked. This input is then fed into the multimodal text decoder to continue predicting the next character in the text description, and so on, until all characters are predicted. This process generates text descriptions from left to right, which is more suitable for text generation tasks and improves the accuracy of the image text descriptions generated by the model.

[0100] Continue to refer to Figure 5 In step S520, a second reference model is obtained, and the model to be optimized is trained based on the second reference model to obtain an image description information generation model.

[0101] In one exemplary implementation, the model parameters of the model to be tuned are fewer than those of the model parameters of the second reference model.

[0102] For example, the second reference model may also include a unimodal image encoder and a multimodal text decoder. However, the unimodal image encoder and multimodal text decoder in the second reference model are large networks, while the unimodal image encoder and multimodal text decoder in the model to be tuned can be small networks as described above. The method for determining the second reference model can refer to the method for determining the first reference model described above, and will not be repeated here.

[0103] Since the single-modal and multimodal text encoders in the target training model are removed from the model to be tuned, the accuracy of the model will be affected. Therefore, in order to ensure the model effect of the final determined image text description generation model, the model to be tuned can be trained to reduce the number of model parameters while ensuring the prediction effect of the model.

[0104] For example, one specific implementation of step S520 may include: obtaining a second reference model; determining a second cross-entropy loss function based on the probability distribution between labeled and unlabeled values ​​in the prediction results of the second reference model and the probability distribution between labeled and unlabeled values ​​in the prediction results of the model to be trained; determining a second relative entropy loss function based on the probability distribution between unlabeled values ​​in the prediction results of the second reference model and the probability distribution between unlabeled values ​​in the prediction results of the model to be trained; determining a second target loss function based on the second cross-entropy loss function and the second relative entropy loss function; determining a second loss function based on the loss between the output of the multimodal text decoder in the model to be tuned and the text description labels corresponding to the input image; and determining a training loss function for the model to be tuned based on the second loss function and the second target loss function, so as to train the model to be tuned according to the training loss function of the model to be tuned to obtain an image description information generation model.

[0105] For example, the second objective loss function can also be understood as the distillation loss function mentioned above, which can be referred to in formula (2) above. This represents the probability distribution between labeled and unlabeled values ​​in the prediction results of the second reference model. Let represent the probability distribution between labeled and unlabeled values ​​in the prediction results of the model to be tuned. This represents the probability distribution among all unlabeled values ​​in the prediction results of the second reference model. Given the probability distribution among all non-label values ​​in the prediction results of the model to be tuned, the above formula (2) can characterize the second objective loss function. The second loss function can be understood as the loss function determined based on the above LM loss. Based on the sum of the second loss function and the second objective loss function, the training loss function of the model to be tuned can be determined. Thus, based on the training loss function of the model to be tuned, the second reference model, and the training dataset determined by merging the above first training dataset and the second training dataset, the model to be tuned is trained.

[0106] Training can be stopped when the value of the training loss function of the model to be tuned is less than the third preset value, thus obtaining the model to be tested and tuned. The model to be tested and tuned is then trained again using test samples. If the model's test performance metrics meet the requirements, the current model to be tested and tuned can be identified as the image description information generation model. If the model's test performance metrics do not meet the requirements, parameters such as the weights of the second loss function and the second objective loss function can be adjusted, and training can be repeated until the model's test performance metrics meet the requirements.

[0107] In other words, in this invention, during the model pre-training phase, i.e., the training phase of the model to be trained, joint training can be performed using a unimodal image encoder, a unimodal text encoder, a multimodal text encoder, and a multimodal text decoder to improve the model's training accuracy. To obtain a smaller model, the unimodal and multimodal text encoders from the pre-training phase can be removed, while the unimodal image encoder and multimodal text decoder obtained during the pre-training phase are further fine-tuned to reduce the number of model parameters while maintaining the model's prediction accuracy.

[0108] Furthermore, in this invention, the idea of ​​distillation is applied simultaneously to both the training phase of the model to be trained and the training phase of the model to be tuned, that is, simultaneously to the pre-training phase and the model tuning phase, which can further guarantee the prediction accuracy of the small model in this invention.

[0109] For example, Figure 10 A framework diagram illustrating another method for determining an image description information generation model according to an embodiment of this disclosure is shown. (See reference...) Figure 10The specific implementation process of determining the image description information generation model can include: pre-training a large model based on image text training data; using the large model as the first reference model and the distillation loss function in formula (2) above, pre-training a small model based on the image text training dataset to obtain the pre-trained small model. Then, removing the unimodal text encoder and multimodal text encoder from the pre-trained large model, retaining only the unimodal image encoder and multimodal text decoder from the pre-trained large model to obtain the large model to be tuned, and performing tuning training on the large model to be tuned based on the image text training data to obtain the tuning result of the large model; at the same time, removing the unimodal text encoder and multimodal text encoder from the pre-trained small model to obtain the small model to be tuned; then, performing tuning training on the small model to be tuned based on the tuning result of the large model and the distillation loss function in formula (2) above to determine the image description information generation model.

[0110] The pre-trained large model can be understood as the first reference model mentioned above, and the large model to be optimized after fine-tuning can be understood as the second reference model mentioned above. In other words, the initial first reference model can be trained firstly based on the training dataset to obtain the first reference model. The initial first reference model includes a unimodal text encoder, a unimodal image encoder, a multimodal text encoder, and a multimodal text decoder; then, the unimodal text encoder and multimodal text encoder in the proposed first reference model are retained, while the unimodal image encoder and multimodal text decoder are retained to obtain the first reference model to be optimized; then, the first reference model to be optimized is trained again based on the training dataset to obtain the second reference model.

[0111] For example, the image description information generation model determined in this embodiment can automatically generate text description information corresponding to the input image. For example, the input... Figure 11 After inputting the image, the generated text description is "There is a person planting rice seedlings in a wide field"; Figure 12 The generated text description following the image is "Two people wearing hats are sitting on a bench by the sea".

[0112] The image description information generation model determined in this disclosure has fewer parameters and generates better image description information compared to other small models. Using other small models for generating image captioning information, CLIP-CAP (Contrastive Language–Image Pre-training prefix for image captioning, a cross-modal pre-trained model based on contrastive image-text learning) and OFA-TINY (Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework, a multimodal, multi-task unified model), as comparison objects, evaluations were conducted on the MSCOCO (Microsoft COCO: Common Objects in Context, a dataset for object recognition) dataset. The comparison results of CLIP-CAP, OFA-TINY, and the small model in this disclosure are shown in Table 1. As can be seen from Table 1, the small model in this disclosure has the fewest parameters, 58% of the parameters of the CLIP-CAP model and 71% of the parameters of the OFA-TINY model. Furthermore, it outperforms the other two models on the CIDEr (Consensus-based image description evaluation, an evaluation metric specifically for evaluating image captioning tasks) performance metric, and also performs better on BLEU-4 (Bilingual) performance. It also ranks highly in EvaluationUnderstudy (a bilingual assessment alternative).

[0113] Table 1 compares the results of three small models used to generate image descriptions.

[0114] Model Number of parameters (M) BLEU-4 CIDEr CLIP-CAP 43 0.335 1.131 OFA-TINY 35 0.359 1.19 The model in this disclosure 25 0.352 1.204

[0115] In one exemplary implementation, because the image description information generation model determined in this disclosure has a small number of model parameters, it can be deployed on terminal devices with limited processing power, such as smartphones. For example, the determined image description information generation model can be converted into ONNX (Open Neural Network Exchange) format, and the converted ONNX model can be deployed on terminal devices such as smartphones to complete the image description information generation task. Of course, the determined image description information generation model can also be converted into other format models for deployment on terminal devices such as smartphones, such as MNN (Lightweight Deep Neural Network Inference Engine) or TFLite (a new model file format based on FlatBuffers).

[0116] In one exemplary application scenario, the image description information generation model defined in this disclosure can be applied in fields such as visual impairment assistance, multimedia retrieval, and assisted monitoring. Taking the field of visual impairment assistance as an example, by clicking the image control in the visual impairment assistance control, users can take a photo or select a picture from their album. After taking a photo or selecting a picture from the album, the image is automatically loaded and displayed, and the image description information generation model defined in this disclosure is used to automatically generate text descriptions of the photos taken in real time or selected from the album. Simultaneously, a voice playback mode can be provided. Clicking the voice playback control can play the generated text descriptions, or the generated text descriptions can be played automatically after they are generated, assisting people with color weakness, color blindness, and other visual impairments to accurately obtain the information expressed by the images.

[0117] In the training phase of the model to be trained, this disclosure provides a loss function through a unimodal text encoder and a multimodal text encoder to improve the training quality of the model. The final image description information generation model does not require a unimodal text encoder and a multimodal text encoder, but only a unimodal image encoder and a multimodal text decoder, which can reduce the number of model parameters and make the model applicable to small mobile terminals such as smartphones. In addition, this disclosure uses a first reference model to train the model to be trained. The model parameters of the first reference model are more than those of the model to be trained, thereby borrowing the idea of ​​distillation to improve the accuracy of the determined image description information generation model.

[0118] It should be noted that the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of this disclosure, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0119] Further reference Figure 13 As shown, the embodiment of this example also provides a device 1300 for determining an image description information generation model. The device 1300 may include: a first acquisition module 1310, a first loss function determination module 1320, a first training module 1330, and an image description information generation model determination module 1340. The system comprises: a first acquisition module 1310 configured to acquire a first reference model and a model to be trained, wherein the model to be trained includes a unimodal image encoder, a unimodal text encoder, a multimodal text encoder, and a multimodal text decoder; a first loss function determination module 1320 configured to determine a first loss function based on the unimodal image encoder, the unimodal text encoder, the multimodal text encoder, and the multimodal text decoder; a first training module 1330 configured to train the model to be trained according to the first loss function and the first reference model to obtain a target model to be trained; and an image description information generation model determination module 1340 configured to determine an image description information generation model based on the unimodal image encoder and the multimodal text decoder in the target model to be trained, wherein the image description information generation model is used to generate text description information corresponding to an image; wherein the model parameters of the model to be trained are fewer than the model parameters of the first reference model; and the input data of the multimodal text encoder and the multimodal text decoder includes the output data of the unimodal image encoder.

[0120] In an exemplary implementation, based on the foregoing embodiments, the first training module 1330 may be specifically configured to: determine a first cross-entropy loss function based on the probability distribution between labeled and unlabeled values ​​in the prediction results of the first reference model and the probability distribution between labeled and unlabeled values ​​in the prediction results of the model to be trained; determine a first relative entropy loss function based on the probability distribution between unlabeled values ​​in the prediction results of the first reference model and the probability distribution between unlabeled values ​​in the prediction results of the model to be trained; determine a first target loss function based on the first cross-entropy loss function and the first relative entropy loss function; and determine a training loss function for the model to be trained based on the first loss function and the first target loss function, so as to train the model to be trained according to the training loss function of the model to be trained to obtain a target model to be trained.

[0121] In one exemplary implementation, based on the foregoing embodiments, the image description information generation model determination module 1340 may include a model to be optimized determination unit and a second acquisition unit, wherein the model to be optimized determination unit is configured to determine the model to be optimized based on the single-modal image encoder and multi-modal text decoder in the target model to be trained; the second acquisition unit is configured to acquire a second reference model and train the model to be optimized according to the second reference model to obtain the image description information generation model; wherein the model parameters of the model to be optimized are less than the model parameters of the second reference model.

[0122] In one exemplary implementation, based on the foregoing embodiments, the step of obtaining a second reference model and training the model to be optimized based on the second reference model to obtain an image description information generation model includes: obtaining the second reference model; determining a second cross-entropy loss function based on the probability distribution between labeled and unlabeled values ​​in the prediction results of the second reference model and the probability distribution between labeled and unlabeled values ​​in the prediction results of the model to be trained; determining a second relative entropy loss function based on the probability distribution between unlabeled values ​​in the prediction results of the second reference model and the probability distribution between unlabeled values ​​in the prediction results of the model to be trained; determining a second target loss function based on the second cross-entropy loss function and the second relative entropy loss function; determining a second loss function based on the loss between the output of the multimodal text decoder in the model to be optimized and the text description labels corresponding to the input image; and determining a training loss function for the model to be optimized based on the second loss function and the second target loss function, so as to train the model to be optimized based on the training loss function to obtain an image description information generation model.

[0123] In an exemplary implementation, based on the foregoing embodiments, when the multimodal text decoder includes multiple layers of transformers, the apparatus further includes a cross-attention calculation module. This module can be specifically configured to: determine, for the current layer transformer of the multimodal text encoder, a cross-attention result between the output data of the unimodal image encoder and the output data of the current layer transformer of the multimodal text decoder; and use the cross-attention result as input data for the next layer transformer of the current layer transformer of the multimodal text decoder to train the model to be tuned.

[0124] In one exemplary implementation, based on the foregoing embodiments, the method for determining the first reference model includes: acquiring a first image-text pair training dataset, training an initial first reference model based on the first image-text pair training dataset to obtain a first reference model to be tuned; acquiring a second image-text pair training dataset, training the first reference model to be tuned based on the second image-text pair training dataset to obtain a first reference model; wherein, the first image-text pair training dataset is determined based on image-text pairs crawled from the network, and the second image-text pair training dataset is determined based on labeled image-text pairs.

[0125] In an exemplary implementation, based on the foregoing embodiments, the first loss function determination module 1320 may be specifically configured to: determine an image-text contrast loss based on the unimodal image encoder and the unimodal text encoder; determine an image-text matching loss based on the multimodal text encoder; determine a language modeling loss based on the loss between the output of the multimodal text decoder and the text description tags corresponding to the input image; and determine a first loss function based on the image-text contrast loss, the image-text matching loss, and the language modeling loss.

[0126] The specific details of each module in the above-mentioned device have been described in detail in the method section of the implementation. For any undisclosed details, please refer to the implementation content of the method section, and therefore will not be repeated here.

[0127] Exemplary embodiments of this disclosure also provide an electronic device for performing a method for determining an image description information generation model. This electronic device may be the terminal 110 described above. Generally, the electronic device may include a processor and a memory, the memory storing executable instructions of the processor, and the processor configured to perform the method for determining the image description information generation model by executing the executable instructions.

[0128] The memory can also be used to store the image description information generation model determined according to the above-mentioned method for determining the image description information generation model. The processor can be configured to process the input image according to the stored image description information generation model via executable instructions to obtain the text description information of the input image.

[0129] The following is based on Figure 14 Taking the mobile terminal 1400 as an example, the construction of this electronic device will be described by way of example. Those skilled in the art will understand that, apart from components specifically designed for mobile purposes, Figure 14 The structure can also be applied to fixed types of equipment.

[0130] like Figure 14As shown, the mobile terminal 1400 may specifically include: a processor 1401, a memory 1402, a bus 1403, a mobile communication module 1404, an antenna 1, a wireless communication module 1405, an antenna 2, a display screen 1406, a camera module 1407, an audio module 1408, a power module 1409, and a sensor module 1410.

[0131] Processor 1401 may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, an encoder, a decoder, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). For example, in this disclosure, a GPU can be used to accelerate the model training process.

[0132] The processor 1401 can be connected to the memory 1402 or other components via the bus 1403.

[0133] The memory 1402 can be used to store computer executable program code, which includes instructions. The processor 1401 executes various functional applications and data processing of the mobile terminal 1400 by running the instructions stored in the memory 1402. The memory 1402 can also store application data, such as images, videos, and other files, and can also store an image description information generation model determined according to the method of this disclosure.

[0134] The communication function of mobile terminal 1400 can be implemented through mobile communication module 1404, antenna 1, wireless communication module 1405, antenna 2, modem processor, and baseband processor. Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Mobile communication module 1404 can provide 2G, 3G, 4G, and 5G mobile communication solutions for mobile terminal 1400. Wireless communication module 1405 can provide wireless communication solutions such as wireless LAN, Bluetooth, and near-field communication for mobile terminal 1400.

[0135] The display screen 1406 is used to implement display functions, such as displaying the user interface, images, and videos. The camera module 1407 is used to implement shooting functions, such as capturing images and videos. The audio module 1408 is used to implement audio functions, such as playing audio and capturing voice. The power module 1409 is used to implement power management functions, such as charging the battery, powering the device, and monitoring the battery status. The sensor module 1410 may include a depth sensor 14101, a pressure sensor 14102, a gyroscope sensor 14103, a barometric pressure sensor 14104, etc., to implement corresponding sensing and detection functions.

[0136] Those skilled in the art will understand that various aspects of this disclosure can be implemented as a system, method, or program product. Therefore, various aspects of this disclosure can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."

[0137] Exemplary embodiments of this disclosure also provide a computer-readable storage medium storing a program product capable of implementing the methods described above. In some possible embodiments, various aspects of this disclosure can also be implemented as a program product including program code that, when run on a terminal device, causes the terminal device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure, such as executing... Figure 2 Any one or more steps in the process.

[0138] The computer-readable medium disclosed herein may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0139] In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wireline, optical fiber, RF, etc., or any suitable combination thereof.

[0140] Furthermore, program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0141] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0142] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for determining an image description information generation model, characterized in that, include: Obtain a first reference model and a model to be trained, wherein the model to be trained includes a unimodal image encoder, a unimodal text encoder, a multimodal text encoder, and a multimodal text decoder; A first loss function is determined based on the single-modal image encoder, the single-modal text encoder, the multimodal text encoder, and the multimodal text decoder; The model to be trained is trained according to the first loss function and the first reference model to obtain the target model to be trained; Based on the single-modal image encoder and multimodal text decoder in the target training model, an image description information generation model is determined, which is used to generate text description information corresponding to the image. Wherein, the model parameters of the model to be trained are fewer than the model parameters of the first reference model; the input data of the multimodal text encoder and the multimodal text decoder include the output data of the unimodal image encoder; The first reference model includes a unimodal image encoder, a unimodal text encoder, a multimodal text encoder, and a multimodal text decoder. The method for determining the first reference model includes: acquiring a first image-text pair training dataset; training an initial first reference model based on the first image-text pair training dataset to obtain a first reference model to be optimized; acquiring a second image-text pair training dataset; and training the first reference model to be optimized based on the second image-text pair training dataset to obtain the first reference model. The first image-text pair training dataset is determined based on image-text pairs crawled from the web, and the second image-text pair training dataset is determined based on labeled image-text pairs. The step of determining the first loss function based on the unimodal image encoder, the unimodal text encoder, the multimodal text encoder, and the multimodal text decoder includes: determining an image-text contrast loss based on the unimodal image encoder and the unimodal text encoder; and determining an image-text matching loss based on the multimodal text encoder. Based on the loss between the output of the multimodal text decoder and the text description labels corresponding to the input image, the language modeling loss is determined; based on the image-text contrast loss, the image-text matching loss, and the language modeling loss, the first loss function is determined. The step of training the model to be trained based on the first loss function and the first reference model to obtain a target model to be trained includes: determining a first cross-entropy loss function based on the probability distribution between labeled and unlabeled values ​​in the prediction results of the first reference model and the probability distribution between labeled and unlabeled values ​​in the prediction results of the model to be trained; determining a first relative entropy loss function based on the probability distribution between unlabeled values ​​in the prediction results of the first reference model and the probability distribution between unlabeled values ​​in the prediction results of the model to be trained; determining a first target loss function based on the first cross-entropy loss function and the first relative entropy loss function; and determining a training loss function for the model to be trained based on the first loss function and the first target loss function, so as to train the model to be trained based on the training loss function of the model to be trained to obtain the target model to be trained. The step of determining the image description information generation model based on the single-modal image encoder and multi-modal text decoder in the target training model includes: determining a model to be optimized based on the single-modal image encoder and multi-modal text decoder in the target training model; obtaining a second reference model; and training the model to be optimized based on the second reference model to obtain the image description information generation model; wherein the model parameters of the model to be optimized are less than the model parameters of the second reference model.

2. The method for determining the image description information generation model according to claim 1, characterized in that, The step of obtaining a second reference model and training the model to be optimized based on the second reference model to obtain an image description information generation model includes: Obtain a second reference model, and determine a second cross-entropy loss function based on the probability distribution between labeled and unlabeled values ​​in the prediction results of the second reference model and the probability distribution between labeled and unlabeled values ​​in the prediction results of the model to be trained. The second relative entropy loss function is determined based on the probability distribution between unlabeled values ​​in the prediction results of the second reference model and the probability distribution between unlabeled values ​​in the prediction results of the model to be trained. Based on the second cross-entropy loss function and the second relative entropy loss function, the second target loss function is determined; The second loss function is determined based on the loss between the output of the multimodal text decoder in the model to be tuned and the text description labels corresponding to the input images; Based on the second loss function and the second target loss function, the training loss function of the model to be tuned is determined, and the model to be tuned is trained according to the training loss function of the model to be tuned to obtain the image description information generation model.

3. The method for determining the image description information generation model according to claim 1, characterized in that, In the case where the multimodal text decoder includes multiple layers of transformers, the method further includes: For the current layer transformer of the multimodal text encoder, determine the cross-attention result between the output data of the single-modal image encoder and the output data of the current layer transformer of the multimodal text decoder; The cross-attention result is used as the input data for the next layer transformer of the current layer transformer of the multimodal text decoder to train the model to be tuned.

4. A device for determining an image description information generation model, characterized in that, include: The first acquisition module is configured to acquire a first reference model and a model to be trained, wherein the model to be trained includes a unimodal image encoder, a unimodal text encoder, a multimodal text encoder, and a multimodal text decoder; The first loss function determination module is configured to determine a first loss function based on the single-modal image encoder, the single-modal text encoder, the multimodal text encoder, and the multimodal text decoder; The first training module is configured to train the model to be trained based on the first loss function and the first reference model to obtain the target model to be trained. The image description information generation model determination module is configured to determine the image description information generation model based on the single-modal image encoder and multi-modal text decoder in the target training model. The image description information generation model is used to generate text description information corresponding to the image. Wherein, the model parameters of the model to be trained are fewer than the model parameters of the first reference model; the input data of the multimodal text encoder and the multimodal text decoder include the output data of the unimodal image encoder, and the first reference model includes a unimodal image encoder, a unimodal text encoder, a multimodal text encoder, and a multimodal text decoder; the method for determining the first reference model includes: obtaining a first image-text pair training dataset, training an initial first reference model based on the first image-text pair training dataset to obtain a first reference model to be optimized; obtaining a second image-text pair training dataset, training the first reference model to be optimized based on the second image-text pair training dataset to obtain a first reference model; wherein, the first image-text pair training dataset is determined based on image-text pairs crawled from the network, and the second image-text pair training dataset is determined based on labeled image-text pairs; The step of determining the first loss function based on the unimodal image encoder, the unimodal text encoder, the multimodal text encoder, and the multimodal text decoder includes: determining an image-text contrast loss based on the unimodal image encoder and the unimodal text encoder; and determining an image-text matching loss based on the multimodal text encoder. Based on the loss between the output of the multimodal text decoder and the text description labels corresponding to the input image, the language modeling loss is determined; based on the image-text contrast loss, the image-text matching loss, and the language modeling loss, the first loss function is determined. The step of training the model to be trained based on the first loss function and the first reference model to obtain a target model to be trained includes: determining a first cross-entropy loss function based on the probability distribution between labeled and unlabeled values ​​in the prediction results of the first reference model and the probability distribution between labeled and unlabeled values ​​in the prediction results of the model to be trained; determining a first relative entropy loss function based on the probability distribution between unlabeled values ​​in the prediction results of the first reference model and the probability distribution between unlabeled values ​​in the prediction results of the model to be trained; determining a first target loss function based on the first cross-entropy loss function and the first relative entropy loss function; and determining a training loss function for the model to be trained based on the first loss function and the first target loss function, so as to train the model to be trained based on the training loss function of the model to be trained to obtain the target model to be trained. The step of determining the image description information generation model based on the single-modal image encoder and multi-modal text decoder in the target training model includes: determining a model to be optimized based on the single-modal image encoder and multi-modal text decoder in the target training model; obtaining a second reference model; and training the model to be optimized based on the second reference model to obtain the image description information generation model; wherein the model parameters of the model to be optimized are less than the model parameters of the second reference model.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 3.

6. An electronic device, characterized in that, include: One or more processors; as well as A memory for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the method as described in any one of claims 1 to 3.

Images