A method for generating image descriptions in minority languages ​​based on large model and retrieval comparison

Abstract

This invention relates to a method for generating image descriptions in less commonly spoken languages ​​based on a large language model and retrieval comparison, belonging to the fields of image and natural language processing technology. First, a parallel corpus dataset from the major language to the less commonly spoken language, as well as image description pairs datasets for both languages, are constructed. Then, a multilingual understanding large language model is trained. Next, image-text alignment is performed, and prompts are generated using vector database retrieval. Finally, descriptions are generated based on the trained model and the generated prompts. This method directly uses the large language model for generation, eliminating the need for a translation step, thus improving generation efficiency. Image description generation is specifically tailored to less commonly spoken languages, and the introduction of retrieval comparison technology further enhances the generation quality and accuracy. This method makes image description generation in less commonly spoken languages ​​more feasible, providing strong technical support for research and applications in related fields.

Description

Technical Field

[0001] This invention relates to a low-resource method for generating image descriptions in minority languages, specifically a method for generating image descriptions in minority languages ​​based on a large language model and retrieval comparison, belonging to the field of image and natural language processing technology. Background Technology

[0002] Image description generation is a task that generates grammatically correct and coherent text that fits the content of an input image. Existing image description generation technologies mainly focus on mainstream languages ​​such as Chinese and English, but research methods for less commonly spoken languages ​​are relatively scarce. Less commonly spoken languages ​​are those used only in certain countries, as opposed to widely used languages ​​like English and French. Examples include German, Japanese, Spanish, Italian, Portuguese, Norwegian, Finnish, Croatian, Slovenian, Czech, Albanian, Bulgarian, Dutch, Estonian, Danish, Russian, Georgian, Belarusian, Armenian, Macedonian, Ethiopian, Hungarian, Greek, Serbian, Slovak, Polish, Romanian, Swedish, Latvian, Lithuanian, Ukrainian, Thai, Arabic, Hindi (Indian languages), Persian (Iranian languages), Hebrew (Israeli languages), Bengali, Indonesian, Malay, Turkish, Filipino, Vietnamese, Lao, Cambodian, Swahili (East Africa), Zulu, Icelandic, Irish, and many more. Existing methods typically involve first generating Chinese and English descriptions, and then translating them into a less common language.

[0003] Since the debut of ChatGPT, large language models have demonstrated tremendous potential in text generation. They can be multilingual experts and reasoning masters. Introducing large language models into image description generation can effectively solve the problem of generating minor languages. Summary of the Invention

[0004] The purpose of this invention is to solve the problem of generating image descriptions in less commonly spoken languages. It creatively proposes a method for generating image descriptions in less commonly spoken languages ​​based on a large model and retrieval comparison. This method directly uses a large language model for generation, eliminating the need for a translation step, thereby improving generation efficiency.

[0005] This method directly targets image description generation for less commonly spoken languages ​​with limited resources. By introducing retrieval and comparison techniques, it further improves the quality and accuracy of the generated descriptions. This method makes image description generation for less commonly spoken languages ​​more feasible and provides strong technical support for research and applications in related fields.

[0006] Beneficial effects

[0007] The method of the present invention has the following advantages compared with the prior art:

[0008] 1. This method fully considers the multilingual capabilities of large language models, training them to become multilingual experts, thereby enabling one-step generation of image descriptions in less common languages ​​and simplifying the generation process.

[0009] 2. This method fully considers historical knowledge, storing existing photo description pairs in a vector database. During generation, historical knowledge is retrieved from the database as a hint for the large model generation, further improving the accuracy of the generation. Attached Figure Description

[0010] Figure 1 This is a schematic diagram of the image-text alignment network training of the present invention;

[0011] Figure 2 This is a schematic diagram of the text alignment network fine-tuning of the present invention combined with the frozen large language model;

[0012] Figure 3 This is a flowchart illustrating the method of the present invention. Detailed Implementation

[0013] The method of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0014] The objective of this invention is achieved through the following technical solutions.

[0015] A method for generating descriptions of images in minority languages ​​based on comparison of large models and retrieval includes the following steps:

[0016] Step 1: Construct parallel corpus datasets from major languages ​​to minor languages, as well as image description pair datasets for major and minor languages.

[0017] Specifically, it includes the following steps:

[0018] First, clarify the target task and define the less commonly spoken language for image description generation. It should be noted that less commonly spoken languages ​​refer to languages ​​with fewer speakers and more limited distribution compared to languages ​​with a large number of speakers and abundant textual resources, such as Tibetan, Mongolian, and Uyghur.

[0019] Then, given the scarcity of existing minority language corpora, we first compile parallel corpora from major languages ​​to minor languages, and then use machine translation to achieve translation from major languages ​​to minor languages, forming a parallel corpus dataset of texts. It should be noted that major languages ​​refer to languages ​​with abundant corpora, such as Chinese, English, and French.

[0020] Finally, we collect image description pair datasets for major languages, translate the major language texts into minor languages, and form image description pair datasets for minor languages. Each image description pair consists of an image and its corresponding description in the minor language.

[0021] This step mainly uses machine translation technology to create parallel corpus datasets from major languages ​​to minor languages, as well as image description pairs datasets for both major and minor languages.

[0022] Step 2: Training a large language model for multilingual understanding.

[0023] Large Language Models (LLMs) are language models trained on massive text corpora, containing billions or more parameters, including but not limited to autoregressive, autoencoder, or encoder-decoder structures. Although LLMs have strong text generation capabilities, their multilingual understanding capabilities have not yet been fully explored due to the small proportion of minority language texts during pre-training.

[0024] In this step, the large language model is trained using the parallel corpus dataset obtained in step 1 and fine-tuning techniques are used. Then, reinforcement learning is performed based on the reward model to enhance the model's capabilities.

[0025] It should be noted that reinforcement learning emphasizes how to act based on the environment in order to maximize expected benefits, which is a research paradigm in the field of machine learning.

[0026] Step 3: Image and text alignment.

[0027] Specifically, it includes the following steps:

[0028] Step 3.1: Pre-train an image-text alignment network on a large language image-text dataset.

[0029] Image-to-text alignment networks (IPNs) are encoder-decoder structures. They involve three learning tasks: image-to-text matching, image-to-text comparison learning, and image-to-text generation. For example... Figure 1 As shown, the image I, the learnable parameters Q, and the descriptive text input T are the three inputs to the image-text alignment network. The output of each layer in the encoder is Z, which can be regarded as the cross-attention fusion of Q and I.

[0030] In image-text matching tasks, it's necessary to determine whether an image and text form a pair. Specifically, in each batch of training, positive and negative samples are constructed for the model to learn from. The loss function for image-text matching is... in Let E represent the image-to-text matching loss, D represent the probability distribution, and y represent the expected value. itmFor real labels, p itm Let H be the cross-entropy for predicting labels. In this task, a bidirectional self-attention mechanism is used, where Z and T observe each other, allowing the image to learn textual information and the text to learn image information.

[0031] In image-text comparison learning, image and text representations are aligned to maximize their mutual information. This is achieved by comparing the image-text similarity of positive and negative pairs. The image-text similarity is calculated using the maximum value of each vector in Z and I. Image-text similarity p i2t (I) is:

[0032]

[0033] in, This represents the image-text similarity at index m, where s represents the softmax function, τ is the learnable temperature coefficient, and T is the text similarity. m This represents the text data with sequence number m; M represents the sequence length.

[0034] Text-image similarity p t2i (T) is:

[0035]

[0036] Among them, Z m This indicates the encoder output when the sequence number is 1.

[0037] The loss function for image-text comparison learning is:

[0038]

[0039] in, This represents the learning loss from image-text comparison; y i2t This represents the truth value for image-to-text conversion, with a value of either 0 or 1; y t2i This represents the truth value for the image-to-text transition, with a value of 0 or 1. In this task, both text and images are single-modal self-attention mechanisms that can only notice their own information.

[0040] In the image description generation task, the model is trained to generate descriptions, and the loss function is:

[0041]

[0042] in, This represents the loss that is generated, where y is the actual output text. as a prefix, This describes the decoding generated by the decoder of the image-text alignment network. In this task, the attention mechanism used is a one-way self-attention mechanism, which decodes the current text and only considers previously generated text and image information.

[0043] Finally, the pre-training loss is:

[0044]

[0045] in, This represents the loss during pre-training.

[0046] Step 3.2: Fine-tune the image-text alignment network for the minority language image-text dataset.

[0047] First, prepare the dataset of image description pairs in minority languages ​​from step 1, the multilingual understanding big language model trained in step 2, and the image-text alignment network from step 3.1.

[0048] In the fine-tuning task, the trained image-text alignment network is aligned with the multilingual understanding big language model, the multilingual understanding big language model is frozen, and the image-text alignment network is trained on the dataset using image descriptions in minority languages. This ensures that the output vector, after passing through a fully connected layer, can be decoded by the multilingual understanding big language model and output as a description in a minority language.

[0049] Specifically, the input image is processed by an image encoder to generate image features and learnable parameters, which are then processed by an image-text alignment network to output a vector. This vector has a different dimension than the input vector of the large language model and needs to be expanded by a fully connected layer to become the input of the multilingual understanding large language model. The multilingual understanding large language model then outputs a description in the minority language. In this task, the loss function is:

[0050]

[0051] Where y is the actual output text, and P(I) is the generated description. Unlike the description generated by decoding in step 3.1, this is a description generated by a multilingual understanding large language model.

[0052] Step 4: Generate suggestions by searching the vector database.

[0053] Step 4.1: Store the descriptions of the images in the minority languages ​​into the vector database.

[0054] First, the image is encoded into vectors using an image encoder. Then, these vectors are stored together with the text in a vector database to form a vector database of graph features.

[0055] Then, when inputting a photo, it is directly entered into the vector database for retrieval, where the search is performed based on vector similarity.

[0056] Step 4.2: Recall Ranking (Top-K).

[0057] In vector databases, vector similarity algorithms, such as Euclidean distance (L2), are used, i.e.:

[0058]

[0059] Where d() represents the Euclidean distance, Let a represent vector a. Let a represent vectors b and a i Let b represent the numerical value of vector a in dimension i. i Let represent the numerical value of vector b in dimension i, and n represent the dimension of the vector space.

[0060] Retrieve the K photos from the database that have the smallest Euclidean distance, and return the descriptions of the K photos.

[0061] Step 4.3: Prompt generation.

[0062] Based on the description returned by the search, a rule-based method generates corresponding prompts according to the question, language requirements, and formatting requirements.

[0063] Step 5: Generate a description based on the trained model and the generated prompts.

[0064] Specifically, using the multilingual understanding big language model trained in step 2 and the image-text alignment network fine-tuned in step 3.2, after inputting an image, prompts are generated in step 4.3. The multilingual understanding big language model combines the image information and prompts output by the image-text alignment network to output the final description.

[0065] Example

[0066] like Figure 1 As shown, it comprises four parts. First, it constructs parallel corpus datasets of major and minor languages ​​and a dataset of image description pairs for minor languages ​​using machine translation technology. Then, it trains a multilingual understanding large language model using the parallel corpus datasets of major and minor languages. Finally, it pre-trains an image-text alignment network using the major language image-text dataset, such as... Figure 2 As shown, the image-text alignment network is then fine-tuned using a minority language image-text dataset, such as... Figure 3 As shown. Then, the image description pairs in the minority languages ​​are stored in the vector database. After inputting a photo, the results of TOP-K are returned through the vector database. The prompts are generated using a rule-based method, and the description is generated by combining the features of the photo.

[0067] Step 1: Construct parallel corpus datasets of major and minor languages, as well as image description pair datasets of major and minor languages.

[0068] By utilizing corpora of major languages, machine translation technology is used to translate them into minor languages, forming a parallel corpus of major and minor languages.

[0069] We collect image description pairs datasets in major languages, translate the text from the major languages ​​into minor languages, and form image description pairs datasets in the minor languages.

[0070] Step 2: Training a large language model for multilingual understanding.

[0071] Using the parallel corpora of major and minor languages ​​obtained in step 1, the large language model is fine-tuned, and then reinforcement learning methods are used to enhance multilingual comprehension capabilities, resulting in a multilingual comprehension large language model.

[0072] Step 3: Image and text alignment method.

[0073] Step 3 includes the following steps:

[0074] Step 3.1: Pre-train an image-text alignment network on a large language image-text dataset.

[0075] Specifically, the image-text alignment network is pre-trained, which includes three sub-tasks: image-text matching, image-text comparison learning, and image-to-text generation. Finally, the image-text alignment network is obtained.

[0076] Step 3.2: Fine-tune the image-text alignment network for the minority language image-text dataset.

[0077] Specifically, the multilingual understanding big language model is frozen, and an image-text alignment network is trained on the dataset using image descriptions in minority languages. This ensures that the output vectors are decoded by the multilingual understanding big language model after passing through a fully connected layer and output as descriptions in minority languages.

[0078] Step 4: Generate suggestions by searching the vector database.

[0079] Step 4 includes the following steps:

[0080] Step 4.1: Store the descriptions of the images in the minority languages ​​into the vector database.

[0081] Step 4.2: Recall Ranking (Top-K).

[0082] Retrieve the description of the image most similar to the input image from a vector database.

[0083] Step 4.3: Prompt generation.

[0084] A rule-based approach is used to generate hints from the descriptions returned by the vector database.

[0085] Step 5: Generate a description based on the trained model and the generated prompts.

[0086] The input image is encoded using an image encoder, passed through an image-text alignment network and a fully connected layer, to generate the input vector for the multilingual understanding model. The input image is then searched in a vector database to retrieve the most similar description, which is then used to generate a prompt that is input into the multilingual understanding model. Finally, the description is generated.

[0087] To illustrate the content and implementation method of this invention, a specific embodiment is provided in this specification. The details introduced in the embodiment are not intended to limit the scope of the claims, but rather to aid in understanding the method described in this invention. Those skilled in the art will understand that various modifications, variations, or substitutions to the steps of the preferred embodiment are possible without departing from the spirit and scope of this invention and its appended claims. Therefore, this invention should not be limited to the content disclosed in the preferred embodiment and the accompanying drawings.

Claims

1. A method for generating descriptions of images in minority languages ​​based on comparison of large models and retrieval, characterized in that, Includes the following steps: Step 1: Construct parallel corpus datasets from major languages ​​to minor languages, as well as image description pair datasets for major and minor languages; Step 2: Training a large language model for multilingual understanding; Using the parallel corpus dataset obtained in step 1, the large language model is trained using fine-tuning techniques, and then reinforcement learning is performed based on the reward model to enhance the model's capabilities. Step 3: Image and text alignment, including the following steps: Step 3.1: Pre-train an image-text alignment network using a large language image-text dataset; The image-text alignment network is an encoder-decoder structure. There are three learning tasks in the image-text alignment network: image-text matching, image-text comparison learning, and image-to-text generation. The inputs include the image I, the learnable parameters Q, and the descriptive text input T. The output of each layer in the encoder is Z, which is regarded as the cross-attention fusion of Q and I. In image-text matching tasks, it is necessary to determine whether an image and text are a pair. In each training batch, positive and negative samples are constructed for the model to learn from. The loss of image-text matching is... in Let E represent the image-to-text matching loss, D represent the probability distribution, and y represent the expected value. itm For real labels, p itm For predicting labels, H is the cross-entropy; in this task, a bidirectional self-attention mechanism is used, that is, Z and T observe each other, the image can learn text information, and the text can also learn image information; In image-text comparison learning, the goal is to align image and text representations to maximize their mutual information. This is achieved by comparing the image-text similarity of positive and negative pairs. The similarity between the image and text is calculated using the maximum value of each vector in Z and I. The image-text similarity p... i2t (I) is: in, This represents the image-text similarity at index m, where s represents the softmax function, τ is the learnable temperature coefficient, and T is the text similarity. m This represents the text data with sequence number m; M represents the sequence length. Text-image similarity p t2i (T) is: Among them, Z m This indicates the encoder output when the sequence number is 1; The loss function for image-text comparison learning is: in, This represents the learning loss from image-text comparison; y i2t This represents the truth value for image-to-text conversion, with a value of either 0 or 1; y t2i The value represents the truth value from image to text, which is either 0 or 1; in this task, both text and image are single-modal self-attention mechanisms that can only notice their own information. In image description generation tasks, this involves training a model to generate descriptions, and the loss function (LOS) is used to describe these descriptions. s The function is: in, This represents the loss that is generated, where y is the actual output text. as a prefix, The description generated by the decoder of the image-text alignment network; in this task, the attention mechanism used is a one-way self-attention mechanism, which decodes the current text and can only see the previously generated text and image information; Finally, the pre-training loss is: in, This represents the loss during pre-training; Step 3.2: Fine-tuning the image-text alignment network for minority language image-text datasets; First, prepare the dataset of image description pairs in minority languages ​​in step 1, the multilingual understanding big language model trained in step 2, and the image-text alignment network in step 3.1; In the fine-tuning task, the trained image-text alignment network is aligned with the multilingual understanding big language model, the multilingual understanding big language model is frozen, and the image-text alignment network is trained on the dataset using image descriptions in minority languages. After the output vector passes through a fully connected layer, it can be decoded by the multilingual understanding big language model and output as a description in a minority language. Step 4: Generate suggestions by searching the vector database; Step 5: Generate a description based on the trained model and the generated prompts.

2. The method for generating image descriptions in minority languages ​​based on comparison of large models and retrieval as described in claim 1, characterized in that, Step 1 includes the following steps: First, clarify the target task and define the language for image description generation; Then, we first organize parallel corpora from major languages ​​to minor languages, and then use machine translation to translate from major languages ​​to minor languages, forming a parallel corpus dataset of texts. Finally, we collect image description pair datasets for major languages, translate the major language texts into minor languages, and form image description pair datasets for minor languages. Each image description pair consists of an image and its corresponding description in the minor language.

3. The method for generating image descriptions in minority languages ​​based on comparison of a large model and retrieval, as described in claim 1, is characterized in that... In step 3.2, the input image is processed by an image encoder to generate image features and learnable parameters, which are then processed by an image-text alignment network to output a vector. The dimension of this vector is different from that of the input vector of the large language model. It needs to be expanded by a fully connected layer to become the input of the multilingual understanding large language model. Then the multilingual understanding large language model outputs the description of the minority language. In this task, the loss function is: Where y is the actual output text, and P(I) is the generated description. This represents the loss during pre-training.

4. The method for generating image descriptions in minority languages ​​based on comparison of large models and retrieval as described in claim 1, characterized in that, Step 4 includes the following steps: Step 4.1: Store the descriptions of the images in the minority languages ​​into the vector database; First, the image is encoded into vectors using an image encoder. Then, these vectors are stored together with the text in a vector database to form a vector database of graph features. Then, when inputting a photo, it is directly entered into the vector database for retrieval, and the search is performed in the vector database based on vector similarity; Step 4.2: Recall and rank Top-K; In vector databases, vector similarity algorithms are used, namely: Where d() represents the Euclidean distance, Let a represent vector a. Let a represent vectors b and a i Let b represent the numerical value of vector a in dimension i. i Let represent the value of vector b in dimension i, and n represent the dimension of the vector space. Retrieve the K photos with the smallest Euclidean distance from the database and return the descriptions of the K photos; Step 4.3: Prompt generation; Based on the description returned by the search, a rule-based method generates corresponding prompts according to the question, language requirements, and formatting requirements.

Images