Image description generation model training, image description generation method and device
By freezing the encoding and decoding layers and using sample conditional labels to guide training, the problems of poor performance and detail omission in existing image description generation models are solved, thereby achieving fine-grained image description generation and reducing training costs.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANGHAI BIREN TECH CO LTD
- Filing Date
- 2023-06-13
- Publication Date
- 2026-07-24
AI Technical Summary
Existing image description generation models perform poorly when generating long sentences, and the generated description text is coarse-grained, ignoring detailed information in the image.
The parameters of the encoding and decoding layers of the existing image description generation model are frozen. The training of other layers is guided by sample condition labels. Lightweight networks or attention modules are used to encode the features of the sample condition labels to generate fine-grained image description text that matches the desired description information.
It improves the generation effect of image description generation model, reduces training cost and number of parameters, and increases the convergence speed of model.
Smart Images
Figure CN116797827B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of multimodal technology, and in particular to an image description generation model training, image description generation method, and apparatus. Background Technology
[0002] Image caption generation is a technique that converts an image into a text description. It is typically implemented using deep learning models, which can extract features from an input image and convert it into natural language text.
[0003] In image caption generation, commonly used models are typically built using CNN-RNN encoder-decoder frameworks and attention-based encoder-decoder frameworks. However, models built using CNN-RNN encoder-decoder frameworks perform poorly in generating long sentences, while models built using attention-based encoder-decoder frameworks generate coarse-grained descriptive text, meaning that the generated descriptive text ignores detailed information in the image. Summary of the Invention
[0004] This invention provides an image description generation model training, image description generation method, and apparatus to address the shortcomings of existing technologies, such as poor generation performance for long sentences or coarse-grained generated description text.
[0005] This invention provides a method for training an image description generation model, comprising:
[0006] Determine the sample condition label corresponding to the sample description image; the sample condition label includes the expected description information of the sample description image;
[0007] Freeze the encoding and decoding layer parameters of the existing image description generation model, and train the other layers of the existing image description generation model based on the sample description image and the sample condition label to obtain the image description generation model.
[0008] According to the image description generation model training method provided by the present invention, the step of training other layers of the existing image description generation model based on the sample description image and the sample condition labels to obtain the image description generation model includes:
[0009] The sample description image is feature-encoded to obtain the sample description image features;
[0010] The sample condition labels are feature-encoded to obtain sample condition label features;
[0011] Based on the sample description image features and the sample condition label features, the other layers of the existing image description generation model are trained to obtain the image description generation model.
[0012] According to the image description generation model training method provided by the present invention, the step of feature encoding the sample condition labels to obtain sample condition label encoded features includes:
[0013] The sample condition labels are input into a lightweight network, which encodes the features of the sample condition labels to obtain adaptive coefficients corresponding to the sample condition label features. The adaptive coefficients are then input into the normalization layer of the existing image description generation model to guide the mapping parameters of the normalization layer.
[0014] or,
[0015] The sample condition labels are input into the attention module added to the existing image description generation model, and the attention module performs feature encoding on the sample condition labels to obtain the sample condition label encoded features.
[0016] According to the image description generation model training method provided by the present invention, the step of encoding the sample condition labels further includes:
[0017] Convert discrete sample condition labels into continuous high-dimensional vectors.
[0018] According to the image description generation model training method provided by the present invention, the expected description information includes at least one of category description information, color description information, foreground description information, and background description information.
[0019] The present invention also provides an image description generation method, comprising:
[0020] Determine the image to be described and its corresponding condition label; the condition label includes the expected description information of the image to be described;
[0021] The image to be described and the conditional label are input into the image description generation model to obtain the image description text output by the image description generation model.
[0022] The image description generation model is trained based on the image description generation model training method described above.
[0023] The present invention also provides an image description generation model training device, comprising:
[0024] The first determining unit is used to determine the sample condition label corresponding to the sample description image; the sample condition label includes the expected description information of the sample description image;
[0025] The model training unit is used to freeze the encoding layer parameters and decoding layer parameters of the existing image description generation model, and to train the other layers of the existing image description generation model based on the sample description image and the sample condition label to obtain the image description generation model.
[0026] The present invention also provides an image description generation apparatus, comprising:
[0027] The second determining unit is used to determine the image to be described and the corresponding condition label; the condition label includes the expected description information of the image to be described;
[0028] An image description unit is used to input the image to be described and the conditional label into an image description generation model to obtain the image description text output by the image description generation model.
[0029] The image description generation model is trained based on the image description generation model training method described above.
[0030] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the image description generation model training method as described above or to implement the image description generation method as described above.
[0031] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image description generation model training method as described above or the image description generation method as described above.
[0032] The present invention also provides a computer program product, comprising a computer program that, when executed by a processor, implements the image description generation model training method as described above or the image description generation method as described above.
[0033] The image description generation model training, image description generation method, and apparatus provided by this invention guide existing image description generation models to learn how to extract desired descriptive information from sample description images through sample condition labels. This enables the trained image description generation model to accurately generate fine-grained image description text that matches the desired descriptive information. Furthermore, embodiments of this invention freeze the encoding and decoding layer parameters of existing image description generation models, thereby reducing the number of parameters required for model training, accelerating model convergence, and lowering training costs. Attached Figure Description
[0034] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0035] Figure 1 This is a flowchart illustrating the training method for the image description generation model provided by the present invention;
[0036] Figure 2 This is a schematic diagram of the structure of the MLP provided by the present invention;
[0037] Figure 3 This is a schematic diagram of the structure of the Lformer provided by the present invention;
[0038] Figure 4 This is a flowchart illustrating the sample condition label encoding method provided by the present invention;
[0039] Figure 5 This is a flowchart illustrating another sample condition tag encoding method provided by the present invention;
[0040] Figure 6 This is a flowchart illustrating the image description generation method provided by the present invention;
[0041] Figure 7 This is a schematic diagram of the image to be described provided by the present invention;
[0042] Figure 8 This is another schematic diagram of an image to be described provided by the present invention;
[0043] Figure 9 This is a schematic diagram of the image description generation model training device provided by the present invention;
[0044] Figure 10 This is a schematic diagram of the image description generation device provided by the present invention;
[0045] Figure 11 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation
[0046] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.
[0047] In image caption generation tasks, models built using the CNN-RNN encoder-decoder framework have limitations when handling fixed-length vectors. RNNs suffer from the vanishing gradient problem over long distances, and while LSTMs have gating structures, their performance is unsatisfactory for generating long sentences due to the increasing complexity and difficulty of current tasks.
[0048] Models built using an attention-based encoder-decoder framework have limited ability to generate descriptive text due to the modal barrier problem commonly found in multimodal image description generation tasks. Furthermore, the limited number of text-image pairs in the dataset itself results in coarse-grained descriptive text.
[0049] Furthermore, although the above problems can be solved by using the current mainstream large-scale vision and language models, the training cost of these large-scale vision and language models is relatively high.
[0050] In response, this invention provides a method for training an image description generation model. Figure 1 This is a flowchart illustrating the image description generation model training method provided by the present invention, as shown below. Figure 1 As shown, the method includes the following steps:
[0051] Step 110: Determine the sample condition label corresponding to the sample description image; the sample condition label includes the expected description information of the sample description image.
[0052] Specifically, sample description images are images used to train the image description generation model. Sample description images can be obtained from various sources, such as public datasets (e.g., coco2014), local datasets, or the Internet.
[0053] Furthermore, the sample condition labels include the expected descriptive information of the sample description image. Expected descriptive information refers to the tendency of the sample description image to describe itself, and this information can include category description information, color description information, foreground description information, and background description information. Specifically, category description information describes the category of information in the sample description image, such as people or animals; color description information describes the color information in the sample description image, such as clothing color or hat color; foreground description information describes the foreground information in the sample description image, such as people, vehicles, or animals; and background description information describes the background information in the sample description image, such as the sky, grass, or buildings. It is understandable that the sample condition labels include the expected descriptive information of the sample description image, meaning that different sample description images will have different sample condition labels.
[0054] Optionally, the sample condition labels of the sample description image can be determined manually, or the foreground description information and background description information in the sample condition labels can be determined by image segmentation of the sample description image, the category description information in the sample condition labels can be determined by target detection of the sample description image, and the color description information in the sample condition labels can be determined by color recognition of the regions corresponding to the foreground description information, background description information and category description information.
[0055] Step 120: Freeze the encoding and decoding layer parameters of the existing image description generation model, and train the other layers of the existing image description generation model based on the sample description images and sample condition labels to obtain the image description generation model.
[0056] Specifically, existing image description generation models can be understood as image description generation models in the existing technology. For example, existing image description generation models can be encoder-decoder models based on CNN-RNN, or encoder-decoder models based on Attention.
[0057] Given that existing image captioning generation models contain a large number of parameters, training the encoding and decoding layers together with other layers presents challenges in terms of memory, communication, and optimization. To address this, this invention freezes the encoding and decoding layer parameters of existing image captioning generation models, setting the encoding and decoding layers to an untrainable state and training the other layers instead. This reduces the number of parameters required for model training, accelerates model convergence, and lowers training costs.
[0058] Furthermore, the sample condition labels include the expected descriptive information of the sample description image. These labels can guide existing image description generation models to learn how to extract the expected descriptive information from the sample description image, thereby ensuring that the generated sample image description text matches the expected descriptive information. For example, if sample description image A corresponds to a girl and a dog on a beach, and the desired generated sample image description text emphasizes the description of the target, target information can be added to the sample condition labels.
[0059] Therefore, sample condition labels can enhance the desired descriptive information in sample descriptive images, thereby enabling the generated sample image descriptive text to match the desired descriptive information.
[0060] Furthermore, image description models can be viewed as visual models, and the process of generating descriptive text requires language training of the visual model. A modal gap exists between the visual training of the visual model and the language training of the language model, causing existing image description models to fail to accurately generate corresponding descriptive text. However, this invention, by introducing sample condition labels, guides existing image description generation models to undergo language training, enabling the resulting image description generation model to accurately generate fine-grained image description text that matches the desired descriptive information.
[0061] The image description generation model training method provided in this invention guides existing image description generation models to learn how to extract desired descriptive information from sample description images through sample condition labels. This enables the trained image description generation model to accurately generate fine-grained image description text that matches the desired descriptive information. Furthermore, this invention freezes the encoding and decoding layer parameters of existing image description generation models, thereby reducing the number of parameters required for model training, accelerating model convergence, and lowering training costs.
[0062] Based on the above embodiments, other layers of the existing image description generation model are trained based on the sample description image and sample condition labels to obtain an image description generation model, including:
[0063] The sample description image is feature-encoded to obtain the sample description image features;
[0064] The sample condition labels are feature-encoded to obtain the sample condition label features;
[0065] Based on the sample description image features and sample condition label features, other layers of the existing image description generation model are trained to obtain the image description generation model.
[0066] Specifically, feature encoding of a sample description image is the process of converting the sample description image into numerical features, thereby obtaining the sample description image features. This can be achieved by using the encoding layer in an existing image description generation model to perform feature encoding on the sample description image.
[0067] Similarly, feature encoding of sample condition labels is the process of converting sample condition labels into numerical features, thereby obtaining sample condition label features. This can be achieved by adding an attention module to an existing image description generation model to perform feature encoding of the sample condition labels, or by using another network (such as a lightweight network). This embodiment of the invention does not specifically limit the approach.
[0068] After obtaining the sample description image features and sample condition label features, the other layers of the existing image description generation model are trained based on the sample description image features and sample condition label features, so that the image description generation model can accurately generate fine-grained image description text that matches the expected description information.
[0069] Based on any of the above embodiments, feature encoding is performed on the sample condition labels to obtain sample condition label encoded features, including:
[0070] The sample condition labels are input into a lightweight network, which encodes the features of the sample condition labels to obtain adaptive coefficients corresponding to the sample condition label features. The adaptive coefficients are then input into the normalization layer of an existing image description generation model to guide the mapping parameters of the normalization layer.
[0071] or,
[0072] The sample condition labels are input into the attention module added to the existing image description generation model. The attention module encodes the sample condition labels to obtain the sample condition label encoded features.
[0073] Specifically, lightweight networks can include MLP (Multilayer Perceptron) and Lformer (Lightweight Transformer), where the structure of MLP is as follows: Figure 2 As shown, the structure of Lformer is as follows: Figure 3 As shown. Adaptive coefficients can include scaling coefficients, shift coefficients, gate coefficients, etc. Using lightweight networks to encode the features of sample condition labels can reduce the number of model parameters and complexity while maintaining model accuracy, thereby improving computational efficiency and inference speed.
[0074] like Figure 4 As shown, the discrete sample condition labels (dog) are converted into continuous high-dimensional vectors (label dense vectors) based on the label embedder. The vector is then passed through a lightweight network (Lformer) to obtain scaling coefficients, shift coefficients, and gate coefficients to control the activation tensors. In other words, the expected descriptive information in the sample condition labels is used to guide the mapping parameters of the normalization layer (Layer Norm).
[0075] When encoding the features of sample condition labels, an attention module can be added to the existing image description generation model. This attention module then encodes the features of the sample condition labels, resulting in the encoded features of the sample condition labels. For example... Figure 5As shown, an attention module (Cross Attention) is added to the Encoder of the existing image description generation model, and the sample condition labels are directly encoded in Cross Attention through the Label Embedder.
[0076] Encoding the sample condition labels based on any of the above embodiments, further including:
[0077] Convert discrete sample condition labels into continuous high-dimensional vectors.
[0078] In some specific implementations, a Label Embedder can be applied to the sample condition labels to convert the continuous sample condition labels into continuous high-dimensional vectors (label dense vectors), thereby embedding the sample condition labels, which are represented by discrete values, into a continuous high-dimensional space, which facilitates model computation.
[0079] Based on any of the above embodiments, the desired descriptive information includes at least one of category descriptive information, color descriptive information, foreground descriptive information, and background descriptive information.
[0080] Specifically, category description information is used to describe the information category in the sample image, such as people, animals, etc.; color description information is used to describe the color information in the sample image, such as clothing color, hat color, etc.; foreground description information is used to describe the foreground information in the sample image, such as people, cars, animals, etc.; and background description information is used to describe the background information in the sample image, such as the background of the sample image, including sky, grass, buildings, etc.
[0081] Based on any of the above embodiments, the present invention also provides a method for training an image description generation model, the method comprising:
[0082] Obtain an image-text pair dataset and add the desired category information from the images to the dataset. Taking the coco2014 caption dataset as an example, the desired sample condition labels can be added to the coco2014 caption dataset based on the coco2014 instance annotations.
[0083] For example, the dataset can be labeled as follows:
[0084] {"caption":"A woman wearing a net on her head cutting a cake.","image":"val2014 / COCO_val2014_000000522418.jpg","image_id":"coco_522418","category_id":1,"name":"person","new_cat_id_map":0}
[0085] Here, "caption" corresponds to the description of the image; "image" is the image path; "image_id" is obtained from the image name; "category_id" is the added category information ID; "name" is the added conditional label information; and "new_cat_id_map" is the re-encoded category ID (because the category_ids labeled by coco2014 are not continuous, so re-encoding is required here).
[0086] The "new_cat_id_map" is read from the above annotations, and the category IDs of the conditional labels are converted into label dense vectors using a Label Embedder. The Label Embedder includes a dropout and embedding layer.
[0087] The label dense vector is input into a lightweight network to obtain scaling, shift, and gate coefficients, which guide the mapping parameters of the normalization layer. Alternatively, Cross Attention can be added to an existing image captioning generation model, where sample conditional labels are directly encoded into Cross Attention via Label Embedder.
[0088] The encoding and decoding layer parameters in the existing image description generation model are frozen. Other layers are trained based on the encoded sample condition labels and the encoded sample description images to obtain the image description generation model.
[0089] Based on any of the above embodiments Figure 6 This is a flowchart illustrating the image description generation method provided by the present invention, as shown below. Figure 6 As shown, the method includes the following steps:
[0090] Step 610: Determine the image to be described and its corresponding condition label; the condition label includes the expected description information of the image to be described;
[0091] Step 620: Input the image to be described and the conditional labels into the image description generation model to obtain the image description text output by the image description generation model;
[0092] The image description generation model is trained based on the image description generation model training method described in any of the above embodiments.
[0093] Specifically, the image to be described refers to the image for which image description text needs to be generated, and the condition labels include the expected description information of the image to be described. The expected description information refers to the descriptive information of the image to be described, which may include category description information, color description information, foreground description information, background description information, etc.
[0094] Since the image description generation model is trained based on the image description generation model training method described in any of the above embodiments, the image description generation model can accurately generate fine-grained image description text that matches the desired description information. That is, after inputting the image to be described and the conditional label into the image description generation model, fine-grained image description text that matches the desired description information can be obtained.
[0095] Experiments show that the image description generation model provided in this embodiment of the invention not only reduces training costs but also improves text description capabilities compared to existing image description generation models. Table 1 compares the performance metrics of the model provided in this embodiment of the invention with those of existing models. As shown in Table 1, Ours is the image description generation method provided in this embodiment of the invention, while Enc-Dec and BLIP are improved image description generation methods in existing technologies. Ours has fewer trained params than Enc-Dec and BLIP, indicating that the image description generation method provided in this embodiment of the invention can reduce training costs. Ours' BLEU@4 is superior to BLIP's BLEU@4, and Ours' CIDEr is superior to the corresponding CIDEr of Enc-Dec and BLIP, indicating that the image description generation method provided in this embodiment of the invention can improve text description capabilities.
[0096] Table 1
[0097]
[0098] In addition, such as Figure 7As shown, the image to be described is "A woman sitting on the beach with a golden dog beside her." If existing image description generation methods are used, the resulting image description text would be "A woman sitting on the beach with a dog." However, if the image description generation method of this embodiment is used, the resulting image description text would be "A woman sitting on the beach with a golden dog." Figure 8 As shown, the content of the image to be described is "A red-haired lady is looking there". If the existing image description generation method is used, the resulting image description text is "A woman is looking there". If the image description generation method of this embodiment is used, the resulting image description text is "A woman with red hair is looking there".
[0099] Therefore, the image description generation method provided in this embodiment of the invention can describe the image content in a way that is biased according to the expected description information in the conditional labels, and obtain fine-grained image description text that matches the expected description information.
[0100] The image description generation model training device provided by the present invention will be described below. The image description generation model training device described below can be referred to in correspondence with the image description generation model training method described above.
[0101] Based on any of the above embodiments, the present invention also provides an image description generation model training device, such as... Figure 9 As shown, the device includes:
[0102] The first determining unit 910 is used to determine the sample condition label corresponding to the sample description image; the sample condition label includes the expected description information of the sample description image.
[0103] The model training unit 920 is used to freeze the encoding layer parameters and decoding layer parameters of the existing image description generation model, and to train other layers of the existing image description generation model based on the sample description images and sample condition labels to obtain the image description generation model.
[0104] The image description generation apparatus provided by the present invention will be described below. The image description generation apparatus described below can be referred to in correspondence with the image description generation method described above.
[0105] Based on any of the above embodiments, the present invention also provides an image description generation apparatus, such as... Figure 10 As shown, the device includes:
[0106] The second determining unit 1010 is used to determine the image to be described and the corresponding condition label; the condition label includes the expected description information of the image to be described.
[0107] The image description unit 1020 is used to input the image to be described and the condition label into the image description generation model to obtain the image description text output by the image description generation model.
[0108] The image description generation model is trained based on the image description generation model training method described in any of the above embodiments.
[0109] Figure 11 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 11 As shown, the electronic device may include a processor 1110, a memory 1120, a communication interface 1130, and a communication bus 1140, wherein the processor 1110, the memory 1120, and the communication interface 1130 communicate with each other through the communication bus 1140. The processor 1110 can call logical instructions in the memory 1120 to execute an image description generation model training method. The method includes: determining sample condition labels corresponding to sample description images; the sample condition labels include the expected description information of the sample description images; freezing the encoding layer parameters and decoding layer parameters of an existing image description generation model; and training other layers of the existing image description generation model based on the sample description images and the sample condition labels to obtain an image description generation model.
[0110] Alternatively, an image description generation method may be performed, the method comprising: determining an image to be described and a corresponding condition label; the condition label including desired description information of the image to be described; inputting the image to be described and the condition label into an image description generation model to obtain image description text output by the image description generation model; the image description generation model being trained based on the image description generation model training method described above.
[0111] Furthermore, the logical instructions in the aforementioned memory 1120 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0112] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, wherein when the program instructions are executed by a computer, the computer is able to execute the image description generation model training method provided by the above methods, the method comprising: determining sample condition labels corresponding to sample description images; the sample condition labels comprising expected description information of the sample description images; freezing the encoding layer parameters and decoding layer parameters of an existing image description generation model; and training other layers of the existing image description generation model based on the sample description images and the sample condition labels to obtain an image description generation model.
[0113] Alternatively, an image description generation method may be performed, the method comprising: determining an image to be described and a corresponding condition label; the condition label including desired description information of the image to be described; inputting the image to be described and the condition label into an image description generation model to obtain image description text output by the image description generation model; the image description generation model being trained based on the image description generation model training method described above.
[0114] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described image description generation model training methods. The method includes: determining sample condition labels corresponding to sample description images; the sample condition labels including expected description information of the sample description images; freezing the encoding layer parameters and decoding layer parameters of an existing image description generation model; and training other layers of the existing image description generation model based on the sample description images and the sample condition labels to obtain an image description generation model.
[0115] Alternatively, an image description generation method may be performed, the method comprising: determining an image to be described and a corresponding condition label; the condition label including desired description information of the image to be described; inputting the image to be described and the condition label into an image description generation model to obtain image description text output by the image description generation model; the image description generation model being trained based on the image description generation model training method described above.
[0116] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0117] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0118] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for training an image description generation model, characterized in that, include: Determine the sample condition labels corresponding to the sample description images; The sample condition label includes the expected descriptive information of the sample description image; Freeze the encoding and decoding layer parameters of the existing image description generation model; The sample conditional labels are converted into continuous high-dimensional vectors using a Label Embedder that includes dropout and embedding layers. The continuous high-dimensional vector is input into a lightweight network to obtain the scaling coefficient, shift coefficient, and gate coefficient corresponding to the sample condition label; The scaling coefficient, the shift coefficient, and the gate coefficient are input into the normalization layer of the existing image description generation model to guide the mapping parameters of the normalization layer. The image description generation model is obtained by training other layers of the existing image description generation model based on the sample description image and the sample condition label.
2. The image description generation model training method according to claim 1, characterized in that, The step of training other layers of the existing image description generation model based on the sample description image and the sample condition labels to obtain the image description generation model includes: The sample description image is feature-encoded to obtain the sample description image features; The sample condition labels are feature-encoded to obtain sample condition label features; Based on the sample description image features and the sample condition label features, the other layers of the existing image description generation model are trained to obtain the image description generation model.
3. The image description generation model training method according to claim 2, characterized in that, The step of feature encoding the sample condition labels to obtain sample condition label encoded features includes: The sample condition labels are input into a lightweight network, which encodes the features of the sample condition labels to obtain adaptive coefficients corresponding to the sample condition label features. The adaptive coefficients are then input into the normalization layer of the existing image description generation model to guide the mapping parameters of the normalization layer. or, The sample condition labels are input into the attention module added to the existing image description generation model, and the attention module performs feature encoding on the sample condition labels to obtain the sample condition label encoded features.
4. The image description generation model training method according to claim 2, characterized in that, The process of encoding the sample condition labels includes, prior to: Convert discrete sample condition labels into continuous high-dimensional vectors.
5. The image description generation model training method according to any one of claims 1 to 4, characterized in that, The desired descriptive information includes at least one of category descriptive information, color descriptive information, foreground descriptive information, and background descriptive information.
6. An image description generation method, characterized in that, include: Determine the image to be described and its corresponding condition labels; The condition label includes the desired descriptive information of the image to be described; The image to be described and the conditional label are input into the image description generation model to obtain the image description text output by the image description generation model. The image description generation model is trained based on the image description generation model training method according to any one of claims 1 to 5.
7. A training device for an image description generation model, characterized in that, include: The first determining unit is used to determine the sample condition label corresponding to the sample description image; The sample condition label includes the expected descriptive information of the sample description image; The model training unit is used to freeze the encoding and decoding layer parameters of the existing image description generation model; The sample conditional labels are converted into continuous high-dimensional vectors using a Label Embedder that includes dropout and embedding layers. The continuous high-dimensional vector is input into a lightweight network to obtain the scaling coefficient, shift coefficient, and gate coefficient corresponding to the sample condition label; The scaling coefficient, the shift coefficient, and the gate coefficient are input into the normalization layer of the existing image description generation model to guide the mapping parameters of the normalization layer. The image description generation model is obtained by training other layers of the existing image description generation model based on the sample description image and the sample condition label.
8. An image description generation apparatus, characterized in that, include: The second determining unit is used to determine the image to be described and the corresponding condition label; The condition label includes the desired descriptive information of the image to be described; An image description unit is used to input the image to be described and the conditional label into an image description generation model to obtain the image description text output by the image description generation model. The image description generation model is trained based on the image description generation model training method according to any one of claims 1 to 5.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the image description generation model training method as described in any one of claims 1 to 5, or the image description generation method as described in claim 6.
10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the image description generation model training method as described in any one of claims 1 to 5, or the image description generation method as described in claim 6.