Two-stage image description generation method, system, device and storage medium

Abstract

The present invention discloses a two-stage image description generation method, system, device and storage medium, wherein the method comprises: S1, obtaining an input image I, and extracting the visual features V of the image through a pre-trained visual encoder; S2, constructing a one-stage draft decoder based on a pre-trained single decoder model, inputting the visual features V into the draft decoder, and generating a draft text description W 1 ; S3, determine the cross-modal feature fusion module, and build a two-stage error correction decoder based on the cross-modal feature fusion module, and combine the visual feature V and the draft text description W 1 Input into the error correction decoder to generate the final text description W 2 ; S4, describe the text W 2 The reference description W paired with the input image I g The comparison is performed, the loss function loss value is calculated, and the overall model parameters are optimized. The present invention achieves significantly higher performance than the original single-stage pre-training model through a two-stage model structure, and can be widely used in the field of natural language processing technology.

Description

Technical Field

[0001] The present invention relates to the fields of multimodality, natural language processing, and pre-training model technology, and in particular to a two-stage image description generation method, system, device, and storage medium. Background Art

[0002] The image captioning task belongs to the field of visual description generation in the multimodal field. Its goal is to generate natural language text describing the content of an image. It is the foundation of current multimodal research tasks and can drive breakthroughs in other multimodal tasks (such as visual dialogue and visual question-answering). It also has considerable practical application scenarios (such as visual assistance for the visually impaired, human-computer interaction, and multimodal retrieval). The goal of image captioning is to find the most effective way to process the input image, extract the features that represent the image, and then, by aligning the features of visual and textual elements, convert the visual features into a descriptive text consisting of a series of words while maintaining language fluency.

[0003] For this task, the mainstream solutions are divided into two major directions: one is to process the features of the two modalities of image and text separately, that is, to use CNN and other object detection networks to extract the visual features of the image, and then use them in the language model of LSTM, Transformer and other structures to generate text descriptions; the other is to expand the model and dataset scale based on the Transformer architecture, uniformly process and align the features of the image and text modalities, and obtain a pre-trained unified model for solving multiple downstream tasks. Although existing research has achieved excellent results in the image description generation task, there are still the following shortcomings: (1) The mainstream solution is mainly based on the encoder-decoder framework, which has the inherent error accumulation and local optimality problems of single decoder generation. (2) The current research on the two-stage decoder structure model is still mainly based on the LSTM structure, in which the multimodal feature fusion is usually relatively simple (splicing, addition, etc.), which cannot fully utilize the complementarity of multimodal features. (3) The two-stage model will increase the parameters, resulting in increased training and fine-tuning costs, and the two-stage decoder in the mainstream solution is often trained from scratch, failing to fully utilize the generalization knowledge of the one-stage pre-trained model. Summary of the Invention

[0004] In order to at least partially solve one of the technical problems existing in the prior art, an object of the present invention is to provide a two-stage image description generation method, system, device and storage medium.

[0005] The technical solution adopted in the present invention is:

[0006] A two-stage image description generation method comprises the following steps:

[0007] S1. Get the input image I and extract the visual features V of the image through the pre-trained visual encoder ViT;

[0008] S2. Based on the existing mainstream pre-trained single decoder model, a one-stage draft decoder is constructed. The visual feature V is input into the draft decoder to generate the draft text description W. 1 ;

[0009] S3. Determine the Cross Modality Fusion (CMF) module and build a two-stage error correction decoder based on the Cross Modality Fusion module to combine the visual features V and the draft text description W 1 Input into the error correction decoder to generate the final text description W 2 ;

[0010] S4. Describe the text W 2 The human-annotated reference description W paired with the input image I g Compare and calculate the loss value of the loss function to optimize the overall parameters of the model.

[0011] Furthermore, the step S1 specifically includes:

[0012] Each input image I in the dataset is fed into the visual encoder (pre-trained ViT-B-32) in turn to obtain the output of the last hidden state of the encoder. Where r is the number of discrete features, d v is the representation dimension of discrete features; remove the [CLS] token with sequence number 0 representing the global feature, and use the remaining r-1 number features as the visual feature V extracted from the input image I.

[0013] Furthermore, the step S2 specifically includes:

[0014] Based on the mainstream single-decoder model structure (Transformer and its variants), a one-stage draft decoder D1 is constructed. The draft decoder D1 consists of three parts: a masked self-attention module, a visual cross-attention module, and a feedforward network module.

[0015] Reference text sequence In turn, it is used as the input of each time step of the draft decoder D1 training, and the feature calculation of self-attention is completed through the mask to obtain the hidden state Where n represents the length of the reference text sequence, which is usually less than 20 words;

[0016] The calculation process of the self-attention feature is as follows, where Q, K, and V all come from the same input in the self-attention:

[0017] fmha (Q,K,V)=Concat(head1,...,head h )W O (1)

[0018] head i =f a (Q i , K i , V i ) (2)

[0019]

[0020] Q i =QW i Q , K i =KW i K , V i =VW i V (4)

[0021] in, d m is the mapping feature dimension of Q and K, d m′ is the feature dimension of the attention-weighted result, d o is the feature dimension of the final output context, and h is the number of attention heads;

[0022] The extracted visual feature V is linearly mapped as K and V for the visual cross attention module, and the calculation result of the mask attention in the previous step is As Q, we can get the result of cross attention

[0023] Will After the input passes through the feedforward network, the hidden state of the third part is obtained Repeat the calculation of the three modules of self-attention module, cross attention module and feedforward network module, and after multiple layers are stacked, the input hidden state H of the final draft decoder is obtained. 1 ;

[0024] H 1 After being mapped to the vocabulary dimension through the linear layer, the probability distribution of each token in the vocabulary is obtained through softmax at all time steps.

[0025] For the probability distribution P 1 Sampling is performed, and the token with the highest probability position is selected as the generated token of the first-stage decoder, and finally the draft text description is obtained. Where l represents the length of the draft description.

[0026] Furthermore, the step S3 specifically includes:

[0027] A two-stage error correction decoder D2 is constructed based on the cross-modal feature fusion module; the error correction decoder D2 includes an input word embedding layer, a masked self-attention module, a cross-modal feature fusion module, a feedforward network module, and an output layer;

[0028] Reference text sequence Embedded into text feature vector After that, it passes through the self-attention module with mask to get the hidden state Where Q, K, V are represented by vector T g Obtained by linear mapping respectively;

[0029] Hidden state As Q, the visual features V, the draft text description W 1 Embedding vector of As K and V respectively, they are input into the cross-modal feature fusion module for calculation to obtain the hidden state output

[0030] Hidden state output After the feedforward network module, multi-layer stacking calculation, and hidden state output mapping to the word list, the probability distribution of the token is obtained, the probability distribution is sampled, and the token with the maximum probability position is selected as the generated token of the second-stage decoder, and finally the draft text description is obtained. Where m represents the length of the sentence.

[0031] Furthermore, the calculation of the cross-modal feature fusion module is as follows:

[0032] The first is the cross attention calculation, in which an additional adapter operation is performed on the cross attention results of visual features:

[0033]

[0034]

[0035] Adapter is a two-layer MLP network with GELU nonlinear activation function. The specific calculation steps are as follows, where mlp-ratio = 0.25,

[0036] h in =XW in (7)

[0037] h out =GELU(h in )Wout (8)

[0038] X′=h out +X (9)

[0039] Among them, X is the adapter input, and the output X′ is obtained after calculation and residual connection;

[0040] The second is GLU gated linear unit activation. GLU activation is required for V′ and T′ to ensure that the result of the cross attention is closely related to the input Q. The calculation process is as follows:

[0041]

[0042]

[0043] Among them, [;] represents the splicing operation, σ represents the sigmoid activation function, ⊙ represents element-wise multiplication;

[0044] Then comes the gate calculation step in the cross-modal feature fusion module. The calculation process is shown in the following formula, where

[0045] V″′=V″⊙σ([Q;V″;T″]W c V ) (12)

[0046] T″′=T″⊙σ([Q;V″;T″]W c T ) (13)

[0047] Finally, the features of the two modalities need to be fused together by splicing + linear mapping to obtain the hidden state output of the cross-modal feature fusion module

[0048]

[0049] in

[0050] Furthermore, the step S4 specifically includes:

[0051] According to W generated by the error correction decoder 2 and reference description W g , calculate the cross entropy loss at each time step:

[0052]

[0053] According to the cross entropy loss, batch training is adopted and the basic learning rate is set to 5e -4, a total of 30 rounds of training were conducted, using the Adam optimizer under the learning rate ramp-up and exponential decay strategy to optimize the model parameters until convergence was achieved.

[0054] Furthermore, the two-stage image description generation method further includes the step of reinforcement learning:

[0055] Taking the final state of the model in the cross entropy phase as the starting point, SCST reinforcement learning optimization is performed, and the loss calculation formula is as follows:

[0056]

[0057] Where W s 2 represents the sampling generation result of the two-stage decoder, r(·) is the reward score function;

[0058] According to the reinforcement learning loss, batch training is adopted, and the learning rate is fixed to 5e -6 , a total of 20 rounds of training were performed, using the Adam optimizer to optimize the model parameters until convergence was achieved.

[0059] Another technical solution adopted in the present invention is:

[0060] A two-stage image description generation system, comprising:

[0061] The visual feature extraction module is used to obtain the input image I and extract the visual features V of the image through the pre-trained visual encoder;

[0062] The first description generation module is used to build a one-stage draft decoder based on the pre-trained single decoder model. According to the visual features V, the draft decoder generates a draft text description W 1 ;

[0063] The second description generation module is used to determine the cross-modal feature fusion module and build a two-stage error correction decoder based on the cross-modal feature fusion module to combine the visual features V and the draft text description W 1 Input into the second-stage error correction decoder to generate the final text description W 2 ;

[0064] Model parameter optimization module, used to describe the text W 2 The reference description W paired with the input image I g Compare and calculate the loss value of the loss function to optimize the overall parameters of the model.

[0065] Another technical solution adopted in the present invention is:

[0066] A two-stage image description generation device, comprising:

[0067] at least one processor;

[0068] at least one memory for storing at least one program;

[0069] When the at least one program is executed by the at least one processor, the at least one processor implements the method described above.

[0070] Another technical solution adopted in the present invention is:

[0071] A computer-readable storage medium stores a program executable by a processor, wherein the program executable by the processor is used to perform the method described above when executed by the processor.

[0072] The beneficial effects of the present invention are as follows: the present invention can utilize the generalized knowledge of the existing single-stage pre-training model, and on the basis of the draft text generated by it, fully align and fuse it with the visual features. Through the two-stage model structure, it can achieve an effect that is significantly higher than the performance of the original single-stage pre-training model, and the generated image description has better evaluation performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following introduction is made to the drawings of the embodiments of the present invention or the related technical solutions in the prior art. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative work.

[0074] Figure 1 This is a structural flow chart of a two-stage image description generation method that introduces pre-trained model knowledge in an embodiment of the present invention;

[0075] Figure 2 is a structural diagram of a cross-modal feature fusion module for a two-stage image description generation method in an embodiment of the present invention;

[0076] Figure 3 This is a flowchart of the steps of a two-stage image description generation method in an embodiment of the present invention. DETAILED DESCRIPTION

[0077] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention and are not to be construed as limiting the present invention. The step numbers in the following embodiments are provided for ease of explanation only and do not limit the order of the steps. The order of execution of the steps in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0078] In the description of the present invention, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, they cannot be understood as limitations on the present invention.

[0079] In the description of the present invention, "several" means one or more, "many" means more than two, "greater than," "less than," and "exceed" are understood to exclude the number itself, while "above," "below," and "within" are understood to include the number itself. The use of "first" and "second" in the description is solely for the purpose of distinguishing technical features and should not be construed as indicating or implying relative importance, implicitly specifying the number of the indicated technical features, or implicitly specifying the order of the indicated technical features.

[0080] Furthermore, in the description of this invention, unless otherwise specified, "plurality" refers to two or more. "And / or" describes the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0081] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, and connecting should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.

[0082] In view of the shortcomings of the existing technology, this embodiment provides a two-stage image description generation method that introduces pre-training model knowledge, such as Figure 1 As shown in the figure, based on the existing pre-trained one-stage model, this method designs and proposes a two-stage image description generation model architecture with broad versatility. In the two-stage decoder, the cross-modality feature fusion module (CMF) is used to fully integrate the draft text and visual features, ultimately effectively improving the performance of the model and having broad practical value.

[0083] like Figure 3 As shown, this embodiment provides a two-stage image description generation method that introduces pre-trained model knowledge, including the following steps:

[0084] S1. Get the input image I and extract the visual features V of the image through the pre-trained visual encoder.

[0085] Input each image I in the dataset into the visual encoder (pre-trained ViT-B-32) in turn, and obtain the output of the last hidden state of the encoder Where r is the number of discrete features, which is 50 in this example, and d v is the representation dimension of the discrete feature, which in this example is 768. Remove the [CLS] token with sequence number 0 representing the global feature, and use the remaining r-1 quantitative features as the visual features extracted from the input image.

[0086] S2. Based on the pre-trained single decoder model, a one-stage draft decoder is constructed. The visual feature V is input into the draft decoder to generate the draft text description W. 1 .

[0087] Based on the mainstream single decoder model structure (Transformer and its variants), a one-stage draft decoder D1 is constructed, which mainly includes three parts: the self-attention module with mask, the cross-attention module for vision, and the feedforward network module. It is used as the input of each time step of the draft decoder training, and the self-attention feature calculation is completed through the mask to obtain the hidden state Where n represents the length of the reference text sequence, which is usually less than 20 words. The specific attention feature calculation process is as follows, where Q, K, and V all come from the same input in self-attention:

[0088] f mha (Q,K,V)=Concat(head1,...,head h )W O (1)

[0089] head i =f a (Q i , K i , V i ) (2)

[0090]

[0091] Q i =QW i Q , Ki =KW i K , V i =VW i V (4)

[0092] in d m is the mapping feature dimension of Q and K, d m′ is the feature dimension of the attention-weighted result, d o Is the feature dimension of the final output context, h is the number of attention heads. The extracted visual features V are linearly mapped as K and V for the visual cross attention module, and the calculation result of the mask attention in the previous step is As Q, we can get the result of cross attention Will After the input passes through the feedforward network, the hidden state of the third part is obtained Repeat the calculation of the above three modules, and after stacking multiple layers, we can obtain the input hidden state H of the final draft decoder 1 . H 1 After being mapped to the vocabulary dimension through the linear layer, the probability distribution of each token in the vocabulary is obtained through softmax at all time steps. For the probability distribution P 1 Sampling is performed, and the token with the highest probability is usually selected as the generated token of the first-stage decoder, and the draft text sequence is finally obtained. Where l represents the length of the draft description.

[0093] S3, determine the cross-modal feature fusion module, and build a two-stage error correction decoder based on the cross-modal feature fusion module, and combine the visual feature V and the draft text description W 1 Input into the error correction decoder to generate the final text description W 2 .

[0094] A two-stage error correction decoder is constructed based on CMF. The specific CMF module structure is as follows: Figure 2 As shown on the right. Embedded into text feature vector After that, we first pass the self-attention layer with mask to get the hidden state Where Q, K, and V are determined by T g Get them by linear mapping respectively. As Q, the visual encoding features V and the draft text W 1 Embedding vector of These are input as K and V respectively into the CMF cross-modal feature fusion module for the following calculations. The first is the cross-attention calculation, where an additional adapter operation is performed on the cross-attention results of the visual features:

[0095]

[0096]

[0097] Adapter is a two-layer MLP network with GELU nonlinear activation function. The specific calculation steps are as follows, where mlp_ratio = 0.25,

[0098] h in =XW in (7)

[0099] h out =GELU(h in )W out (8)

[0100] X′=h out +X (9)

[0101] The second is GLU gated linear unit activation. GLU activation is required for V′ and T′ to ensure that the result of the cross attention is closely related to the input Q. The calculation process is as follows:

[0102]

[0103]

[0104] Where [;] represents the splicing operation, σ represents the sigmoid activation function, and ⊙ represents element-wise multiplication. Next comes the gated calculation step in CMF. Through gated weighting, the model in the second stage can learn to distinguish the benefits of draft information and visual information for the current token, and learn how to identify errors in the draft caption to generate better descriptions. The calculation process is shown in the following formula, where

[0105] V″′=V″⊙σ([Q;V″;T″]W c V ) (12)

[0106] T″′=T″⊙σ([Q;V″;T″]W c T ) (13)

[0107] Finally, the features of the two modalities need to be fused together, which is achieved here by splicing + linear mapping to obtain the hidden state output of the CMF module in

[0108]

[0109] After the CMF module is calculated, similar to the S2 step, there are also steps such as feedforward network layer, multi-layer stacking calculation, mapping the hidden state output to the word list, sampling to obtain token output, etc., and finally the text description sequence of the two-stage decoder is obtained. Where m represents the length of the sentence.

[0110] S4. Describe the text W 2 The reference description W paired with the input image I g Compare and calculate the loss value of the loss function to optimize the overall parameters of the model.

[0111] According to W generated by the two-stage decoder 2 and reference description W g , calculate the cross entropy loss at each time step:

[0112]

[0113] According to the above cross entropy loss, batch training is adopted and the basic learning rate is set to 5e -4 , a total of 30 rounds of training, using the Adam optimizer with a learning rate ramp-up and exponential decay strategy, to optimize the model parameters until convergence is achieved. Then, using the final state of the model in the cross entropy phase as the starting point, SCST reinforcement learning optimization is performed. The loss calculation formula is as follows, where W s 2 represents the sample generation result of the two-stage decoder, r(·) is the reward score function, here we use CIDEr:

[0114]

[0115] According to the above reinforcement learning loss, batch training is adopted, and the learning rate is fixed to 5e -6 , a total of 20 rounds of training were performed, using the Adam optimizer to optimize the model parameters until convergence was achieved.

[0116] Training and validation testing were performed on the MS COCO dataset, which contains 123,287 images, each with five manually annotated text description sentences. Using the mainstream Karpathy partitioning method, the dataset was divided into training, validation, and test sets in a ratio of 113,287:5000:5000. The training and test samples had the same category distribution, and the evaluation metric was CIDEr, a method for calculating the n-gram cosine similarity between two text sentences. Compared with mainstream technical solutions for image description generation tasks, the experimental results of this example are shown in Table 1 below:

[0117] Table 1

[0118] Model XE-CIDEr RL-CIDEr Att2in 101.3 111.4 Up-Down 113.5 120.1 Up-Down+CMA-DM 115.1 125.0 AAT 115.5 126.7 M2* 111.6 125.0 M2*+CMF 121.9 126.1 Transformer (baseline) 110.6 125.3 CMF-TS(ours) 121.8 127.3

[0119] In summary, the present invention has the following advantages and beneficial effects compared to the prior art:

[0120] (1) The two-stage image description generation method designed by the present invention, which introduces pre-trained model knowledge, can be adapted to the mainstream single-decoder model and has strong scalability and reproducibility.

[0121] (2) The two-stage image description generation method designed by the present invention introduces pre-trained model knowledge, which fully utilizes the knowledge of the first-stage pre-trained model and optimizes and improves its results to generate better image descriptions, which can significantly improve the performance of relevant evaluation indicators.

[0122] (3) The two-stage image description generation method designed by the present invention introduces pre-trained model knowledge, and proposes a cross-modal feature fusion (CMF) module based on multi-head attention, gated linear units, and gating mechanisms. It can effectively align the multimodal features of vision and text, and can be extended to existing technologies involving multimodal feature fusion, with strong applicability.

[0123] This embodiment also provides a two-stage image description generation system, including:

[0124] The visual feature extraction module is used to obtain the input image I and extract the visual features V of the image through the pre-trained visual encoder;

[0125] The first description generation module is used to build a one-stage draft decoder based on the pre-trained single decoder model. According to the visual features V, the draft decoder generates a draft text description W 1 ;

[0126] The second description generation module is used to determine the cross-modal feature fusion module and build a two-stage error correction decoder based on the cross-modal feature fusion module to combine the visual features V and the draft text description W 1Input into the second-stage error correction decoder to generate the final text description W 2 ;

[0127] Model parameter optimization module, used to describe the text W 2 The reference description W paired with the input image I g Compare and calculate the loss value of the loss function to optimize the overall parameters of the model.

[0128] A two-stage image description generation system of this embodiment can execute a two-stage image description generation method provided by the method embodiment of the present invention, can execute any combination of implementation steps of the method embodiment, and has the corresponding functions and beneficial effects of the method.

[0129] This embodiment further provides a two-stage image description generation device, comprising:

[0130] at least one processor;

[0131] at least one memory for storing at least one program;

[0132] When the at least one program is executed by the at least one processor, the at least one processor implements the following Figure 3 The method shown.

[0133] A two-stage image description generation device of this embodiment can execute a two-stage image description generation method provided by the method embodiment of the present invention, can execute any combination of implementation steps of the method embodiment, and has the corresponding functions and beneficial effects of the method.

[0134] The present application also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs Figure 3 The method shown.

[0135] This embodiment also provides a storage medium storing instructions or programs that can execute a two-stage image description generation method provided by an embodiment of the method of the present invention. When the instructions or program are run, any combination of implementation steps of the method embodiment can be executed, and the corresponding functions and beneficial effects of the method can be obtained.

[0136] In some optional embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided in an exemplary manner for the purpose of providing a more comprehensive understanding of the technology. The disclosed method is not limited to the operation and logic flow presented herein. Optional embodiments are contemplated in which the order of the various operations is changed and the sub-operations described as a part of a larger operation are performed independently.

[0137] Furthermore, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise indicated, one or more of the functions and / or features described may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It will also be understood that a detailed discussion of the actual implementation of each module is not necessary for understanding the present invention. More specifically, given the properties, functions, and internal relationships of the various functional modules in the devices disclosed herein, the actual implementation of the module will be understood within the ordinary skill of an engineer. Therefore, a person skilled in the art using ordinary skill will be able to implement the present invention set forth in the claims without undue experimentation. It will also be understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0138] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0139] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0140] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.

[0141] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0142] In the above description of this specification, reference to the terms "one embodiment / example," "another embodiment / example," or "certain embodiments / examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0143] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

[0144] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A two-stage image description generation method, characterized in that: The following steps are involved: S1. Get the input image I and extract the visual features V of the image through the pre-trained visual encoder; S2. Based on the pre-trained single decoder model, a one-stage draft decoder is constructed. The visual feature V is input into the draft decoder to generate the draft text description W. 1 ; S3, determine the cross-modal feature fusion module, and build a two-stage error correction decoder based on the cross-modal feature fusion module, and combine the visual feature V and the draft text description W 1 Input into the error correction decoder to generate the final text description W 2 ; S4. Describe the text W 2 The reference description W paired with the input image I g Compare and calculate the loss value of the loss function, and then optimize the overall parameters of the model; The step S3 specifically includes: A two-stage error correction decoder D2 is constructed based on the cross-modal feature fusion module; the error correction decoder D2 includes an input word embedding layer, a masked self-attention module, a cross-modal feature fusion module, a feedforward network module, and an output layer; Reference description Embedded into text feature vector After that, the hidden state is obtained through the self-attention module with mask Hidden state As Q, the visual features V, the draft text description W 1 Embedding vector of As K and V respectively, they are input into the cross-modal feature fusion module for calculation to obtain the hidden state output Hidden state output After the feedforward network module, multi-layer stacking calculation, and hidden state output mapping to the word list, the probability distribution of the token is obtained, the probability distribution is sampled, and the token with the maximum probability position is selected as the generated token of the second-stage decoder, and finally the draft text description is obtained. Where m represents the length of the sentence.

2. A two-stage image description generation method according to claim 1, characterized in that: The step S1 specifically includes: Input each input image I in the dataset into the visual encoder in turn to obtain the output of the last hidden state of the encoder Where r is the number of discrete features, d v is the representation dimension of discrete features; remove the [CLS] token with sequence number 0 representing the global feature, and use the remaining r-1 number features as the visual feature V extracted from the input image I.

3. A two-stage image description generation method according to claim 1, characterized in that: The step S2 specifically includes: Based on the mainstream single decoder model structure, a one-stage draft decoder D1 is constructed. The draft decoder D1 consists of three parts: a masked self-attention module, a visual cross-attention module, and a feedforward network module. Reference description In turn, it is used as the input of each time step of the draft decoder D1 training. The feature calculation of self-attention is completed through masking to obtain the hidden state Where n represents the length of the reference text sequence; The calculation process of the self-attention feature is as follows: f mha (Q,K,V)=Concat(head1,…,head h )W O (1) head i =f a (Q i ,K i ,V i ) (2) Q i =QW i Q ,K i =KW i K ,V i =VW i V (4) in, d m is the mapping feature dimension of Q and K, d m′ is the feature dimension of the attention-weighted result, d o is the feature dimension of the final output context, and h is the number of attention heads; The extracted visual feature V is linearly mapped as K and V for the visual cross attention module, and the calculation result of the mask attention in the previous step is As Q, we can get the result of cross attention Will After the input passes through the feedforward network, the hidden state of the third part is obtained Repeat the calculation of the three modules of self-attention module, cross attention module and feedforward network module, and after multiple layers are stacked, the input hidden state H of the final draft decoder is obtained. 1 ; H 1 After being mapped to the vocabulary dimension through the linear layer, the probability distribution of each token in the vocabulary is obtained through softmax at all time steps. For the probability distribution P 1 Sampling is performed, and the token with the highest probability position is selected as the generated token of the first-stage decoder, and finally the draft text description is obtained. Where l represents the length of the draft description.

4. A two-stage image description generation method according to claim 1, characterized in that: The calculation of the cross-modal feature fusion module is as follows: The first is the cross attention calculation, in which an additional adapter operation is performed on the cross attention results of visual features: Adapter is a two-layer MLP network with GELU nonlinear activation function. The specific calculation steps are as follows: h in =XW in (7) h out =GELU(h in )W out (8) X′=h out +X (9) where X is the adapter input, and after calculation and residual connection, the output is X′; Next is the GLU gated linear unit activation. GLU activation is required for V′ and T′ to ensure that the result of the cross attention is closely related to the input Q. The calculation process is as follows: Among them, [;] represents the splicing operation, σ represents the sigmoid activation function, ⊙ represents element-wise multiplication; Then comes the gate calculation step in the cross-modal feature fusion module. The calculation process is shown in the following formula, where Finally, the features of the two modalities need to be fused together by splicing + linear mapping to obtain the hidden state output of the cross-modal feature fusion module in 5. The two-stage image description generation method according to claim 1, characterized in that: The step S4 specifically includes: According to W generated by the error correction decoder 2 and reference description W g , calculate the cross entropy loss at each time step: According to the cross entropy loss, batch training is adopted and the basic learning rate is set to 5e -4 , a total of 30 rounds of training were conducted, using the Adam optimizer under the learning rate ramp-up and exponential decay strategy to optimize the model parameters until convergence was achieved.

6. A two-stage image description generation method according to claim 5, characterized in that: The two-stage image description generation method also includes the step of reinforcement learning: Taking the final state of the model in the cross entropy phase as the starting point, SCST reinforcement learning optimization is performed, and the loss calculation formula is as follows: Where W s 2 represents the sampling generation result of the two-stage decoder, r(·) is the reward score function; According to the reinforcement learning loss, batch training is adopted, and the learning rate is fixed to 5e -6 , a total of 20 rounds of training were performed, using the Adam optimizer to optimize the model parameters until convergence was achieved.

7. A two-stage image description generation system, characterized in that: include: The visual feature extraction module is used to obtain the input image I and extract the visual features V of the image through the pre-trained visual encoder; The first description generation module is used to build a one-stage draft decoder based on the pre-trained single decoder model. According to the visual features V, the draft decoder generates a draft text description W 1 ; The second description generation module is used to determine the cross-modal feature fusion module and build a two-stage error correction decoder based on the cross-modal feature fusion module to combine the visual features V and the draft text description W 1 Input into the second-stage error correction decoder to generate the final text description W 2 ; Model parameter optimization module, used to describe the text W 2 The reference description W paired with the input image I g Compare and calculate the loss value of the loss function, and then optimize the overall parameters of the model; The second description generating module is specifically configured to perform the following steps: A two-stage error correction decoder D2 is constructed based on the cross-modal feature fusion module; the error correction decoder D2 includes an input word embedding layer, a masked self-attention module, a cross-modal feature fusion module, a feedforward network module, and an output layer; Reference description Embedded into text feature vector After that, the hidden state is obtained through the self-attention module with mask Hidden state As Q, the visual features V, the draft text description W 1 Embedding vector of As K and V respectively, they are input into the cross-modal feature fusion module for calculation to obtain the hidden state output Hidden state output After the feedforward network module, multi-layer stacking calculation, and hidden state output mapping to the word list, the probability distribution of the token is obtained, the probability distribution is sampled, and the token with the maximum probability position is selected as the generated token of the second-stage decoder, and finally the draft text description is obtained. Where m represents the length of the sentence.

8. A two-stage image description generation device, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a program executable by a processor, characterized in that: The processor-executable program is used to perform the method according to any one of claims 1 to 6 when executed by the processor.

Images