Construction method of image description model, image description method and equipment

Abstract

The invention discloses a construction method of an image description model, an image description method and equipment, and belongs to the technical field of image processing. A DT model comprising multiple layers of DT module cascading is designed, it is guaranteed that the denoising training process in a next layer of DT module is constrained by a prediction result of an upper layer of DT module through the multiple layers of DT module cascading, and on the basis of self-constraint of a diffusion model, the denoising training process of the next layer of DT module is optimized. Predicted sentence structure information and denoised comprehensive semantic information serve as enhanced constraint conditions to train a denoising process, and the method is a diffusion network non-autoregression image description method based on structure and semantic information condition constraint. The problems that in an existing non-autoregression image description method, dependency information is lacked between words, and generated sentences have serious word repetition and missing are solved, and the image can be rapidly and accurately described. In addition, the invention further provides an optimization strategy of multiple iterations of the P candidate models, so that the sample quality and the training efficiency can be greatly improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and more specifically, relates to a method for constructing an image description model, an image description method and a device. Background Art

[0002] For the fields of computer vision and natural language, the task of image description aims to automatically describe the content of an image using appropriate sentences. This task spans the fields of computer vision and natural language processing and has a wide range of application scenarios, such as helping the visually impaired understand the environment and alerting drivers of self-driving cars to potential risks.

[0003] Existing image description methods mainly include autoregressive and non-autoregressive methods. The autoregressive image description method is a method that generates image descriptions through an autoregressive model. The generation of each word in the described sentence needs to rely on the information of the previously generated words. The autoregressive model refers to a method that predicts the next part by serializing part of the data. The autoregressive image description first extracts features from the image and uses the image feature vector to initialize the hidden state of the autoregressive model. The model gradually generates description words based on the current latent state and the previously generated words. It relies on the step-by-step generation of text and has a slow generation speed. The non-autoregressive method generates the entire description sequence at one time, and the sentence described does not need to use the words of the sentence that has been generated. It can generate text in parallel and is faster, but the generation quality is often not as good as the autoregressive method and cannot accurately describe the image. Summary of the invention

[0004] In view of the above defects or improvement needs of the prior art, the present invention provides a method for constructing an image description model, an image description method and a device, so as to solve the technical problem that the prior art cannot describe images quickly and accurately.

[0005] In order to achieve the above object, in a first aspect, the present invention provides a method for constructing an image description model, comprising:

[0006] S1, extract the regional visual features of each image in the training set and map them into a visual embedding tensor of preset size; encode the visual embedding tensor into visual context information; use a cross-modal retrieval model to obtain the comprehensive semantic information x0 of the visual embedding tensor; after simulating bit encoding of x0, further perform position encoding and time encoding to obtain the semantic condition y; perform T-step denoising operation on x0 to obtain the potential state sequence x1, x2, …, x T ; T is the preset total number of diffusion steps; let the current iteration number epoch = 1;

[0007] S2, from x1, x2, …, x TIn the sequence z1, z2, …, z is sampled L times from the front to the back, and the sequence z1, z2, …, z L ; In the subsequent training process, z t As the input of the DT model at time step t, and z l Before z l-1 is input into the DT model; t = L, L-1, ..., 1; l = L, L-1, ..., 2; 2 ≤ L ≤ T; the DT model includes I cascaded DT modules; I is a positive integer; each DT module includes: a multi-head self-attention layer, a cross-attention layer, a sentence decoder, a linear classifier, and a stacked semantic encoder;

[0008] S3. Perform L time-step training on the DT model: At each time step t, perform in the i-th DT module: transform y, z t , the comprehensive semantic information after de-noising of the i-1th DT module at time step t The comprehensive semantic information after denoising of the i-th DT module at the previous time step After concatenation to obtain the semantic latent code, it passes through the stacked semantic encoder and multi-head self-attention layer in sequence to obtain the self-attention semantic prior information; the cross-attention layer is used to obtain the cross-attention signal of the visual context information and the self-attention semantic prior information, and the sentence decoder is used to decode the self-attention semantic prior information with the cross-attention signal as a constraint to obtain the comprehensive semantic information after denoising of the i-th DT module at time step t. Then the corresponding image description information is obtained through the linear classifier By minimizing The difference loss from x0, and The difference loss with the corresponding true description information is used to train the i-th DT module; where i = 1, 2, …, I; Image description information includes: sentence structure information;

[0009] S4, determine whether the epoch reaches the preset number of iterations M. If so, take the current DT model as the optimal DT model; otherwise, add one to the epoch and go to S2 to start the next iteration;

[0010] S5. Remove the semantic encoders of each DT module in the optimal DT model and the linear classifiers in the first I-1 DT modules, and connect the output of the sentence decoder of the j-th DT module to the multi-head self-attention layer of the j+1-th DT module to construct an image description model including the optimal DT model; j=1,2,…,I-1; in the application process of the image description model, the external input of the multi-head self-attention layer of the first DT module therein is random noise, and the external input of the cross-attention layer of each DT module therein is the visual context information of the image to be described.

[0011] Further preferably, the number of the DT models is P; P≥1; the S3 includes: performing L time steps of training on the P DT models respectively;

[0012] The method for constructing the above image description model also includes:

[0013] After the first iteration is completed and before the next iteration begins, the image description information obtained at the last time step of the current P DT models is used to form a prediction sample pool; the image description information with the highest NLP evaluation index is selected from the prediction sample pool as the optimal image description information for the first iteration, and is saved together with its corresponding DT model;

[0014] After the completion of the rth iteration and before the start of the r+1th iteration, the image description information obtained at the last time step of the current P DT models is combined with the optimal image description information in the previous iteration to form a new prediction sample pool. The image description information with the highest NLP evaluation index is selected from the new prediction sample pool as the optimal image description information in the rth iteration and saved together with its corresponding DT model; r = 2, 3, …, M;

[0015] The DT model corresponding to the optimal image description information in the Mth iteration is taken as the optimal DT model.

[0016] More preferably, The difference loss from x0 includes: Mean squared error loss and / or cross entropy loss between x0 and x0.

[0017] Further preferably, the method for constructing the above-mentioned image description model further comprises: adjusting the value of L after each iteration is completed and before the next iteration starts.

[0018] Further preferably, the image description information includes: the number of sentences and sentence structure information of each sentence; the sentence structure information includes: the number of words in the sentence, word attributes and word positions.

[0019] Further preferably, the method for constructing the image description model further includes: the following operations performed between S4 and S5:

[0020] A1, determine whether the NLP evaluation index of the image description information output by the last time step of the optimal DT model is greater than the first preset threshold. If so, go to A2; otherwise, set the current iteration number epoch = 1 and go to S2;

[0021] A2. Strengthen learning of the optimal DT model;

[0022] A3, judging whether the NLP evaluation index of the image description information outputted by the last time step of the optimal DT model after the completion of reinforcement learning is greater than the second preset threshold value, if so, go to A4; otherwise, go to A2;

[0023] A4. End the training and go to S5.

[0024] Further preferably, the above A2 includes:

[0025] A plurality of sentence structure information is obtained by sampling the image description information outputted by the current optimal DT model at the last time step, and a plurality of word selections are selected from each sentence structure information to obtain a plurality of word selection combinations; an advantage function is set for each word selection position of each word selection combination, and the optimal DT model is reinforced by learning the result with the goal of maximizing the weighted sum of the advantage functions of each word selection position in each word selection combination;

[0026] Among them, the kth word position s of the word combination u k The advantage function is:

[0027]

[0028] is the NLP evaluation index for the word combination u; the reverse behavior baseline Used to reflect word selection position k The contribution value of the selected words at Expresses expectation; -k Indicates the position s of the word in u except for the word k The combination of words in all other positions except π k Indicates the preset word selection strategy; u' k Indicates the position s of the word in u except for the word k Word selection in positions other than ; To -k with u' k The combined NLP evaluation index is expressed as u' k Replace the word position s in u kNLP evaluation index of the word combination after the word selection.

[0029] In a second aspect, the present invention provides an image description method for use in an application phase, comprising:

[0030] Extracting regional visual features of the image to be described and mapping them into a visual embedding tensor of a preset size; encoding the visual embedding tensor into visual context information of the image to be described;

[0031] Input the visual context information and random noise of the image to be described into the image description model to obtain the corresponding image description information;

[0032] The above-mentioned image description model is constructed by using the method for constructing an image description model provided by the first aspect of the present invention.

[0033] In a third aspect, the present invention provides an electronic device, comprising: a memory and a processor, wherein the memory stores a computer program, and the processor executes the method provided in the first aspect and / or the second aspect of the present invention when executing the computer program.

[0034] In a fourth aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method provided in the first aspect and / or the second aspect of the present invention is performed.

[0035] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects:

[0036] 1. The present invention provides a method for constructing an image description model, and designs a DT model including a cascade of multiple DT modules. In the training process of each time step t, the semantic condition y, the Self-condition comprehensive semantic information z that has not been denoised at time step t are converted into the semantic condition y. t , the comprehensive semantic information after denoising of the previous DT module at time step t, and the semantic latent code obtained by splicing the comprehensive semantic information after denoising of the current DT module at the previous time step, combined with visual context information to predict image description information and denoised comprehensive semantic information; wherein, the image description information includes sentence structure information; the present invention ensures that the denoising training process in the next-layer DT module will be constrained by the prediction results of the previous-layer DT module through multi-layer cascading DT modules, and on the basis of the self-constraint of the diffusion model, the predicted sentence structure information and the denoised comprehensive semantic information are used as enhanced constraint conditions to train the denoising process. It is a diffusion network non-autoregressive image description method based on sentence structure information and comprehensive semantic information conditional constraints, which solves the problems of lack of dependency information between words and serious word duplication and missing in the generated sentences in the existing non-autoregressive image description methods, and can quickly and accurately describe images.

[0037] 2. Furthermore, the method for constructing an image description model provided by the present invention proposes an optimization strategy for multiple iterations of P candidate models. During the training process, P DT models are trained separately in each iteration, and the currently obtained P image description information and the optimal image description information obtained in the previous iteration are combined to form a prediction sample pool, and the image description information with the highest NLP evaluation index is selected as the optimal image description information under the current iteration. At the same time, the corresponding DT model is selected as the optimal DT model, which can ensure that the predicted prediction sample pool has a higher proportion of positive samples, suppress the probability of generating low-quality image description information, thereby greatly improving sample quality and training efficiency.

[0038] 3. Furthermore, the method for constructing the image description model provided by the present invention takes into account that only by minimizing The difference loss from x0, and Due to the difference loss with the corresponding true description information, the performance improvement of training the i-th DT module has a certain upper limit, so the reinforcement learning training stage is introduced to further improve the performance of the image description model.

[0039] 4. Furthermore, in the method for constructing an image description model provided by the present invention, in the process of reinforcement learning of the DT model, a separate advantage function is defined for each word position to measure the contribution of a specific position, thereby correcting the deviation of the reward value of each word in the sentence, which can better balance the contribution value of the word to the sentence as a whole, thereby better encouraging the final generated image description sentence to have more words that conform to the sentence structure characteristics. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 An overall flow chart of a method for constructing an image description model provided by an embodiment of the present invention;

[0041] Figure 2 The overall architecture diagram of the image description based on multi-layer DT module cascade provided by the embodiment of the present invention;

[0042] Figure 3 A DT model denoising training process using cross entropy loss provided in an embodiment of the present invention;

[0043] Figure 4 A schematic diagram of an optimization strategy for multiple iterations of P candidate DT models provided in an embodiment of the present invention;

[0044] Figure 5 A flowchart of the DT model training of joint cross entropy and reinforcement learning provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0046] Diffusion Transformer (DT) is a machine learning model for generating data. It is mainly used in the field of computer vision for image generation, image editing, image insertion drawing, image segmentation, etc. The diffusion model has the advantages of high-quality generation, progressive generation, solid theoretical foundation, strong versatility, and stable training. The diffusion model mainly includes two steps: first, a fixed forward diffusion process, that is, gradually adding Gaussian noise to the data until pure noise is obtained; second, a trainable reverse denoising process, gradually denoising from pure noise until the initial data is obtained. The diffusion model trains the model by gradually adding noise to the data and then denoising it, so as to obtain noise-free data.

[0047] DT uses the Transformer model for diffusion networks. The Transformer model is a deep learning model in the field of natural language processing. It is widely used in machine translation, text generation, summary generation and other tasks. Its core is the attention mechanism. It uses the Encoder-Decoder architecture and can provide high-quality output in multiple generation tasks. However, there are still problems in using diffusion networks for image description. For example, the denoised comprehensive semantic information obtained by Self-condition is not ideal for predicting sentence structure, it cannot guide the training of the denoising process of the diffusion model well, the serial training method makes it difficult to find the optimal candidate DT model in time, and it cannot measure the contribution of words in a specific position to the reward of the entire sentence.

[0048] In view of the above problems, the present invention constrains the denoising process by adding comprehensive semantic information and sentence structure information as conditions. Specifically, in a first aspect, the present invention provides a method for constructing an image description model, comprising:

[0049] S1, extract the regional visual features of each image in the training set and map them into a visual embedding tensor of preset size; encode the visual embedding tensor into visual context information; use a cross-modal retrieval model to obtain the comprehensive semantic information x0 of the visual embedding tensor; after simulating bit encoding of x0, further perform position encoding and time encoding to obtain the semantic condition y; perform T-step denoising operation on x0 to obtain the potential state sequence x1, x2, …, x T ; T is the preset total number of diffusion steps; let the current iteration number epoch = 1;

[0050] S2, from x1, x2, …, x T In the sequence z1, z2, …, z is sampled L times from the front to the back, and the sequence z1, z2, …, z L ; In the subsequent training process, zt is used as the input of the DT model at time step tt, and z l Before z l-1 is input into the DT model; t = L, L-1, ..., 1; l = L, L-1, ..., 2; 2 ≤ L ≤ T; the DT model includes I cascaded DT modules; I is a positive integer; each DT module includes: a multi-head self-attention layer, a cross-attention layer, a sentence decoder, a linear classifier, and a stacked semantic encoder;

[0051] In an optional implementation, the semantic encoder is a Transformer encoder; and the sentence decoder is a Transformer decoder.

[0052] S3. Perform L time-step training on the DT model: At each time step t, perform in the i-th DT module: transform y, z t , the comprehensive semantic information after de-noising of the i-1th DT module at time step t The comprehensive semantic information after denoising of the i-th DT module at the previous time step After concatenation to obtain the semantic latent code, it passes through the stacked semantic encoder and multi-head self-attention layer in sequence to obtain the self-attention semantic prior information; the cross-attention layer is used to obtain the cross-attention signal of the visual context information and the self-attention semantic prior information, and the sentence decoder is used to decode the self-attention semantic prior information with the cross-attention signal as a constraint to obtain the comprehensive semantic information after denoising of the i-th DT module at time step t. Then the corresponding image description information is obtained through the linear classifier By minimizing The difference loss from x0, and The difference loss with the corresponding true description information is used to train the i-th DT module; where i = 1, 2, …, I;

[0053] In an optional implementation manner, the image description information includes: the number of sentences and sentence structure information of each sentence; the sentence structure information includes: the number of words in the sentence, word attributes and word positions.

[0054] S4, determine whether the epoch reaches the preset number of iterations M. If so, take the current DT model as the optimal DT model; otherwise, add one to the epoch and go to S2 to start the next iteration;

[0055] In an optional implementation manner, the preset number of iterations M is 60.

[0056] S5. Remove the semantic encoders of each DT module in the optimal DT model and the linear classifiers in the first I-1 DT modules, and connect the output of the sentence decoder of the j-th DT module to the multi-head self-attention layer of the j+1-th DT module to construct an image description model including the optimal DT model; j=1,2,…,I-1; in the application process of the image description model, the external input of the multi-head self-attention layer of the first DT module therein is random noise, and the external input of the cross-attention layer of each DT module therein is the visual context information of the image to be described.

[0057] In an optional implementation manner, the number of the DT models is P; P ≥ 1; the S3 includes: performing L time steps of training on the P DT models respectively;

[0058] The method for constructing the above image description model also includes:

[0059] After the first iteration is completed and before the next iteration begins, the image description information obtained at the last time step of the current P DT models is used to form a prediction sample pool; the image description information with the highest NLP evaluation index is selected from the prediction sample pool as the optimal image description information for the first iteration, and is saved together with its corresponding DT model;

[0060] After the completion of the rth iteration and before the start of the r+1th iteration, the image description information obtained at the last time step of the current P DT models is combined with the optimal image description information in the previous iteration to form a new prediction sample pool. The image description information with the highest NLP evaluation index is selected from the new prediction sample pool as the optimal image description information in the rth iteration and saved together with its corresponding DT model; r = 2, 3, …, M;

[0061] The DT model corresponding to the optimal image description information in the Mth iteration is taken as the optimal DT model.

[0062] In an optional implementation, the value of P is 4.

[0063] In an optional implementation manner, the method for constructing the image description model further includes: adjusting the value of L after each iteration is completed and before the next iteration begins.

[0064] In an optional implementation manner, the method for constructing the image description model further includes: the following operations performed between S4 and S5:

[0065] A1, determine whether the NLP evaluation index of the image description information output by the last time step of the optimal DT model is greater than the first preset threshold. If so, go to A2; otherwise, set the current iteration number epoch = 1 and go to S2;

[0066] A2. Strengthen learning of the optimal DT model;

[0067] A3, judging whether the NLP evaluation index of the image description information outputted by the last time step of the optimal DT model after the completion of reinforcement learning is greater than the second preset threshold value, if so, go to A4; otherwise, go to A2;

[0068] A4. End the training and go to S5.

[0069] In an optional implementation manner, the NLP evaluation indicator adopts the CIDEr score, and the first preset threshold and the second preset threshold are 130 and 133 respectively.

[0070] In an alternative embodiment,

[0071] The above A2 includes: obtaining multiple sentence structure information from the sampling of the image description information output by the current optimal DT model in the last time step, and selecting multiple words from each sentence structure information to obtain multiple word selection combinations; setting an advantage function for each word selection position of each word selection combination, and performing reinforcement learning on the optimal DT model with the goal of maximizing the weighted sum of the advantage functions of each word selection position in each word selection combination;

[0072] Among them, the kth word position s of the word combination u k The advantage function is:

[0073]

[0074] is the NLP evaluation index for the word combination u; the reverse behavior baseline Used to reflect word selection position k The contribution value of the selected words at Expresses expectation; -k Indicates the position s of the word in u except for the word k The combination of words in all other positions except π k Indicates the preset word selection strategy; u' k Indicates the position s of the word in u except for the word k Word selection in positions other than ; To -k with u' k The combined NLP evaluation index is expressed as u' k Replace the word position s in uk NLP evaluation index of the word combination after the word selection.

[0075] It should be noted that the NLP evaluation indicators mentioned above may be: CIDEr score, BLEU score, METOR score, ROUGE score, accuracy, precision, recall rate, etc. In the present invention, the CIDEr score is preferably used.

[0076] In order to further illustrate the method for constructing the image description model provided by the present invention, a specific embodiment is described in detail below:

[0077] like Figure 1 As shown, it is an overall flow chart of the method for constructing the image description model provided in the embodiment; the method for constructing the image description model provided in this embodiment mainly includes four key technologies: denoising of comprehensive semantic information in the cascade DT, cross entropy training based on sentence structure information and comprehensive semantic information, multiple iterations of optimization strategy of P candidate DT models obtained by parallel training of P GPUs, and reinforcement learning that defines a separate advantage function for each position as a baseline function. The key technical essentials are described as follows:

[0078] (1) Denoising of comprehensive semantic information in cascaded DT: First, optimize the DT module. In the past, only the noisy comprehensive semantic information in the current DT layer at the current time step was used in the DT model training process. The denoising quality of this method that only uses the noisy comprehensive semantic information in a certain layer of DT module is poor. Therefore, we propose to stack multiple DT modules and generate high-fidelity comprehensive semantic information through model training with multiple DT layers and multiple time steps. The method proposed in the present invention uses the semantic structure information predicted by the previous time step of the current DT layer and the current time step of the previous DT layer in the denoising training process of the DT module, so that the final generated image description is more in line with the natural sentence structure.

[0079] DT training mainly consists of two steps: the denoising process and the denoising process. In the denoising process, the initial comprehensive semantic information x0 is denoised for T steps to obtain the noisy comprehensive semantic information x1, x2, …, x T , usually T = 50. Denoising is the purpose of DT module training. T Randomly sample L noisy comprehensive semantic information in z1,z2,…,z L , and x0 is recorded as z0. De-noise them through DT training until z0 is restored. For the i-th layer DT module, use the semantic condition y, the self-condition comprehensive semantic information (word token) z of the i-th layer DT module at time step t that has not been denoised t , the comprehensive semantic information after de-noising of the i-1th DT module at time step t The comprehensive semantic information after denoising of the i-th DT module at the previous time step By concatenating semantic latent codes. Since all four are of dimension bs×20×512, the dimension of semantic latent code is bs×80×512. Semantic latent code obtains semantic prior information through stacked Transformer-based semantic encoders, and then obtains self-attention semantic prior information through multi-head self-attention layers. Then the visual context information and self-attention semantic prior information are sent to the cross-attention layer to output the attention signal. The self-attention semantic prior information and the attention signal are input into the sentence decoder. The output of the sentence decoder is the noisy comprehensive semantic information z at the current time step t. t Comprehensive semantic information after denoising Reuse Predicting the structural information of the sentence can provide more powerful constraint information for the denoising process training of the diffusion model: comprehensive semantic information and sentence structure information after denoising.

[0080] (2) Cross-entropy training based on sentence structure information and comprehensive semantic information: The cross-entropy loss training stage can be regarded as a coarse-grained model optimization. The purpose of DT training is to use the comprehensive semantic information z after adding noise. t As a condition, z is obtained through DT training t The comprehensive semantic information after denoising d t , which is the approximate value of z0. t It is a code that has been fixed in the process of adding noise, and no additional information is required. Such a process is called self-conditioning of the denoising process. Since the denoising process based on self-constraint lacks strong constraints, there is error accumulation in training. Therefore, the model in the present invention uses the semantic condition y output by the semantic information module of the current time step of the i-th layer DT module and the self-condition comprehensive semantic information z of the i-th layer DT module at the current time step that has not been denoised. t , the i-1th layer DT module at the current time step Comprehensive semantic information after denoising The i-th layer DT module is Comprehensive semantic information after denoising Generate semantic latent code (i.e., the comprehensive semantic information after denoising) and The potential state of x0 The semantic potential code is obtained by stacking semantic encoders to obtain semantic prior information. After passing through the multi-head self-attention layer, the self-attention semantic prior information and the attention signal in the visual encoding module are input into the sentence decoder together. That is, the self-attention semantic prior information is constrained by the attention signal to obtain the comprehensive semantic information after denoising. Send it to the linear classifier to predict the sentence structure information b predicted by the i-th layer DT module at the current time step j .

[0081] The loss function is used to measure the difference between the model prediction results and the actual results. In actual training, the comprehensive semantic information after denoising at time step t is and the mean square error between the initial comprehensive semantic information z0 Comprehensive semantic information after denoising at time step t The cross entropy loss function between the initial comprehensive semantic information z0 The semantic structure information loss function of the current time step t and the previous t steps in the denoising process Sum to get the total loss function in:

[0082]

[0083] in, represents the comprehensive semantic information after denoising at the tth step of the i-th DT module; J is the number of sentences in the image description information obtained at time step t; Q represents the number of words in the j-th sentence at time step t; c jq is the attribute of the qth word in the jth sentence at time step t; b j is the sentence structure information of the jth sentence at time step t; represents the CIDEr score function; p θ (·) represents the noise probability distribution during the denoising process. and The smaller the value, the better the denoising result. The closer to z0 (i.e. x0), the more noise z is input during prediction. t The easier it is to predict the sentence structure. Used for measurement The difference loss with the corresponding true description information, The smaller it is, the closer the predicted image description information is to the true description information.

[0084] (3) Reinforcement learning that defines a separate advantage function for each word position as a baseline function: In reinforcement learning, the baseline function is a measure of the average quality of the sampled sentences. This method uses the expected reward of the sample as the baseline, and sentences that are higher than the expected reward of the sample are encouraged to be generated. It is used to reduce the variance when estimating the reward function, perform bias correction on the reward value of the sentence, and improve the stability of the strategy and learning efficiency. In the past, a reward mechanism based on greedy decoding was used to allow each word position to receive the same team reward, which was unable to measure the contribution of the word at a specific position to the reward of the entire sentence. Such a baseline function cannot reflect the value of semantic structure information. Therefore, in this embodiment, a separate advantage function A is defined for each word position. k (s k ,u) is used to measure the contribution of words selected at a specific position to the semantic structure information, where s k It represents the position of the kth word selection, and u represents the word selection combination. It rewards Subtract the inverse baseline R at a specific location k (s k ,u -k ) to reflect the position s k The contribution value of the selected word. -k Indicates except position s k The word combination of all other positions except k (s k ,u) actually reflects the change of position s k The contribution value of the selected word at position s. k We traverse all possible word choices while keeping the word combinations at other positions unchanged, and calculate the inverse behavior baseline by marginalizing the reward. u' k Indicates word combinations at other positions, π k Indicates word selection strategy, represents the CIDEr score function. In order to simplify the calculation, we actually only consider the K words with the highest probability to approximate the expected calculation in the above equation. The inverse behavior baseline is a measure used to measure whether the word selection violates the semantic structure information. If the position s is determined k The word selection at the position conforms to the semantic structure information B given in the denoising training phase, then the advantage function A k (s k ,u) expand a certain ratio, otherwise reduce a certain ratio. Through the effect of the weighted advantage function, the reward of the word position that meets the structural prediction information in the sentence can be increased, thereby encouraging the generation of words that meet the structural characteristics.

[0085] (4) Multiple iterations of optimization strategy for P candidate DT models: The traditional self-critical strategy only establishes a sample pool with relatively high quality, and through multiple rounds of epoch optimization training, the traditional strategy of making the model prediction set as close to the sample pool as possible has obvious disadvantages: First, with the increase of epochs, the proportion of positive samples in the sample pool gradually decreases, and the loss gradient of the reward function quickly approaches zero. Secondly, the oscillation process of the model can optimize one part while weakening another part. Under the strategy of completely abandoning the overall performance index without breakthrough, these fluctuations are meaningless. In the optimization strategy of this embodiment, when the model is trained and optimized to the next epoch, P candidate next-generation models will be generated. There are differences between each candidate DT model. After the reasoning process, each prediction sample pool is obtained. For the same image, starting from the second round of iteration, there will be P+1 predicted sample sentences. From them, the sentences with higher NLP indicators such as accuracy, precision, and recall compared with the true label are selected as the sample sentences of the image in the sample pool, so that the target sample pool maintains a high proportion of positive samples, thereby suppressing the probability of generating low-quality sentences, and thus selecting the optimal DT model.

[0086] The above process is described in detail below with detailed diagrams:

[0087] like Figure 2 The figure shows the overall architecture diagram of the natural language image description model training based on the diffusion transformation of structure and semantic information in one embodiment of the present application. The purpose of the present invention is to use comprehensive semantic information and semantic structure information to constrain denoising training during model training, and adopt optimization strategies such as multiple iterations of P candidate DT models and advantage functions to improve the quality of the prediction sample pool.

[0088] In the implementation, the steps that should be included are: generating comprehensive semantic information, generating semantic latent codes, generating cross-attention signals, predicting sentence structure information, cross-entropy loss training of the diffusion model based on sentence structure information and comprehensive semantic information, optimization strategy of multiple iterations of P candidate DT models, and reinforcement learning that defines a separate advantage function for each word position as a baseline function.

[0089] 1. Generate comprehensive semantic information

[0090] like Figure 2As shown in the figure, the I-layer DT that generates semantic latent coding and sentence structure information includes: a visual semantic interaction module and a sentence structure prediction module. The visual semantic interaction module mainly includes: a cross-attention layer; the sentence structure prediction module mainly includes: a stacked semantic encoder, a multi-head self-attention layer, a sentence decoder and a linear classifier. The original image is subjected to regional visual feature extraction using target detection models such as R-CNN, Faster-RCNN, and YOLO, and the visual features of different regions in the image are extracted, mainly including the visual features of the foreground area; the regional visual features are first mapped to a fixed-size visual embedding vector through a fully connected layer, and the dimension is from bs×50×2048 to bs×50×512, where the training set is divided into batches to train the DT model, and bs is the batch size; the visual embedding vector is then encoded to obtain dimensionally unchanged visual context information as the input of the cross-attention layer in the visual semantic interaction module. At the same time, the cross-modal retrieval large model is used to output the visual embedding tensor as comprehensive semantic information x0 through analog bit encoding. The comprehensive semantic information includes nouns that may be contained in the image and the deduced verbs.

[0091] 2. Generating Semantic Latent Codes

[0092] By converting the word information of the comprehensive semantic information x0 into binary, the simulated bit encoding of the comprehensive semantic information x0 is realized. After obtaining the tensor after the simulated bit encoding, further position encoding and time encoding are performed to obtain the semantic condition y. Specifically, the simulated bit encoding is to set a serial number for each word. For example, the serial number of the word 'woman' in the vocabulary is 3, and the simulated bit encoding is a 14-bit information, the upper 12 bits of which are -1 and the lower two bits are 1, that is, the comprehensive semantic information is output through the simulated bit encoding. The dimension is bs×20×14 embedding tensor. After the tensor after the simulated bit encoding passes through a double embedding layer including full connection, position encoding, and time encoding, a semantic condition y with a dimension of bs×20×512 embedding tensor is output. In the case of I-layer cascaded DT modules (I is 3 in this embodiment), for the i-th layer DT module, the semantic condition y and the Self-condition potential state z of the current time step of the i-th layer DT module with a dimension of bs×20×512 are used. t , the comprehensive semantic information denoised by the i-1th layer DT module with dimension bs×20×512 at the current time step t The comprehensive semantic information denoised by the i-th DT module at the previous time step The semantic latent code with the dimension of bs×80×512 is generated by splicing. The semantic latent code is passed through the Transformer-based semantic encoder stacked in the sentence structure prediction module to obtain the semantic prior information with the dimension of bs×80×512. The features of the first 20 highly reliable dimensions related to the word token are sampled from the semantic prior information to obtain the semantic prior information with the dimension of bs×20×512, and then the self-attention semantic prior information is obtained through the multi-head self-attention layer.

[0093] 3. Generating Cross-Attention Signals

[0094] The regional visual features of the original image are obtained through Faster-RCNN, and the regional visual features are mapped into a visual embedding tensor of a fixed size. The visual embedding tensor is then encoded into visual context information. The visual context information and self-attention semantic prior information are passed through the cross-attention layer to obtain the cross-attention signal.

[0095] 4. Predicting sentence structure information

[0096] like Figure 2 As shown in Figure 1, the sentence structure prediction module contains a stacked Transformer-based semantic encoder, a multi-head self-attention layer, a Transformer-based sentence decoder, and two linear classifiers. The cross-attention signal and the self-attention semantic prior information are fed into the Transformer-based sentence decoder to obtain the current time step z with a dimension of bs×20×512. t Comprehensive semantic information after denoising Will The sentence structure information in the image description information is sent to two linear classifiers for prediction. The sentence is divided into smaller grammatical structures by the constituency parser. The predicted sentence structure information and the real sentence structure information label (Ground truth) are parsed into a tree structure and B = {b1, b2, ..., b J} records the sentence structure information of its J sentences, among which b j ={c j1 ,c j2 ,…,c jQ}, Q represents the number of words in the jth sentence, c jq Represents the attribute of the qth word in the jth sentence. The classification network can be trained using this sentence structure information as a label. The predicted sentence structure information and the comprehensive semantic information together form a semantic conditional signal with strong constraints to guide the denoising training of the diffusion network.

[0097] 5. Diffusion model cross entropy loss training based on sentence structure information and comprehensive semantic information

[0098] The training of the DT module mainly includes two steps: the denoising process and the denoising process. In the denoising process, the initial comprehensive semantic information x0 with a dimension of bs×20×512 is denoised for T steps to obtain the noisy comprehensive semantic information x0,x1,…,x T , generally T = 50, so that it changes from a tensor that conforms to a certain characteristic distribution law to a Gaussian noise that conforms to the normal distribution. Figure 3 As shown, denoising is the purpose of DT module training. In each batch in the same epoch, in x1, x2, ..., x T Randomly sample L noisy comprehensive semantic information in z1,z2,…,z L , and x0 is recorded as z0. Through DT training, they are denoised until z0 is restored. The noisy comprehensive semantic information z of the previous time step t+1 After DT training, the denoising result is obtained, which is recorded as the comprehensive semantic information d after denoising in the previous time step t+1 The purpose of DT training is to use the comprehensive semantic information z after adding noise t As a condition, train DT to obtain the comprehensive semantic information d after denoising t .z t It is a fixed code in the denoising process and does not require additional information. Such a denoising process is defined as having The Markov chain of Gaussian transitions is learned at the beginning, where I is the noise variance. Since this denoising process that relies only on self-constraints for training has error accumulation, the result is low generation quality. Therefore, the comprehensive semantic information and sentence structure information obtained are used as enhanced conditions to constrain the denoising process. At the same time, in order to solve the error accumulation problem, the semantic condition y output by the semantic information module at the time step t of the i-th layer DT module and the noisy Self-condition comprehensive semantic information z at the current time step of the i-th layer DT module are t , the comprehensive semantic information of the current time step denoised by the i-1th layer DT module The comprehensive semantic information denoised at the previous time step of the i-th layer DT module Then the joint distribution of the DT module denoising training process of the i-th layer changes from self-constrained It becomes a comprehensive semantic and sentence structure information constraint for noise addition and denoising In this stage of training, the sum of mean square error, cross entropy loss function, and sentence structure information loss function is used as the standard to judge the performance of model training.

[0099] 6. Optimization strategy for multiple iterations of P candidate DT models

[0100] In the diffusion model denoising training phase, the gradient of the expected reward is optimized by sampling sentences from the model. These samples are regarded as short-term goals in the training process, and the quality of the samples directly determines the optimization effect. The traditional self-critical optimization strategy does not update the sample pool at every step. Its strategy is to first establish a relatively high-quality sample pool, and through multiple rounds of epoch optimization training, make the model's prediction set as close to the sample pool as possible. The traditional optimization strategy will cause the proportion of positive samples in the sample pool to gradually decrease, and the loss gradient of the reward function will quickly approach zero. Figure 4 As shown, this embodiment proposes an optimization strategy for multiple iterations of P candidate DT models trained on P GPUs, generating P candidate DT models in each step. Since the sampling positions of comprehensive semantic information on each GPU are most likely different, there are differences between the candidate DT models. Each model obtains its own prediction sample pool through the reasoning process. For the same image, there are a total of P+1 predicted sample sentences including the original sample pool. Subsequently, synchronous optimization is performed to select the sentence with the highest NLP index compared with the true label from the P+1 sample sentences as the sampling sentence of the image in the sample pool, so as to determine the optimal DT model.

[0101] 7. Define a separate advantage function for each word position as a reinforcement learning baseline function

[0102] In the cross entropy loss training phase, the model uses the loss function of semantic structure information. To supervise the denoising process, this makes the sentences generated by the model more structured, but the performance improvement of cross entropy training has a certain upper limit. Figure 5 As shown, the training of the model is therefore divided into two stages: the cross-entropy training stage and the reinforcement learning training stage. On the other hand, since nouns contribute the most to sentence-level rewards, if the value of semantic structure information cannot be reflected in the gradient loss, the optimization training process will gradually destroy the original structural features, which will lead to the semantic structure information supervision in the cross-entropy training stage being in vain. Here, a typical example of sentence structure destruction is that reinforcement learning optimization training tends to cause the model to generate sentences consisting of a large number of nouns and a small number of conjunctions, which encroaches on the space of verbs. However, traditional sentence-level rewards based on greedy decoding allow each word position to receive the same team reward, and cannot measure the contribution of words in a specific position to the team reward. In order to solve the above problems, this embodiment defines a baseline function that can reflect the value of structural information to a certain extent, which can specify a separate advantage function for each position. It reflects the position k The contribution value of the selected word. k The weighted advantage function The role of the word is to improve the position reward in the sentence structure information, thereby encouraging the final image description sentence to have more words that meet the sentence structure characteristics.

[0103] In summary, the previous methods that only use DT modules for image description are not enough to generate high-quality image description information. The present invention provides a natural language description method for images that constrains diffusion transformation based on sentence structure information and comprehensive semantic information generated by visual context information and comprehensive semantic information. t , the comprehensive semantic information after de-noising of the i-1th DT module at time step t The comprehensive semantic information after denoising of the i-th DT module at the previous time step concatenate to generate semantic latent coding. Since semantic latent coding contains denoising and Therefore, the semantic potential code generates semantic prior information after the stacked semantic encoder, and then obtains the self-attention semantic prior information through the multi-head self-attention layer; the attention signal is obtained by using the self-attention semantic prior information and the visual context information, and the attention signal and the self-attention semantic prior information are input into the sentence decoder together to obtain the noisy comprehensive semantic information z t After denoising ( Not z t z before adding noise t-1 , but an approximation to z0), and then use The image description information is predicted, and the denoising process is constrained by sentence structure information and comprehensive semantic information. On this basis, an optimization strategy of multiple iterations of P candidate DT models is adopted to replace the self-critical training optimization strategy, and a new sentence-based baseline reward is proposed to ensure that the value of semantic structure information can still be reflected in the fine-grained tuning training process. In practice, the present invention finds that adding comprehensive semantic information and sentence structure information to train the denoising process can speed up the convergence speed. The optimization strategy of multiple iterations of P GPUs to simultaneously train P candidate DT models can ensure that the predicted sentence sample pool has a higher proportion of positive samples, thereby greatly improving sample quality and training efficiency. In addition, the proposed baseline function of sample-based expected reward can better balance the contribution of word position to the word itself and the sentence as a whole.

[0104] The present invention aims to solve the problem of lack of dependency information between words and serious word duplication and missing in generated sentences in non-autoregressive image description methods by improving the training process of diffusion network, so as to generate higher quality image description. Since the denoising process that relies solely on the denoising algorithm in the DT module for self-regulation and optimization has error accumulation, the present invention, based on the self-constraint of the diffusion model, sends the semantic potential coding into the stacked semantic encoder to obtain semantic prior information, sends the visual context information and the self-attention semantic prior information into the cross-attention layer to obtain the attention signal, and the attention signal and the self-attention semantic prior information are input into the sentence decoder together to obtain the denoised comprehensive semantic information, and the denoised comprehensive semantic information is sent to the linear classifier to predict the sentence structure information, and then the sentence structure information and the denoised comprehensive semantic information are used as enhanced constraint conditions to train the denoising process. Since the visual context information, the comprehensive semantic information, and the semantic condition information, the semantic prior information and the sentence structure information generated by them are added to the denoising process, the convergence of the denoising process can be accelerated while eliminating redundancy. After the previous methods generated the initial sentence sample pool, the optimization strategies adopted, such as self-critical optimization, did not update the sample pool at every step, which would cause the proportion of positive samples in the sample pool to gradually decrease. In the training, our method adopts an optimization strategy of training P candidate DT models on P GPUs for multiple iterations. Each step will generate P candidate next-generation models. Each model obtains its own sample pool after inference, and selects sentences with higher NLP (natural language processing) indicators, such as accuracy, precision, and recall, than the true label as the sampled sentences of the image in the sample pool, thereby improving the sample quality and training efficiency. A separate advantage function is defined for each position as the baseline function, and the reward value of each word in the sentence is corrected for deviation, so as to better balance the contribution of the word to the sentence as a whole.

[0105] In a second aspect, the present invention provides an image description method for use in an application phase, comprising:

[0106] Extracting regional visual features of the image to be described and mapping them into a visual embedding tensor of a preset size; encoding the visual embedding tensor into visual context information of the image to be described;

[0107] Input the visual context information and random noise of the image to be described into the image description model to obtain the corresponding image description information;

[0108] The above-mentioned image description model is constructed by using the method for constructing an image description model provided by the first aspect of the present invention.

[0109] In this process, the external input of the multi-head self-attention layer of the first DT module of the optimal DT model in the image description model is random noise, and the external input of the cross-attention layer of each DT module is the visual context information of the image to be described.

[0110] In the first DT module, random noise is passed through a multi-head self-attention layer to obtain the corresponding self-attention semantic prior information; a cross-attention layer is used to obtain the cross-attention signal of the visual context information of the image to be described and the self-attention semantic prior information, and the sentence decoder is used to decode the self-attention semantic prior information with the cross-attention signal as a constraint to obtain the denoised comprehensive semantic information d of the first DT module. 1 , and input to the multi-head self-attention layer of the next DT module;

[0111] In the i-th DT module, the denoised comprehensive semantic information d output by the i-1-th DT module i-1 The corresponding self-attention semantic prior information is obtained through the multi-head self-attention layer; the cross-attention layer is used to obtain the cross-attention signal of the visual context information of the image to be described and the self-attention semantic prior information, and the sentence decoder is used to decode the self-attention semantic prior information with the cross-attention signal as a constraint to obtain the denoised comprehensive semantic information d of the i-th DT module i ; i=2,3,…,I;

[0112] In the I-th DT module, based on the denoised comprehensive semantic information d I Get image description information of the image to be described.

[0113] The related technical solution is the same as the method for constructing the image description model provided in the first aspect of the present invention, and will not be described in detail here.

[0114] In a third aspect, the present invention provides an electronic device, comprising: a memory and a processor, wherein the memory stores a computer program, and the processor executes the method provided in the first aspect and / or the second aspect of the present invention when executing the computer program.

[0115] The related technical solutions are the same as the method for constructing the image description model provided by the first aspect of the present invention and the image description method provided by the second aspect of the present invention, and will not be described in detail here.

[0116] In a fourth aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method provided in the first aspect and / or the second aspect of the present invention is performed.

[0117] The related technical solutions are the same as the method for constructing the image description model provided by the first aspect of the present invention and the image description method provided by the second aspect of the present invention, and will not be described in detail here.

[0118] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for constructing an image description model, characterized in that: include: S1, extract the regional visual features of each image in the training set and map them into a visual embedding tensor of a preset size; Encode the visual context information into a visual embedding tensor; A cross-modal retrieval model is used to obtain the comprehensive semantic information x0 of the visual embedding tensor; after simulating bit encoding of x0, position encoding and time encoding are further performed to obtain the semantic condition y; a T-step noise operation is performed on x0 to obtain the potential state sequence x1, x2, …, x T ; T is the preset total number of diffusion steps; let the current iteration number epoch = 1; S2, from x1, x2, …, x T In the sequence z1, z2, …, z is sampled L times from the front to the back, and the sequence z1, z2, …, z L ; In the subsequent training process, z t As the input of the DT model at time step t, and z l Before z l-1 is input into the DT model; t = L, L-1, …, 1; l = L, L-1, …, 2; 2≤L≤T; the DT model includes I cascaded DT modules; I is a positive integer; Each of the DT modules includes: a multi-head self-attention layer, a cross-attention layer, a sentence decoder, a linear classifier, and a stacked semantic encoder; S3. Perform L time-step training on the DT model: At each time step t, perform in the i-th DT module: transform y, z t , the comprehensive semantic information after de-noising of the i-1th DT module at time step t The comprehensive semantic information after denoising of the i-th DT module at the previous time step After concatenation to obtain the semantic latent code, it passes through the stacked semantic encoder and multi-head self-attention layer in sequence to obtain the self-attention semantic prior information; the cross-attention layer is used to obtain the cross-attention signal of the visual context information and the self-attention semantic prior information, and the sentence decoder is used to decode the self-attention semantic prior information with the cross-attention signal as a constraint to obtain the comprehensive semantic information after denoising of the i-th DT module at time step t. Then the corresponding image description information is obtained through the linear classifier By minimizing The difference loss from x0, and The difference loss with the corresponding true description information is used to train the i-th DT module; where i = 1, 2, …, I; The image description information includes: sentence structure information; S4, judging whether the epoch reaches the preset number of iterations M, if so, taking the current DT model as the optimal DT model; otherwise, adding one to the epoch and going to S2 to start the next iteration; S5. Remove the semantic encoders of each DT module in the optimal DT model and the linear classifiers in the first I-1 DT modules, and connect the output end of the sentence decoder of the j-th DT module to the multi-head self-attention layer of the j+1-th DT module to construct an image description model including the optimal DT model; j=1,2,…,I-1; during the application of the image description model, the external input of the multi-head self-attention layer of the first DT module therein is random noise, and the external input of the cross-attention layer of each DT module therein is the visual context information of the image to be described.

2. The method for constructing an image description model according to claim 1, characterized in that: The number of the DT models is P; P≥1; S3 includes: performing L time steps of training on the P DT models respectively; The method for constructing the image description model also includes: After the first iteration is completed and before the next iteration begins, the image description information obtained at the last time step of the current P DT models is used to form a prediction sample pool; the image description information with the highest NLP evaluation index is selected from the prediction sample pool as the optimal image description information for the first iteration, and is saved together with its corresponding DT model; After the completion of the rth iteration and before the start of the r+1th iteration, the image description information obtained at the last time step of the current P DT models is combined with the optimal image description information in the previous iteration to form a new prediction sample pool. The image description information with the highest NLP evaluation index is selected from the new prediction sample pool as the optimal image description information in the rth iteration and saved together with its corresponding DT model; r = 2, 3, …, M; The DT model corresponding to the optimal image description information in the Mth iteration is used as the optimal DT model.

3. The method for constructing an image description model according to claim 1, characterized in that: Said The difference loss from x0 includes: Mean squared error loss and / or cross entropy loss between x0 and x0.

4. The method for constructing an image description model according to any one of claims 1 to 3, characterized in that: Also includes: After each iteration is completed and before the next iteration begins, adjust the value of L.

5. The method for constructing an image description model according to any one of claims 1 to 3, characterized in that: Also includes: The following operations are performed between the S4 and the S5: A1, determine whether the NLP evaluation index of the image description information output by the last time step of the optimal DT model is greater than the first preset threshold. If so, go to A2; otherwise, set the current iteration number epoch=1, and go to S2; A2. Strengthen learning of the optimal DT model; A3, judging whether the NLP evaluation index of the image description information outputted by the last time step of the optimal DT model after the completion of reinforcement learning is greater than the second preset threshold value, if so, go to A4; otherwise, go to A2; A4. End the training and go to S5.

6. The method for constructing an image description model according to claim 5, characterized in that: The image description information includes: the number of sentences and sentence structure information of each sentence; the sentence structure information includes: the number of words in the sentence, word attributes and word positions.

7. The method for constructing an image description model according to claim 6, characterized in that: The A2 includes: A plurality of sentence structure information is obtained by sampling the image description information outputted by the current optimal DT model at the last time step, and a plurality of word selections are selected from each sentence structure information to obtain a plurality of word selection combinations; an advantage function is set for each word selection position of each word selection combination, and the optimal DT model is reinforced by learning the result with the goal of maximizing the weighted sum of the advantage functions of each word selection position in each word selection combination; Among them, the kth word position s of the word combination u k The advantage function is: is the NLP evaluation index for the word combination u; the reverse behavior baseline Used to reflect word selection position k The contribution value of the selected words at Expresses expectation; -k Indicates the position s of the word in u except for the word k The combination of words in all other positions except π k Indicates the preset word selection strategy; u' k Indicates the position s of the word in u except for the word k Word selection in positions other than ; To -k with u' k The combined NLP evaluation index is expressed as u' k Replace the word position s in u k NLP evaluation index of the word combination after the word selection.

8. An image description method, characterized in that: For the application phase, including: Extracting regional visual features of the image to be described and mapping them into a visual embedding tensor of a preset size; encoding the visual embedding tensor into visual context information of the image to be described; Input the visual context information and random noise of the image to be described into the image description model to obtain the corresponding image description information; The image description model is constructed by using the image description model construction method described in any one of claims 1 to 7.

9. An electronic device, characterized in that: include: A memory and a processor, wherein the memory stores a computer program, and the processor executes the method according to any one of claims 1 to 8 when executing the computer program.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is performed.

Images