A method for constructing an image description model, an image description method and an apparatus

Abstract

The application discloses a kind of construction methods of image description model, image description method and equipment, belong to image processing technical field;The application designs a kind of DT model including multilayer DT module cascade, through multilayer DT module cascade to guarantee the denoising training process in next layer DT module is constrained by the prediction result of last layer DT module, and on the basis of diffusion model self-constraint, predicted sentence structure information and denoised comprehensive semantic information are used as enhanced constraint condition to train denoising process, it is a kind of diffusion network non-autoregressive image description method based on structure and semantic information conditional constraint, solve the problem that there is no dependency information between words in existing non-autoregressive image description method, generated sentence exists serious word repetition and the problem of missing, can quickly and accurately describe image.In addition, the application also proposes the optimization strategy of multiple iterations of P candidate models, can greatly improve sample quality and training efficiency.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and more particularly relates to a method for constructing an image description model, an image description method and equipment. BACKGROUND

[0002] For the field of computer vision and natural language, the task of image description aims to automatically describe the content of an image with a suitable sentence, which task spans the fields of computer vision and natural language processing and has a wide range of application scenarios, such as helping the visually impaired to understand the environment, reminding the driver of an autonomous vehicle of potential risks, etc.

[0003] Existing image description methods mainly include two categories: autoregressive and non-autoregressive. Autoregressive image description is a method of generating image descriptions through an autoregressive model, in which the generation of each word in the description sentence needs to rely on the information of the previously generated words. Autoregressive model refers to a method of predicting the next part through a part of the serialized data. Autoregressive image description first extracts features from the image, uses the image feature vector to initialize the hidden state of the autoregressive model, and the model generates description words step by step according to the current latent state and the previously generated words. It relies on the step-by-step generation of text, and the generation speed is slow. The non-autoregressive method generates the entire description sequence at one time, and the sentence in the description does not need to use the generated sentence words. It can generate text in parallel, and is faster, but the generation quality is often not as good as the autoregressive method, and it cannot accurately describe the image. SUMMARY

[0004] In view of the above defects or improvement needs of the prior art, the present application provides a method for constructing an image description model, an image description method and equipment, to solve the technical problem that the prior art cannot quickly and accurately describe the image.

[0005] To achieve the above purpose, in a first aspect, the present application provides a method for constructing an image description model, comprising:

[0006] S1, extracting the regional visual features of each image in the training set and mapping them into a visual embedding tensor of a preset size; encoding the visual embedding tensor into visual context information; using a cross-modal retrieval model to obtain comprehensive semantic information x0 of the visual embedding tensor; after simulating the bit encoding of x0, further performing position encoding and time encoding to obtain semantic conditions y; performing T-step noise operation on x0 to obtain the latent state sequence x1, x2, …, x T of x0; T is a preset total diffusion step number; let the current iteration number epoch = 1;

[0007] S2, obtaining the semantic conditions y from x1, x2, …, x TThe sequence is obtained by sampling L times in a forward-to-back order, resulting in the sequence z1, z2, ..., z. L In subsequent training, z t As the input to the DT model at time step t, and z l Prior to z l-1 The input is fed into the DT model; t = L, L-1, ..., 1; l = L, L-1, ..., 2; 2 ≤ L ≤ T; the DT model consists of I cascaded DT modules; I is a positive integer; each DT module includes: a multi-head self-attention layer, a cross-attention layer, a sentence decoder, a linear classifier, and a stacked semantic encoder;

[0008] S3. Train the DT model for L time steps: At each time step t, in the i-th DT module, perform the following: Set y, z t The complete semantic information of the (i-1)th DT module after denoising at time step t The denoised complete semantic information of the i-th DT module in the previous time step After concatenation to obtain the semantic latent code, it is sequentially passed through a stacked semantic encoder and a multi-head self-attention layer to obtain self-attention semantic prior information. A cross-attention layer is used to obtain the cross-attention signal of visual context information and self-attention semantic prior information. The sentence decoder decodes the self-attention semantic prior information with the cross-attention signal as a constraint to obtain the denoised comprehensive semantic information of the i-th DT module at time step t. Then, the corresponding image description information is obtained through a linear classifier. By minimizing The difference loss with x0, and The difference loss between the actual description information and the actual description information is used to train the i-th DT module; where i = 1, 2, ..., I; Image description information includes: sentence structure information;

[0009] S4. Determine if the epoch has reached the preset iteration number M. If so, use the current DT model as the optimal DT model; otherwise, increment the epoch by one and go to S2 to start the next iteration.

[0010] S5, remove the semantic encoder of each DT module in the optimal DT model, and the linear classifier in the first I-1 DT modules, and connect the output end of the sentence decoder of the jth DT module with the multi-head self-attention layer of the j+1th DT module, to construct an image description model comprising the optimal DT model; j=1, 2, …, I-1; in the application process of the image description model, the external input of the multi-head self-attention layer of the first DT module is random noise, and the external input of the cross-attention layer of each DT module is the visual context information of the image to be described.

[0011] Further preferably, the number of the DT models is P; P≥1; and S3 comprises: performing training of L time steps on the P DT models respectively.

[0012] The method for constructing the image description model further comprises:

[0013] After the first iteration is completed and before the next iteration starts, the image description information obtained at the last time step of the current P DT models is used to form a prediction sample pool; the image description information with the highest NLP evaluation index in the prediction sample pool is selected as the optimal image description information under the first iteration, and the corresponding DT model is saved together;

[0014] After the rth iteration is completed and before the r+1th iteration starts, the image description information obtained at the last time step of the current P DT models is used to form a new prediction sample pool together with the optimal image description information under the last iteration; the image description information with the highest NLP evaluation index in the new prediction sample pool is selected as the optimal image description information under the rth iteration, and the corresponding DT model is saved together; r=2, 3, …, M.

[0015] The DT model corresponding to the optimal image description information under the Mth iteration is used as the optimal DT model.

[0016] Further preferably, The difference loss with x0 comprises: The mean square error loss and / or the cross-entropy loss between x0.

[0017] Further preferably, the method for constructing the image description model further comprises: adjusting the value of L after each iteration is completed and before the next iteration starts.

[0018] Further preferably, the image description information comprises: the number of sentences and the sentence structure information of each sentence; and the sentence structure information comprises: the number of words in the sentence, the word attributes and the word positions.

[0019] More preferably, the method for constructing the above image description model further includes the following operations performed between S4 and S5:

[0020] A1. Determine whether the NLP evaluation index of the image description information output by the last time step of the trained optimal DT model is greater than the first preset threshold. If yes, go to A2; otherwise, set the current iteration number epoch = 1 and go to S2.

[0021] A2. Perform reinforcement learning on the optimal DT model;

[0022] A3. Determine whether the NLP evaluation index of the image description information output by the optimal DT model after reinforcement learning is greater than the second preset threshold. If yes, go to A4; otherwise, go to A2.

[0023] A4. End training and proceed to S5.

[0024] More preferably, A2 includes:

[0025] Multiple sentence structure information is obtained by sampling the image description information output by the last time step of the current best DT model, and multiple words are selected from each sentence structure information to obtain multiple word combinations; an advantage function is set for each word position of each word combination, and the optimal DT model is subjected to reinforcement learning with the goal of maximizing the weighted sum of the advantage functions of each word position in each word combination;

[0026] Wherein, the k-th word selection position s of word selection combination u k The dominance function is:

[0027]

[0028] The NLP evaluation metric for word combination u; the inverse is the baseline. Used to reflect the word selection position s k The contribution value of word selection at each location; Indicates expectation; u -k Indicates the position s of the word selection in u. k Combinations of words in all other positions except those mentioned above; π k Indicates a preset word selection strategy; u' k Indicates the position s of the word selection in u. k Word selection in other positions besides; To make u -k with u' k The combined NLP evaluation metrics represent the use of u' k Replace the word selection position s in u kThe NLP evaluation index of the word combination after the word selection of the place.

[0029] In a second aspect, the present application provides an image description method for the application stage, comprising:

[0030] extracting the regional visual features of the image to be described and mapping them into a visual embedding tensor of a preset size; and encoding the visual embedding tensor into visual context information of the image to be described;

[0031] inputting the visual context information of the image to be described and random noise into an image description model to obtain corresponding image description information;

[0032] The image description model is constructed by using the construction method of the image description model provided in the first aspect of the present application.

[0033] In a third aspect, the present application provides an electronic device, comprising a memory and a processor, the memory storing a computer program, and the processor executing the computer program to execute the method provided in the first aspect and / or the second aspect of the present application.

[0034] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and the computer program is executed by a processor to execute the method provided in the first aspect and / or the second aspect of the present application.

[0035] Overall, the above technical solutions conceived by the present application can achieve the following beneficial effects:

[0036] 1. The present application provides a construction method of an image description model, which designs a DT model comprising a cascade of multiple DT modules. In the training process at each time step t, the semantic condition y, the self-condition comprehensive semantic information z t that has not been denoised at time step t, the comprehensive semantic information denoised by the previous DT module at time step t, and the comprehensive semantic information denoised by the current DT module at the previous time step are spliced to obtain semantic latent encoding, which is combined with visual context information to predict image description information and denoised comprehensive semantic information. The image description information includes sentence structure information. The present application ensures that the denoising training process in the next layer of DT modules is constrained by the prediction results of the previous layer of DT modules through the cascade of multiple DT modules. On the basis of the self-constraint of the diffusion model, the predicted sentence structure information and denoised comprehensive semantic information are used as enhanced constraint conditions to train the denoising process. It is a diffusion network non-autoregressive image description method based on sentence structure information and comprehensive semantic information constraint, which solves the problems of lack of dependency information between words, serious word repetition and deletion in generated sentences in existing non-autoregressive image description methods, and can quickly and accurately describe images.

[0037] 2. Furthermore, the image description model construction method provided by this invention proposes a multi-iteration optimization strategy for P candidate models. During the training process, P DT models are trained in each iteration, and the currently obtained P image description information and the optimal image description information obtained in the previous iteration are used together to form a prediction sample pool. The image description information with the highest NLP evaluation index is selected as the optimal image description information in the current iteration, and the corresponding DT model is selected as the optimal DT model. This ensures that the prediction sample pool has a higher proportion of positive samples, suppresses the probability of generating low-quality image description information, and thus greatly improves the sample quality and training efficiency.

[0038] 3. Furthermore, the image description model construction method provided by the present invention takes into account only minimizing The difference loss with x0, and The performance improvement of training the i-th DT module by the difference loss relative to the corresponding real description information has a certain upper limit. Therefore, a reinforcement learning training phase is introduced to further improve the performance of the image description model.

[0039] 4. Furthermore, the image description model construction method provided by this invention defines a separate advantage function for each word position to measure the contribution of a specific position during the reinforcement learning process of the DT model, thereby correcting the bias of the reward value of each word in the sentence, which can better balance the contribution value of words to the whole sentence, and thus better encourage the final generated image description sentence to have more words that conform to the sentence structure features. Attached Figure Description

[0040] Figure 1 A flowchart illustrating the overall process of constructing an image description model according to an embodiment of the present invention;

[0041] Figure 2 This is an overall architecture diagram of image description based on multi-layer DT module cascade provided in an embodiment of the present invention;

[0042] Figure 3 This is the denoising training process of the DT model using cross-entropy loss provided in the embodiments of the present invention;

[0043] Figure 4 This is a schematic diagram of the optimization strategy for multiple iterations of P candidate DT models provided in an embodiment of the present invention.

[0044] Figure 5 The flowchart illustrates the training process of the DT model using joint cross-entropy and reinforcement learning, as provided in this embodiment of the invention. Detailed Implementation

[0045] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application. In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0046] Diffusion Transformer (DT) is a machine learning model for generating data, which is mainly used for image generation, image editing, image insertion drawing, image segmentation, etc. in the field of computer vision. Diffusion model has the advantages of high-quality generation, progressive generation, solid theoretical foundation, strong universality, stable training, etc. Diffusion model mainly includes two steps: 1. Fixed forward diffusion process, that is, gradually adding Gaussian noise to the data until pure noise is obtained; 2. Trainable reverse denoising process, gradually denoising from pure noise until the initial data is obtained. Diffusion model trains the model through the process of gradually adding noise to the data and then denoising, so as to obtain noise-free data.

[0047] DT uses Transformer model in diffusion network, and Transformer model is a deep learning model in natural language processing field, which is widely used in machine translation, text generation, abstract generation and other tasks, and its core is attention mechanism. It adopts Encoder-Decoder architecture and can provide high-quality output in multiple generation tasks. However, there are still problems in using diffusion network for image description, such as that the denoising comprehensive semantic information obtained by using Self-condition is not ideal for predicting sentence structure, etc., the diffusion model denoising process cannot be well guided during training, the training mode in series is difficult to find the optimal candidate DT model in time, and the contribution of words at a specific position to the reward of the entire sentence cannot be measured.

[0048] In view of the above problems, the present application adds comprehensive semantic information and sentence structure information as conditions to constrain the denoising process. Specifically, in the first aspect, the present application provides a method for constructing an image description model, comprising:

[0049] S1, extracting the regional visual features of each image in the training set and mapping them into visual embedding tensors of a preset size; encoding the visual embedding tensors into visual context information; using a cross-modal retrieval model to obtain comprehensive semantic information x0 of the visual embedding tensors; after simulating the bit encoding of x0, further performing position encoding and time encoding to obtain semantic conditions y; performing T-step noise operation on x0 to obtain the latent state sequence x1, x2, …, x T ; T is a preset total diffusion step number; let the current iteration number epoch = 1;

[0050] S2, sampling from x1, x2, …, x T in the front-to-back order L times to obtain a sequence z1, z2, …, z L ; in the subsequent training process, ztis input into the DT model as the input of the DT model at time step tt, and z l is input into the DT model prior to z l-1 ; t = L, L-1, …, 1; l = L, L-1, …, 2; 2≤L≤T; the DT model comprises I cascaded DT modules; I is a positive integer; each DT module comprises: a multi-head self-attention layer, a cross-attention layer, a sentence decoder, a linear classifier, and a stacked semantic encoder;

[0051] In an optional embodiment, the semantic encoder is a Transformer encoder, and the sentence decoder is a Transformer decoder.

[0052] S3, performing L time steps of training on the DT model: at each time step t, in the i-th DT module, performing: concatenating y, z t , the comprehensive semantic information of the i-1-th DT module at time step t after denoising the comprehensive semantic information of the i-th DT module at the previous time step after denoising to obtain semantic latent encoding, then sequentially passing through the stacked semantic encoder and the multi-head self-attention layer to obtain self-attention semantic prior information; obtaining a cross-attention signal of the visual context information and the self-attention semantic prior information by using the cross-attention layer, and decoding the self-attention semantic prior information by using the sentence decoder to obtain the comprehensive semantic information of the i-th DT module at time step t after denoising , and then obtaining the corresponding image description information by using the linear classifier. by minimizing the difference loss of x0and and the difference loss of the corresponding real description information, to train the i-th DT module; wherein, i = 1, 2, …, I;

[0053] In an optional embodiment, the image description information comprises: the number of sentences and the sentence structure information of each sentence; the sentence structure information comprises: the number of words in the sentence, the word attributes, and the word positions.

[0054] S4, judging whether the epoch reaches a preset iteration number M, if yes, taking the current DT model as the optimal DT model; otherwise, increasing the epoch by one, and turning to S2 to start the next iteration;

[0055] In an optional implementation, the preset number of iterations M is 60.

[0056] S5, remove the semantic encoder of each DT module in the optimal DT model, and the linear classifier in the first I-1 DT modules, and connect the output end of the sentence decoder of the jth DT module with the multi-head self-attention layer of the j+1th DT module, to construct an image description model comprising the optimal DT model; j=1, 2, …, I-1; in the application process of the image description model, the external input of the multi-head self-attention layer of the first DT module inside is random noise, and the external input of the cross-attention layer of each DT module inside is the visual context information of the image to be described.

[0057] In an optional implementation, the number of the DT models is P; P≥1; and the S3 comprises: performing training of L time steps on the P DT models respectively.

[0058] The method for constructing the image description model further comprises:

[0059] After the first iteration is completed and before the next iteration starts, the image description information obtained at the last time step of the current P DT models is used to form a prediction sample pool; the image description information with the highest NLP evaluation index in the prediction sample pool is selected as the optimal image description information under the first iteration, and the DT model corresponding to the image description information is saved together;

[0060] After the rth iteration is completed and before the r+1th iteration starts, the image description information obtained at the last time step of the current P DT models is used to form a new prediction sample pool together with the optimal image description information under the last iteration; the image description information with the highest NLP evaluation index in the new prediction sample pool is selected as the optimal image description information under the rth iteration, and the DT model corresponding to the image description information is saved together; r=2, 3, …, M.

[0061] The DT model corresponding to the optimal image description information under the Mth iteration is used as the optimal DT model.

[0062] In an optional implementation, the value of P is 4.

[0063] In an optional implementation, the method for constructing the image description model further comprises: adjusting the value of L after each iteration is completed and before the next iteration starts.

[0064] In an optional implementation, the method for constructing the image description model further comprises: performing the following operation between S4 and S5:

[0065] A1. Determine whether the NLP evaluation index of the image description information output by the last time step of the trained optimal DT model is greater than the first preset threshold. If yes, go to A2; otherwise, set the current iteration number epoch = 1 and go to S2.

[0066] A2. Perform reinforcement learning on the optimal DT model;

[0067] A3. Determine whether the NLP evaluation index of the image description information output by the optimal DT model after reinforcement learning is greater than the second preset threshold. If yes, go to A4; otherwise, go to A2.

[0068] A4. End training and proceed to S5.

[0069] In one optional implementation, the NLP evaluation metric is the CIDEr score, and the first preset threshold and the second preset threshold are set to 130 and 133, respectively.

[0070] In one alternative implementation,

[0071] The above A2 includes: obtaining multiple sentence structure information from the image description information output by the last time step of the current best DT model, and selecting multiple words from each sentence structure information to obtain multiple word combinations; setting an advantage function for each word position in each word combination, and performing reinforcement learning on the best DT model with the goal of maximizing the weighted sum of the advantage functions of each word position in each word combination;

[0072] Wherein, the k-th word selection position s of word selection combination u k The dominance function is:

[0073]

[0074] The NLP evaluation metric for word combination u; the inverse is the baseline. Used to reflect the word selection position s k The contribution value of word selection at each location; Indicates expectation; u -k Indicates the position s of the word selection in u. k Combinations of words in all other positions except those mentioned above; π k Indicates a preset word selection strategy; u' k Indicates the position s of the word selection in u. k Word selection in other positions besides; To make u -k with u' k The combined NLP evaluation metrics represent the use of u' k Replace the word selection position s in uk NLP evaluation metrics for word selection combinations after word selection.

[0075] It should be noted that the NLP evaluation metrics mentioned above can be: CIDEr score, BLEU score, METOR score, ROUGE score, accuracy, precision, recall, etc. In this invention, the CIDEr score is preferred.

[0076] To further illustrate the method for constructing the image description model provided by this invention, a specific embodiment is described in detail below:

[0077] like Figure 1 The diagram shows the overall flowchart of the image description model construction method provided in this embodiment. The image description model construction method provided in this embodiment mainly includes four key technologies: denoising based on comprehensive semantic information in cascaded DT, cross-entropy training based on sentence structure information and comprehensive semantic information, a multi-iteration optimization strategy of parallel training with P GPUs to obtain P candidate DT models, and reinforcement learning that defines a separate advantage function as a baseline function for each position. The key technical points are described as follows:

[0078] (1) Denoising of comprehensive semantic information in cascaded DT: First, the DT module is optimized. In the past, the DT model training process only used the noisy comprehensive semantic information of the current DT layer at the current time step. This denoising quality of using only the noisy comprehensive semantic information of a certain DT module is poor. Therefore, we propose stacking multiple DT modules and generating high-fidelity comprehensive semantic information through model training with multiple DT layers and multiple time steps. The method proposed in this invention uses the semantic structure information predicted by the previous time step of the current DT layer and the current time step of the previous DT layer in the denoising training process of the DT module, so that the final generated image description is more in line with the natural sentence structure.

[0079] DT training mainly includes two steps: noise addition and noise removal. The noise addition process involves adding noise for T steps to the initial comprehensive semantic information x0, resulting in noisy comprehensive semantic information x1, x2, ..., x... T Typically, T = 50. Denoising is the goal of DT module training, in x1, x2, ..., x... T Let L noisy, complete semantic information samples be randomly selected from the sample, denoted as z1, z2, ..., z. L Let x0 be denoised as z0. Denoising is performed on them through DT training until z0 is recovered. For the i-th layer DT module, the semantic condition y and the undenoised full semantic information (word token) z of the i-th layer DT module at time step t are used. t The complete semantic information of the (i-1)th DT module after denoising at time step t The denoised complete semantic information of the i-th DT module in the previous time step The semantic latent encoding is constructed by concatenating semantic latent codes. Since all four components have a dimension of bs×20×512, the semantic latent encoding has a dimension of bs×80×512. Semantic latent encoding obtains semantic prior information through stacked Transformer-based semantic encoders, and then passes it through a multi-head self-attention layer to obtain self-attention semantic prior information. The visual context information and self-attention semantic prior information are then fed into a cross-attention layer to output an attention signal. The self-attention semantic prior information and the attention signal are input into a sentence decoder, and the output of the sentence decoder is the noisy, comprehensive semantic information z at the current time step t. t Denoising complete semantic information Use again Predicting sentence structure information can provide stronger constraints for training the diffusion model's denoising process: comprehensive semantic information and sentence structure information after denoising.

[0080] (2) Cross-entropy training based on sentence structure information and comprehensive semantic information: The cross-entropy loss training stage can be regarded as coarse-grained model optimization. The purpose of DT training is to use the noisy comprehensive semantic information z t Given the condition, z is obtained through DT training. t Denoising complete semantic information d t That is, the approximation value of z0. t The encoding is fixed during the noise addition process and does not require additional information; this process is called self-conditioning in the denoising process. Since self-conditioning-based denoising lacks strong constraints, error accumulation occurs during training. Therefore, the model in this invention, based on self-conditioning, uses the semantic condition y output by the semantic information module of the i-th layer DT module at the current time step, and the complete semantic information z of the un-denoised Self-condition of the i-th layer DT module at the current time step. t The (i-1)th layer DT module at the current time step Denoising complete semantic information The i-th layer DT module at the previous time step Comprehensive semantic information after denoising Generate semantic latent encoding (i.e., about the denoised full semantic information) and The potential state of x0 The semantic latent encoding is obtained by stacking the semantic encoder, obtaining semantic prior information, and after passing through the multi-head self-attention layer, the self-attention semantic prior information and the attention signal in the visual encoding module are input into the sentence decoder, that is, the self-attention semantic prior information is constrained by the attention signal to obtain the denoised comprehensive semantic information is input into a linear classifier to predict the sentence structure information b predicted by the i-th DT module at the current time step j .

[0081] The loss function is used to measure the difference between the predicted result of the model and the actual result. In actual training, the mean square error between the comprehensive semantic information x denoised at time step t and the initial comprehensive semantic information z0 The comprehensive semantic information x denoised at time step t and the initial comprehensive semantic information z0 The semantic structure information loss function of the current time step t and the previous t steps in the denoising process is summed to obtain the total loss function Wherein:

[0082]

[0083] Wherein, denotes the comprehensive semantic information denoised at time step t of the i-th DT module; J is the number of sentences in the image description information obtained at time step t; Q represents the number of words in the j-th sentence at time step t; c jq is the attribute of the q-th word in the j-th sentence at time step t; b j is the sentence structure information of the j-th sentence at time step t; denotes the CIDEr score function; p θ (·) represents the noise probability distribution in the denoising process. The smaller the , the closer the denoised to z0 (i.e. x0), and the easier it is to predict the sentence structure by inputting the noise z t during prediction. is used to measure the difference loss between and the corresponding real description information, The smaller the

[0084] (3) Each word position defines a separate advantage function as reinforcement learning of baseline function: In reinforcement learning, the baseline function is a measure of the average quality of the sampled sentence, and the method takes the expected reward of the sample as the baseline. The sentence is encouraged to be generated if it is higher than the expected reward of the sample. It is used to reduce the variance when estimating the reward function, to correct the bias of the reward value of the sentence, and to improve the stability and learning efficiency of the strategy. The reward mechanism based on greedy decoding in the past makes each word position accept the same team reward, which cannot measure the contribution of the word at a specific position to the overall sentence reward. Such baseline function cannot reflect the value of semantic structure information, so in this embodiment, a separate advantage function A k (s k ,u) is defined for each word position to measure the contribution of the word at a specific position to the semantic structure information, where s k represents the position of the kth word, and u represents the combination of the selected words. It reflects the contribution value of the word at position s by subtracting the inverse behavior baseline R k (s k ,u -k ) from the common team reward k . Wherein, u -k represents the combination of selected words at all positions except position s k . In this way, the change of A k (s k ,u) actually reflects the contribution value of the word at position s k . Formally, when position s k traverses all possible selected words while keeping the combination of selected words at other positions unchanged, the inverse behavior baseline R u' k represents the combination of selected words at other positions, π k represents the selected word strategy, represents the CIDEr score function. In order to simplify the calculation, we actually only consider the top K selected words to approximate the expected calculation in the above equation. The inverse behavior baseline is a measure of whether the selected word violates the semantic structure information. If it is judged that the selected word at position s k meets the semantic structure information B given in the denoising training stage, the advantage function A k (s k ,u) expands by a certain ratio, otherwise it shrinks by a certain ratio. Through the effect of weighting the advantage function, the reward of the word position that meets the structure prediction information in the sentence can be improved, so as to encourage the generation of words that meet the structure characteristics.

[0085] (4) Optimization strategy of multiple iterations of P candidate DT models: The traditional self-critical strategy only establishes a sample pool with relatively high quality, and through multiple rounds of epoch optimization training, the model prediction set is as close as possible to the traditional strategy of the sample pool. The obvious disadvantages of the strategy are: first, with the increase of epoch, the proportion of positive samples in the sample pool gradually decreases, and the loss gradient of the reward function quickly tends to zero. Secondly, the oscillation process of the model can optimize part of it, while weakening another part. In the strategy of completely abandoning the overall performance index without breakthrough, these fluctuations are meaningless. In the optimization strategy of the embodiment, the model generates P candidate next-generation models when training optimization to the next epoch. Each candidate DT model is different, and the inference process obtains its own predicted sample pool. For the same image, from the second iteration, there are P+1 predicted sample sentences, from which the sentence with higher NLP indicators such as accuracy, precision, and recall compared to the true label is selected as the sampling sentence in the sample pool, so that the target sample pool maintains a high proportion of positive sample quantity, thereby suppressing the probability of generating low-quality sentences, and thus selecting the optimal DT model.

[0086] The above process will be described in detail below in conjunction with detailed diagrams:

[0087] As Figure 2 shown is the overall architecture diagram of the natural language image description model training based on the structure and semantic information diffusion transformation in one embodiment of the application. The purpose of the application is to utilize comprehensive semantic information and semantic structure information to constrain the denoising training during model training, and to use P candidate DT model multiple iteration, advantage function and other optimization strategies to improve the quality of the predicted sample pool.

[0088] In implementation, the steps that should be included are: generating comprehensive semantic information, generating semantic latent encoding, generating cross-attention signal, predicting sentence structure information, diffusion model cross-entropy loss training based on sentence structure information and comprehensive semantic information, optimization strategy of multiple iterations of P candidate DT models, and defining a separate advantage function for each word position as the reinforcement learning of the baseline function.

[0089] I. Generating comprehensive semantic information

[0090] As Figure 2As shown, the I-layer DT generating semantic latent encoding and sentence structure information all include: a visual semantic interaction module and a sentence structure prediction module. The visual semantic interaction module mainly includes: a cross-attention layer; the sentence structure prediction module mainly includes: a stacked semantic encoder, a multi-head self-attention layer, a sentence decoder and a linear classifier. The region visual feature of the original image is extracted by using a target detection model such as R-CNN, Faster-RCNN, YOLO, etc., and the visual feature of different regions in the image is obtained, mainly including the visual feature of the foreground region; the region visual feature is first mapped to a fixed size visual embedding vector through a fully connected layer, and the dimension is from bs×50×2048 to bs×50×512, wherein the training set is divided into batches to train the DT model, and bs is the size of the batch; the visual embedding vector is then encoded to obtain visual context information with a fixed dimension, which is used as the input of the cross-attention layer in the visual semantic interaction module. At the same time, by using a cross-modal retrieval large model, the visual embedding tensor is output as comprehensive semantic information x0 by analog bit encoding. The comprehensive semantic information includes nouns and derived verbs that may be contained in the image.

[0091] II. Generating semantic latent encoding

[0092] By converting the word information of comprehensive semantic information x0 into binary, analog bit encoding of comprehensive semantic information x0 is realized, and after obtaining the tensor after analog bit encoding, position encoding and time encoding are further performed to obtain semantic condition y. Specifically, analog bit encoding is to set a serial number for each word, for example, the serial number of the word 'woman' in the word library is 3, and the analog bit encoding is a 14-bit information, of which the high 12 bits are -1 and the low two bits are 1, that is, the comprehensive semantic information is output as a dimension of bs×20×14 embedding tensor through analog bit encoding. After the tensor after analog bit encoding passes through a double embedding layer including full connection, position encoding and time encoding, a semantic condition y of a dimension of bs×20×512 embedding tensor is output. In the case of I-layer cascaded DT module (I takes a value of 3 in this embodiment), for the i-th layer DT module, the semantic condition y, the self-condition latent state z t , the comprehensive semantic information denoised by the i-1-th layer DT module at the current time step t the comprehensive semantic information denoised by the i-th layer DT module at the last time step The semantic latent encoding with a dimension of bs x 80 x 512 is generated by concatenation. The semantic latent encoding is obtained by a stacked Transformer-based semantic encoder in the sentence structure prediction module, and has a dimension of bs x 80 x 512 semantic prior information. The first 20 high-reliability dimension features related to the word token are sampled from the semantic prior information, and the semantic prior information with a dimension of bs x 20 x 512 is obtained. Then, the multi-head self-attention layer is used to obtain the self-attention semantic prior information.

[0093] III. Generating cross-attention signals

[0094] The region visual features of the original image are obtained by Faster-RCNN, and the region visual features are mapped into a fixed-size visual embedding tensor. Then, the visual embedding tensor is encoded into visual context information. The visual context information and the self-attention semantic prior information are input into the cross-attention layer to obtain the cross-attention signal.

[0095] IV. Predicting sentence structure information

[0096] As shown in Figure 2 , the sentence structure prediction module includes a stacked Transformer-based semantic encoder, a multi-head self-attention layer, a Transformer-based sentence decoder, and two linear classifiers. The cross-attention signal and the self-attention semantic prior information are input into the Transformer-based sentence decoder to obtain the current time step z t Denoised comprehensive semantic information The is input into the two linear classifiers to predict the sentence structure information in the image description information. The sentence is divided into a small syntax structure selection parser. The predicted sentence structure information and the real sentence structure information label (ground truth) are parsed into a tree structure, and B = {b1, b2, …, b J} records the sentence structure information of the J sentences, where b j = {c j1 ,c j2 ,…,c jQ}, Q represents the number of words in the jth sentence, and c jq represents the attribute of the qth word in the jth sentence. The sentence structure information is used as a label to train the classification network. The predicted sentence structure information and the comprehensive semantic information together form a semantic conditional signal with strong constraints to guide the denoising training of the diffusion network.

[0097] V. Diffusion model cross-entropy loss training based on sentence structure information and comprehensive semantic information

[0098] The training of the DT module mainly includes two steps: a noise-adding process and a noise-reducing process. The noise-adding process involves adding noise to an initial comprehensive semantic information x0 with dimensions bs×20×512 in T steps, resulting in noisy comprehensive semantic information x0, x1, ..., x... T Typically, T = 50, transforming it from a tensor conforming to a certain characteristic distribution into Gaussian noise conforming to a normal distribution. For example... Figure 3 As shown, denoising is the goal of DT module training. In each batch within the same epoch, at x1, x2, ..., x T Let L noisy, complete semantic information samples be randomly selected from the sample, denoted as z1, z2, ..., z. L Let x0 be denoted as z0. Denoising is performed on these values ​​using DT training until z0 is recovered. The noisy, comprehensive semantic information z0 from the previous time step is then used. t+1 The denoising result obtained after DT training is denoised as the complete semantic information d after denoising in the previous time step. t+1 The goal of DT training is to obtain the full semantic information z after adding noise. t As a condition, train DT to obtain the denoised full semantic information d. t z t This is a fixed encoding during the noise addition process, requiring no additional information; such a noise reduction process is defined as having [a certain characteristic]. The initial learning process uses a Markov chain with Gaussian transitions, where I represents the noise variance. However, this denoising process, relying solely on self-constraints for training, suffers from error accumulation, resulting in low-quality generation. Therefore, the obtained comprehensive semantic information and sentence structure information are used as enhancement conditions to constrain the denoising process. Furthermore, to address the error accumulation problem, the semantic condition y output by the semantic information module at time step t of the i-th layer DT module, and the noisy self-condition comprehensive semantic information z of the current time step of the i-th layer DT module are used. t Comprehensive semantic information of the current time step denoising of the (i-1)th layer DT module The complete semantic information of the previous time step denoising of the i-th layer DT module The loss function is calculated by fusing different components. Therefore, the joint distribution of the denoising training process of the DT module in the i-th layer is from self-constrained... This transforms into a comprehensive constraint of semantic and sentence structure information, including noise addition and denoising. In this training phase, the sum of mean squared error, cross-entropy loss function, and sentence structure information loss function is used as the standard to judge the performance of the model training.

[0099] VI. Optimization strategy for multiple iterations of P candidate DT models

[0100] In the diffusion model denoising training phase, the gradient of the expected reward is optimized by sampling sentences from the model, which are considered short-term goals in the training process, and the quality of the samples directly determines the optimization effect. The traditional self-critical optimization strategy does not update the sample pool at every step, and its strategy is simply to establish a relatively high-quality sample pool first, and through multiple rounds of optimization training, the model's prediction set is as close to the sample pool as possible. The traditional optimization strategy will cause the proportion of positive samples in the sample pool to gradually decrease, and the reward function loss gradient will quickly tend to zero. As shown in Figure 4 , the embodiment proposes an optimization strategy for multiple iterations of P candidate DT models trained on P GPUs. At each step, P candidate DT models are generated. Since the sampling positions of comprehensive semantic information on each GPU are likely to be different, the candidate DT models differ from each other. Each model obtains its own predicted sample pool through the inference process. For the same image, there are P+1 predicted sample sentences including the original sample pool, and then the synchronization optimization is performed. From the P+1 sample sentences, the sentence with the highest NLP index compared to the true label is selected as the sampling sentence in the sample pool for the image, to determine the optimal DT model.

[0101] Seven, define a separate advantage function for each word position as the reinforcement learning of the baseline function

[0102] In the cross-entropy loss training phase, the model uses the loss function of semantic structure information to supervise the denoising process, which makes the sentences generated by the model more structured, but the performance improvement of cross-entropy training has a certain upper limit. As shown in Figure 5 , therefore, the model training is divided into two stages: the cross-entropy training stage and the reinforcement learning training stage. On the other hand, since nouns have the greatest contribution to the sentence-level reward, if the value of the semantic structure information cannot be reflected in the gradient loss, the optimization training process will gradually destroy the original structure features, and thus the semantic structure information supervision in the cross-entropy training stage will be wasted. Here, a typical case of sentence structure destruction is that reinforcement learning optimization training tends to make the model generate sentences composed of a large number of nouns and a small number of conjunctions, which encroaches on the space of verbs. However, the traditional sentence-level reward based on greedy decoding allows each word position to receive the same team reward, and cannot measure the contribution of the word at a specific position to the team reward. In order to solve the above problems, the embodiment defines a baseline function that can reflect the value of the structure information to some extent, which can specify a separate advantage function for each position, which reflects the contribution value of the word at position s k . Further, the weighted advantage function of A k reflects the contribution value of the word at position s t .The role of the sentence structure information is to improve the position reward in the sentence structure information, so as to encourage the generated image description sentence to have more words meeting the sentence structure characteristics.

[0103] Therefore, the previous method is insufficient to generate high-quality image description information by only using the DT module for image description. The present application provides an image natural language description method based on the sentence structure information and the comprehensive semantic information generated based on the visual context information and the comprehensive semantic information, to constrain the diffusion transformation. The semantic condition y, the i-th layer DT module, the self-condition comprehensive semantic information (word token) z t , the i-1-th layer DT module, the denoised comprehensive semantic information at time step t , the i-th layer DT module, the denoised comprehensive semantic information at the previous time step are spliced to generate semantic latent encoding. Since the semantic latent encoding contains the prior information of the denoised and , the semantic latent encoding generates semantic prior information after passing through the stacked semantic encoder, and then obtains self-attention semantic prior information through the multi-head self-attention layer; the attention signal is obtained by using the self-attention semantic prior information and the visual context information, and the attention signal and the self-attention semantic prior information are input into the sentence decoder to obtain the denoised comprehensive semantic information z t ( not z t z before being added with noise t-1 , but an approximation of z0), and the image description information is predicted by using , the denoising process is constrained by using the sentence structure information and the comprehensive semantic information. On this basis, a multiple iteration optimization strategy of P candidate DT models is used to replace the self-critical training optimization strategy, and a new sentence-based baseline reward is proposed to ensure that the value of the semantic structure information can still be reflected in the fine-grained tuning training process. In practice, the present application finds that adding comprehensive semantic information and sentence structure information to train the denoising process can accelerate the convergence speed. The multiple iteration optimization strategy of P candidate DT models trained by P GPUs at the same time can ensure that the predicted sentence sample pool has a higher proportion of positive sample quantity, thereby greatly improving the sample quality and training efficiency. In addition, the proposed sample-based expected reward baseline function can better balance the contribution value of the word position to the word itself and the whole sentence.

[0104] ​The present application aims to solve the problem of lack of dependency information between words in non-autoregressive image description methods and the existence of serious word repetition and omission in generated sentences by improving the training process of diffusion networks, so that higher quality image descriptions can be generated. Since there is error accumulation in the self-adjusting and optimizing denoising process relying only on the denoising algorithm in the DT module, the present application sends the semantic latent code into the stacked semantic encoder to obtain semantic prior information on the basis of self-constraint of the diffusion model, inputs the visual context information and the self-attention semantic prior information into the cross-attention layer to obtain the attention signal, inputs the attention signal and the self-attention semantic prior information into the sentence decoder to obtain the denoised comprehensive semantic information, inputs the denoised comprehensive semantic information into the linear classifier to predict the sentence structure information, and then uses the sentence structure information and the denoised comprehensive semantic information as enhanced constraint conditions to train the denoising process. Since the visual context information, comprehensive semantic information, semantic conditional information, semantic prior information and sentence structure information generated by them are added to the denoising process, the convergence of the denoising process can be accelerated while eliminating redundancy. After generating the initial sentence sample pool in the previous method, the optimization strategies such as self-critical optimization are not updated at every step, which will cause the proportion of positive samples in the sample pool to gradually decrease. In the training process, the present application adopts an optimization strategy of simultaneously training P candidate DT models for multiple iterations using P GPUs, generates P candidate next-generation models at each step, selects the sentences with higher NLP (natural language processing) indicators such as accuracy, precision and recall compared with the true label as the sampling sentences of the image in the sample pool from the respective sample pools obtained by inference of each model, thereby improving the sample quality and training efficiency. A separate advantage function is defined for each position as the baseline function to correct the deviation of the reward value of each word in the sentence, thereby better balancing the contribution value of the word to the whole sentence.

[0105] In a second aspect, the present application provides an image description method for the application stage, comprising:

[0106] extracting the regional visual features of the image to be described and mapping them into a visual embedding tensor of a preset size; and encoding the visual embedding tensor into visual context information of the image to be described;

[0107] inputting the visual context information of the image to be described and random noise into an image description model to obtain corresponding image description information;

[0108] The image description model is constructed by using the construction method of the image description model provided in the first aspect of the present application.

[0109] In the process, the external input of the multi-head self-attention layer of the first DT module of the optimal DT model in the image description model is random noise, and the external input of the cross-attention layer of each DT module is the visual context information of the image to be described.

[0110] In the first DT module, the random noise obtains corresponding self-attention semantic prior information through the multi-head self-attention layer; the cross-attention layer obtains a cross-attention signal of the visual context information of the image to be described and the self-attention semantic prior information, and decodes the self-attention semantic prior information by taking the cross-attention signal as a constraint through the sentence decoder to obtain the denoised comprehensive semantic information d 1 of the first DT module, and input into the multi-head self-attention layer of the next DT module.

[0111] In the i-th DT module, the denoised comprehensive semantic information d i-1 of the (i-1)-th DT module is input into the multi-head self-attention layer to obtain corresponding self-attention semantic prior information; the cross-attention layer obtains a cross-attention signal of the visual context information of the image to be described and the self-attention semantic prior information, and decodes the self-attention semantic prior information by taking the cross-attention signal as a constraint through the sentence decoder to obtain the denoised comprehensive semantic information d i of the i-th DT module; i=2, 3, …, I.

[0112] In the I-th DT module, the image description information of the image to be described is obtained based on the denoised comprehensive semantic information d I of the I-th DT module through a linear classifier.

[0113] The related technical solutions are the same as the construction method of the image description model provided in the first aspect of the present application, and will not be repeated here.

[0114] In a third aspect, the present application provides an electronic device, comprising a memory and a processor, the memory stores a computer program, and the processor executes the computer program to execute the method provided in the first aspect and / or the second aspect of the present application.

[0115] The related technical solutions are the same as the construction method of the image description model provided in the first aspect of the present application and the image description method provided in the second aspect of the present application, and will not be repeated here.

[0116] In a fourth aspect, the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to execute the method provided in the first aspect and / or the second aspect of the present application.

[0117] The related technical solutions are the same as the construction method of the image description model provided by the first aspect of the present application and the image description method provided by the second aspect of the present application, which are not described herein.

[0118] Those skilled in the art can understand that the above description is only the preferred embodiment of the present application, and is not used to limit the present application, and any modification, equivalent replacement and improvement made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A method for constructing an image description model, characterized in that, include: S1. Extract the regional visual features of each image in the training set and map them to a visual embedding tensor of a preset size. Encode the visual embedding tensor into visual context information; A cross-modal retrieval model is used to obtain the comprehensive semantic information x0 of the visual embedding tensor; after simulated bit encoding of x0, positional encoding and temporal encoding are further performed to obtain the semantic condition y; T-step noise addition operation is performed on x0 to obtain the latent state sequence x1, x2, ..., x0. T T is the preset total number of diffusion steps; let the current iteration number epoch = 1; S2, from x1, x2, ..., x T The sequence is obtained by sampling L times in a forward-to-back order, resulting in the sequence z1, z2, ..., z. L In subsequent training, z t As the input to the DT model at time step t, and z l Prior to z l-1 The inputs are fed into the DT model; t = L, L-1, ..., 1; l = L, L-1, ..., 2; 2≤L≤T; The DT model consists of I cascaded DT modules; I is a positive integer; Each of the DT modules includes: a multi-head self-attention layer, a cross-attention layer, a sentence decoder, a linear classifier, and a stacked semantic encoder; S3. Train the DT model for L time steps: At each time step t, in the i-th DT module, perform the following: Set y, z t The complete semantic information of the (i-1)th DT module after denoising at time step t The denoised complete semantic information of the i-th DT module in the previous time step After concatenation to obtain the semantic latent code, it is sequentially passed through a stacked semantic encoder and a multi-head self-attention layer to obtain self-attention semantic prior information. A cross-attention layer is used to obtain the cross-attention signal of visual context information and self-attention semantic prior information. The sentence decoder decodes the self-attention semantic prior information with the cross-attention signal as a constraint to obtain the denoised comprehensive semantic information of the i-th DT module at time step t. Then, the corresponding image description information is obtained through a linear classifier. By minimizing The difference loss with x0, and The difference loss between the actual description information and the actual description information is used to train the i-th DT module; where i = 1, 2, ..., I; The image description information includes: sentence structure information; S4. Determine whether the epoch has reached the preset iteration number M. If so, take the current DT model as the optimal DT model; otherwise, increment the epoch by one and go back to S2 to start the next iteration. S5. Remove the semantic encoders of each DT module in the optimal DT model, as well as the linear classifiers in the first I-1 DT modules, and connect the output of the sentence decoder of the j-th DT module to the multi-head self-attention layer of the j+1-th DT module to construct an image description model including the optimal DT model; j = 1, 2, ..., I-1; during the application of the image description model, the external input of the multi-head self-attention layer of the first DT module is random noise, and the external input of the cross-attention layer of each DT module is the visual context information of the image to be described.

2. The method for constructing an image description model according to claim 1, characterized in that, The number of DT models is P; P≥1; S3 includes: training each of the P DT models for L time steps; The method for constructing the image description model also includes: After the first iteration is completed and before the next iteration begins, the image description information obtained at the last time step of the current P DT models is used to form a prediction sample pool. The image description information with the highest NLP evaluation index is selected from the prediction sample pool as the optimal image description information for the first iteration and is saved together with its corresponding DT model. After the r-th iteration is completed and before the (r+1)-th iteration begins, the image description information obtained at the last time step of the current P DT models is combined with the optimal image description information from the previous iteration to form a new prediction sample pool. The image description information with the highest NLP evaluation index is selected from the new prediction sample pool as the optimal image description information for the r-th iteration and is saved together with its corresponding DT model; r = 2, 3, ..., M. The DT model corresponding to the optimal image description information under the Mth iteration is taken as the optimal DT model.

3. The method for constructing an image description model according to claim 1, characterized in that, The The difference loss from x0 includes: Mean squared error loss and / or cross-entropy loss between x0 and x0.

4. The method for constructing an image description model according to any one of claims 1-3, characterized in that, Also includes: After each iteration is completed and before the next iteration begins, adjust the value of L.

5. The method for constructing an image description model according to any one of claims 1-3, characterized in that, Also includes: The following operations are performed between S4 and S5: A1. Determine whether the NLP evaluation index of the image description information output by the last time step of the trained optimal DT model is greater than the first preset threshold. If yes, proceed to A2; otherwise, set the current iteration number epoch = 1 and proceed to S2. A2. Perform reinforcement learning on the optimal DT model; A3. Determine whether the NLP evaluation index of the image description information output by the optimal DT model after reinforcement learning is greater than the second preset threshold. If yes, proceed to A4; otherwise, proceed to A2. A4. End training and proceed to S5.

6. The method for constructing an image description model according to claim 5, characterized in that, The image description information includes: the number of sentences and the sentence structure information of each sentence; the sentence structure information includes: the number of words in the sentence, word attributes, and word positions.

7. The method for constructing an image description model according to claim 6, characterized in that, A2 includes: Multiple sentence structure information is obtained by sampling the image description information output by the last time step of the current best DT model, and multiple words are selected from each sentence structure information to obtain multiple word combinations; an advantage function is set for each word position of each word combination, and the optimal DT model is subjected to reinforcement learning with the goal of maximizing the weighted sum of the advantage functions of each word position in each word combination; Wherein, the k-th word selection position s of word selection combination u k The dominance function is: The NLP evaluation metric for word combination u; the inverse is the baseline. Used to reflect the word selection position s k The contribution value of word selection at each location; Indicates expectation; u -k Indicates the position s of the word selection in u. k Combinations of words in all other positions except those mentioned above; π k Indicates a preset word selection strategy; u' k Indicates the position s of the word selection in u. k Word selection in other positions besides; To make u -k with u' k The combined NLP evaluation metrics represent the use of u' k Replace the word selection position s in u k NLP evaluation metrics for word selection combinations after word selection.

8. An image description method, characterized in that, For the application phase, including: Extract the visual features of the regions in the image to be described and map them into a visual embedding tensor of a preset size; encode the visual embedding tensor into the visual context information of the image to be described; The visual context information and random noise of the image to be described are input into the image description model to obtain the corresponding image description information; The image description model is constructed using the image description model construction method described in any one of claims 1-7.

9. An electronic device, characterized in that, include: A memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the method described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, It stores a computer program, which, when executed by a processor, performs the method described in any one of claims 1-8.

Images