Image description text processing method and device, electronic equipment, storage medium and computer program product

Abstract

The invention relates to an image description text processing method and device, electronic equipment, a storage medium and a computer program product. The processing method comprises the steps of performing word segmentation processing on an initial description text of an original image to obtain a word segmentation result of the initial description text; inputting a word segmentation result into a text encoder to obtain a discrete vector and a continuous vector; performing noise adding and denoising processing of a preset number of steps on the continuous vector, updating a discrete vector of the current step by using the predicted noise of the current step and the random noise added in the current step in each step in the preset number of steps, and taking the updated discrete vector as a discrete vector of the next step, taking a continuous vector corresponding to the discrete vector of the next step as a continuous vector of the next step; and determining a final description text of the original image based on the updated discrete vector of each step in response to the completion of the noise adding and denoising processing of the preset step number.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing, and in particular to a method and device for processing image description text, an electronic device, a storage medium, and a computer program product. Background Art

[0002] Prompt engineering is a technique that improves the performance of generative models by optimizing input instructions. With the rapid development of large models, such as large language models and text-to-graph models, prompt engineering has become a key tool for unlocking their potential. Whether generating high-quality natural language text or creating realistic images, prompt engineering plays a crucial role.

[0003] Currently, prompting word engineering methods can be roughly divided into two categories: hard prompting and soft prompting. However, the design of hard prompting relies on manual experience and is difficult to optimize. Soft prompting, on the other hand, relies on the embedding space of a specific model and is generally difficult to transfer between models of different architectures. Summary of the Invention

[0004] The present disclosure provides a method and device for processing image description text, an electronic device, a storage medium, and a computer program product, to at least solve the problems in the related art of high difficulty in optimizing prompt words and the lack of universality between models of different architectures.

[0005] According to a first aspect of an embodiment of the present disclosure, a method for processing image description text is provided, comprising: performing word segmentation processing on an initial description text of an original image to obtain a word segmentation result of the initial description text; inputting the word segmentation result into a text encoder to obtain a discrete vector and a continuous vector, wherein the discrete vector is a discrete embedding vector output by a word unit embedding layer of the text encoder, and the continuous vector is an embedding vector finally output by the text encoder; performing noise addition and denoising processing on the continuous vector for a predetermined number of steps, wherein, at each step in the predetermined number of steps, the discrete vector of the current step is updated using the predicted noise of the current step and the random noise added to the current step, and the updated discrete vector is used as the discrete vector of the next step, and the continuous vector corresponding to the discrete vector of the next step is used as the continuous vector of the next step; in response to the completion of the noise addition and denoising processing for the predetermined number of steps, the final description text of the original image is determined based on the updated discrete vector of each step.

[0006] Optionally, the continuous vector is subjected to noise addition and denoising processing for a predetermined number of steps, including: at each step in the predetermined number of steps, performing the following processing: performing noise addition processing on the continuous vector of the current step based on the random noise of the current step to obtain the noise data of the current step, wherein, when the current step is the first step of the predetermined number of steps, the continuous vector of the current step is a continuous vector; inputting the noise data of the current step and the continuous vector of the current step into the diffusion model to obtain the predicted noise of the current step; using the predicted noise of the current step, the random noise of the current step, the discrete vector of the current step and the original image, updating the discrete vector of the current step and using the updated discrete vector as the discrete vector of the next step; inputting the discrete vector of the next step into the text encoder to obtain the continuous vector of the next step.

[0007] Optionally, the discrete vector of the current step is updated using the predicted noise of the current step, the random noise of the current step, the discrete vector of the current step and the original image, including: determining the noise prediction loss based on the predicted noise of the current step and the random noise of the current step; determining the semantic loss based on the discrete vector of the current step and the original image; determining the total loss based on the noise prediction loss and the semantic loss; and updating the discrete vector of the current step by minimizing the total loss.

[0008] Optionally, after updating the discrete vector of the current step using the predicted noise of the current step and the random noise added to the current step, the method includes: for each word vector in the updated discrete vector, obtaining the distance between the current word vector and each word vector in the predetermined vocabulary; determining the word vector with the smallest distance in the predetermined vocabulary as the final vector of the current word vector; and determining the final updated discrete vector of the current step based on all final vectors.

[0009] Optionally, based on the discrete vectors updated in each step, the final description text of the original image is determined, including: sampling part of the discrete vectors from the discrete vectors updated in each step; for each discrete vector in the partial discrete vectors, using the continuous vector and diffusion model corresponding to the current discrete vector to perform denoising processing on any random noise for a predetermined number of steps to obtain an estimated image corresponding to the original image; based on the discrete vector corresponding to the estimated image with the highest image quality, the final description text of the original image is determined.

[0010] Optionally, the processing method also includes: obtaining image information associated with the original image; generating a control condition vector based on the image information; wherein, inputting the noisy data of the current step and the continuous vector of the current step into the diffusion model to obtain the predicted noise of the current step, including: inputting the control condition vector, the noisy data of the current step and the continuous vector of the current step into the diffusion model to obtain the predicted noise of the current step.

[0011] Optionally, generating a control condition vector based on image information includes: processing the image information using a control condition model to obtain a control condition vector, wherein the control condition model includes an encoder, and the encoder has the same structure as the encoder in the diffusion model; wherein the control condition vector, the noisy data of the current step, and the continuous vector of the current step are input into the diffusion model to obtain the predicted noise of the current step, including: inputting the noisy data of the current step and the continuous vector of the current step into the encoder of the diffusion model to obtain the encoded feature vector; and inputting the encoded feature vector and the control condition vector into the decoder of the diffusion model to obtain the predicted noise.

[0012] Optionally, the noisy data of the current step and the continuous vector of the current step are input into the encoder of the diffusion model to obtain a coding feature vector, including: for each layer in the encoder of the diffusion model, performing the following processing: obtaining the upper-layer control condition vector output by the upper coding layer of the coding layer corresponding to the current coding layer in the encoder of the control condition model; inputting the upper-layer coding feature vector and the upper-layer control condition vector into the current coding layer to obtain the coding feature vector output by the current coding layer, wherein the upper-layer coding feature vector is the coding feature vector output by the upper coding layer of the current coding layer, and when the current coding layer is the first layer of the encoder of the diffusion model, the input of the current coding layer is the noisy data of the current step and the continuous vector of the current step.

[0013] Optionally, the image information is processed using a control condition model to obtain a control condition vector, including: for each layer in the encoder of the control condition model, performing the following processing: obtaining the upper-layer coding feature vector output by the previous coding layer of the coding layer corresponding to the current coding layer in the encoder of the diffusion model; inputting the upper-layer control condition vector and the upper-layer coding feature vector into the current coding layer to obtain the control condition vector output by the current coding layer, wherein the upper-layer control condition vector is the control condition vector output by the previous coding layer of the current coding layer, and when the current coding layer is the first layer of the encoder of the control condition model, the input of the current coding layer is the image information, the noise-added data of the current step, and the continuous vector of the current step.

[0014] Optionally, the control condition model further includes an intermediate layer, which has the same structure as the intermediate layer in the diffusion model. The processing method also includes: for each layer of the intermediate layer of the diffusion model, performing the following processing: obtaining the first intermediate output of the previous layer output of the layer corresponding to the current layer in the intermediate layer of the control condition model; inputting the second intermediate output and the first intermediate output into the current layer to obtain the intermediate output of the current layer output, wherein the second intermediate output is the intermediate output of the previous layer output of the current layer, and when the current layer is the first layer of the intermediate layer of the diffusion model, the input of the current layer is the output of the encoder of the diffusion model. According to a second aspect of an embodiment of the present disclosure, a device for processing image description text is provided, comprising: a word segmentation unit, configured to perform word segmentation processing on the initial description text of the original image to obtain a word segmentation result of the initial description text; an encoding unit, configured to input the word segmentation result into a text encoder to obtain a discrete vector and a continuous vector, wherein the discrete vector is a discrete embedding vector output by the word unit embedding layer of the text encoder, and the continuous vector is an embedding vector finally output by the text encoder; a denoising unit, configured to perform denoising and denoising processing on the continuous vector for a predetermined number of steps, wherein, in each step in the predetermined number of steps, the discrete vector of the current step is updated using the predicted noise of the current step and the random noise added to the current step, and the updated discrete vector is used as the discrete vector of the next step, and the continuous vector corresponding to the discrete vector of the next step is used as the continuous vector of the next step; a text determination unit, configured to determine the final description text of the original image based on the updated discrete vector of each step in response to the completion of the denoising and denoising processing for the predetermined number of steps.

[0015] Optionally, the denoising unit is further configured to perform the following processing for each step in the predetermined number of steps: performing noise processing on the continuous vector of the current step based on the random noise of the current step to obtain the noisy data of the current step, wherein, when the current step is the first step of the predetermined number of steps, the continuous vector of the current step is a continuous vector; inputting the noisy data of the current step and the continuous vector of the current step into the diffusion model to obtain the predicted noise of the current step; using the predicted noise of the current step, the random noise of the current step, the discrete vector of the current step and the original image, updating the discrete vector of the current step and using the updated discrete vector as the discrete vector of the next step; inputting the discrete vector of the next step into the text encoder to obtain the continuous vector of the next step.

[0016] Optionally, the denoising unit is further configured to determine a noise prediction loss based on the predicted noise of the current step and the random noise of the current step; determine a semantic loss based on the discrete vector of the current step and the original image; determine a total loss based on the noise prediction loss and the semantic loss; and update the discrete vector of the current step by minimizing the total loss.

[0017] Optionally, the denoising unit is further configured to, after updating the discrete vector of the current step using the predicted noise of the current step and the random noise added to the current step, obtain the distance between the current word vector and each word vector in the predetermined vocabulary for each word vector in the updated discrete vector; determine the word vector with the smallest distance in the predetermined vocabulary as the final vector of the current word vector; and determine the final updated discrete vector of the current step based on all final vectors.

[0018] Optionally, the text determination unit is further configured to sample partial discrete vectors from the discrete vectors updated at each step; for each discrete vector in the partial discrete vectors, perform denoising processing on any random noise for a predetermined number of steps using the continuous vector and diffusion model corresponding to the current discrete vector to obtain an estimated image corresponding to the original image; and determine the final description text of the original image based on the discrete vector corresponding to the estimated image with the highest image quality.

[0019] Optionally, the processing device also includes: an acquisition unit configured to acquire image information associated with the original image; a generation unit configured to generate a control condition vector based on the image information; wherein the denoising unit is further configured to input the control condition vector, the noise-added data of the current step and the continuous vector of the current step into the diffusion model to obtain the predicted noise of the current step.

[0020] Optionally, the generation unit is further configured to process the image information using a control condition model to obtain a control condition vector, wherein the control condition model includes an encoder, and the encoder has the same structure as the encoder in the diffusion model; wherein the denoising unit is further configured to input the noisy data of the current step and the continuous vector of the current step into the encoder of the diffusion model to obtain an encoded feature vector; and input the encoded feature vector and the control condition vector into the decoder of the diffusion model to obtain predicted noise.

[0021] Optionally, the denoising unit is further configured to perform the following processing for each layer in the encoder of the diffusion model: obtaining an upper-layer control condition vector output by the previous coding layer of the coding layer corresponding to the current coding layer in the encoder of the control condition model; inputting the upper-layer coding feature vector and the upper-layer control condition vector into the current coding layer to obtain the coding feature vector output by the current coding layer, wherein the upper-layer coding feature vector is the coding feature vector output by the previous coding layer of the current coding layer, and when the current coding layer is the first layer of the encoder of the diffusion model, the input of the current coding layer is the noisy data of the current step and the continuous vector of the current step.

[0022] Optionally, the generation unit is further configured to perform the following processing for each layer in the encoder of the control condition model: obtaining the upper-layer coding feature vector output by the previous coding layer of the coding layer corresponding to the current coding layer in the encoder of the diffusion model; inputting the upper-layer control condition vector and the upper-layer coding feature vector into the current coding layer to obtain the control condition vector output by the current coding layer, wherein the upper-layer control condition vector is the control condition vector output by the previous coding layer of the current coding layer, and when the current coding layer is the first layer of the encoder of the control condition model, the input of the current coding layer is the image information, the noise-added data of the current step, and the continuous vector of the current step.

[0023] Optionally, the control condition model further includes an intermediate layer, which has the same structure as the intermediate layer in the diffusion model. The denoising unit is also configured to perform the following processing for each layer of the intermediate layer of the diffusion model: obtain the first intermediate output of the previous layer output of the layer corresponding to the current layer in the intermediate layer of the control condition model; input the second intermediate output and the first intermediate output into the current layer to obtain the intermediate output of the current layer output, wherein the second intermediate output is the intermediate output of the previous layer output of the current layer, and when the current layer is the first layer of the intermediate layer of the diffusion model, the input of the current layer is the output of the encoder of the diffusion model. According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: at least one processor; and at least one memory storing computer-executable instructions, wherein the computer-executable instructions, when executed by the at least one processor, prompt the at least one processor to execute the above-mentioned method for processing image description text.

[0024] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to execute the above-mentioned method for processing image description text.

[0025] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, comprising computer instructions, which implement the above-mentioned method for processing image description text when executed by a processor.

[0026] The technical solutions provided by the embodiments of the present disclosure bring at least the following beneficial effects: According to the image description text processing method and device, electronic device, storage medium and computer program product disclosed in the present invention, the initial description text to be optimized is converted into a discrete vector, and the discrete vector is automatically learned and updated using multiple noise addition and denoising processes. The final description text is determined based on the updated discrete vector, thereby optimizing the initial description text; the present invention combines the automation characteristics of soft prompts with the portability and interpretability of hard prompts, that is, the present invention is used to automatically generate high-quality description text, so that the generated results based on the description text are more in line with user needs, while avoiding the high cost and low efficiency of manual debugging of the description text, thereby improving the controllability and adaptability of the generation task. Therefore, the present invention solves the problem in the related art that prompt word optimization is difficult and not universal between models of different architectures.

[0027] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute an improper limitation of the present disclosure.

[0029] Figure 1 is a flowchart of a method for processing image description text according to an exemplary embodiment of the present disclosure; Figure 2 is a schematic diagram of the overall process of the present disclosure according to an exemplary embodiment of the present disclosure; Figure 3 is a schematic diagram of a controllable diffusion model architecture according to an exemplary embodiment of the present disclosure; Figure 4 is a schematic diagram of a system flow of semantic optimization according to an exemplary embodiment of the present disclosure; Figure 5 is a diagram comparing effects of adding a semantic optimization module and not adding a semantic optimization module according to an exemplary embodiment of the present disclosure; Figure 6 is a block diagram of a device for processing image description text according to an exemplary embodiment of the present disclosure; Figure 7 is a diagram illustrating a computing environment coupled with a user interface according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0031] It should be noted that the terms "first," "second," and the like in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the numbers used in this manner are interchangeable where appropriate so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation methods described in the following examples do not represent all implementation methods consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with certain aspects of the present disclosure as detailed in the appended claims.

[0032] It should be noted that the phrase "at least one of the several items" in this disclosure includes three types of parallel situations: "any one of the several items", "a combination of any multiple of the several items", and "all of the several items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. For another example, "performing at least one of step 1 and step 2" means the following three parallel situations: (1) performing step 1; (2) performing step 2; and (3) performing both step 1 and step 2.

[0033] In recent years, large language models (such as GPT and Llama) and text-to-graph models (such as DALL E and StableDiffusion) have demonstrated remarkable capabilities in generative tasks. Trained on massive amounts of data, these models are capable of understanding and generating complex linguistic and visual content. However, effectively guiding these models to produce desired results has become a core challenge. Cue word engineering is a key technology for addressing this challenge.

[0034] Currently, prompt engineering methods can be roughly divided into two categories. The first category is hard prompting, which uses manually designed, interpretable tag sequences to guide model behavior. For example, in a text-to-image model, users can generate highly consistent images through carefully designed prompts (such as "a cat in a spacesuit walking on the moon"). The second category is soft prompting, which consists of continuous language embedding vectors that cannot be mapped to any human-readable tags to implicitly optimize the model's generation results.

[0035] However, although hard prompts have unique advantages in flexibility, portability, and interpretability, such as the wording of hard prompts can be manually adjusted to adapt to different styles or task requirements, and can be reused between different models, the design of hard prompts relies on manual experience and is difficult to optimize; and soft prompts, because they rely on the embedding space of a specific model, are usually difficult to migrate between models of different architectures.

[0036] In response to the above problems, the present disclosure provides a method for processing image description text, namely, performing word segmentation processing on the initial description text of the original image to obtain the word segmentation result of the initial description text, inputting the word segmentation result into a text encoder to obtain a discrete vector and a continuous vector, and then performing a predetermined number of steps of denoising and denoising on the continuous vector, wherein, at each step in the predetermined number of steps, the discrete vector of the current step is updated using the predicted noise of the current step and the random noise added to the current step, and the updated discrete vector is used as the discrete vector of the next step, and the continuous vector corresponding to the discrete vector of the next step is used as the continuous vector of the next step; after the predetermined number of steps of denoising and denoising are completed, the final description text of the original image can be determined based on the updated discrete vector of each step. The generated result of the final description text determined in the above manner meets the user's needs while avoiding the high cost and low efficiency of manually debugging the description text. For example, in the text-to-image task, the present disclosure can automatically generate high-quality prompt words, so that the generated result based on the prompt words is more in line with the user's needs while avoiding the high cost and low efficiency of manually debugging the prompt words.

[0037] Hereinafter, a method and apparatus for processing image description text, an electronic device, a storage medium, and a computer program product according to exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0038] Figure 1 is a flowchart of a method for processing image description text according to an exemplary embodiment of the present disclosure. Figure 1 As shown, the method for processing image description text includes the following steps: In step S101 , the initial description text of the original image is segmented to obtain a segmentation result of the initial description text.

[0039] As an example, the initial description can summarize the main semantic content of the original image. Specifically, the initial description can be derived from a user's observation of the original image. Alternatively, the original image can be processed using an existing multimodal large model to generate the initial description. Specifically, the original image is fed into the multimodal large model, which generates a compact and accurate description that serves as a prompt for the original image, also known as the initial description. For example, if the original image is a landscape painting, the description generated by the multimodal large model might read: "A sunny meadow, with blue sky and white clouds in the distance, and lush trees."

[0040] It should be noted that the above-mentioned multimodal large model can be GPT-4, Llama (such as Llama-3.2-vision), etc., and this disclosure is not limited to this. As long as the above-mentioned multimodal large model can effectively process the multimodal information between images and texts and extract accurate descriptive information in the context of the image, it will be fine.

[0041] As an example, after obtaining the initial description text of the original image, the initial description text can be segmented by a word segmenter according to a predetermined vocabulary and converted into corresponding word indexes (Idx), and these word indexes are used as the segmentation results of the initial description text.

[0042] It should be noted that the key to the above embodiment is to ensure the accuracy and compactness of the initial description text to ensure semantic guidance during the subsequent diffusion process. The above-mentioned word segmenter can be a tokenizer of the CLIP model, which is a multimodal model based on contrastive learning and is known as Contrastive Language-Image Pre-training.

[0043] return Figure 1 In step S102, the word segmentation result is input into the text encoder to obtain a discrete vector and a continuous vector, wherein the discrete vector is the discrete embedding vector output by the word embedding layer of the text encoder, and the continuous vector is the embedding vector finally output by the text encoder.

[0044] As an example, after obtaining the corresponding word index, the token embedding layer (token_embedding) in the text encoder can be input to obtain a discrete vector, that is, the token embedding layer passes these indexes through a predetermined vocabulary. Mapping to text embedding vector , that is, the above discrete vector, where, is the vocabulary size of the model where the text encoder resides, represents the dimension of the vector, Represents the number of word vectors in a discrete vector. The dimension of this discrete vector varies depending on the model settings of the text encoder. This discrete vector serves as the basis for subsequent generation and optimization, and is also key data in the subsequent generation and optimization process. After obtaining the discrete vector, the subsequent Transformer layer in the text encoder further generates a continuous vector corresponding to the discrete vector and passes the continuous vector to the subsequent diffusion model.

[0045] It should be noted that the key to the above embodiment is to ensure that the corresponding embedding vector can effectively guide the diffusion model during the generation process, helping the diffusion model to make more accurate noise predictions, thereby maintaining semantic consistency and image quality in each generation step of the diffusion model. The above text encoder can also be the text encoder of the CLIP model.

[0046] return Figure 1 In step S103, the continuous vector is subjected to a predetermined number of steps of denoising and denoising. In each step of the predetermined number of steps, the discrete vector of the current step is updated using the predicted noise of the current step and the random noise added to the current step, and the updated discrete vector is used as the discrete vector of the next step, and the continuous vector corresponding to the discrete vector of the next step is used as the continuous vector of the next step.

[0047] As an example, the diffusion model includes a predetermined number of denoising layers (such as a Unet network), each of which performs one step of denoising processing, and the predetermined number of the denoising layers corresponds to the predetermined number of steps.

[0048] As an example, after obtaining the discrete vector and the continuous vector, the two vectors will then participate in the noise prediction process of the diffusion model, and the corresponding noise prediction loss value will be calculated based on the predicted noise and the actual noise in the prediction process. The discrete vector is then updated using the gradient of the loss to achieve optimization of the discrete vector.

[0049] As an example, the random noise added in the current step is the actual noise data added in the current step during the forward noise addition process.

[0050] According to an exemplary embodiment of the present disclosure, at each step in the predetermined number of steps, the following processing can be performed: the continuous vector of the current step and the random noise of the current step are input into the current noise layer of the diffusion model to obtain the noise data of the current step, wherein when the current step is the first step of the predetermined number of steps, the continuous vector of the current step is a continuous vector; the noisy data of the current step and the continuous vector of the current step are input into the diffusion model to obtain the predicted noise of the current step; the discrete vector of the current step is updated using the predicted noise of the current step, the random noise of the current step, the discrete vector of the current step, and the original image, and the updated discrete vector is used as the discrete vector of the next step; the discrete vector of the next step is input into the text encoder to obtain the continuous vector of the next step. Through this embodiment, a semantic loss is introduced between the discrete vector and the original image, so that the discrete vector can be updated more accurately based on the noise prediction loss and the semantic loss.

[0051] As an example, after obtaining the discrete vector and the continuous vector, the two vectors will then participate in the noise prediction process of the diffusion model, and calculate the corresponding noise prediction loss value based on the predicted noise and the actual noise in the prediction process, as well as the semantic loss value of the discrete vector and the original image. The discrete vector is then updated through the gradient of the two loss values to achieve optimization of the discrete vector.

[0052] As an example, the forward noise addition process can be performed for a predetermined number of steps using the following formula:

[0053] in, is the noise data of the current step, t represents the current step, The random noise added to the current step (e.g., random noise that conforms to the standard Gaussian distribution), that is, the actual noise data added to the current step, , Indicates the i The degree of noise addition can be set as needed.

[0054] According to an exemplary embodiment of the present disclosure, updating the discrete vector of the current step using the predicted noise of the current step, the random noise of the current step, the discrete vector of the current step, and the original image may include: determining a noise prediction loss based on the predicted noise of the current step and the random noise of the current step; determining a semantic loss based on the discrete vector of the current step and the original image; determining a total loss based on the noise prediction loss and the semantic loss; and updating the discrete vector of the current step by minimizing the total loss. Through this embodiment, a relatively accurate discrete vector can be obtained by minimizing the noise prediction loss and the semantic loss.

[0055] As an example, a composite loss function can be used to simultaneously consider noise prediction loss and semantic loss to ensure the quality of the subsequent generated image and the semantic consistency of the description text, where the noise prediction loss ensures that the difference between the model's prediction of noise and the actual noise is minimized, and the semantic loss ensures that the discrete vector With the original image The similarity between them is maximized, that is, the semantic loss is minimized, thereby ensuring that the image semantics is consistent with the description text.

[0056] The specific composite loss function can be as follows:

[0057] in, represents the diffusion model, represents the CLIP model, is the current step, represents the noise data of the current step, represents the updated discrete vector, represents the original image, Represents the random noise added in the current step, represents the prediction noise of the current step, represents a continuous vector, Represents the loss function of the diffusion model, which is used to calculate the noise prediction loss. Represents the loss function of the CLIP model, which is used to calculate the semantic loss. and represent the weight coefficients of noise prediction loss and semantic loss respectively.

[0058] Update the discrete vector by calculating the gradient of the above composite loss function , thereby improving the quality of subsequent generated images and the semantic consistency of description texts.

[0059] According to an exemplary embodiment of the present disclosure, after updating the discrete vector of the current step using the predicted noise of the current step and the random noise added to the current step, for each word vector in the updated discrete vector, the distance between the current word vector and each word vector in the predetermined vocabulary can be obtained; the word vector with the smallest distance in the predetermined vocabulary is determined as the final vector of the current word vector; and based on all final vectors, the final updated discrete vector of the current step is determined. Through this embodiment, each updated discrete vector is compared with the word vector in the predetermined vocabulary, preventing the discrete vector from deviating from the word space during the update process, resulting in the inability to decode the correct semantics later.

[0060] As an example, in order to prevent the discrete vector from deviating from the word space after the update, resulting in the inability to decode the correct semantics, each time the discrete vector is updated, The projection function updates the discrete vector Mapping back to a predefined vocabulary, i.e. text embedding space , thus in Found in The closest vector, specifically, to For each word vector in , we can calculate the current word vector and The Euclidean distance between each word vector in , and select Replace the word vector with the smallest median distance This ensures that the optimization process always maintains semantic consistency after each update of the discrete vector.

[0061] It should be noted that through The updated discrete vector from the projection function is then fed back into the diffusion model to participate in noise prediction, which is the next step in the denoising process. As the iterations progress, the discrete vector is gradually optimized, and the semantic quality and accuracy of the image generated based on it are continuously improved.

[0062] According to an exemplary embodiment of the present disclosure, the processing method of the present disclosure can also obtain image information associated with the original image; generate a control condition vector based on the image information; wherein the method includes: inputting the control condition vector, the noisy data of the current step, and the continuous vector of the current step into the diffusion model to obtain the predicted noise of the current step. Through this embodiment, image information associated with the original image is introduced into the denoising process, assisting the diffusion model in denoising random noise, generating a relatively accurate image, increasing the accuracy of denoising, and improving the quality of the generated image.

[0063] As an example, the above-mentioned image information is in different forms of information in different application scenarios, such as a compressed image, a depth image, an edge image, a segmentation image, etc., which is not limited in the present disclosure.

[0064] For example, taking image compression restoration as an example, the original image can be input into the image compression network to obtain a very blurred compressed image, which is also the compressed image of the original image. The compressed image is the image information, which can be used as a control condition to participate in subsequent restoration tasks.

[0065] For example, taking image style adjustment as an example, the original image can be input into the image processing network to obtain a depth map. The depth map, that is, the image information, can be used as a control condition to participate in subsequent adjustment tasks, that is, to adjust other images to images in the style of the original image.

[0066] It should be noted that the working principle of the diffusion model is to generate a clear target image from the initial noise through gradual denoising. However, the traditional diffusion model lacks fine control capabilities and cannot accurately generate an expected image based on the input text description, especially in scenarios with highly customized and diverse image content. Therefore, this embodiment introduces image information associated with the original image into the denoising process to assist the diffusion model in denoising processing and improve the quality of the generated image.

[0067] According to an exemplary embodiment of the present disclosure, generating a control condition vector based on image information may include: processing the image information using a control condition model to obtain a control condition vector, wherein the control condition model includes an encoder having the same structure as the encoder in the diffusion model; wherein the control condition vector, the noise-added data of the current step, and the continuous vector of the current step are input into the current denoising layer in the diffusion model to obtain the predicted noise of the current step, including: inputting the noise-added data of the current step and the continuous vector of the current step into the encoder of the diffusion model to obtain an encoded feature vector; and inputting the encoded feature vector and the control condition vector into the decoder of the diffusion model to obtain the predicted noise. Through this embodiment, a control condition model is introduced, which includes an encoder with the same structure as the encoder in the diffusion model, thereby converting image information associated with the original image into a vector and introducing a denoising process.

[0068] As an example, the control condition model described above replicates the encoder block of the diffusion model. However, to reduce the additional computational burden and prevent the control condition model from overly interfering with the diffusion model, the control condition model's parameters can be compressed and optimized. For example, the number of input channels per layer can be reduced to 1 / 5 of the original. This approach significantly reduces the number of parameters in the control condition model while retaining sufficient expressive power to effectively control the generation process. The core goal of the control condition unit is to improve the controllability of the diffusion model with a very small number of parameters. As an example, by inputting image information into the control condition model, a control condition vector can be obtained, and the control condition vector and the encoded feature vector of the diffusion model are input into the decoder of the diffusion model together to obtain the predicted noise, wherein the number of input channels of each layer in the control condition model is less than the number of corresponding layers in the diffusion model.

[0069] According to an exemplary embodiment of the present disclosure, inputting the noise-added data of the current step and the continuous vector of the current step into the encoder of the diffusion model to obtain a coding feature vector may include: performing the following processing for each layer in the encoder of the diffusion model: obtaining an upper-layer control condition vector output by the coding layer of the previous coding layer corresponding to the current coding layer in the encoder of the control condition model; inputting the upper-layer coding feature vector and the upper-layer control condition vector into the current coding layer to obtain a coding feature vector output by the current coding layer, wherein the upper-layer coding feature vector is the coding feature vector output by the coding layer of the previous coding layer of the current coding layer, and when the current coding layer is the first layer of the encoder of the diffusion model, the input of the current coding layer is the noise-added data of the current step and the continuous vector of the current step. This embodiment implements dual-path control signal transmission, that is, the output of the control condition model can not only affect the decoder of the diffusion model, but also affect the encoder of the diffusion model, thereby guiding information flow across the entire network, ensuring that the control condition vector forms a consistent control effect throughout the network and reducing cross-layer information loss, so that the output of the control condition model can better assist the diffusion model in predicting noise.

[0070] As an example, in existing diffusion methods, image information can usually only be injected at certain specific stages (such as the decoding stage), which causes a lag in the image information in the time dimension, thereby affecting the final generation quality. In order to solve the problem of control instability caused by the lag introduced by image information in traditional methods, the architectural design of the control condition unit in this embodiment is refined and optimized to ensure that the guidance of the diffusion generation process can be effectively enhanced while the amount of computation is controllable. Specifically, this embodiment adds dual-path transmission content, that is, the output of the control condition model is not only input into the decoder of the diffusion model, but also into the encoder of the diffusion model, so that the control condition vector corresponding to the image information can be dynamically adjusted at different stages of the diffusion process, achieving more efficient cross-layer information flow. In other words, by optimizing the information interaction mechanism, the control condition vector can guide image generation more accurately and efficiently.

[0071] Specifically, for each layer in the encoder of the control condition model, the control condition vector output by the current encoding layer will undergo zero convolution processing. The result of this processing is not only sent to the corresponding layer in the decoder of the diffusion model, but also to the corresponding layer in the encoder of the diffusion model, and fused with the output of the previous layer in the encoder of the diffusion model to ensure that the image information forms a consistent control effect in the entire network and reduce the loss of information across layers (decoder and encoder), thereby guiding the information flow throughout the entire network.

[0072] It should be noted that in order to ensure that the image information does not affect the original generation ability of UNet in the diffusion model, zero-setting convolution can be used before the control condition vector is input into the diffusion model. This allows the control condition vector to gradually learn its influence on the generation process in a manner that is approximately zero-initialized, thereby avoiding excessive intervention in the early stages of training.

[0073] According to an exemplary embodiment of the present disclosure, processing image information using a control condition model to obtain a control condition vector may include: performing the following processing for each layer in the encoder of the control condition model: obtaining an upper-layer coding feature vector output by the previous coding layer of the coding layer corresponding to the current coding layer in the encoder of the diffusion model; inputting the upper-layer control condition vector and the upper-layer coding feature vector into the current coding layer to obtain a control condition vector output by the current coding layer, wherein the upper-layer control condition vector is the control condition vector output by the previous coding layer of the current coding layer, and when the current coding layer is the first layer of the encoder of the control condition model, the input of the current coding layer is the image information, the noise-added data of the current step, and the continuous vector of the current step. Through this embodiment, dynamic feedback optimization is achieved, that is, the output of the encoder of the diffusion model can also affect the encoder of the control condition model, so that the control condition vector can also be adaptively adjusted to adapt to the dynamic changes of different diffusion step sizes, avoiding the problem of delayed propagation of the control signal in existing methods.

[0074] As an example, the output of the encoding layer of the diffusion model is also transmitted back to the control condition model. That is, in each layer of the diffusion model's encoder, the encoded feature vector output by the current encoding layer is not only propagated to the corresponding layer in the diffusion model's decoding layer, but also fed back to the corresponding layer in the control condition model's encoder after undergoing zero-set convolution. This encoded feature vector is fused with the output of the corresponding layer in the control condition model's encoder and serves as the input to the next layer in the control condition model's encoder. This allows the control condition vector of each layer in the control condition model's encoder to be adaptively adjusted to accommodate the dynamic changes of different diffusion step sizes, avoiding the problem of delayed control signal propagation in existing methods.

[0075] Compared with the existing technology, the information interaction method adopted in the present invention, that is, information interaction is carried out between the encoders of the diffusion model and the control condition model, so that the control condition vector of each layer can be updated based on the latest generated features, thereby reducing the lag in the information transmission process and improving the accuracy of control.

[0076] According to an exemplary embodiment of the present disclosure, the control condition model further includes an intermediate layer, and the intermediate layer has the same structure as the intermediate layer in the diffusion model. For each layer of the intermediate layer of the diffusion model, the following processing is performed: the first intermediate output of the previous layer output of the layer corresponding to the current layer in the intermediate layer of the control condition model is obtained; the second intermediate output and the first intermediate output are input to the current layer to obtain the intermediate output of the current layer output, wherein the second intermediate output is the intermediate output of the previous layer output of the current layer, and when the current layer is the first layer of the intermediate layer of the diffusion model, the input of the current layer is the output of the encoder of the diffusion model. Through this embodiment, the intermediate layer output further included in the control condition unit can further affect the intermediate layer of the diffusion model, making denoising more accurate, thereby obtaining a better description text.

[0077] As an example, the control condition model described above also replicates the diffusion model's middle block. Similarly, to reduce the additional computational burden and prevent the control condition model from overly interfering with the diffusion model, the parameters of the middle block can be compressed and optimized. For example, the number of input channels per layer can be reduced to one-fifth of the original. This approach significantly reduces the number of parameters in the middle block while retaining sufficient expressive power to effectively control the generation process.

[0078] As an example, for each layer in the middle layer of the control condition model, the output of the current layer will be sent to the corresponding layer in the middle layer of the diffusion model and fused with the output of the previous layer in the middle layer of the diffusion model to ensure that the image information forms a consistent control effect in the entire network.

[0079] In step S104 , in response to the completion of the predetermined number of steps of denoising and denoising processing, a final description text of the original image is determined based on the discrete vector updated in each step.

[0080] As an example, for each discrete vector, a corresponding image can be generated using a diffusion model, and the discrete vector with the best image quality is selected and converted to obtain the final description text, which is not limited in this disclosure.

[0081] According to an exemplary embodiment of the present disclosure, determining the final description text of the original image based on the discrete vectors updated at each step may also include: sampling a portion of the discrete vectors from the discrete vectors updated at each step; for each discrete vector in the portion of the discrete vectors, performing a predetermined number of steps of denoising on any random noise using the continuous vector corresponding to the current discrete vector and a diffusion model to obtain an estimated image corresponding to the original image; and determining the final description text of the original image based on the discrete vector corresponding to the estimated image with the highest image quality. In this embodiment, selecting a portion of the discrete vectors can reduce the amount of computation; and selecting the discrete vector corresponding to the estimated image with the highest image quality can ensure the accuracy of the description text.

[0082] As an example, in an iteration of a predetermined number of steps, a generation test can be performed after every certain number of iterations, such as after every 100 iterations, and different indicators (such as LPIPS, CLIP Score, PSNR, etc.) are used to measure the quality of the final generated image, and then the discrete vector corresponding to the optimal indicator is selected to obtain the final description text.

[0083] Specifically, the generation test is achieved by: inputting the continuous vector and random noise corresponding to the current discrete vector into the diffusion model, performing denoising on the random noise for a predetermined number of steps, and obtaining an estimated image corresponding to the original image, that is, after the diffusion model outputs the predicted noise of the current step, using the denoising formula to remove the predicted noise from the denoised data of the previous step to obtain the denoised data of the current step, and inputting the denoised data of the step and the above-mentioned continuous vector into the diffusion model again to obtain the next predicted noise, and performing denoising again until the denoising for the predetermined number of steps is completed to obtain the estimated image. It should be noted that when the current step is the first step, the denoised data of the previous step is random noise; using indicators such as LPIPS, CLIP Score, PSNR, etc. to measure the image quality of the estimated image, and then selecting the estimated image with the highest measurement index, and subsequently using the discrete vector corresponding to the estimated image to convert it into the final description text.

[0084] In order to facilitate understanding of the present disclosure, Figure 2 、 Figure 3 and Figure 4 Provide a description of the system.

[0085] Figure 2 A schematic diagram showing the overall process of the present disclosure is shown in FIG. Figure 2 As shown, taking the image restoration scenario as an example, the overall process of the present disclosure mainly includes three modules: a text extraction module, a controllable diffusion module and a semantic optimization module. Among them, the text extraction model is used to obtain the initial description text (prompt) of the original image, the controllable diffusion module is used to combine the description text information and control condition information to control the diffusion model to generate an image, and the semantic optimization model is used to optimize the initial description text in combination with the controllable diffusion module.

[0086] The following are the three models introduced separately: 1) Text extraction module Existing multimodal large language models (such as GPT-4, Llama-3.2-vision, etc.) can be used to extract the initial description text of the original image. That is, the original image is input into the multimodal large language model, and the multimodal large language model will output the initial description text of the original image.

[0087] After obtaining the initial description text, the initial description text can be segmented by the word segmenter according to the predetermined vocabulary and converted into corresponding word indexes (Idx). Then, these indexes can be embedded in the predetermined vocabulary through the word embedding layer (token_embedding) in the text encoder. Mapping to a discrete vector. After obtaining the discrete vector, the subsequent Transformer layer in the text encoder further generates a continuous vector corresponding to the discrete vector and passes the continuous vector to the subsequent controllable diffusion module. It should be noted that this part can also be considered part of the semantic optimization module, but for ease of understanding, it is described here in the text extraction module.

[0088] 2) Controllable diffusion module The controllable diffusion module can include a diffusion model and a control condition model. The diffusion model is used to generate high-quality reconstructed images and can be constructed using Stable Diffusion, a concept not limited in this disclosure. The control condition model is used to introduce control information into the diffusion model. It should be noted that existing diffusion models lack controllability during the generation process, making it difficult to accurately guide image reconstruction. Therefore, this disclosure introduces a control condition model to enhance the controllability of the diffusion model.

[0089] The above control information can be Figure 2 The preprocessing network in

[15] can be used to obtain the original image. The functions of the preprocessing network vary in different scenarios. For example, in image compression restoration, the preprocessing network can be an image compression network. In this case, the original image is input into the image compression network to obtain a compressed image of the original image. This compressed image can be used as a control condition for subsequent restoration tasks.

[0090] After obtaining the continuous vector and the compression graph, the continuous vector is subjected to a noise addition process for a predetermined number of steps, and then subjected to a denoising process for a predetermined number of steps.

[0091] Figure 3 The controllable diffusion model architecture diagram is shown, which mainly shows the t-th step denoising process, such as Figure 3 As shown, in the denoising process, the noise data of the current step can be Perform denoising, Input the diffusion model (i.e. Unet) to predict the prediction noise of the current step. It should be noted that the diffusion model and the continuous vector of the current step are also input at this time. Figure 3 It is not shown that the continuous vector is obtained based on the discrete vector updated in the current step; in the denoising process, the compressed image is also and Input the control condition model to obtain the control condition vector of the current step, which is then input into the diffusion model. It should be noted that the continuous vector of the current step is also input into the control condition model. Figure 3 Not shown.

[0092] Moreover, if Figure 3 As shown in the figure, the output of each layer in the encoder of the control condition model is not only fed into the corresponding layer in the decoder of the diffusion model, but also fed into the corresponding layer in the encoder of the diffusion model and fused with the output of the previous layer in the encoder of the diffusion model, ensuring that the image information forms a consistent control effect in the entire network and reducing the loss of information across layers (decoder and encoder), thereby guiding the information flow throughout the entire network.

[0093] Furthermore, if Figure 3 As shown, the output of the encoding layer of the diffusion model is also transmitted back to the control condition model, that is, the output of each layer in the encoder of the diffusion model is not only sent to the corresponding layer in the decoding layer of the diffusion model, but also fed back to the corresponding layer in the encoder of the control condition model. The feedback encoding feature vector is fused with the output of the corresponding layer in the encoder of the control condition model and serves as the input of the next layer of the corresponding layer in the encoder of the control condition model. This allows the control condition vector of each layer in the encoder of the control condition model to be adaptively adjusted to adapt to the dynamic changes of different diffusion step sizes, avoiding the problem of delayed propagation of the control signal in existing methods.

[0094] Finally, if Figure 3 As shown in the figure, to ensure that the compressed graph does not affect the original generation ability of the UNet in the diffusion model, zero-setting convolution can be used before the control condition vector is input into the diffusion model. This allows the control condition vector to gradually learn its influence on the generation process in a manner that is approximately zero-initialized, thereby avoiding excessive interference in the early stages of training. In addition, zero-setting convolution can also be used before the output of the encoding layer of the diffusion model is fed back to the corresponding layer in the encoder of the control condition model.

[0095] It should be noted that before performing the above processing, the controllable diffusion module needs to be trained accordingly in advance for different tasks. First, obtain the data set required for training. For example, a high-resolution LSDIR data set can be used. In addition, the multimodal large language model Llava can be used to generate text prompt words for each image to provide richer training supervision signals. In addition, for different tasks, different control conditions need to be extracted from the data set, such as depth map extraction, edge map extraction, etc., which are not limited in this disclosure. Secondly, taking image compression artifact repair as an example, the control condition extracts the compressed image at this time. The original image and the compressed image corresponding to the original image are mapped to the latent space through VAE and then input into the controllable diffusion module for noise prediction. The loss function used in the training process can be as follows:

[0096] in, is the real noise, To predict the noise, is the noise data of step t, For text prompt words, is the control condition (compression graph), To control the parameters of the network.

[0097] 3) Semantic Optimization Module In order to further improve the control accuracy and effect of the controllable diffusion module in the image generation process, the present disclosure introduces a semantic optimization module, which aims to achieve more accurate and effective image generation control by optimizing the description text of the original image. The core goal of the semantic optimization module is to obtain the best text description of the original image to ensure that the generated image is more semantically in line with user expectations. To this end, the present disclosure combines the controllable diffusion module to update the discrete vector of the description text. In this process, no additional module needs to be trained, that is, the parameters of the controllable diffusion module are fixed during the optimization process, avoiding the huge resource consumption brought about by training a large multimodal model; and the optimization goal is to minimize the noise prediction loss and semantic loss. The semantic loss can adopt the loss function of CLIP to maintain semantic consistency during the optimization process and improve the final embedding quality.

[0098] Figure 4 Show the system flow of semantic optimization, such as Figure 4 As shown in the figure, the process mainly includes the following parts: First, obtain the discrete vector and continuous vector of the initial description text, that is, Figure 4 The learnable embedding vector shown in FIG is obtained by the process that has been discussed in detail before and will not be discussed here.

[0099] Secondly, at the beginning of the optimization, the above continuous vector is involved in Figure 3The denoising process shown in Figure 1 assists in predicting the prediction noise at each step. In each denoising process, the noise prediction loss can be calculated using the predicted noise of the current step and the real noise added at the current step. The semantic loss can also be calculated using the discrete vector of the current step and the original image. By minimizing the noise prediction loss and the semantic loss, the discrete vector is updated, i.e. Figure 4 The updated discrete vector is obtained by back propagation in , and the updated discrete vector can make the text description more consistent with the semantics of the generated image.

[0100] Again, after obtaining the updated discrete vector After that, you can also use The function projects the updated discrete vector back into the vocabulary , ensuring that each updated text vector meets the expectations of the language model and Found in The closest vector is used as the final updated discrete vector. After obtaining the final updated discrete vector, the previous operation can be repeated to perform multiple iterations of denoising and updating until the maximum number of iterations is reached, at which point the update is stopped.

[0101] Finally, as described above, during the entire image generation process, the discrete vectors are continuously updated through iterative optimization. After a certain number of iterations (for example, 100 times), a generation test can be performed to evaluate the quality of the generated image. For example, commonly used image quality assessment indicators such as LPIPS, CLIP Score, and PSNR can be used to select the discrete vector with the best image quality as the optimal discrete vector. Based on the optimal discrete vector, the final description text is converted, which can well guide the generation process.

[0102] It can be seen that the semantic optimization module of this embodiment accurately controls the image generation process by optimizing discrete vectors. This module can be deeply integrated with the controllable diffusion module. That is, the controllable diffusion module can deeply fuse the continuous vector corresponding to the discrete vector with the generated features through its UNet network and attention mechanism, effectively assisting noise prediction, thereby ensuring the semantic consistency of the generated image.

[0103] It should be noted that the optimization process can also be adjusted according to the test results to ensure that the generated image reaches the optimal state in terms of quality and semantic consistency. That is, each generated image will be compared with the predetermined target, and the discrete vector will be continuously adjusted according to the comparison result until the desired result is achieved.

[0104] After multiple rounds of optimization and testing, the final description text is used by the controllable diffusion module to generate an image with high semantic consistency with the original image. This image not only meets the user's visual needs but also maintains semantic consistency with the input text description. This high-quality output image can be further adjusted to meet user requirements, such as changing color, style, or other details to meet personalized needs. The adjusted image is then displayed to the user as the final generated result.

[0105] In order to prove the feasibility of the present invention, the present invention also takes image compression repair as an example to conduct effect testing. Figure 5 As shown in Figure 3, the addition of the semantic optimization module can make the result closer to the original image and avoid the generation of erroneous information by obtaining the best semantic description of the image.

[0106] In summary, the present disclosure has built a semantic optimization framework based on a multimodal large model. The architecture includes a text extraction module, a controllable diffusion module, and a semantic optimization module. First, the text extraction module extracts the initial description text from the original image through the multimodal large model; secondly, the initial description text is segmented and a discrete vector is obtained through a text encoder. Then, in the semantic optimization module, the continuous vector corresponding to the discrete vector participates in the denoising and denoising process of the controllable diffusion module. During the denoising process, the discrete vector is optimized (the parameters of the controllable diffusion model remain fixed), and the optimized discrete vector is mapped into a compact text, that is, the final description text; finally, the final description text can generate an image that is highly consistent with the original image through the controllable diffusion module. Through this framework, the semantic expression ability of text information can be improved and the image reconstruction quality can be optimized. Moreover, a cross-modal information interaction mechanism can be effectively constructed to enhance the text and image features, providing strong support for subsequent optimization and reconstruction.

[0107] It should be noted that the final description text obtained by the present disclosure contains compact and accurate semantic information, which can provide high-precision semantic optimization and control for the diffusion model. It can be used for tasks such as controllable text images and multimodal image compression. It can also be used in various scenarios that require high-quality and diversified image generation, which is not limited by the present disclosure.

[0108] Figure 6 1 is a block diagram of a device for processing image description text according to an exemplary embodiment of the present disclosure. Figure 6 The processing device includes a word segmentation unit 60, an encoding unit 62, a denoising unit 64 and a text determination unit 66.

[0109] The word segmentation unit 60 is configured to perform word segmentation processing on the initial description text of the original image to obtain the word segmentation result of the initial description text; the encoding unit 62 is configured to input the word segmentation result into the text encoder to obtain a discrete vector and a continuous vector, wherein the discrete vector is the discrete embedding vector output by the word unit embedding layer of the text encoder, and the continuous vector is the embedding vector finally output by the text encoder; the denoising unit 64 is configured to perform denoising and denoising processing on the random noise for a predetermined number of steps, wherein, in each step in the predetermined number of steps, the discrete vector of the current step is updated using the predicted noise of the current step and the random noise added to the current step, and the updated discrete vector is used as the discrete vector of the next step, and the continuous vector corresponding to the discrete vector of the next step is used as the continuous vector of the next step; the text determination unit 66 is configured to determine the final description text of the original image based on the updated discrete vector of each step in response to the completion of the denoising and denoising processing for the predetermined number of steps.

[0110] According to an exemplary embodiment of the present disclosure, the denoising unit 64 is further configured to perform the following processing for each step in the predetermined number of steps: performing noise processing on the continuous vector of the current step based on the random noise of the current step to obtain the noisy data of the current step, wherein, when the current step is the first step of the predetermined number of steps, the continuous vector of the current step is a continuous vector; inputting the noisy data of the current step and the continuous vector of the current step into the diffusion model to obtain the predicted noise of the current step; using the predicted noise of the current step, the random noise of the current step, the discrete vector of the current step and the original image, updating the discrete vector of the current step and using the updated discrete vector as the discrete vector of the next step; inputting the discrete vector of the next step into the text encoder to obtain the continuous vector of the next step.

[0111] According to an exemplary embodiment of the present disclosure, the denoising unit 64 is further configured to determine a noise prediction loss based on the predicted noise of the current step and the random noise of the current step; determine a semantic loss based on the discrete vector of the current step and the original image; determine a total loss based on the noise prediction loss and the semantic loss; and update the discrete vector of the current step by minimizing the total loss.

[0112] Optionally, the denoising unit 64 is further configured to, after updating the discrete vector of the current step using the predicted noise of the current step and the random noise added to the current step, obtain the distance between the current word vector and each word vector in the predetermined vocabulary for each word vector in the updated discrete vector; determine the word vector with the smallest distance in the predetermined vocabulary as the final vector of the current word vector; and determine the final updated discrete vector of the current step based on all final vectors.

[0113] Optionally, the text determination unit 66 is further configured to sample partial discrete vectors from the discrete vectors updated at each step; for each discrete vector in the partial discrete vectors, perform denoising processing on any random noise for a predetermined number of steps using the continuous vector and diffusion model corresponding to the current discrete vector to obtain an estimated image corresponding to the original image; and determine the final description text of the original image based on the discrete vector corresponding to the estimated image with the highest image quality.

[0114] Optionally, the processing device also includes: an acquisition unit configured to acquire image information associated with the original image; a generation unit configured to generate a control condition vector based on the image information; wherein the denoising unit 64 is further configured to input the control condition vector, the noise-added data of the current step and the continuous vector of the current step into the diffusion model to obtain the predicted noise of the current step.

[0115] Optionally, the generation unit is further configured to process the image information using a control condition model to obtain a control condition vector, wherein the control condition model includes an encoder, and the encoder has the same structure as the encoder in the diffusion model; wherein the denoising unit is further configured to input the noisy data of the current step and the continuous vector of the current step into the encoder of the diffusion model to obtain an encoded feature vector; and input the encoded feature vector and the control condition vector into the decoder of the diffusion model to obtain predicted noise.

[0116] Optionally, the denoising unit 64 is further configured to perform the following processing for each layer in the encoder of the diffusion model: obtaining the upper-layer control condition vector output by the previous coding layer of the coding layer corresponding to the current coding layer in the encoder of the control condition model; inputting the upper-layer coding feature vector and the upper-layer control condition vector into the current coding layer to obtain the coding feature vector output by the current coding layer, wherein the upper-layer coding feature vector is the coding feature vector output by the previous coding layer of the current coding layer, and when the current coding layer is the first layer of the encoder of the diffusion model, the input of the current coding layer is the continuous vector of the current step and the noisy data of the current step.

[0117] Optionally, the generation unit is further configured to perform the following processing for each layer in the encoder of the control condition model: obtaining the upper-layer coding feature vector output by the previous coding layer of the coding layer corresponding to the current coding layer in the encoder of the diffusion model; inputting the upper-layer control condition vector and the upper-layer coding feature vector into the current coding layer to obtain the control condition vector output by the current coding layer, wherein the upper-layer control condition vector is the control condition vector output by the previous coding layer of the current coding layer, and when the current coding layer is the first layer of the encoder of the control condition model, the input of the current coding layer is image information, the continuous vector of the current step and the noise-added data of the current step.

[0118] Optionally, the control condition model further includes an intermediate layer, which has the same structure as the intermediate layer in the diffusion model. The denoising unit 64 is also configured to perform the following processing for each layer of the intermediate layer of the diffusion model: obtain the first intermediate output of the previous layer output of the layer corresponding to the current layer in the intermediate layer of the control condition model; input the second intermediate output and the first intermediate output into the current layer to obtain the intermediate output of the current layer output, wherein the second intermediate output is the intermediate output of the previous layer output of the current layer, and when the current layer is the first layer of the intermediate layer of the diffusion model, the input of the current layer is the output of the encoder of the diffusion model.

[0119] Figure 7 A computing environment 710 is shown coupled to a user interface 750. The computing environment 710 may be part of a data processing server. The computing environment 710 includes a processor 720, a memory 730, and an input / output (I / O) interface 740.

[0120] The processor 720 generally controls the overall operation of the computing environment 710, such as operations associated with display, data acquisition, data communication, and image processing. The processor 720 may include one or more processors for executing instructions to perform all or some of the steps in the above-described methods. In addition, the processor 720 may include one or more modules that facilitate interaction between the processor 720 and other components. The processor may be a central processing unit (CPU), a microprocessor, a single-chip microcomputer, a graphics processing unit (GPU), etc.

[0121] The memory 730 is configured to store various types of data to support the operation of the computing environment 710. The memory 730 may include predetermined software 732. Examples of such data include instructions for any application or method operating on the computing environment 710, video data sets, image data, etc. The memory 730 may be implemented using any type of volatile or non-volatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0122] I / O interface 740 provides an interface between processor 720 and peripheral interface modules (e.g., keyboard, click wheel, buttons, etc.). Buttons may include, but are not limited to, a home button, a start scan button, and a stop scan button. I / O interface 740 may be coupled to an encoder and a decoder.

[0123] In an embodiment, the computing environment 710 may be implemented by one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs), FPGAs, GPUs, controllers, microcontrollers, microprocessors, or other electronic components for performing the above methods.

[0124] According to an embodiment of the present disclosure, an electronic device may be provided, which includes at least one memory and at least one processor, wherein a set of computer-executable instructions is stored in the at least one memory, and when the computer-executable instruction set is executed by the at least one processor, a method for processing image description text according to an embodiment of the present disclosure is executed.

[0125] As an example, the electronic device may be a PC, a tablet device, a personal digital assistant, a smartphone, or other device capable of executing the aforementioned set of instructions. Here, the electronic device 1000 is not necessarily a single electronic device, but may also be any collection of devices or circuits capable of executing the aforementioned instructions (or instruction sets) individually or in combination. The electronic device may also be part of an integrated control system or system manager, or may be configured as a portable electronic device that is interconnected locally or remotely (e.g., via wireless transmission) via an interface.

[0126] In addition, the electronic device may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device may be connected to each other via a bus and / or a network.

[0127] According to an embodiment of the present disclosure, a computer-readable storage medium may also be provided, wherein when the instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to execute the method for processing image description text according to an embodiment of the present disclosure. Examples of computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as a multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device configured to store a computer program and any associated data, data files and data structures in a non-transitory manner and provide the computer program and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be executed in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0128] According to an embodiment of the present disclosure, a computer program product is also provided, comprising computer instructions. When executed by a processor, the computer instructions implement the above-described method for processing image description text. In an embodiment, a computer program product is also provided, comprising, for example, multiple programs in memory 730. The multiple programs can be executed by processor 720 in computing environment 710 to perform the above-described method. For example, the computer program product can include a non-transitory computer-readable storage medium.

[0129] Unless otherwise specifically stated, the order of steps of the method according to the present disclosure is intended to be illustrative only, and the steps of the method according to the present disclosure are not limited to the order specifically described above, but can be changed according to actual circumstances. In addition, at least one of the steps of the method according to the present disclosure can be adjusted, combined, or deleted according to actual needs.

[0130] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the appended claims.

[0131] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for processing image description text, characterized in that: include: Performing word segmentation processing on the initial description text of the original image to obtain a word segmentation result of the initial description text; Inputting the word segmentation result into a text encoder to obtain a discrete vector and a continuous vector, wherein the discrete vector is a discrete embedding vector output by the word unit embedding layer of the text encoder, and the continuous vector is an embedding vector finally output by the text encoder; Performing a predetermined number of steps of noise addition and denoising processing on the continuous vector, wherein, in each step of the predetermined number of steps, the discrete vector of the current step is updated using the predicted noise of the current step and the random noise added to the current step, and the updated discrete vector is used as the discrete vector of the next step, and the continuous vector corresponding to the discrete vector of the next step is used as the continuous vector of the next step; In response to the completion of the predetermined number of steps of denoising and denoising processing, a final description text of the original image is determined based on the discrete vector updated in each step.

2. The processing method according to claim 1, characterized in that The performing noise addition and denoising processing on the continuous vector for a predetermined number of steps includes: At each of the predetermined number of steps, the following processing is performed: performing noise processing on the continuous vector of the current step based on the random noise of the current step to obtain the noisy data of the current step, wherein when the current step is the first step of the predetermined number of steps, the continuous vector of the current step is the continuous vector; Inputting the noisy data of the current step and the continuous vector of the current step into a diffusion model to obtain the predicted noise of the current step; Using the predicted noise of the current step, the random noise of the current step, the discrete vector of the current step and the original image, the discrete vector of the current step is updated and the updated discrete vector is used as the discrete vector of the next step; The discrete vector of the next step is input into the text encoder to obtain the continuous vector of the next step.

3. The processing method according to claim 2, characterized in that The updating of the discrete vector of the current step by using the predicted noise of the current step, the random noise of the current step, the discrete vector of the current step and the original image comprises: determining a noise prediction loss based on the predicted noise of the current step and the random noise of the current step; Determining a semantic loss based on the discrete vector of the current step and the original image; Determining a total loss based on the noise prediction loss and the semantic loss; By minimizing the total loss, the discrete vector of the current step is updated.

4. The processing method according to claim 1, wherein After updating the discrete vector of the current step using the predicted noise of the current step and the random noise added to the current step, the method includes: For each word vector in the updated discrete vector, obtaining the distance between the current word vector and each word vector in the predetermined vocabulary; determining the word vector with the smallest distance in the predetermined vocabulary as the final vector of the current word vector; Based on all final vectors, the final updated discrete vector of the current step is determined.

5. The processing method according to claim 2, characterized in that The step of determining the final description text of the original image based on the discrete vector updated in each step includes: Sample part of the discrete vector from the discrete vector updated at each step; For each discrete vector in the partial discrete vectors, performing denoising processing on any random noise for the predetermined number of steps using the continuous vector corresponding to the current discrete vector and the diffusion model to obtain an estimated image corresponding to the original image; The final description text of the original image is determined based on the discrete vector corresponding to the estimated image with the highest image quality.

6. The processing method according to claim 2, characterized in that The processing method further comprises: Acquiring image information associated with the original image; generating a control condition vector based on the image information; The step of inputting the noise-added data of the current step and the continuous vector of the current step into a diffusion model to obtain the predicted noise of the current step includes: The control condition vector, the noise-added data of the current step, and the continuous vector of the current step are input into the diffusion model to obtain the predicted noise of the current step.

7. The processing method according to claim 6, characterized in that The generating of a control condition vector based on the image information includes: Processing the image information using a control condition model to obtain the control condition vector, wherein the control condition model includes an encoder having the same structure as the encoder in the diffusion model; The step of inputting the control condition vector, the noise-added data of the current step, and the continuous vector of the current step into a diffusion model to obtain the predicted noise of the current step includes: Inputting the noisy data of the current step and the continuous vector of the current step into the encoder of the diffusion model to obtain an encoded feature vector; The encoded feature vector and the control condition vector are input into a decoder of the diffusion model to obtain the predicted noise.

8. The processing method according to claim 7, characterized in that The step of inputting the noise-added data of the current step and the continuous vector of the current step into the encoder of the diffusion model to obtain the encoded feature vector comprises: For each layer in the encoder of the diffusion model, the following processing is performed: Obtaining an upper layer control condition vector output by an upper coding layer of a coding layer corresponding to a current coding layer in an encoder of the control condition model; Inputting the upper layer coding feature vector and the upper layer control condition vector into the current coding layer to obtain the coding feature vector output by the current coding layer, The upper-layer coding feature vector is a coding feature vector output by the coding layer above the current coding layer. When the current coding layer is the first layer of the encoder of the diffusion model, the input of the current coding layer is the noisy data of the current step and the continuous vector of the current step.

9. The processing method according to claim 7, characterized in that The process of processing the image information using the control condition model to obtain the control condition vector includes: For each layer in the encoder of the control-conditional model, the following processing is performed: Obtaining an upper-layer coding feature vector output by an upper coding layer of a coding layer corresponding to a current coding layer in an encoder of the diffusion model; Inputting the upper layer control condition vector and the upper layer coding feature vector into the current coding layer to obtain the control condition vector output by the current coding layer, Among them, the upper-layer control condition vector is the control condition vector output by the previous coding layer of the current coding layer. When the current coding layer is the first layer of the encoder of the control condition model, the input of the current coding layer is the image information, the noise-added data of the current step and the continuous vector of the current step.

10. The processing method according to claim 7, characterized in that: The control condition model further includes an intermediate layer, and the intermediate layer has the same structure as the intermediate layer in the diffusion model. The processing method further includes: For each intermediate layer of the diffusion model, the following processing is performed: Obtain a first intermediate output of a previous layer output of a layer corresponding to the current layer in the intermediate layer of the control condition model; Input the second intermediate output and the first intermediate output into the current layer to obtain the intermediate output of the current layer, The second intermediate output is the intermediate output of the previous layer output of the current layer. When the current layer is the first layer of the intermediate layer of the diffusion model, the input of the current layer is the output of the encoder of the diffusion model.

11. A device for processing image description text, characterized in that: include: A word segmentation unit is configured to perform word segmentation processing on the initial description text of the original image to obtain a word segmentation result of the initial description text; an encoding unit configured to input the word segmentation result into a text encoder to obtain a discrete vector and a continuous vector, wherein the discrete vector is a discrete embedding vector output by a word unit embedding layer of the text encoder, and the continuous vector is an embedding vector finally output by the text encoder; a denoising unit configured to perform denoising and denoising processing on the continuous vector for a predetermined number of steps, wherein, at each step in the predetermined number of steps, the discrete vector of the current step is updated using the predicted noise of the current step and the random noise added to the current step, and the updated discrete vector is used as the discrete vector of the next step, and the continuous vector corresponding to the discrete vector of the next step is used as the continuous vector of the next step; The text determination unit is configured to determine the final description text of the original image based on the discrete vector updated in each step in response to the completion of the predetermined number of steps of denoising and denoising processing.

12. An electronic device, characterized in that: include: at least one processor; at least one memory storing computer-executable instructions, When the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to execute the method for processing image description text according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that When the instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to perform the method for processing image description text according to any one of claims 1 to 10.

14. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the method for processing image description text according to any one of claims 1 to 10 is implemented.

Images