Image conversion method, system, computer device and storage medium

Abstract

The present application relates to an image conversion method, system, computer device and storage medium. It includes: obtaining the image description text of the original image, and determining the original text embedding vector and the target text embedding vector according to the target description text and the image description text through a text encoder; determining the fused text embedding vector according to the original text embedding vector and the target text embedding vector through a deep neural network model; and performing image conversion on the original image according to the fused text embedding vector through a stable diffusion model to determine the target image. The above scheme can keep the characteristic structure of the original image as much as possible during the image conversion process of the original image, improve the accuracy of the image conversion result, and make the acquired target image have the desired visual perception.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image conversion method, system, computer device and storage medium. Background Art

[0002] In recent years, the rapid development of deep learning and artificial intelligence technology has had an impact on the field of image translation. As an important branch of computer vision, image translation aims to transform the content of an image into another image with different style, content or field, while preserving the content and structure of the source image as much as possible. The social application of image translation technology is becoming more and more extensive, and its importance is becoming increasingly prominent.

[0003] Domain transfer image translation is an important subtask of image translation, which aims to convert an image from the source domain to the target domain while keeping the content structure information of the original image unchanged. Currently, when the original image is transferred from the source domain to the target domain, a deep neural network is often used to convert the original image from the source domain to the target domain based on the image features of the original image and the description information of the target image. There is a defect that the generated target image is inconsistent with the visual perception of the original image. Therefore, how to ensure the accuracy of the image conversion result when the original image is transferred from the domain so that the obtained target image has the desired visual perception is a problem that needs to be solved. Summary of the invention

[0004] Based on this, it is necessary to provide a facial expression image conversion method, system, computer device and storage medium that can ensure the accuracy of the image conversion result when performing domain migration on the original image, so that the acquired target image has the desired visual perception, which is a problem that needs to be solved.

[0005] In a first aspect, the present application provides an image conversion method, the method comprising:

[0006] Get the image description text of the original image;

[0007] Determine, by a text encoder, a target text embedding vector according to the target description text, and determine an original text embedding vector according to the image description text;

[0008] Determine a fused text embedding vector according to the original text embedding vector and the target text embedding vector through a deep neural network model;

[0009] The original image is transformed according to the fused text embedding vector by using a stable diffusion model to determine a target image.

[0010] In one embodiment, obtaining the image description text of the original image includes:

[0011] Inputting an original image into an image description generator to determine a sub-image of the original image;

[0012] Converting the sub-image into an image embedding vector through a linear embedding vector layer of a neural network model;

[0013] Determining, by a visual encoder in an image description generator, image features of the original image according to the image embedding vector;

[0014] The image description text of the original image is determined according to the image features by a visual decoder in the image description generator.

[0015] In one embodiment, determining a target text embedding vector according to a target description text and determining an original text embedding vector according to the image description text by a text encoder includes:

[0016] Determine an original word sequence of the image description text and a target word sequence of the target description text;

[0017] Determine an original input embedding vector according to the original word-gram sequence, and determine a target input embedding vector according to the target word-gram sequence;

[0018] Determine an original embedding vector matrix according to the original input embedding vector, and determine a target embedding vector matrix according to the target input embedding vector;

[0019] Through the text encoder, an original text embedding vector is determined according to the original embedding vector matrix, and a target text embedding vector is determined according to the target embedding vector matrix.

[0020] In one embodiment, determining an original input embedding vector according to the original word-gram sequence, and determining a target input embedding vector according to the target word-gram sequence, comprises:

[0021] Determine an original word embedding vector and an original position embedding vector corresponding to the original word-meta sequence, and a target word embedding vector and a target position embedding vector corresponding to the target word-meta sequence;

[0022] Determine an original input embedding vector of the original word-meta sequence according to the original word embedding vector and the original position embedding vector;

[0023] A target input embedding vector for a target word sequence is determined according to the target word embedding vector and the target position embedding vector.

[0024] In one embodiment, determining a fused text embedding vector according to the original text embedding vector and the target text embedding vector through a deep neural network model includes:

[0025] Inputting the target text embedding vector into the deep neural network model, performing feature extraction on the target text embedding vector through the multi-head self-attention layer of the deep neural network model, and generating semantic features of the target text;

[0026] Inputting the original text embedding vector and the target text embedding vector into the cross attention layer of the deep neural network model to generate fused semantic information of the target text and fused semantic information of the image description text;

[0027] A fused text embedding vector of the original text embedding vector and the target text embedding vector is determined according to the fused semantic information of the target text and the fused semantic information of the image description text through the feedforward network layer of the deep learning network.

[0028] In one embodiment, performing image conversion on the original image according to the fused text embedding vector through a stable diffusion model to determine a target image includes:

[0029] Performing semantic segmentation on the original image according to the image description text through a multimodal pre-training model and an image classification model to determine a segmented image;

[0030] Determining a mask image of the segmented image according to pixel information of the segmented image;

[0031] The mask image is input into a stable diffusion model, a feature space vector is generated based on the fused text embedding vector, and the feature space vector is subjected to denoising processing by a denoising diffusion probability model to determine a target space vector;

[0032] The original image is converted according to the target space vector to determine a target image.

[0033] In one embodiment, determining a mask image of the segmented image according to pixel information of the segmented image includes:

[0034] Determining an image confidence of the segmented image according to pixel information of the segmented image, and determining a confidence matrix according to the image confidence;

[0035] Inversely process the matrix elements in the confidence matrix to determine the target matrix;

[0036] A mask map is determined according to the target matrix and the segmented image.

[0037] In a second aspect, the present application further provides an image conversion system, the image conversion system comprising:

[0038] Original image input interface, used to input original images;

[0039] Description text input interface, used to input target description text;

[0040] A target image display interface is used to display a target image; the target image is generated by: using a text encoder, determining an original text embedding vector and a target text embedding vector according to the target description text of the original image and the image description text; using a deep neural network model, determining a fused text embedding vector according to the original text embedding vector and the target text embedding vector; using a stable diffusion model, performing image conversion on the original image according to the fused text embedding vector to determine the target image.

[0041] In a third aspect, the present application further provides a computer device, the computer device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the following steps when executing the computer program:

[0042] Get the image description text of the original image;

[0043] Determine, by a text encoder, a target text embedding vector according to the target description text, and determine an original text embedding vector according to the image description text;

[0044] Determine a fused text embedding vector according to the original text embedding vector and the target text embedding vector through a deep neural network model;

[0045] The original image is transformed according to the fused text embedding vector by using a stable diffusion model to determine a target image.

[0046] In a fourth aspect, the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0047] Get the image description text of the original image;

[0048] Determine, by a text encoder, a target text embedding vector according to the target description text, and determine an original text embedding vector according to the image description text;

[0049] Determine a fused text embedding vector according to the original text embedding vector and the target text embedding vector through a deep neural network model;

[0050] The original image is transformed according to the fused text embedding vector by using a stable diffusion model to determine a target image.

[0051] The above-mentioned image conversion method, system, computer device and storage medium obtain the image description text of the original image, determine the target text embedding vector according to the target description text through the text encoder, and determine the original text embedding vector according to the image description text; determine the fused text embedding vector according to the original text embedding vector and the target text embedding vector through the deep neural network model; and perform image conversion on the original image according to the fused text embedding vector through the stable diffusion model to determine the target image. The problem that the image process is directly dominated by the text condition and the content and structural semantics of the original image are ignored, and the generated target image is inconsistent with the visual perception of the original image is solved. In the above-mentioned scheme, when performing image conversion on the original image, the fused text embedding vector is determined based on the image description text of the original image and the target description text of the target image to be generated, so that the fused text embedding vector can represent the text features of the image description text and the text features of the target description text, and the fused text embedding vector is used to guide the stable diffusion model to perform image conversion on the original image, which can keep the characteristic structure of the original image as much as possible during the image conversion of the original image, improve the accuracy of the image conversion result, and make the obtained target image have the expected visual perception. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 A diagram showing an application environment of an image conversion method in an embodiment;

[0053] Figure 2 is a schematic flow chart of an image conversion method in one embodiment;

[0054] Figure 3 is a flow chart of an image conversion method in another embodiment;

[0055] Figure 4 is a flow chart of an image conversion method in another embodiment;

[0056] Figure 5 is a flow chart of an image conversion method in another embodiment;

[0057] Figure 6 is a structural block diagram of an image conversion device in one embodiment;

[0058] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0060] The image conversion method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The server 104 obtains the image description text of the original image, and determines the original text embedding vector and the target text embedding vector according to the target description text and the image description text through the text encoder; through the deep neural network model, the fusion text embedding vector is determined according to the original text embedding vector and the target text embedding vector; through the stable diffusion model, the original image is converted according to the fusion text embedding vector, the target image is determined, and the target image is sent to the terminal 102 through the communication network. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablets, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 can be implemented with an independent server or a server cluster consisting of multiple servers.

[0061] In one embodiment, Figure 2 As shown, an image conversion method is provided. This embodiment takes the method applied to a terminal as an example for illustration. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0062] S210: Obtain image description text of the original image.

[0063] The original image refers to the image that needs to be domain migrated, and the image description text refers to the text information describing the image content in the original image. The image description text can be a phrase or sentence that describes the main object in the original image.

[0064] S220, determining a target text embedding vector according to the target description text through a text encoder, and determining an original text embedding vector according to the image description text.

[0065] The target description text refers to the description text of the target image that needs to be converted from the original image when performing image conversion on the original image. The original text embedding vector refers to the text embedding vector corresponding to the original image, and the target text embedding vector refers to the text embedding vector corresponding to the target text. The text embedding vector is a fixed-size real number vector obtained by converting the text. These vectors can capture the semantic information in the text, so that semantically similar texts have similar vector representations in the embedding vector space. ‌

[0066] Specifically, the original text embedding vector and the target text embedding vector may be determined according to the target description text and the image description text by using a text encoder and natural language processing technology.

[0067] S230. Determine a fused text embedding vector according to the original text embedding vector and the target text embedding vector through a deep neural network model.

[0068] Specifically, the original text embedding vector and the target text embedding vector can be input into a deep neural network model, and the original text embedding vector and the target text embedding vector can be fused through a semantic fusion network layer in the deep neural network model to determine a fused text embedding vector of the original text embedding vector and the target text embedding vector.

[0069] S240, performing image conversion on the original image according to the fused text embedding vector through a stable diffusion model to determine a target image.

[0070] Among them, the stable diffusion model is an image generation technology based on deep learning. It can generate high-quality and diverse images by simulating the gradual diffusion and recovery process of image data in noise. This model is similar to an intelligent painter that can "draw" realistic images from random noise, providing a powerful tool for image creation, editing and multi-field applications. For example, if a user inputs a text description or keyword of "summer beach" into the stable diffusion model, the model can capture the core features of this concept, such as blue sky, white clouds, golden sand and sparkling sea, and then generate a series of summer beach scene images that match it, each unique and rich in details.

[0071] Specifically, the fused text embedding vector is used as a guiding condition to guide the pre-trained stable diffusion model to perform image transformation on the original image to generate the target image.

[0072] In the above image conversion method, the image description text of the original image is obtained, and the target text embedding vector is determined according to the target description text through the text encoder, and the original text embedding vector is determined according to the image description text; the fused text embedding vector is determined according to the original text embedding vector and the target text embedding vector through the deep neural network model; and the original image is converted according to the fused text embedding vector through the stable diffusion model to determine the target image. The problem that the image process is directly dominated by the text condition and the content and structural semantics of the original image are ignored, and the generated target image is inconsistent with the visual perception of the original image is solved. In the above scheme, when the original image is converted, the fused text embedding vector is determined based on the image description text of the original image and the target description text of the target image to be generated, so that the fused text embedding vector can represent the text features of the image description text and the text features of the target description text, and the fused text embedding vector is used to guide the stable diffusion model to convert the original image. In the process of image conversion of the original image, the characteristic structure of the original image can be kept as much as possible, the accuracy of the image conversion result is improved, and the obtained target image has the expected visual perception.

[0073] In one embodiment, obtaining the image description text of the original image includes:

[0074] An original image is input into an image description generator to determine a sub-image of the original image; the sub-image is converted into an image embedding vector through a linear embedding vector layer of a neural network model; the image features of the original image are determined according to the image embedding vector through a visual encoder in the image description generator; and the image description text of the original image is determined according to the image features through a visual decoder in the image description generator.

[0075] Among them, the image description generator consists of two parts: a visual encoder and a visual text decoder. The visual encoder has a total of 12 layers, each of which consists of layer normalization, multi-head self-attention layer, layer normalization, and a multi-layer perceptron block. The multi-head self-attention layer is used to perform self-attention calculations on the image segments to capture the dependencies between different segments, so that the global information of the entire image can be encoded into the representation of each segment. In addition, with its ability to handle the relationship between long-distance pixels, it can better capture the semantic information of the image. The multi-layer perceptron block further learns the complex features of the image and adjusts the output dimension by extracting features from each image segment to meet subsequent processing. At the same time, the residual connection is used to better transmit information, reduce information loss, and alleviate the impact of gradient disappearance, so that the changes and important features of the input data can be learned more effectively.

[0076] The above scheme obtains the image description text of the original image through the neural network model and the image description generator, and can directly obtain accurate image description text based on the original image, saving manpower and improving the efficiency of obtaining image description text.

[0077] In one embodiment, Figure 3 As shown, determining the original text embedding vector and the target text embedding vector according to the target description text and the image description text includes:

[0078] S310: Determine an original word sequence of the image description text and a target word sequence of the target description text.

[0079] Among them, the word sequence is also called token sequence, which refers to the symbols used to represent words or phrases in the process of natural language processing.

[0080] Specifically, the image description text is segmented to determine the token sequence of the image description text, that is, the original word sequence; the target description text is segmented to determine the token sequence of the target description text, that is, the target word sequence.

[0081] S320: Determine an original input embedding vector according to the original word-unit sequence, and determine a target input embedding vector according to the target word-unit sequence.

[0082] S330. Determine an original embedding vector matrix according to the original input embedding vector, and determine a target embedding vector matrix according to the target input embedding vector.

[0083] Specifically, the original input embedding vectors are integrated to determine the original embedding vector matrix, and the target input embedding vectors are integrated to determine the target embedding vector matrix.

[0084] For example, the original embedding vector matrix is ​​shown in formula (1):

[0085] (1)

[0086] in, is the original embedding vector matrix.

[0087] S340, determining an original text embedding vector according to an original embedding vector matrix through a text encoder, and determining a target text embedding vector according to a target embedding vector matrix.

[0088] Specifically, the original embedding vector matrix is ​​input into the text encoder, and the vector relationship between each original input embedding vector in the original embedding vector matrix is ​​determined by the text encoder, and the multi-layer perceptron based on the attention mechanism is used to determine the vector relationship between each original input embedding vector in the original embedding vector matrix, and the original embedding vector is layer-normalized according to the vector relationship between each original input embedding vector, so as to determine the original text embedding vector. The target embedding vector matrix is ​​input into the text encoder, and the vector relationship between each target input embedding vector in the original embedding vector matrix is ​​determined by the text encoder, and the multi-layer perceptron based on the attention mechanism is used to determine the vector relationship between each target input embedding vector in the original embedding vector matrix, and the target embedding vector is layer-normalized according to the vector relationship between each target input embedding vector, so as to determine the target text embedding vector.

[0089] For example, the calculation formula of the original text embedding vector is shown in formula (2):

[0090] (2)

[0091] in, is the original text embedding vector, LN represents layer normalization, MLP represents multi-layer perceptron, and Attention represents attention mechanism. Represents the output data of the previous text encoder.

[0092] The above scheme determines the original input embedding vector based on the original word sequence, and determines the target input embedding vector based on the target word sequence, determines the original embedding vector matrix based on the original input embedding vector, and determines the target embedding vector matrix based on the target input embedding vector, determines the original text embedding vector based on the original embedding vector matrix, and determines the target text embedding vector based on the target embedding vector matrix, which can improve the reliability of the original text embedding vector and the target text embedding vector.

[0093] In one embodiment, determining an original input embedding vector according to an original word-gram sequence and determining a target input embedding vector according to a target word-gram sequence includes:

[0094] Determine the original word embedding vector and the original position embedding vector corresponding to the original word-meta sequence, as well as the target word embedding vector and the target position embedding vector corresponding to the target word-meta sequence; determine the original input embedding vector of the original word-meta sequence based on the original word embedding vector and the original position embedding vector; determine the target input embedding vector of the target word-meta sequence based on the target word embedding vector and the target position embedding vector.

[0095] Specifically, the obtained original word-meta sequence and target word-meta sequence are respectively mapped to a specific index according to a predefined vocabulary, so as to determine the token embedding vector and position embedding vector of the original word-meta sequence according to the mapped index, that is, the original word embedding vector and the original position embedding vector of the original word-meta sequence. The original word embedding vector of the original word-meta sequence and the position embedding vector of the original word-meta sequence are added together to determine the original input embedding vector of the original word-meta sequence. The target word embedding vector and the target position embedding vector of the target word-meta sequence are determined according to the mapped index. The target word embedding vector and the target position embedding vector of the target word-meta sequence are added together to determine the target input embedding vector of the target word-meta sequence.

[0096] Exemplarily, the calculation formula of the original input embedding vector is shown in formula (3):

[0097] (3)

[0098] in, is the i-th original input embedding vector, is the token embedding vector of the original word sequence, is the position embedding vector of the original word sequence.

[0099] The above scheme adds the original word embedding vector and the original position embedding vector to determine the original input embedding vector of the original word sequence, and adds the target word embedding vector and the target position embedding vector to determine the target input embedding vector of the target word sequence. This allows the original input embedding vector to represent the token embedding vector and the position embedding vector of the original word sequence, and the target input embedding vector to represent the token embedding vector and the position embedding vector of the target word sequence, thereby improving the reliability of the original input embedding vector and the target input embedding vector.

[0100] In one embodiment, Figure 4 As shown, through the deep neural network model, the fused text embedding vector is determined according to the original text embedding vector and the target text embedding vector, including:

[0101] S410, input the target text embedding vector into the deep neural network model, and determine the semantic features of the target text through the multi-head self-attention layer of the deep neural network model.

[0102] It should be noted that, in general, the descriptive information of the target text is more detailed. Therefore, inputting the target text embedding vector into the deep neural network model can better capture the complex semantic features of the target text through the multi-head self-attention layer of the deep neural network model, which is convenient for subsequent better understanding of the semantics of the target text.

[0103] S420, inputting the original text embedding vector and the semantic features of the target text into the cross-attention layer of the deep neural network model to generate fused semantic information of the image description text, and inputting the target text embedding vector and the semantic features of the target text into the cross-attention layer of the deep neural network model to generate fused semantic information of the target text.

[0104] Specifically, the original text embedding vector, the target text embedding vector and the semantic features of the target text are input into the cross-attention layer of the deep neural network model, the original text embedding vector is used as the query statement query, and the semantic features of the target text are used as the key value key-value. The cross-attention layer of the deep neural network model is guided by the semantic features of the target text to generate the fused semantic information of the image description text corresponding to the original text embedding vector and the fused semantic information of the target text corresponding to the target text embedding vector.

[0105] S430. Determine a fused text embedding vector of the original text embedding vector and the target text embedding vector according to the fused semantic information of the target text and the fused semantic information of the image description text through a feedforward network layer of the deep learning network.

[0106] The above scheme determines the fused semantic information of the target text and the original image text according to the semantic features of the original text embedding vector and the target text through the cross-attention layer of the deep neural network model, and then determines the fused text embedding vector of the original text embedding vector and the target text embedding vector according to the fused semantic information of the target text and the fused semantic information of the image description text in the feedforward network layer of the deep learning network, which can improve the accuracy of the original text embedding vector and the target text embedding vector.

[0107] In one embodiment, Figure 5 As shown, through the stable diffusion model, the original image is transformed according to the fused text embedding vector to determine the target image, including:

[0108] S501. Perform semantic segmentation on the original image according to the image description text through a multimodal pre-training model and an image classification model to determine a segmented image.

[0109] Among them, the multimodal pre-training model is the CLIP (Contrastive Language-Image Pre-Training) model. The CLIP model is a multimodal pre-training neural network. Its core idea is to use a large amount of paired data of images and texts for pre-training to learn the alignment relationship between images and texts. The image classification model is the ViT (‌Vision Transformer, image classification) model. The core idea of ​​the ViT model is to use the attention mechanism of the Transformer to model the relationship between image blocks. The attention mechanism allows the model to assign different weights according to the correlation between different elements in the input sequence. Through a multi-layer self-attention mechanism, the ViT model is able to encode and capture the relationship between image blocks, thereby achieving a global understanding of the image.

[0110] It should be noted that an image is composed of many pixels, and image semantic segmentation is to group pixels according to the different semantic meanings expressed in the image.

[0111] Specifically, the Transformer decoder in the multimodal pre-trained model determines the multimodal embedding vector according to the image description text of the original image, and the multimodal embedding vector is the CLIP embedding vector. The original image is input into the image classification model, the activation function of the middle layer of the image classification model is extracted, and the activation function of the middle layer is mapped to the decoder, and the decoder outputs the segmented image of the original image according to the text vector of the original image, the activation function of the middle layer and the multimodal embedding vector. On this basis, the original image is segmented into a foreground area and a background area according to the semantic information of the image description text. The foreground area refers to the main area that needs to be translated in the image conversion task, and the background area refers to the retainable area that does not need to be converted.

[0112] Exemplarily, the formula for determining the multimodal embedding vector is shown in formula (4):

[0113] (4)

[0114] in, is the multimodal embedding vector, represents the labeling function of the multimodal pre-trained model, is the mapping function of the multimodal embedding vector, that is, the function that maps the word sequence of the image description text to the multimodal embedding vector, is the position encoding function of the multimodal pre-trained model, and i is the position index of the word sequence of the image description text when the word sequence of the image description text is mapped to the multimodal embedding vector.

[0115] S502: Determine a mask image of the segmented image according to pixel information of the segmented image.

[0116] Specifically, the confidence of the segmented image is determined according to the pixel information of the segmented image, and the segmented image with the confidence is used as a mask image of the segmented image.

[0117] S503: input the mask image into the stable diffusion model, generate a feature space vector based on the fused text embedding vector, perform noise reduction processing on the feature space vector through the denoising diffusion probability model, and determine the target space vector.

[0118] It should be noted that the denoising diffusion probability model is a generative model that generates data by simulating diffusion and reverse diffusion in physical processes. This process consists of two main parts: the forward process and the reverse process, where the forward process is the denoising process and the reverse process is the denoising process. In the denoising process, the denoising diffusion probability model gradually introduces noise into the data until the data becomes completely noisy. This process can be regarded as a Markov chain, in which each step is deterministic and eventually reaches an equilibrium state, that is, the distribution of the data becomes a standard Gaussian distribution. For example, a user can gradually add random noise to a clear picture. As time goes by, the user continues to add noise, and the originally clear picture will become more and more blurred. Finally, the originally clear picture will completely become a piece of white noise. The denoising process is the inverse operation of the denoising process, and the goal is to recover the original data from the noise. This process is achieved by gradually predicting and removing noise. In practical applications, a neural network is usually used to predict the noise in each step, and then the predicted noise is subtracted from the current noisy image to restore the image of the previous time step. For example, if the user has a picture composed entirely of noise, it is actually obtained by gradually adding noise to a clear picture. The denoising process is to reverse this process, just like gradually blowing away the fog to make the picture clear again. The user can start from the noise, try to guess and remove some of the noise, and restore the next picture. By repeating this process and removing the noise step by step, the original clear picture can be restored.

[0119] Specifically, the mask map is input into the stable diffusion model as a structural condition. The stable diffusion model is used to convert the mask map from the pixel space corresponding to the mask map to a size matching the potential representation of the original image based on the fused text embedding vector. The convolutional neural network is then used to extract features from the converted mask map to determine the feature space vector corresponding to the mask map.

[0120] It should be noted that the convolutional neural network consists of a convolutional layer with a 3×3 convolution kernel and a 2×2 step size. The potential representation of the image is to compress the image data into a smaller representation through the autoencoder technology. The feature space vector is denoised through the denoising diffusion probability model to determine the target space vector.

[0121] S504: Perform image conversion on the original image according to the target space vector to determine the target image.

[0122] The above scheme uses a multimodal pre-trained model and an image classification model to perform semantic segmentation on the original image to determine the segmented image, uses a stable diffusion model, takes the fused text embedding vector as a condition, guides the stable diffusion model to translate the mask image, determines the feature space vector, and guides the original image to perform image translation through the denoised feature space vector to determine the target image, which can improve the accuracy of the target image.

[0123] In one embodiment, determining a mask image of the segmented image according to pixel information of the segmented image includes:

[0124] The image confidence of the segmented image is determined according to the pixel information of the segmented image, and the confidence matrix is ​​determined according to the image confidence; the matrix elements in the confidence matrix are inversely processed to determine the target matrix; and the mask map is determined according to the target matrix and the segmented image.

[0125] It should be noted that, since the semantic segmentation of the image is based on the description of the original image, the segmentation result shows that the foreground part is more prominent. In order to introduce the background area of ​​the segmentation map into the image translation process, it is necessary to let the background information retention module focus on the background part of the segmentation map rather than the foreground part. The image confidence of the segmented image can be determined based on the average pixel value of the pixels contained in the segmented image. The segmentation result of the original image is a mask map with confidence. The size of the confidence matrix is ​​related to the mask. Figure 1 It is necessary to transform the confidence matrix of the mask image to obtain a mask image with a new confidence to assist the subsequent image translation process to better preserve the background structure. That is, by inversely processing each matrix element of the confidence matrix and determining the target matrix based on the inversely processed matrix elements, the mask image can meet the requirements of subsequent processing. At this time, the mask image after semantic segmentation processing will have a higher confidence in the background area, while the pixels in the foreground area that need to be translated will be a low-confidence mask image.

[0126] Exemplarily, the matrix elements in the confidence matrix are inversely processed to determine the formula for the target matrix as shown in formula (5):

[0127] (5)

[0128] in, is the target matrix i Line j Matrix elements of columns, Represents the confidence matrix i Line j Matrix elements of the columns.

[0129] It should be noted that, based on the above embodiments, the backbone network U-Net of the stable diffusion model is composed of paired encoders and decoders of sizes 8×8, 16×16, 32×32 and 64×64.

[0130] The denoising diffusion probability model mainly calculates the loss function by using the backbone network to predict the noise of the sampled denoised image at a specific time step in the reverse process and the noise randomly selected from the Gaussian distribution. The calculation formula of the loss function is shown in formula (6):

[0131] (6)

[0132] in, refers to the denoised sample at time step t, Loss function calculated for the noise diffusion probabilistic model.

[0133] The model loss function of the stable diffusion model has a relatively good effect on the text image task, but does not meet the requirements of the image conversion task. Therefore, when training the stable diffusion model, the model loss function of the stable diffusion model is used as part of the total loss function in the model training process to ensure that the model output image of the stable diffusion model matches the fusion semantics corresponding to the fused text embedding vector, thereby strengthening the guiding role of the fused text embedding vector in the image conversion process. For example, the model loss function of the stable diffusion model is shown in formula (7):

[0134] (7)

[0135] in, is the model loss function of the stable diffusion model, and y refers to the text condition; Function that converts text into conditional vectors.

[0136] Furthermore, the mask image includes an image corresponding to the foreground area and an image corresponding to the background area. The mask image corresponding to the background area is used as a structural condition to retain the background area during the image conversion process. The model loss function of the stable diffusion model is adjusted based on the loss function determined by the noise diffusion probability model to determine the noise prediction loss function. The formula of the noise prediction loss function is shown in formula (8):

[0137] (8)

[0138] in, is the potential representation at time step t; refers to the target text condition; Refers to the background structure conditions.

[0139] In addition, in order to ensure that the target image and the target description text are aligned, the image encoder and text encoder of the multimodal pre-training model are used to extract the text features of the target description text and the image features of the target image obtained after the original image is converted. The degree of alignment between the target description text and the target image is determined according to the feature difference between the text features of the target description text and the image features of the target image, that is, the model loss function of the multimodal pre-training model. The calculation formula of the model loss function of the multimodal pre-training model is shown in formula (9):

[0140] (9)

[0141] in, is the model loss function of the multimodal pre-trained model, image encoder for multimodal pre-trained models, A text encoder for a multimodal pre-trained model, is the target image, Provides a descriptive text for the target.

[0142] In order to ensure that the background area of ​​the target image is as consistent as possible with the background area of ​​the original image, the background loss function is introduced as part of the total loss function of the model. The calculation formula of the background loss function is shown in formula (10):

[0143] (10)

[0144] in, represents the background loss function, Represents the pixel value of the target image at (i, j), represents the pixel value of the original image at (i, j), W represents the width range of the image pixels, and H represents the height range of the image pixels.

[0145] According to the noise prediction loss function, the background loss function and the model loss function of the multimodal pre-training model, the total loss function when training the stable diffusion model is determined as shown in formula (11):

[0146] (11)

[0147] in, is the total loss function, is the weight of the noise prediction loss function, is the weight of the model loss function of the multimodal pre-trained model, is the weight of the background loss function.

[0148] Exemplarily, based on the above embodiment, the image conversion method includes:

[0149] An original image is input into an image description generator to determine a sub-image of the original image; the sub-image is converted into an image embedding vector through a linear embedding vector layer of a neural network model; the image features of the original image are determined according to the image embedding vector through a visual encoder in the image description generator; and the image description text of the original image is determined according to the image features through a visual decoder in the image description generator.

[0150] The image description text is segmented to determine the token sequence of the image description text, that is, the original word sequence; the target description text is segmented to determine the token sequence of the target description text, that is, the target word sequence. According to the predefined vocabulary, the obtained original word sequence and target word sequence are respectively mapped to a specific index to determine the token embedding vector and position embedding vector of the original word sequence according to the mapped index, that is, the original word embedding vector and the original position embedding vector of the original word sequence. The original word embedding vector of the original word sequence and the position embedding vector of the original word sequence are added to determine the original input embedding vector of the original word sequence. The target word embedding vector and the target position embedding vector of the target word sequence are determined according to the mapped index. The target word embedding vector and the target position embedding vector of the target word sequence are added to determine the target input embedding vector of the target word sequence. The original input embedding vectors are integrated to determine the original embedding vector matrix, and the target input embedding vectors are integrated to determine the target embedding vector matrix. The original embedding vector matrix is ​​input into the text encoder. Through the text encoder, the multi-layer perceptron based on the attention mechanism determines the vector relationship between each original input embedding vector in the original embedding vector matrix. The original embedding vector is layer-normalized according to the vector relationship between each original input embedding vector to determine the original text embedding vector. The target embedding vector matrix is ​​input into the text encoder. Through the text encoder, the multi-layer perceptron based on the attention mechanism determines the vector relationship between each target input embedding vector in the original embedding vector matrix. The target embedding vector is layer-normalized according to the vector relationship between each target input embedding vector to determine the target text embedding vector.

[0151] The target text embedding vector is input into the deep neural network model, and the semantic features of the target text are determined through the multi-head self-attention layer of the deep neural network model. The original text embedding vector, the target text embedding vector and the semantic features of the target text are input into the cross-attention layer of the deep neural network model, the original text embedding vector is used as the query statement query, and the semantic features of the target text are used as the key-value. The cross-attention layer of the deep neural network model is guided by the semantic features of the target text to generate the fused semantic information of the image description text corresponding to the original text embedding vector, and the fused semantic information of the target text corresponding to the target text embedding vector. Through the feedforward network layer of the deep learning network, the fused text embedding vector of the original text embedding vector and the target text embedding vector is determined according to the fused semantic information of the target text and the fused semantic information of the image description text.

[0152] Through the Transformer decoder in the multimodal pre-trained model, the multimodal embedding vector is determined according to the image description text of the original image. The multimodal embedding vector is the CLIP embedding vector. The original image is input into the image classification model, the activation function of the middle layer of the image classification model is extracted, and the activation function of the middle layer is mapped to the decoder. The decoder outputs the segmented image of the original image according to the text vector of the original image, the activation function of the middle layer and the multimodal embedding vector. On this basis, the original image is segmented into a foreground area and a background area according to the semantic information of the image description text. The foreground area refers to the main area that needs to be translated in the image conversion task, and the background area refers to the retainable area that can be saved without image conversion.

[0153] The image confidence of the segmented image is determined according to the pixel information of the segmented image, and the confidence matrix is ​​determined according to the image confidence; the matrix elements in the confidence matrix are inversely processed to determine the target matrix; the mask map is determined according to the target matrix and the segmented image. The mask map is input into the stable diffusion model as a structural condition, and the mask map is converted from the pixel space corresponding to the mask map to a size matching the potential representation of the original image based on the fused text embedding vector through the stable diffusion model, and then the converted mask map is feature extracted through the convolutional neural network to determine the feature space vector corresponding to the mask map. The original image is guided to perform image translation through the denoised feature space vector to determine the target image.

[0154] In the above-mentioned image conversion method, the image description text of the original image is obtained, and the original text embedding vector and the target text embedding vector are determined according to the target description text and the image description text through the text encoder; the fused text embedding vector is determined according to the original text embedding vector and the target text embedding vector through the deep neural network model; and the target image is determined according to the fused text embedding vector by the stable diffusion model. The problem that the image process is directly dominated by the text condition and the content and structural semantics of the original image are ignored, and the generated target image is inconsistent with the visual perception of the original image is solved. In the above-mentioned scheme, when the original image is converted, the fused text embedding vector is determined based on the image description text of the original image and the target description text of the target image to be generated, so that the fused text embedding vector can represent the text features of the image description text and the text features of the target description text, and the fused text embedding vector is used to guide the stable diffusion model to convert the original image, so that the characteristic structure of the original image can be kept as much as possible during the image conversion of the original image, and the accuracy of the image conversion result is improved, so that the obtained target image has the expected visual perception.

[0155] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0156] Based on the same inventive concept, the embodiment of the present application also provides an image conversion device for implementing the above-mentioned image conversion method. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above-mentioned method, so the specific limitations in one or more image conversion device embodiments provided below can refer to the limitations of the image conversion method above, and will not be repeated here.

[0157] In one embodiment, an image conversion system is provided, comprising:

[0158] Original image input interface, used to input original images;

[0159] Description text input interface, used to input target description text;

[0160] The target image display interface is used to display the target image; the target image is generated by: using a text encoder, determining the original text embedding vector and the target text embedding vector according to the target description text and the image description text of the original image; using a deep neural network model, determining the fused text embedding vector according to the original text embedding vector and the target text embedding vector; using a stable diffusion model, performing image conversion on the original image according to the fused text embedding vector to determine the target image.

[0161] In one embodiment, Figure 6 As shown, an image conversion device is provided, comprising: an image description text determination module 601, a target text embedding determination module 602, a fusion text embedding determination module 603 and a target image determination module 604, wherein:

[0162] An image description text determination module 601 is used to obtain an image description text of an original image;

[0163] A target text embedding determination module 602 is used to determine a target text embedding vector according to a target description text through a text encoder, and to determine an original text embedding vector according to an image description text;

[0164] A fused text embedding determination module 603 is used to determine a fused text embedding vector according to the original text embedding vector and the target text embedding vector through a deep neural network model;

[0165] The target image determination module 604 is used to perform image conversion on the original image according to the fused text embedding vector through a stable diffusion model to determine the target image.

[0166] Exemplarily, the image description text determination module 601 is specifically used for:

[0167] Input the original image into the image description generator to determine the sub-image of the original image;

[0168] The sub-image is converted into an image embedding vector through a linear embedding vector layer of a neural network model;

[0169] Determine the image features of the original image based on the image embedding vector through the visual encoder in the image description generator;

[0170] The image description text of the original image is determined based on the image features through the visual decoder in the image description generator.

[0171] Exemplarily, the target text embedding determination module 602 is specifically used for:

[0172] Determine an original word sequence of the image description text and a target word sequence of the target description text;

[0173] Determine an original input embedding vector based on the original word-unit sequence, and determine a target input embedding vector based on the target word-unit sequence;

[0174] Determine an original embedding vector matrix based on the original input embedding vector, and determine a target embedding vector matrix based on the target input embedding vector;

[0175] Through the text encoder, the original text embedding vector is determined according to the original embedding vector matrix, and the target text embedding vector is determined according to the target embedding vector matrix.

[0176] Exemplarily, the target text embedding determination module 602 is further specifically configured to:

[0177] Determine the original word embedding vector and the original position embedding vector corresponding to the original word-meta sequence, and the target word embedding vector and the target position embedding vector corresponding to the target word-meta sequence;

[0178] Determine an original input embedding vector of the original word-unit sequence according to the original word embedding vector and the original position embedding vector;

[0179] The target input embedding vector of the target word sequence is determined according to the target word embedding vector and the target position embedding vector.

[0180] Exemplarily, the fusion text embedding determination module 603 is specifically used for:

[0181] The target text embedding vector is input into the deep neural network model, and the feature of the target text embedding vector is extracted through the multi-head self-attention layer of the deep neural network model to generate the semantic features of the target text;

[0182] Input the original text embedding vector and the semantic features of the target text into the cross attention layer of the deep neural network model to generate the fused semantic information of the image description text; input the target text embedding vector and the semantic features of the target text into the cross attention layer of the deep neural network model to generate the fused semantic information of the target text;

[0183] Through the feedforward network layer of the deep learning network, a fused text embedding vector of the original text embedding vector and the target text embedding vector is determined according to the fused semantic information of the target text and the fused semantic information of the image description text.

[0184] Exemplarily, the target image determination module 604 is specifically used for:

[0185] Through the multimodal pre-training model and image classification model, the original image is semantically segmented according to the image description text to determine the segmented image;

[0186] Determine a mask image of the segmented image according to pixel information of the segmented image;

[0187] The mask image is input into the stable diffusion model, a feature space vector is generated based on the fused text embedding vector, and the feature space vector is denoised by the denoising diffusion probability model to determine the target space vector;

[0188] The original image is transformed according to the target space vector to determine the target image.

[0189] Exemplarily, the target image determination module 604 is further specifically configured to:

[0190] Determining an image confidence of the segmented image according to pixel information of the segmented image, and determining a confidence matrix according to the image confidence;

[0191] Inversely process the matrix elements in the confidence matrix to determine the target matrix;

[0192] Determine the mask map based on the target matrix and the segmentation image.

[0193] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 7 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be realized through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, an image conversion method is realized. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0194] Those skilled in the art will understand that Figure 7The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0195] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0196] Step 1: Obtain image description text of the original image;

[0197] Step 2: Determine the target text embedding vector based on the target description text through the text encoder, and determine the original text embedding vector based on the image description text;

[0198] Step 3: Determine the fused text embedding vector based on the original text embedding vector and the target text embedding vector through a deep neural network model;

[0199] Step 4: Through the stable diffusion model, the original image is transformed according to the fused text embedding vector to determine the target image.

[0200] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0201] Step 1: Obtain image description text of the original image;

[0202] Step 2: Determine the target text embedding vector based on the target description text through the text encoder, and determine the original text embedding vector based on the image description text;

[0203] Step 3: Determine the fused text embedding vector based on the original text embedding vector and the target text embedding vector through a deep neural network model;

[0204] Step 4: Through the stable diffusion model, the original image is transformed according to the fused text embedding vector to determine the target image.

[0205] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0206] Step 1: Obtain image description text of the original image;

[0207] Step 2: Determine the target text embedding vector based on the target description text through the text encoder, and determine the original text embedding vector based on the image description text;

[0208] Step 3: Determine the fused text embedding vector based on the original text embedding vector and the target text embedding vector through a deep neural network model;

[0209] Step 4: Through the stable diffusion model, the original image is transformed according to the fused text embedding vector to determine the target image.

[0210] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0211] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded vector non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0212] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0213] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An image conversion method, characterized in that: include: Get the image description text of the original image; Determine, by a text encoder, a target text embedding vector according to the target description text, and determine an original text embedding vector according to the image description text; Determine a fused text embedding vector according to the original text embedding vector and the target text embedding vector through a deep neural network model; determine a fused text embedding vector according to the original text embedding vector and the target text embedding vector through a deep neural network model, including: inputting the target text embedding vector into the deep neural network model, performing feature extraction on the target text embedding vector through the multi-head self-attention layer of the deep neural network model, and generating semantic features of the target text; inputting the original text embedding vector and the semantic features of the target text into the cross-attention layer of the deep neural network model to generate fused semantic information of the image description text, inputting the target text embedding vector and the semantic features of the target text into the cross-attention layer of the deep neural network model to generate fused semantic information of the target text; determine a fused text embedding vector of the original text embedding vector and the target text embedding vector according to the fused semantic information of the target text and the fused semantic information of the image description text through the feedforward network layer of the deep learning network; The original image is transformed according to the fused text embedding vector by using a stable diffusion model to determine a target image.

2. The method according to claim 1, characterized in that Get the image description text of the original image, including: Inputting an original image into an image description generator to determine a sub-image of the original image; Converting the sub-image into an image embedding vector through a linear embedding vector layer of a neural network model; Determining, by a visual encoder in an image description generator, image features of the original image according to the image embedding vector; The image description text of the original image is determined according to the image features by a visual decoder in the image description generator.

3. The method according to claim 1, characterized in that Determining a target text embedding vector according to the target description text, and determining an original text embedding vector according to the image description text, including: Determine an original word sequence of the image description text and a target word sequence of the target description text; Determine an original input embedding vector according to the original word-gram sequence, and determine a target input embedding vector according to the target word-gram sequence; Determine an original embedding vector matrix according to the original input embedding vector, and determine a target embedding vector matrix according to the target input embedding vector; Through the text encoder, an original text embedding vector is determined according to the original embedding vector matrix, and a target text embedding vector is determined according to the target embedding vector matrix.

4. The method according to claim 3, characterized in that Determining an original input embedding vector according to the original word-gram sequence, and determining a target input embedding vector according to the target word-gram sequence, including: Determine an original word embedding vector and an original position embedding vector corresponding to the original word-meta sequence, and a target word embedding vector and a target position embedding vector corresponding to the target word-meta sequence; Determine an original input embedding vector of the original word-meta sequence according to the original word embedding vector and the original position embedding vector; A target input embedding vector for a target word sequence is determined according to the target word embedding vector and the target position embedding vector.

5. The method according to claim 1, characterized in that The method of performing image conversion on the original image according to the fused text embedding vector by using a stable diffusion model to determine a target image includes: Performing semantic segmentation on the original image according to the image description text through a multimodal pre-training model and an image classification model to determine a segmented image; Determining a mask image of the segmented image according to pixel information of the segmented image; Inputting the mask image into the stable diffusion model, generating a feature space vector based on the fused text embedding vector, performing noise reduction processing on the feature space vector through a denoising diffusion probability model, and determining a target space vector; The original image is converted according to the target space vector to determine a target image.

6. The method according to claim 5, characterized in that Determining a mask image of the segmented image according to pixel information of the segmented image includes: Determining an image confidence of the segmented image according to pixel information of the segmented image, and determining a confidence matrix according to the image confidence; Inversely process the matrix elements in the confidence matrix to determine the target matrix; A mask map is determined according to the target matrix and the segmented image.

7. An image conversion system, characterized in that: The image conversion system comprises: Original image input interface, used to input original images; Description text input interface, used to input target description text; A target image display interface is used to display a target image; the target image is generated by: determining an original text embedding vector and a target text embedding vector according to a target description text of an original image and the image description text through a text encoder; determining a fused text embedding vector according to the original text embedding vector and the target text embedding vector through a deep neural network model; determining a fused text embedding vector according to the original text embedding vector and the target text embedding vector through a deep neural network model, including: inputting the target text embedding vector into the deep neural network model, extracting features of the target text embedding vector through a multi-head self-attention layer of the deep neural network model, and generating semantic features of the target text ; Input the original text embedding vector and the semantic features of the target text into the cross-attention layer of the deep neural network model to generate fused semantic information of the image description text; input the target text embedding vector and the semantic features of the target text into the cross-attention layer of the deep neural network model to generate fused semantic information of the target text; determine the fused text embedding vector of the original text embedding vector and the target text embedding vector according to the fused semantic information of the target text and the fused semantic information of the image description text through the feedforward network layer of the deep learning network; perform image conversion on the original image according to the fused text embedding vector through the stable diffusion model to determine the target image.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images