Image style translation method, system, computer device and storage medium

Abstract

The present application relates to an image style translation method, system, computer device and storage medium. It includes: inputting a source image into a stable diffusion model to determine the image potential representation of the source image; inputting the source image into a target encoder to determine the original potential vector; processing the original potential vector through an instance normalization layer of a deep neural network model to determine a normalized potential vector; determining a time step embedding vector of the source image, and determining a target potential vector through the deep neural network according to the normalized potential vector and the time step embedding vector; extracting style features of a target reference image based on the time step embedding vector through a deep learning image recognition model to determine the style embedding vector of the target reference image; and generating a target translation image through a stable diffusion model according to the image potential representation, the target potential vector and the style embedding vector. The accuracy of image style translation of the source image is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image style translation method, system, computer device and storage medium. Background Art

[0002] Image style translation, as an important branch in the field of computer vision, aims to transform the content of an image into another image with different style, content or domain, while preserving the content and structure of the source image as much as possible. Image style translation technology has a wide range of applications in industries such as medicine, entertainment, urban planning and autonomous driving.

[0003] In recent years, the stylized image style translation method based on deep learning has been more and more widely used in many fields. In particular, the multimodal style transfer method based on the diffusion model has become a research hotspot. This method combines the powerful generation ability of the diffusion model and the richness of multimodal information, provides richer guidance for style transfer, and further improves the generation quality of stylized translation. With the development of image style translation technology, the social application of image style translation technology is becoming more and more extensive, and its importance is becoming increasingly significant. In the medical field, image style translation technology is used to translate the style of medical images into a format that is easier to analyze, assisting doctors in making more accurate disease diagnoses. In the entertainment industry, this technology can generate works of art with a specific style, providing users with a unique visual experience. At present, when translating the image style of the source image, the image description text of the target translation image is often used to guide the image style translation of the source image. However, the inventors found that the new image obtained by translating the source image according to the image description text provided by the user often cannot fully meet the user's expected requirements. Therefore, how to improve the accuracy of image style translation of the source image so that the obtained target translation image meets the user's expected requirements is a problem that needs to be solved. Summary of the invention

[0004] Based on this, it is necessary to provide an image style translation method, system, computer device and storage medium that can improve the accuracy of image style translation of source images and make the acquired target translation image meet the user's expected requirements in response to the above technical problems.

[0005] In a first aspect, the present application provides an image style translation method, the method comprising:

[0006] Get the image latent representation, normalized latent vector, and time-step embedding vector of the source image;

[0007] Determining a target latent vector based on the normalized latent vector and the time step embedding vector through a deep neural network model;

[0008] Extracting style features of a target reference image based on the time step embedding vector through a deep learning image recognition model to determine a style embedding vector of the target reference image;

[0009] A target translated image is generated according to the image latent representation, the target latent vector and the style embedding vector by a stable diffusion model.

[0010] In one embodiment, obtaining an image latent representation, a normalized latent vector, and a time step embedding vector of a source image includes:

[0011] Inputting a source image into a stable diffusion model, and converting the source image into a candidate latent representation in a latent space using a variational autoencoder of a backbone part of the stable diffusion model;

[0012] Performing noise processing on the candidate latent representation to determine the image latent representation of the source image;

[0013] Inputting the source image into a target encoder to determine an original latent vector;

[0014] Processing the original latent vector through an instance normalization layer of a deep neural network model to determine a normalized latent vector;

[0015] The time step of the source image is obtained, and the time step of the source image is converted into a high-dimensional position encoding vector, and the high-dimensional position encoding vector is converted into a time step embedding vector through a fully connected layer of a deep neural network model.

[0016] In one embodiment, determining a target latent vector according to the normalized latent vector and the time step embedding vector by a deep neural network model includes:

[0017] Processing the time step embedding vector by a multi-layer perceptron to determine a weight matrix and a bias vector;

[0018] Performing element-by-element multiplication of the normalized latent vector and the weight matrix to obtain multiplied data, and performing element-by-element addition of the multiplied data and the bias vector to obtain a candidate latent vector;

[0019] The candidate latent vector is input into a point-by-point convolutional layer of a deep neural network to determine a target latent vector.

[0020] In one embodiment, the candidate latent vector is input into a point-by-point convolutional layer of a deep neural network to determine a target latent vector, including:

[0021] Inputting the candidate potential vector into a point-by-point convolution layer in a deep neural network to determine a convolution processing vector;

[0022] Performing activation processing on the convolution processing vector through an activation layer function to determine an activation processing vector;

[0023] The activation processing vector is input into a point-by-point convolutional layer in the deep neural network to determine a target latent vector.

[0024] In one embodiment, extracting style features of a target reference image based on the time step embedding vector by using a deep learning image recognition model to determine the style embedding vector of the target reference image includes:

[0025] A deep learning image recognition model is used to extract style features of the target reference image and determine the style feature vector of the target reference image;

[0026] The style feature vector and the time step embedding vector of the source image are dimensionally matched, and the style embedding vector of the target reference image is determined according to the matching result.

[0027] In one embodiment, a deep learning image recognition model is used to extract style features of a target reference image to determine a style feature vector of the target reference image, including:

[0028] Inputting a target reference image into a deep learning image recognition model, processing the target reference image through a target activation function layer corresponding to each target convolution block in the deep learning image recognition model, and determining a target style feature image corresponding to each target convolution block, respectively; wherein the number of the target convolution blocks is five;

[0029] The convolution layer filters in the deep learning image recognition model are used to extract feature vectors of the target style feature images corresponding to each target convolution block, and the style feature vector of the target reference image is determined.

[0030] In one embodiment, generating a target translated image according to the image latent representation, the target latent vector and the style embedding vector by using the stable diffusion model includes:

[0031] Concatenating the image potential representation and the target potential vector through a generative backbone in a deep neural network model to determine a concatenated potential vector;

[0032] Determine a denoised latent vector by using the style embedding vector to guide the concatenated latent vector through a decoder of a stable diffusion model backbone;

[0033] The denoised latent vector is input into a decoder of a stable diffusion model backbone to determine a target translation image.

[0034] In a second aspect, the present application further provides an image style translation system, the image style translation system comprising:

[0035] Source image input interface, used to input source images;

[0036] Reference image input interface, used to input reference images;

[0037] A target translation image display interface is used to display a target translation image; the target translation image is generated by: obtaining an image potential representation, a normalized potential vector and a time step embedding vector of a source image; determining a target potential vector according to the normalized potential vector and the time step embedding vector through a deep neural network model; extracting style features of a target reference image based on the time step embedding vector through a deep learning image recognition model to determine a style embedding vector of the target reference image; and generating a target translation image according to the image potential representation, the target potential vector and the style embedding vector through a stable diffusion model.

[0038] In a third aspect, the present application further provides a computer device, the computer device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the following steps when executing the computer program:

[0039] Get the image latent representation, normalized latent vector, and time-step embedding vector of the source image;

[0040] Determining a target latent vector based on the normalized latent vector and the time step embedding vector through a deep neural network model;

[0041] Extracting style features of a target reference image based on the time step embedding vector through a deep learning image recognition model to determine a style embedding vector of the target reference image;

[0042] A target translated image is generated according to the image latent representation, the target latent vector and the style embedding vector by the stable diffusion model.

[0043] In a fourth aspect, the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0044] Get the image latent representation, normalized latent vector, and time-step embedding vector of the source image;

[0045] Determining a target latent vector based on the normalized latent vector and the time step embedding vector through a deep neural network model;

[0046] Extracting style features of a target reference image based on the time step embedding vector through a deep learning image recognition model to determine a style embedding vector of the target reference image;

[0047] A target translated image is generated according to the image latent representation, the target latent vector and the style embedding vector by the stable diffusion model.

[0048] The above-mentioned image style translation method, system, computer device and storage medium obtain the image latent representation, normalized latent vector and time step embedding vector of the source image; determine the target latent vector according to the normalized latent vector and the time step embedding vector through a deep neural network model; extract the style feature of the target reference image based on the time step embedding vector through a deep learning image recognition model to determine the style embedding vector of the target reference image; generate the target translation image according to the image latent representation, the target latent vector and the style embedding vector through a stable diffusion model. The problem that when the image style translation of the source image is guided by the image text description of the target translation image, the new image obtained after the translation process may not fully meet the user's expected requirements due to the inherent fuzziness or ambiguity of the text description is solved. The above-mentioned scheme, when performing image style translation on the source image, first performs image processing on the source image to obtain the target latent vector corresponding to the image latent representation and the time step embedding vector of the source image, and then guides the image latent representation and the target latent vector to perform styled image conversion according to the style embedding vector of the target reference image, and determines the target translation image corresponding to the source image, which can improve the accuracy of image style translation of the source image and make the obtained target translation image meet the user's expected requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A diagram showing an application environment of an image style translation method in one embodiment;

[0050] Figure 2 is a schematic diagram of a flow chart of an image style translation method in one embodiment;

[0051] Figure 3 is a flowchart of an image style translation method in another embodiment;

[0052] Figure 4 is a flowchart of an image style translation method in another embodiment;

[0053] Figure 5 A flowchart of an image style translation method in another embodiment is shown in FIG.

[0054] Figure 6 is a structural block diagram of an image style translation device in one embodiment;

[0055] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0057] As shown in the background technology, in the prior art, the new image obtained by translating the source image according to the image description text provided by the user often fails to fully meet the user's expected requirements. The inventors have found that the reason for this problem is that in the prior art, when performing image style translation on the source image, a deep learning network is often used to perform image style translation based on the image description text of the new image that the user expects to obtain. In the stylized image style translation task, the image text description has inherent fuzziness or ambiguity, which inevitably leads to the fact that the translated new image often fails to fully meet the user's expected requirements.

[0058] Based on this, an embodiment of the present application provides an image style translation method, which performs image processing on a source image to obtain an image latent representation of the source image and a target latent vector corresponding to a time-step embedding vector, and then guides the image latent representation and the target latent vector to perform stylized image conversion based on the style embedding vector of the target reference image, and determines a target translation image corresponding to the source image. This solves the problem that when the source image is translated for image style guidance through an image text description of the target translation image, the new image obtained through the translation process may not fully meet the user's expected requirements due to the inherent ambiguity or ambiguity of the text description.

[0059] Specifically, the image style translation method can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The server 104 inputs the source image into the stable diffusion model to determine the image potential representation of the source image; inputs the source image into the target encoder to determine the original potential vector; processes the original potential vector through the instance normalization layer of the deep neural network model to determine the normalized potential vector; determines the time step embedding vector of the source image, and determines the target potential vector according to the normalized potential vector and the time step embedding vector through the deep neural network; extracts the style features of the target reference image based on the time step embedding vector through the deep learning image recognition model, and determines the style embedding vector of the target reference image; generates a target translation image through the stable diffusion model according to the image potential representation, the target potential vector and the style embedding vector, and sends the target translation image to the terminal 102 through the communication network. The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0060] In one embodiment, Figure 2 As shown, a method for image style translation is provided. This embodiment uses the method applied to a terminal as an example for illustration. It is understandable that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0061] S210, obtaining an image latent representation, a normalized latent vector, and a time step embedding vector of a source image.

[0062] The source image refers to the image that needs to be translated. The image potential representation refers to the internal knowledge learned through training data in the machine learning model. This knowledge is stored in the model architecture in the form of weights and biases, representing a compressed form of the input data and containing the most important features required for the model to perform the task. The potential representation is the result of the model trying to understand the potential structure or pattern in the data.

[0063] Specifically, the source image is input into the pre-trained stable diffusion model, and the source image is compressed from the pixel dimension space to the latent space by the variational autoencoder of the stable diffusion model to determine the image latent representation of the source image. The source image is input into the target encoder with the same structure as the variational autoencoder of the pre-trained stable diffusion model, and the pixels of the source image are compressed into points in the latent space by the target encoder, and the points in the latent space obtained after compression are used as the original latent vector. The original latent vector is processed by the instance normalization layer of the deep neural network model to determine the normalized latent vector, and at the same time, the time step embedding vector of the source image is obtained.

[0064] The target encoder is an encoder with the same structure as the variational autoencoder of the pre-trained stable diffusion model. The ‌original latent vector‌ refers to a low-dimensional vector used to represent image features in image generation models. These low-dimensional vectors are usually located in a low-dimensional space called the latent space, where each image corresponds to a point. In the latent space, similar points may represent images with similar content or style.

[0065] S220. Determine a target latent vector according to the normalized latent vector and the time step embedding vector through a deep neural network.

[0066] When processing the source image, the time range is divided into continuous small time intervals, each of which is called a time step. A time step embedding vector is a unique vector assigned to each time step in a sequence in time series analysis. This vector is used to preserve the order information of the time steps, so that the model can understand the relative position relationship between different time steps in the time series.

[0067] S230. Using a deep learning image recognition model, extract style features of the target reference image based on the time step embedding vector, and determine the style embedding vector of the target reference image.

[0068] The target reference image refers to the target reference image of the target translated image that is desired to be obtained when performing image style translation on the source image, that is, the image style of the target reference image is consistent with the image style of the target translated image that the user expects to obtain.

[0069] S240, generating a target translated image according to the image latent representation, the target latent vector and the style embedding vector through a stable diffusion model.

[0070] Specifically, the image latent representation, the target latent vector and the style embedding vector are input into the stable diffusion model. Through the decoder of the stable diffusion model, the style embedding vector is used to guide the image latent representation and the target latent vector to implicitly stylize the image and generate the target translated image.

[0071] In the above image style translation method, the image latent representation, normalized latent vector and time step embedding vector of the source image are obtained; the target latent vector is determined according to the normalized latent vector and the time step embedding vector through a deep neural network model; the style feature of the target reference image is extracted based on the time step embedding vector through a deep learning image recognition model to determine the style embedding vector of the target reference image; the target translation image is generated according to the image latent representation, the target latent vector and the style embedding vector through a stable diffusion model. The problem that when the image style translation of the source image is guided by the image text description of the target translation image, the new image obtained after the translation process may not fully meet the user's expected requirements due to the inherent ambiguity or ambiguity of the text description is solved. In the above scheme, when the source image is translated into an image style, the source image is first processed to obtain the target latent vector corresponding to the image latent representation and the time step embedding vector of the source image, and then the image latent representation and the target latent vector are guided to perform styled image conversion according to the style embedding vector of the target reference image, and the target translation image corresponding to the source image is determined, which can improve the accuracy of the image style translation of the source image and make the obtained target translation image meet the user's expected requirements.

[0072] In one embodiment, obtaining an image latent representation, a normalized latent vector, and a time step embedding vector of a source image includes:

[0073] The source image is input into the stable diffusion model, and the variational autoencoder of the backbone of the stable diffusion model is used to convert the source image into a candidate latent representation in the latent space; the candidate latent representation is denoised to determine the image latent representation of the source image; the source image is input into the target encoder to determine the original latent vector; the original latent vector is processed through the instance normalization layer of the deep neural network model to determine the normalized latent vector; the time step of the source image is obtained and converted into a high-dimensional position encoding vector, and the high-dimensional position encoding vector is converted into a time step embedding vector through the fully connected layer of the deep neural network model.

[0074] Among them, the variational autoencoder of the backbone part of the stable diffusion model can be a variational autoencoder with a sampling factor of 8.

[0075] Specifically, the source image is input into the stable diffusion model, and the variational autoencoder in the backbone of the stable diffusion model compresses the source image from the pixel dimension space to the latent space to obtain a candidate latent representation in the latent space, and then a series of noise addition processes are performed on the candidate latent representation to determine the image latent representation of the source image. The original latent vector is input into the instance normalization layer of the deep neural network model, and the mean and variance of each feature channel of the original latent vector in the instance normalization layer are calculated. The normalized latent vector is determined by the mean and variance corresponding to each feature channel obtained, and then the time step of the source image is obtained, and the time step of the source image is processed by sine transform and cosine transform, and the time step of the source image is converted into a high-dimensional position encoding vector. The high-dimensional position encoding vector is mapped to a suitable dimensional space through the fully connected layer of the deep neural network model to convert the high-dimensional position encoding vector into a time step embedding vector. Exemplarily, the normalized latent vector is obtained as shown in formula (1):

[0076] (1)

[0077] in, is the original potential vector with position coordinates (h, w) in the feature channel identified as c, is the mean of the original latent vector in the feature channel identified as c, is the variance of the original latent vector in the feature channel identified as c, and are learnable parameters.

[0078] The above scheme performs noise processing on the candidate latent representations of the source image in the latent space to determine the image latent representation of the source image, so that the image latent representation of the source image can characterize the image features of the source image while facilitating subsequent image conversion of the source image.

[0079] In one embodiment, determining a target latent vector according to the normalized latent vector and the time step embedding vector by the deep neural network includes:

[0080] The time step embedding vector is processed by a multi-layer perceptron to determine the weight matrix and the bias vector; the normalized latent vector is multiplied element-by-element with the weight matrix to obtain the multiplied data, and the multiplied data is added element-by-element with the bias vector to obtain the candidate latent vector; the candidate latent vector is input into the point-by-point convolution layer of the deep neural network to determine the target latent vector.

[0081] Among them, the multilayer perceptron is a feedforward artificial neural network model with a multilayer structure composed of multiple neurons. It is usually used to solve classification and regression problems. The basic structure of the multilayer perceptron includes an input layer, an output layer and at least one or more hidden layers.

[0082] Specifically, the time step embedding vector of the source image is determined, and the time step embedding vector is processed by a multi-layer perceptron to obtain a weight matrix and a bias vector. The calculation formula for calculating the candidate latent vector based on the normalized latent vector, the weight matrix and the bias vector is shown in formula (2):

[0083] (2)

[0084] Among them, z is the candidate potential vector, z IN is the normalized latent vector, w is the weight matrix, and b is the bias vector.

[0085] The above scheme calculates the candidate latent vector according to the normalized latent vector, the weight matrix and the bias vector, processes the candidate latent vector according to the point-by-point convolution layer of the deep neural network, determines the target latent vector, and can improve the reliability of the target latent vector.

[0086] In one embodiment, Figure 3 As shown, the candidate latent vector is input into the point-by-point convolution layer of the deep neural network to determine the target latent vector, including:

[0087] S310, inputting the candidate potential vector into a point-by-point convolution layer in a deep neural network to determine a convolution processing vector.

[0088] Specifically, the candidate latent vector is input into a 1×1 convolution layer in a deep neural network, i.e., a point-by-point convolution layer, to degrade the image style information corresponding to the target latent vector and determine the convolution processing vector.

[0089] S320, performing activation processing on the convolution processing vector through an activation layer function to determine an activation processing vector.

[0090] Specifically, the convolution processing vector is input into the activation layer function to enhance the image content features in the convolution processing vector and determine the activation processing vector.

[0091] S330: Input the activation processing vector into the point-by-point convolution layer in the deep neural network to determine the target potential vector.

[0092] Specifically, the activation processing vector is input into the point-by-point convolution layer of the deep neural network. The channel dimension of the activation processing vector is reduced during the point-by-point convolution process, thereby eliminating the style information of the source image in the activation processing vector, so as to optimize and adjust the representation of the source image content by the activation processing vector and determine the target latent vector.

[0093] The above scheme, in the process of processing the candidate latent vector to determine the target latent vector, can retain the content information of the source image contained in the candidate latent vector, and at the same time remove the style information of the source image in the candidate latent vector, which can improve the accuracy of subsequent image style translation of the source image.

[0094] In one embodiment, Figure 4 As shown, through the deep learning image recognition model, the style feature of the target reference image is extracted based on the time step embedding vector, and the style embedding vector of the target reference image is determined, including:

[0095] S410: Use a deep learning image recognition model to extract style features of the target reference image and determine a style feature vector of the target reference image.

[0096] The deep learning image recognition model may be a VGG19 model, which is a convolutional neural network model including 16 convolutional layers and 3 fully connected layers. Style feature vector is a mathematical representation method used to describe and quantify different style features. In the fields of image processing, computer vision, and machine learning, style feature vectors are often used to capture style features in images or designs, such as color, texture, and shape.

[0097] Specifically, a target reference image having the same style as a target translation image of a source image is determined, the target reference image is input into a deep learning image recognition model to extract style features of the target reference image, determine the style features of the target reference image, vectorize the style features of the target reference image, and determine a style feature vector of the target reference image.

[0098] S420, dimensionally matching the style feature vector and the time step embedding vector of the source image, and determining the style embedding vector of the target reference image according to the matching result.

[0099] Specifically, the dimension of the style feature vector and the time step embedding vector of the source image are matched to convert the style feature vector into a style embedding vector that matches the size of the time step embedding vector.

[0100] In the above scheme, the deep learning image recognition model extracts style features of the target reference image, and can capture the micro-style of the target reference image, such as the brush strokes and image texture of the target reference image, and can also capture the macro-style of the target reference image, such as the overall tone and style genre of the target reference image, thereby improving the integrity and effectiveness of the style feature vector, and determining the style embedding vector of the target reference image based on the result of matching the style feature vector with the time step embedding vector dimension, which can further improve the accuracy of the source image style translation when guiding the source image to perform image style translation based on the style embedding vector.

[0101] In one embodiment, a deep learning image recognition model is used to extract style features of a target reference image to determine a style feature vector of the target reference image, including:

[0102] The target reference image is input into the deep learning image recognition model, and the target reference image is processed by the target activation function layer corresponding to each target convolution block in the deep learning image recognition model, and the target style feature image corresponding to each target convolution block is determined respectively; wherein the number of target convolution blocks is five; the feature vector of the target style feature image corresponding to each target convolution block is extracted respectively by the convolution layer filter in the deep learning image recognition model, and the style feature vector of the target reference image is determined.

[0103] Among them, the number of target convolution blocks is five, namely the first convolution block, the second convolution block, the third convolution block, the fourth convolution block and the fifth convolution block; the target activation function layer corresponding to the target convolution block can be selected according to actual needs. For example, the target activation function layer corresponding to the target convolution block can be the second activation function layer of the first convolution block, the second activation function layer of the second convolution block, the fourth activation function layer of the third convolution block, the fourth activation function layer of the fourth convolution block and the fourth activation function layer of the fifth convolution block.

[0104] Exemplarily, the target reference image can be input into a deep learning image recognition model, and the first style feature image of the target reference image can be determined by the second activation function layer of the first convolution block in the deep learning image recognition model; the second style feature image of the target reference image can be determined by the second activation function layer of the second convolution block in the deep learning image recognition model; the third style feature image of the target reference image can be determined by the fourth activation function layer of the third convolution block in the deep learning image recognition model; the fourth style feature image of the target reference image can be determined by the fourth activation function layer of the fourth convolution block in the deep learning image recognition model; the fifth style feature image of the target reference image can be determined by the fourth activation function layer of the fifth convolution block in the deep learning image recognition model; and feature vectors of the first style feature image, the second style feature image, the third style feature image and the fourth style feature image can be extracted by the convolution layer filters in the deep learning image recognition model to determine the style feature vector of the target reference image.

[0105] Among them, the convolutional layer filter in the deep learning image recognition model is a signal extraction method based on a neural network. It trains the neural network to learn the specific features of the signal, thereby extracting the components of interest from the original signal.

[0106] The above scheme processes the target reference image according to the target activation function layer corresponding to each target convolution block in the deep learning image recognition model, determines the target style feature image corresponding to each target convolution block, and determines the style feature vector of the target reference image according to the target style feature image, so as to improve the reliability of the style feature vector.

[0107] Exemplarily, the target reference image is input into a deep learning image recognition model, and ReLU (Rectified Linear Unit Layer, activation function) layers of different depths in the deep learning image recognition model can extract style information at different levels. The first style feature image of the target reference image is determined by the second activation function layer of the first convolution block in the deep learning image recognition model, and the number of filters of the convolution layer connected before the second activation function layer of the first convolution block is 64; the second style feature image of the target reference image is determined by the second activation function layer of the second convolution block in the deep learning image recognition model, and the number of filters of the convolution layer connected before the second activation function layer of the second convolution block is 128; the third style feature image of the target reference image is determined by the fourth activation function layer of the third convolution block in the deep learning image recognition model, and the number of filters of the convolution layer connected before the fourth activation function layer of the third convolution block is 256; the fourth style feature image of the target reference image is determined by the fourth activation function layer of the fourth convolution block in the deep learning image recognition model, and the number of filters of the convolution layer connected before the fourth activation function layer of the fourth convolution block is 512; the fifth style feature image of the target reference image is determined by the fourth activation function layer of the fifth convolution block in the deep learning image recognition model, and the number of filters of the convolution layer connected before the fourth activation function layer of the fifth convolution block is 512. The convolutional layer filters in the deep learning image recognition model are used to extract feature vectors of the first style feature image, the second style feature image, the third style feature image, and the fourth style feature image to determine the style feature vector of the target reference image, and the dimension of the style feature vector is 2944.

[0108] In one embodiment, Figure 5 As shown, through the stable diffusion model, the target translation image is generated according to the image potential representation, the target potential vector and the style embedding vector, including:

[0109] S510, concatenating the image potential representation and the target potential vector through a generation backbone in a deep neural network model to determine a concatenated potential vector.

[0110] The generative backbone in a deep neural network refers to the part of the network used for feature extraction, which is usually located at the front end of the network and is used to extract image information and generate feature maps for use by subsequent networks. The concatenated latent vector is used as the initial latent vector for determining the target translation image.

[0111] S520, using a stable diffusion model, using the style embedding vector to guide the concatenated latent vector to perform denoising processing, and determining the denoised latent vector.

[0112] Specifically, the style embedding vector of the target reference image is input into the stable diffusion model. In the process of denoising the spliced ​​latent vector through the stable diffusion model, the style embedding vector of the target reference image can guide the spliced ​​latent vector to learn the style feature information of the target reference image, and the spliced ​​latent vector after denoising is used as the denoised latent vector.

[0113] S530: Input the denoised latent vector into the decoder of the stable diffusion model backbone to determine the target translation image.

[0114] In the above scheme, in the process of generating the target translation image through the stable diffusion model, the style embedding vector of the target reference image is used to guide the splicing latent vector to learn the style information of the target reference image, and the splicing latent vector can maintain the content and structure information of the source image, thereby generating an accurate target translation image.

[0115] It should be noted that, in order to achieve a better image style translation effect, when the stable diffusion model is pre-trained, the stable diffusion model is iteratively trained based on the content loss function and style loss function of the stable diffusion model. Wherein, when the stable diffusion model is pre-trained, the stable diffusion model to be trained can be used to perform image style translation on the sample image to be translated, and when the sample image to be translated is stylized, the stylized translation reference image used can be the sample reference image. That is, the sample image to be translated and the target reference image are input into the stable diffusion model to be trained. When the stable diffusion model performs image style translation on the sample image to be translated, the sample image to be translated is converted into a latent representation of the image to be translated in the latent space, and the latent representation of the image to be translated is subjected to noise processing to determine the latent representation of the image to be translated of the sample image to be translated, and then the latent vector of the image to be translated of the sample image to be translated is determined according to the sample image to be translated, and the style embedding vector of the sample reference image is determined. By guiding the latent representation of the image to be translated and the latent vector of the image to be translated to perform image conversion through the style embedding vector of the sample reference image, the sample stylized translation image corresponding to the sample image to be translated can be obtained.

[0116] The content loss function is determined by the difference between the noise actually added to the input image by the stable diffusion model in the process of processing the input image to determine the image potential representation of the input image, and the noise added to the input image by the stable diffusion model backbone network prediction. The style loss function refers to the mean difference data and standard deviation difference data between the sample stylized translation image and the sample reference image in the feature space. The mean difference data and standard deviation difference data can be represented by the L2 norm. When the content loss function meets the preset content loss function conditions, and the style loss function meets the preset style loss function conditions, it can be determined that the training of the stable diffusion model is completed.

[0117] Exemplarily, based on the above embodiment, the image style translation method includes:

[0118] The source image is input into the stable diffusion model. The variational autoencoder in the backbone of the stable diffusion model compresses the source image from the pixel dimension space to the latent space, obtains the candidate latent representation in the latent space, and then performs a series of noise addition processing on the candidate latent representation to determine the image latent representation of the source image.

[0119] The source image is input into a target encoder having the same structure as the variational autoencoder of the pre-trained stable diffusion model, and the pixels of the source image are compressed into points in the latent space through the target encoder, and the points in the latent space obtained after compression are used as the original latent vectors.

[0120] Determine the time-step embedding vector of the source image, process the time-step embedding vector through a multi-layer perceptron, determine the weight matrix and the bias vector; perform element-by-element multiplication of the normalized latent vector and the weight matrix to obtain the multiplied data, and perform element-by-element addition of the multiplied data and the bias vector to obtain the candidate latent vector. Input the candidate latent vector into the 1×1 convolution layer, i.e., the point-by-point convolution layer, in the deep neural network to degrade the image style information corresponding to the target latent vector and determine the convolution processing vector. Input the convolution processing vector into the activation layer function to enhance the image content features in the convolution processing vector and determine the activation processing vector. Input the activation processing vector into the point-by-point convolution layer of the deep neural network again, and reduce the channel dimension of the activation processing vector in the process of point-by-point convolution, thereby eliminating the style information of the source image in the activation processing vector, so as to optimize and adjust the representation of the source image content by the activation processing vector and determine the target latent vector.

[0121] Determine a target reference image having the same style as a target translation image of a source image, input the target reference image into a deep learning image recognition model, determine a first style feature image of the target reference image through the second activation function layer of the first convolution block in the deep learning image recognition model, and the number of filters of the convolution layer connected before the second activation function layer of the first convolution block is 64; determine a second style feature image of the target reference image through the second activation function layer of the second convolution block in the deep learning image recognition model, and the number of filters of the convolution layer connected before the second activation function layer of the second convolution block is 128; The fourth activation function layer of the convolution block determines the third style feature image of the target reference image, and the number of filters of the convolution layer connected before the fourth activation function layer of the third convolution block is 256; the fourth style feature image of the target reference image is determined by the fourth activation function layer of the fourth convolution block in the deep learning image recognition model, and the number of filters of the convolution layer connected before the fourth activation function layer of the fourth convolution block is 512; the fifth style feature image of the target reference image is determined by the fourth activation function layer of the fifth convolution block in the deep learning image recognition model, and the number of filters of the convolution layer connected before the fourth activation function layer of the fifth convolution block is 512. Feature vectors are extracted from the first style feature image, the second style feature image, the third style feature image, and the fourth style feature image through the convolution layer filters in the deep learning image recognition model to determine the style feature vector of the target reference image, and the dimension of the style feature vector is 2944. The style feature vector and the time step embedding vector of the source image are dimensionally matched to convert the style feature vector into a style embedding vector that matches the size of the time step embedding vector.

[0122] The image latent representation and the target latent vector are concatenated through the generative backbone in the deep neural network model to determine the concatenated latent vector. The style embedding vector of the target reference image is input into the stable diffusion model. In the process of denoising the concatenated latent vector through the stable diffusion model, the style embedding vector of the target reference image can guide the concatenated latent vector to learn the style feature information of the target reference image. The concatenated latent vector after denoising is used as the denoised latent vector. The denoised latent vector is input into the decoder of the backbone part of the stable diffusion model to determine the target translation image.

[0123] In the above-mentioned image style translation method, the source image is input into a stable diffusion model to determine the image potential representation of the source image; the source image is input into a target encoder to determine the original potential vector; the original potential vector is processed by an instance normalization layer of a deep neural network model to determine the normalized potential vector; the time step embedding vector of the source image is determined, and the target potential vector is determined by the deep neural network based on the normalized potential vector and the time step embedding vector; the style feature of the target reference image is extracted based on the time step embedding vector by a deep learning image recognition model to determine the style embedding vector of the target reference image; and the target translation image is generated based on the image potential representation, the target potential vector and the style embedding vector by a stable diffusion model. The problem that when the image style translation of the source image is guided by the image text description of the target translation image, the new image obtained after the translation process may not fully meet the user's expected requirements due to the inherent ambiguity or ambiguity of the text description is solved. The above scheme, when performing image style translation on a source image, first performs image processing on the source image to obtain the image latent representation of the source image and the target latent vector corresponding to the time step embedding vector, and then guides the image latent representation and the target latent vector to perform styled image conversion based on the style embedding vector of the target reference image, and determines the target translation image corresponding to the source image. This can improve the accuracy of image style translation of the source image and make the obtained target translation image meet the user's expected requirements.

[0124] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0125] Based on the same inventive concept, the embodiment of the present application also provides an image style translation device for implementing the image style translation method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more image style translation device embodiments provided below can refer to the limitations of the image style translation method above, and will not be repeated here.

[0126] In one embodiment, an image style translation system is provided, including a source image input interface, a reference image input interface, and a target translation image display interface:

[0127] Source image input interface, used to input source images;

[0128] Reference image input interface, used to input reference images;

[0129] The target translation image display interface is used to display the target translation image; the target translation image is generated by: obtaining the image latent representation, normalized latent vector and time step embedding vector of the source image; determining the target latent vector based on the normalized latent vector and the time step embedding vector through a deep neural network model; extracting the style features of the target reference image based on the time step embedding vector through a deep learning image recognition model, and determining the style embedding vector of the target reference image; generating the target translation image based on the image latent representation, the target latent vector and the style embedding vector through a stable diffusion model.

[0130] In one embodiment, Figure 6 As shown, an image style translation device is provided, comprising: an image potential representation determination module 601, a target potential vector determination module 602, a style embedding vector determination module 603 and a target translation image determination module 604, wherein:

[0131] An image latent representation determination module 601 is used to obtain an image latent representation, a normalized latent vector and a time step embedding vector of a source image;

[0132] A target latent vector determination module 602 is used to determine a target latent vector according to the normalized latent vector and the time step embedding vector through a deep neural network model;

[0133] A style embedding vector determination module 603 is used to extract style features of a target reference image based on a time step embedding vector through a deep learning image recognition model, and determine a style embedding vector of the target reference image;

[0134] The target translation image determination module 604 is used to generate a target translation image according to the image latent representation, the target latent vector and the style embedding vector through a stable diffusion model.

[0135] Exemplarily, the image potential representation determination module 601 is specifically used for:

[0136] Inputting a source image into a stable diffusion model, and converting the source image into a candidate latent representation in a latent space using a variational autoencoder of a backbone part of the stable diffusion model;

[0137] Performing noise processing on the candidate latent representation to determine the image latent representation of the source image;

[0138] Inputting the source image into a target encoder to determine an original latent vector;

[0139] Processing the original latent vector through an instance normalization layer of a deep neural network model to determine a normalized latent vector;

[0140] The time step of the source image is obtained and converted into a high-dimensional position encoding vector, and the high-dimensional position encoding vector is converted into a time step embedding vector through the fully connected layer of the deep neural network model.

[0141] Exemplarily, the target potential vector determination module 602 is specifically used for:

[0142] The time step embedding vector is processed by a multi-layer perceptron to determine the weight matrix and bias vector;

[0143] The normalized latent vector is multiplied element by element by the weight matrix to obtain the multiplied data, and the multiplied data is added element by element to the bias vector to obtain the candidate latent vector;

[0144] The candidate latent vectors are input into the point-wise convolutional layer of a deep neural network to determine the target latent vector.

[0145] Exemplarily, the target potential vector determination module 602 is further specifically configured to:

[0146] Input the candidate latent vector into a point-wise convolution layer in a deep neural network to determine a convolution processing vector;

[0147] Perform activation processing on the convolution processing vector through the activation layer function to determine the activation processing vector;

[0148] The activation processing vector is fed into a point-wise convolutional layer in a deep neural network to determine the target latent vector.

[0149] Exemplarily, the style embedding vector determination module 603 is specifically used for:

[0150] A deep learning image recognition model is used to extract style features of the target reference image and determine the style feature vector of the target reference image;

[0151] The style feature vector and the time step embedding vector of the source image are dimensionally matched, and the style embedding vector of the target reference image is determined based on the matching results.

[0152] Exemplarily, the style embedding vector determination module 603 is further specifically used for:

[0153] Input the target reference image into the deep learning image recognition model, process the target reference image through the target activation function layer corresponding to each target convolution block in the deep learning image recognition model, and determine the target style feature image corresponding to each target convolution block respectively; wherein the number of target convolution blocks is five;

[0154] The convolutional layer filters in the deep learning image recognition model are used to extract feature vectors of the target style feature images corresponding to each target convolution block, and the style feature vector of the target reference image is determined.

[0155] Exemplarily, the target translation image determination module 604 is specifically used for:

[0156] Concatenate the image latent representation and the target latent vector through a generative backbone in a deep neural network model to determine a concatenated latent vector;

[0157] Through the decoder of the backbone part of the stable diffusion model, the style embedding vector is used to guide the concatenated latent vector for denoising, and the denoised latent vector is determined;

[0158] The denoised latent vector is fed into the decoder of the backbone part of the stable diffusion model to determine the target translation image.

[0159] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 7 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be realized through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, an image style translation method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0160] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0161] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0162] Step 1: Obtain the image latent representation, normalized latent vector and time step embedding vector of the source image;

[0163] Step 2: a target latent vector determination module, used to determine the target latent vector according to the normalized latent vector and the time step embedding vector through a deep neural network model;

[0164] Step 3: a style embedding vector determination module is used to extract style features of a target reference image based on a time step embedding vector through a deep learning image recognition model, and determine a style embedding vector of the target reference image;

[0165] Step 4: A target translation image determination module is used to generate a target translation image based on the image latent representation, the target latent vector and the style embedding vector through a stable diffusion model.

[0166] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0167] Step 1: Obtain the image latent representation, normalized latent vector and time step embedding vector of the source image;

[0168] Step 2: a target latent vector determination module, used to determine the target latent vector according to the normalized latent vector and the time step embedding vector through a deep neural network model;

[0169] Step 3: a style embedding vector determination module is used to extract style features of a target reference image based on a time step embedding vector through a deep learning image recognition model, and determine a style embedding vector of the target reference image;

[0170] Step 4: A target translation image determination module is used to generate a target translation image based on the image latent representation, the target latent vector and the style embedding vector through a stable diffusion model.

[0171] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0172] Step 1: Obtain the image latent representation, normalized latent vector and time step embedding vector of the source image;

[0173] Step 2: a target latent vector determination module, used to determine the target latent vector according to the normalized latent vector and the time step embedding vector through a deep neural network model;

[0174] Step 3: a style embedding vector determination module is used to extract style features of a target reference image based on a time step embedding vector through a deep learning image recognition model, and determine a style embedding vector of the target reference image;

[0175] Step 4: A target translation image determination module is used to generate a target translation image based on the image latent representation, the target latent vector and the style embedding vector through a stable diffusion model.

[0176] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0177] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded vector non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, etc., but are not limited to this.

[0178] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0179] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An image style translation method, characterized in that: include: Get the image latent representation, normalized latent vector, and time-step embedding vector of the source image; Determining a target latent vector based on the normalized latent vector and the time step embedding vector through a deep neural network model; Extracting style features of a target reference image based on the time step embedding vector through a deep learning image recognition model to determine a style embedding vector of the target reference image; generating a target translated image according to the image latent representation, the target latent vector and the style embedding vector by a stable diffusion model; The determining a target latent vector according to the normalized latent vector and the time step embedding vector by a deep neural network model includes: Processing the time step embedding vector by a multi-layer perceptron to determine a weight matrix and a bias vector; Performing element-by-element multiplication of the normalized latent vector and the weight matrix to obtain multiplied data, and performing element-by-element addition of the multiplied data and the bias vector to obtain a candidate latent vector; The candidate latent vector is input into a point-by-point convolutional layer of a deep neural network to determine a target latent vector.

2. The method according to claim 1, characterized in that Get the image latent representation, normalized latent vector, and time-step embedding vector of the source image, including: Inputting a source image into a stable diffusion model, and converting the source image into a candidate latent representation in a latent space using a variational autoencoder of a backbone part of the stable diffusion model; Performing noise processing on the candidate latent representation to determine the image latent representation of the source image; Inputting the source image into a target encoder to determine an original latent vector; Processing the original latent vector through an instance normalization layer of a deep neural network model to determine a normalized latent vector; The time step of the source image is obtained, and the time step of the source image is converted into a high-dimensional position encoding vector, and the high-dimensional position encoding vector is converted into a time step embedding vector through a fully connected layer of a deep neural network model.

3. The method according to claim 1, characterized in that Inputting the candidate latent vector into a point-by-point convolutional layer of a deep neural network to determine a target latent vector includes: Inputting the candidate potential vector into a point-by-point convolution layer in a deep neural network to determine a convolution processing vector; Performing activation processing on the convolution processing vector through an activation layer function to determine an activation processing vector; The activation processing vector is input into a point-by-point convolutional layer in the deep neural network to determine a target latent vector.

4. The method according to claim 1, characterized in that: Extracting style features of a target reference image based on the time step embedding vector by using a deep learning image recognition model to determine a style embedding vector of the target reference image includes: A deep learning image recognition model is used to extract style features of the target reference image and determine the style feature vector of the target reference image; The style feature vector and the time step embedding vector of the source image are dimensionally matched, and the style embedding vector of the target reference image is determined according to the matching result.

5. The method according to claim 4, characterized in that A deep learning image recognition model is used to extract style features of the target reference image and determine the style feature vector of the target reference image, including: Inputting a target reference image into a deep learning image recognition model, processing the target reference image through a target activation function layer corresponding to each target convolution block in the deep learning image recognition model, and determining a target style feature image corresponding to each target convolution block, respectively; wherein the number of the target convolution blocks is five; The convolution layer filters in the deep learning image recognition model are used to extract feature vectors of the target style feature images corresponding to each target convolution block, and the style feature vector of the target reference image is determined.

6. The method according to claim 1, characterized in that Generating a target translated image according to the image latent representation, the target latent vector and the style embedding vector by the stable diffusion model, including: Concatenating the image potential representation and the target potential vector through a generative backbone in a deep neural network model to determine a concatenated potential vector; Determine a denoised latent vector by using the style embedding vector to guide the concatenated latent vector through a decoder of a stable diffusion model backbone; The denoised latent vector is input into a decoder of a stable diffusion model backbone to determine a target translation image.

7. An image style translation system, characterized in that: The image style translation system comprises: Source image input interface, used to input source images; Reference image input interface, used to input reference images; A target translation image display interface is used to display a target translation image; the target translation image is generated by: obtaining an image potential representation, a normalized potential vector and a time step embedding vector of a source image; determining a target potential vector according to the normalized potential vector and the time step embedding vector through a deep neural network model; extracting style features of a target reference image based on the time step embedding vector through a deep learning image recognition model to determine a style embedding vector of the target reference image; generating a target translation image according to the image potential representation, the target potential vector and the style embedding vector through a stable diffusion model; wherein determining the target potential vector according to the normalized potential vector and the time step embedding vector through a deep neural network model comprises: processing the time step embedding vector through a multi-layer perceptron to determine a weight matrix and a bias vector; performing element-by-element multiplication of the normalized potential vector and the weight matrix to obtain multiplied data, and performing element-by-element addition of the multiplied data and the bias vector to obtain a candidate potential vector; and inputting the candidate potential vector into a point-by-point convolution layer of a deep neural network to determine a target potential vector.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images