Image processing method, device, equipment, medium and program product

Abstract

The embodiment of the present application discloses an image processing method, apparatus, device, medium and program product, wherein the method comprises: obtaining a first image to be processed and style hint information; performing image coding processing on the first image to obtain image coding features; using an attention algorithm to perform a linear transformation on the first image and the style hint information, and embedding the image coding features into the linear transformation to generate a target semantic feature; generating a second image based on the target semantic feature. The embodiment of the present application can embed image coding features in the image generation process, thereby reducing the image generation cost and improving the image quality.

Description

Technical Field

[0001] The present application relates to the field of computer technology, in particular to the field of artificial intelligence, and specifically to an image processing method, an image processing device, a computer equipment, a computer-readable storage medium, and a computer program product. Background Art

[0002] Text drawing refers to the process of drawing pictures based on the text entered by the user.

[0003] Currently, the diffusion model is supported to realize text-generated images. The existing text-generated image process is to use a large amount of image data to fine-tune the model after randomly initializing the model parameters of the pre-trained diffusion model, so that new images can be generated based on the fine-tuned diffusion model guided by the text used during training. The existing inference stage must use a large amount of image data to fine-tune the model, resulting in a high cost for image generation, and there are problems such as poor text concreteness when only using the text-guided model to generate images. Summary of the invention

[0004] The embodiments of the present application provide an image processing method, apparatus, device, medium and program product, which can embed image coding features in the image generation process, reduce the image generation cost and improve the image concreteness after style conversion.

[0005] On the one hand, an embodiment of the present application provides an image processing method, the method comprising:

[0006] Acquire a first image to be processed and style prompt information; the first image has a first style, and the style indication information is used to indicate that the first image is converted from the first style to a second style;

[0007] Performing image coding processing on the first image to obtain image coding features, where the image coding features are used to characterize content characteristics of image content included in the first image;

[0008] An attention algorithm is used to perform a linear transformation on the first image and the style prompt information, and the image encoding feature is embedded into the linear transformation to generate a target semantic feature; the target semantic feature has content characteristics of the image content included in the first image, image characteristics of the first image, and style conversion characteristics of the style prompt information;

[0009] A second image is generated based on the target semantic feature; the second image has a second style, and the image content included in the second image is the same as the image content included in the first image.

[0010] On the other hand, an embodiment of the present application provides an image processing device, the device comprising:

[0011] An acquisition unit, used to acquire a first image to be processed and style prompt information; the first image has a first style, and the style indication information is used to indicate that the first image is converted from the first style to a second style;

[0012] A processing unit, configured to perform image coding processing on the first image to obtain image coding features, where the image coding features are used to characterize content characteristics of image content included in the first image;

[0013] The processing unit is further used to use an attention algorithm to perform a linear transformation on the first image and the style prompt information, and embed the image coding feature into the linear transformation to generate a target semantic feature; the target semantic feature has content characteristics of the image content included in the first image, image characteristics of the first image, and style conversion characteristics of the style prompt information;

[0014] The processing unit is further used to generate a second image based on the target semantic feature; the second image has a second style, and the image content included in the second image is the same as the image content included in the first image.

[0015] In one implementation, the linear transformation includes a linear projection transformation process; and the processing unit is used to embed the image coding feature into the linear transformation to generate the target semantic feature, specifically for:

[0016] Acquire initial semantic features obtained by performing linear projection transformation processing during the linear transformation process; wherein the linear transformation is used to extract key information from the first image and the style hint information;

[0017] The image encoding features and the initial semantic features are combined to generate the target semantic features.

[0018] In one implementation, the linear transformation includes a first sub-linear transformation, a second sub-linear transformation, and a third sub-linear transformation, and the first sub-linear transformation, the second sub-linear transformation, and the third sub-linear transformation all include linear projection transformation processing; the target semantic feature includes a first sub-semantic feature, a second sub-semantic feature, and a third sub-semantic feature;

[0019] The processing unit is used to combine the image coding features and the initial semantic features to generate target semantic features, specifically for:

[0020] The image coding feature and the initial semantic feature obtained by performing the linear projection transformation in the first sub-linear transformation process are combined to generate a first sub-semantic feature;

[0021] The image coding feature and the initial semantic feature obtained by performing the linear projection transformation in the second sub-linear transformation process are combined to generate a second sub-semantic feature;

[0022] The image coding feature and the initial semantic feature obtained by performing the linear projection transformation in the third sub-linear transformation process are combined to generate the third sub-semantic feature;

[0023] A target semantic feature is generated based on the first sub-semantic feature, the second sub-semantic feature, and the third sub-semantic feature.

[0024] In one implementation, the first sub-linear transformation and the second sub-linear transformation are used to perform text semantic extraction on the style hint information, and the first sub-linear transformation and the second sub-linear transformation are performed to obtain the style conversion characteristics of the style hint information; the style conversion characteristics of the style conversion hint information are the initial semantic characteristics obtained by performing the linear projection transformation process in the first sub-linear transformation process and the second sub-linear transformation process; the style conversion characteristics are used to characterize the style parameters required for converting from the first style to the second style;

[0025] The third sub-linear transformation is used to extract image features from the first image, and the image characteristics of the first image are obtained by executing the third sub-linear transformation; the image characteristics of the first image are initial semantic features obtained by executing the linear projection transformation in the process of the third sub-linear transformation; the image characteristics are used to characterize the image data of the first image, and the image data includes at least one of the following: resolution, size data and pixel data.

[0026] In one implementation, the processing unit is further configured to:

[0027] Performing feature dimensionality reduction processing on the image coding features to obtain a feature matrix corresponding to the image coding features; the dimension of the feature matrix is ​​smaller than the dimension of the image coding features;

[0028] The processing unit is used to embed the image coding features into the linear transformation to generate the target semantic features, specifically for:

[0029] The feature matrix corresponding to the image encoding features is embedded into the linear transformation to generate the target semantic features.

[0030] In one implementation, the feature matrix is ​​formed by sequentially multiplying the first submatrix, the second submatrix, and the third submatrix; the row and column information of the image coding feature is m×n, where m and n are positive integers; and the processing unit is used to perform feature dimensionality reduction processing on the image coding feature to obtain the feature matrix corresponding to the image coding feature, specifically for:

[0031] According to the row and column information of the image coding features, the image coding features are matrix decomposed to obtain two orthogonal matrices and a diagonal matrix corresponding to the image coding features; wherein, the row and column information of one of the two orthogonal matrices is m×m, and the row and column information of the other orthogonal matrix is ​​n×n; the row and column information of the diagonal matrix is ​​n×n, and the elements other than the main diagonal in the diagonal matrix are 0, the values ​​of the elements on the main diagonal are used to represent the importance of the content characteristics, and the elements on the main diagonal are arranged from large to small according to the values;

[0032] Selecting r elements from the n elements on the main diagonal of the diagonal matrix in descending order of value to form a second submatrix, and constructing the first submatrix and the third submatrix based on the two orthogonal matrices and the second submatrix;

[0033] The first sub-matrix, the second sub-matrix and the third sub-matrix are multiplied in sequence to obtain a feature matrix corresponding to the image coding feature.

[0034] In one implementation, the image processing method is performed by a pre-trained target image generation model, the target image generation model includes an attention algorithm layer; the pre-training process of the target image generation model includes:

[0035] Obtain an image-text pair from a pre-training data set; the image-text pair includes a sample image and a text description for describing the sample image;

[0036] Matrixing the sample images in the image-text pair using the initial image generation model to obtain predicted image coding features; the matrixing includes image coding, or image coding and feature dimensionality reduction; and,

[0037] The initial image generation model is used to perform semantic extraction processing on the text description in the image-text pair to obtain the predicted semantic features;

[0038] The predicted image encoding features are embedded into the attention algorithm layer, and the predicted image corresponding to the sample image is generated by combining the predicted semantic features;

[0039] Based on the predicted image and the sample image, the initial image generation model is pre-trained to obtain a pre-trained target image generation model.

[0040] In one implementation, the image content included in the first image includes at least one of the following: a face, a person, an animal, and an object; the image content has not been learned in the pre-training stage;

[0041] The method of obtaining style prompt information includes any of the following: a default setting; or, generated based on an input operation for a second style in a style selection interface of an object; wherein the style selection interface includes one or more candidate styles, and the input operation is a selection operation of selecting the second style from one or more candidate styles; or, the style selection interface includes a style input area, and the input operation is an input operation of inputting the second style in the style input area.

[0042] In one implementation, the process of constructing a pre-training data set includes:

[0043] Selecting a first number of sample images from a public data set to form a first data set; and,

[0044] Searching a second number of sample images from the Internet to form a second data set; the sum of the first number and the second number is greater than a quantity threshold;

[0045] Data processing is performed on the sample images in the first data set and the second data set respectively to obtain a pre-training data set for pre-training the initial image generation model.

[0046] In one implementation, the initial image generation model includes a text description module; any sample image in the first data set or the second data set is represented as a target sample image; a processing unit is used to perform data processing on the sample images in the first data set and the second data set respectively, and when a pre-training data set for pre-training the initial image generation model is obtained, it is specifically used to:

[0047] Performing image quality processing on the target sample image to obtain a processed target sample image; the image quality processing at least includes: square mapping processing, image enhancement processing and size normalization processing; and,

[0048] A text description module is used to perform text generation processing on the target sample image to generate a target text description corresponding to the target sample image;

[0049] The target sample image and the target text description corresponding to the target sample image constitute an image-text pair.

[0050] In one implementation, the initial image generation model includes a first image encoding module, a second image encoding module and a feature fusion module; the feature fusion module includes trainable parameters of the initial image generation model; the matrix processing includes image encoding processing; the processing unit is used to perform matrix processing on the sample image in the image-text pair using the initial image generation model to obtain the predicted image coding feature, specifically for:

[0051] Using a first image encoding module to perform image encoding processing on a sample image in an image-text pair to obtain a first predicted sub-image feature; and

[0052] Using a second image encoding module to perform image encoding processing on a sample image in the image-text pair to obtain a second predicted sub-image feature;

[0053] A feature fusion module is used to perform feature fusion on the first predicted sub-image feature and the second predicted sub-image feature to obtain a predicted image coding feature corresponding to the image-text pair.

[0054] On the other hand, an embodiment of the present application provides a computer device, the computer device comprising:

[0055] A processor for loading and executing computer programs;

[0056] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the image processing method is implemented.

[0057] On the other hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program, and the computer program is suitable for being loaded by a processor and executing the above-mentioned image processing method.

[0058] On the other hand, an embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the above-mentioned image processing method is implemented.

[0059] In an embodiment of the present application, after obtaining the first image to be processed and the style hint information for indicating that the first image is to be converted from the first style to the second style, on the one hand, the first image can be directly subjected to image coding processing to obtain the image coding features of the first image, and the image coding features can be used to characterize the content characteristics of the image content included in the first image. On the other hand, an attention algorithm can be used to perform a linear transformation on the first image and the style hint information. Furthermore, the image coding features of the first image are embedded in the linear transformation, so that the target semantic features can be extracted during the linear transformation process, and the target semantic features not only have the content characteristics of the image content included in the first image, but also include the image characteristics of the first image (such as some basic image data of the image, such as resolution, image size, etc.) and the style conversion characteristics of the style hint information (such as the style parameters of the second style, etc.). In this way, a second image can be generated based on target semantic features that include both image features (such as content features and image features of image content) and text features (such as style conversion features). The second image has a high image quality, and the high image quality is specifically reflected in: the image content included in the second image and the image content included in the first image have a high similarity, and the second image has a second style indicated by the style prompt information, that is, the style conversion is successfully achieved. In the above-mentioned style conversion process, the image coding features of the first image are directly embedded in the linear transformation process. Compared with the need for dozens or even hundreds of images to train the model to learn image features, it not only improves the concreteness of the generated second image with the second style, but also effectively reduces the cost required to generate the second image. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0061] Figure 1 It is a structural diagram of an LDM model;

[0062] Figure 2a is a structural schematic diagram of an initial image generation model provided by an exemplary embodiment of the present application;

[0063] Figure 2b is a structural schematic diagram of a target image generation model provided by an exemplary embodiment of the present application;

[0064] Figure 3 is a structural schematic diagram of an image processing system provided by an exemplary embodiment of the present application;

[0065] Figure 4 is a flowchart of an image processing method provided by an exemplary embodiment of the present application;

[0066] Figure 5a is a schematic diagram of an interface for uploading a first image provided by an exemplary embodiment of the present application;

[0067] Figure 5b is a schematic diagram of another interface for uploading a first image provided by an exemplary embodiment of the present application;

[0068] Figure 6 It is a structural diagram of a VAE model;

[0069] Figure 7 It is a schematic diagram of the structure of a TransformLayer module;

[0070] Figure 8 is a flowchart of another image processing method provided by an exemplary embodiment of the present application;

[0071] Fig. 9 is a schematic diagram of collecting sample images from the Internet provided by an exemplary embodiment of the present application;

[0072] Fig.10 It is a structural diagram of a BLIP model;

[0073] Fig.11 It is a schematic diagram of a BLIP model provided by an exemplary embodiment of the present application that generates a corresponding target text description according to a target sample image;

[0074] Fig.12 is a schematic diagram of image quality processing provided by an exemplary embodiment of the present application;

[0075] Fig.13 is a schematic diagram of a pre-training process provided by an exemplary embodiment of the present application;

[0076] Fig.14a is a schematic diagram of an SVD matrix decomposition provided by an exemplary embodiment of the present application;

[0077] Fig.14b is a schematic diagram of reducing the dimension of an SVD matrix provided by an exemplary embodiment of the present application;

[0078] Fig.15 is a structural schematic diagram of an image processing device provided by an exemplary embodiment of the present application;

[0079] Fig.16It is a structural diagram of a computer device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0080] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0081] The embodiment of the present application proposes an image processing solution based on the Wensheng graph diffusion model. Among them, the Wensheng graph diffusion model belongs to the field of artificial intelligence (AI). Artificial intelligence is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence technology is a comprehensive discipline that involves a wide range of fields, including both hardware-level technology and software-level technology. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, pre-trained model technology, operation / interaction system, mechatronics, etc. Among them, the pre-trained model is also called a large model or a basic model. After fine-tuning, it can be widely used in downstream tasks in various major directions of artificial intelligence.

[0082] Further, the Vincent graph diffusion model (or simply Vincent graph or diffusion model, etc.) is a pre-trained model that supports inputting text to control the generation of pictures (or images) during the pre-training diffusion model. In detail, the diffusion model includes two main processes: forward diffusion and backward diffusion; in the forward diffusion stage, the image is contaminated by gradually introduced noise until the image becomes completely random noise; in the backward process, a series of Markov chains are used to gradually remove the prediction noise at each time step (or timestamp, such as 1 second), thereby recovering data from Gaussian noise to generate an image.

[0083] Furthermore, there are many types of diffusion models. In the embodiment of the present application, the diffusion model is introduced as a logical data model (Latent Diffusion Models, LDM) as an example. Among them, the logical data model can be called a Stable Diffusion model, which diffuses in latent space rather than pixel space, which can save memory. At the same time, combined with the text semantic feedback from the Transformer, it can produce diversified and highly detailed images while retaining the semantic structure of the data. For example, the main structural diagram of the LDM model is as follows Figure 1As shown in the figure, it mainly consists of three parts, namely, the text encoder TextEncoder, the image encoder VAE Encoder and the denoising model Unet. In the process of pre-training the LDM model, the image encoder VAE Encoder needs to add noise to the image. The denoising model Unet continuously removes noise under the condition that the text encoder TextEncoder encodes the text, and restores the original image through the image decoder VAE De coder. After the LDM model is pre-trained, it is necessary to use a large number of text-image pairs as fine-tuning training data to fine-tune the pre-trained LDM model; during the fine-tuning training process, the LDM model uses the text in the text-image pair as a condition and the noisy image in the text-image pair as input to denoise the image, thereby achieving the purpose of restoring the image from the noise. After the pre-training and fine-tuning described above, the LDM model can generate images with corresponding meanings according to the text description.

[0084] From the above description of the diffusion model, it can be seen that the traditional diffusion model needs to adopt pre-training and fine-tuning to realize model training, and the number of training data (such as text image pairs) required for pre-training and fine-tuning is large, and there are problems such as high training cost and poor text concreteness. Among them, concreteness can refer to the real existence form and state of things or phenomena, as well as their specific manifestations and characteristics; things or phenomena with strong concreteness are easy to be directly perceived and understood by people, while things or phenomena with weak concreteness are often not easy to be directly perceived and understood by people. The embodiment of the present application uses the Wenshengtu diffusion model as the base model to provide a new image processing scheme, which can generate a lightweight auxiliary matrix based on the active image (i.e., a low-rank matrix with a small number of parameters, which supports injection into the large model in a pluggable manner to achieve the purpose of fine-tuning the large model) to realize personalized style conversion. Personalized style conversion can be simply understood as generating a second image with a second style based on an active image "first image" with a first style, and the image content included in the generated second image is the same or similar to the image content included in the first image. Taking a first image with a first style including a face as an example, a personalized style conversion is performed on the first image to obtain a second image with a second style, and the face included in the second image and the face included in the first image have a high degree of similarity. From the visual effect, it can be seen that the object to which the face included in the second image belongs is the same object as the object to which the face included in the first image belongs.

[0085] As mentioned above, the embodiment of the present application uses the StableDiffusion model of the Vincent graph diffusion model as the base model, and mainly realizes improvements from two aspects: data and model, in the hope that when new image content (such as new characters or faces) is introduced into the model, the cost of model training can be reduced, the generalization ability of the model can be improved, the shortcomings of poor text concreteness can be compensated, and the fidelity of the generated image can be improved (that is, the similarity between the image content included in the generated image and the image content included in the original image, the higher the similarity, the higher the fidelity). For the sake of ease of explanation, the embodiment of the present application refers to the model that uses the Vincent graph diffusion model as the base model to implement the image processing solution as an image generation model, the model before pre-training is called the initial image generation model, and the pre-trained model is called the target image generation model. Among them, the image processing solution mainly includes: a pre-training stage for the initial image generation model, and a model inference stage for using the target image generation model to realize personalized style conversion. The following is a brief introduction to the solution process of these two stages:

[0086] (1) Pre-training stage.

[0087] In the pre-training stage, the embodiment of the present application supports the autonomous construction of 100,000-level high-quality data without manual annotation to pre-train the initial image generation model; and the target image generation model pre-trained with 100,000-level data has good model generalization ability, and can directly perform model inference by introducing new image content without fine-tuning the model (i.e., no training is required), thereby avoiding the training cost caused by fine-tuning. Among them, the structural diagram of the initial image generation model in the pre-training stage can be seen in Figure 2a ;like Figure 2a The improvements shown mainly include two aspects: data and model structure.

[0088] Regarding data optimization, the embodiment of the present application supports the use of a BLIP model to generate text descriptions for images, thereby improving the similarity between sample images and text descriptions in image-text pairs in a pre-training data set; it also supports normalizing images to ensure the quality and consistency of sample images in the pre-training data set.

[0089] For the optimization of the model structure, the embodiment of the present application supports the use of a pre-trained StableDiffusion model that can realize the text image function as a base model, directly using the pre-trained image coding model CLIP ImageEncoder, the image coding model VAE Encoder, the multimodal pre-trained model BLIP, the pre-trained text model CLIP TextEncoder, the matrix conversion module (SVD conversion module) and the model parameters of the attention mechanism (such as the cross attention mechanism Cross Attention), and only needs to update the model parameters for the image coding model TransformLayer during the pre-training process, and only needs to retain the model parameters of this part after the pre-training is completed. In order to make up for the shortcomings of poor concreteness of the text and improve the fidelity of the generated image, it also supports the extraction of the image coding features of the first image to be processed, and the features are embedded in the model in a pluggable manner, specifically embedded in the attention mechanism included in the model, and image generation is achieved by directly analyzing high-quality image coding features, thereby improving the image quality of the generated image.

[0090] (2) Model inference stage.

[0091] Considering that the embodiment of the present application uses 100,000-level high-quality pre-training data when pre-training the initial image generation model, and the image coding features are directly embedded into the model for image feature learning during the pre-training stage. This makes the target image generation model pre-trained by the embodiment of the present application have a high generalization ability, which can be directly used for model inference, that is, the pre-trained target image generation model is directly used to perform personalized style conversion on the first image to be processed, and the second image can be generated, and while ensuring that the second image has a different image style from the first image, it is ensured that the image content included in the second image is the same as the image content included in the first image (or the image content has a high degree of similarity).

[0092] Among them, the structural diagram of the target image generation model can be found in Figure 2b ;like Figure 2b As shown, in the model inference stage, the model parameters of the pre-trained image coding model TransformLayer are first loaded into the computer device. Then, the computer device obtains the first image to be processed and the style prompt information; wherein, the first image has a first style, and the style prompt information is used to indicate that the first image is converted from the first style to the second style. In this way, the target image generation model with the loaded TransformLayer model parameters can be used to perform image coding processing on the first image to obtain the image coding features of the first image, and the image coding features can be used to characterize the content characteristics of the image content included in the first image. Optionally, the image coding features can be matrix transformed (such as Figure 2b SVD conversion as shown) to convert the image coding features into a feature matrix with a small number of parameters. Then, the image coding features are embedded into a linear transformation for the first image and the style hint information (specifically, the feature matrix obtained by feature conversion can be inserted into the target image generation model) to generate target semantic features, so that the target semantic features can simultaneously have the content characteristics of the image content included in the first image, the image characteristics of the first image, and the style conversion characteristics of the style hint information. Finally, a second image is generated based on the target semantic features, the second image has a second style, and the image content included in the second image is the same as the image content included in the first image.

[0093] Furthermore, for the same first image to be processed, if the user wants to perform multiple style conversions on the first image, then considering that the image encoding features of the first image have been migrated to the target image generation model during the first style conversion process. Therefore, for any subsequent style conversion except the first style conversion in the multiple style conversions, it is only necessary to change the style prompt information to achieve multiple style conversions. This improves the conversion speed and efficiency of multiple style conversions to a certain extent, and can improve the user's image style conversion experience in image application scenarios (such as scenes of personalized character style conversion, which can specifically include converting real images containing real people into cartoon images).

[0094] It should be noted that the above Figure 2a and Figure 2b The model structures shown are exemplary, and the model structures may be transformed in actual applications. For example, the type and number of modules included in the model structure may be transformed; for example, when the image coding features do not need to be converted into a feature matrix with a small parameter amount, the model structure may not even include an SVD conversion module; and so on. The embodiments of the present application do not limit the specific model structures of the initial image generation model and the target image generation model.

[0095] It has been found in practice that the image processing solution provided by the embodiment of the present application has obvious advantages when performing image style conversion. The following is an example of comparing the solution of the present application with the existing mainstream multi-task model solution to illustrate the advantages of the embodiment of the present application, where:

[0096] The existing mainstream Wenshengtu scheme needs to go through the following two steps: 1) After obtaining the pre-trained diffusion model, if it is necessary to perform style conversion on an image containing new image content (such as a face), after randomly initializing the model parameters of the diffusion model, it is necessary to use dozens or even hundreds of image-text pairs (image-text) containing the image content to fine-tune the diffusion model so that the diffusion model can learn the content characteristics of the image content. 2) After fine-tuning the model, the text used during training (specifically fine-tuning) can be used to guide the diffusion model to generate images. It can be seen that the traditional Wenshengtu scheme requires dozens or even hundreds of images for fine-tuning for each new image content, and training at least hundreds of times. It takes a long time (such as dozens of minutes on GPUA100 to achieve fine-tuning) to restore the content characteristics of the image content, resulting in high training costs. Moreover, when there is little training data for fine-tuning (e.g., only one image), randomly initializing model parameters to train image content can easily lead to overfitting due to the small amount of data, resulting in different text input conditions only outputting images of the original style (i.e., the original training image), and it is difficult to change the style of the image through text. In addition, generating images based only on the text used in training will cause problems such as poor text concreteness, as the text itself is difficult to clearly describe the image content (such as the appearance of an object, such as texture, color, and light).

[0097] However, the embodiment of the present application provides a new image processing scheme, which can directly perform image coding processing on the first image to extract the image coding features of the first image, and also supports embedding the image coding features into the linear transformation process for the first image and the style prompt information, so as to make up for the shortcomings of the text concreteness in the traditional scheme, so that the image coding features of the first image can be taken into account when the linear transformation is performed on the first image and the style prompt information, and ensure that the target semantic features generated by the linear transformation can simultaneously have the content characteristics of the image content included in the first image, the image characteristics of the first image and the style conversion characteristics of the style prompt information. In this way, it can effectively ensure that the image quality of the second image generated based on the target semantic features is better, and the fidelity of the generated image is guaranteed; the better image quality is mainly reflected in that the second image has the second style indicated by the style prompt information, and the image content included is the same as the image content included in the first image. In addition, the embodiment of the present application does not require manual annotation, and independently constructs 100,000-level high-quality data to pre-train the model, so that the pre-trained model has a strong model generalization ability, and can directly realize the style conversion of the image including the new image content without fine-tuning the model when introducing new image content, which greatly reduces the model training cost.

[0098] In practical applications, the embodiment of the present application supports the use of an image generation platform with a Wensheng map function to perform style conversion on a first image to generate a second image, that is, the image generation platform deploys the image processing solution provided by the embodiment of the present application; in this way, when the user / object has a need for personalized style conversion of the first image, the corresponding second image can be automatically generated based on the first image through the image generation platform. Among them, the image generation platform can refer to an application with a Wensheng map function. An application can refer to a computer program for completing one or more specific tasks; by classifying the application according to different dimensions (such as the operation mode and function of the application), the types of the same application under different dimensions can be obtained. For example: according to the operation mode of the application, the application may include but is not limited to: a client installed in the terminal, a small program that can be used without downloading and installing (as a subprogram of the client), a Web (World Wide Web, Global Wide Area Network) application opened through a browser, etc. For another example: according to the functional type of the application, the application may include but is not limited to: IM (Instant Messagin g, instant messaging) application, content interaction application, etc. Instant messaging applications refer to applications for instant messaging and social interaction based on the Internet. Instant messaging applications may include, but are not limited to: social applications with communication functions, map applications with social interaction functions, game applications, etc. Content interaction applications refer to applications that can realize content interaction, such as sharing platforms, personal spaces, news, videos, and app stores.

[0099] Furthermore, the image generation platform can also be a plug-in included in the above-mentioned application that supports the text image function. For example, if the application is an instant messaging application in the form of a client, then the image generation platform can be an image plug-in included in the instant messaging application. At this time, through the image plug-in, an object (such as any object using the instant messaging application) can directly perform image style conversion in the instant messaging application during the process of using the instant messaging application for social interaction, without the need for application jump (such as jumping from the instant messaging application to an independent image application).

[0100] To facilitate understanding of the image processing solution provided in the embodiment of the present application, Figure 3 The image processing system shown in the figure is used to exemplify an image processing scenario of the image processing solution. Figure 3 As shown, the image processing system includes a terminal 301 and a server 302. The embodiment of the present application does not limit the number and naming of the terminals and servers.

[0101] Among them, terminal 301 may refer to a terminal device used by an object with image style conversion requirements. Terminal devices may include, but are not limited to: smart phones (such as smart phones with Android systems deployed, or smart phones with Internetworking Operating System (IOS) deployed, etc.), tablet computers, portable personal computers, vehicle-mounted devices, head-mounted devices, smart homes and other devices. The embodiment of the present application does not limit the type of terminal devices, which is explained here. The above-mentioned image generation platform can be deployed in terminal 301, so that when the object needs image style conversion, it can open the image generation platform through the terminal 301 to perform style conversion and other operations. Server 302 is a background server corresponding to terminal 301, which is used to interact with terminal 301 to provide computing and application service support for the image generation platform deployed in terminal 301. Server 302 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms. The terminal 301 and the server 302 may be directly or indirectly connected via wired or wireless communication, which is not limited in this application.

[0102] The image processing solution provided in the embodiment of the present application can be executed by a computer device, which is equipped with a target image generation model pre-trained in the embodiment of the present application; thus, the computer device can be used to call the target image generation model to perform image style conversion in the model inference stage. The computer device can be Figure 3 The terminal or server in the system shown, that is, the embodiment of the present application supports the execution of the image processing solution by either the terminal or the server, or by the terminal and the server together.

[0103] Taking the case where the terminal and the server jointly execute the image processing solution as an example, it is assumed that a pre-trained target image generation model is deployed in the server, and an image generation platform with image style conversion capability (i.e., the aforementioned Wenshengtu function) is installed in the terminal, which can be a small program with a personalized avatar style conversion function. For the user, if the user has a need to perform style conversion on the first image (such as a photo containing his or her own or another person's face) during the use of the terminal, the user can open the small program installed in the terminal through the terminal and only provide a single first image to the small program. In this way, the terminal can transmit the first image to the server, so that the server can call the deployed target image generation model based on directly performing image coding processing on the first image to obtain the image coding features of the first image, and embed the image coding features into the target image generation model, so that the target image generation model can combine the image coding features, the first image and the style prompt information (default or user selection) to generate the second image. Finally, the server returns the generated second image to the terminal, and outputs the newly generated second image to the object through the terminal. Of course, the embodiment of the present application supports the user to select multiple style prompt information in batches after providing a first image, indicating that the user wants to perform multiple style conversions on the single first image; in this implementation, the server can perform style conversions on the first image according to each style prompt information, obtain the second image corresponding to each style prompt information, and return it to the terminal for display. The embodiment of the present application does not limit the number of style prompt information selected by the user, and the following description takes the number of style prompt information as 1 as an example, which is specially explained here.

[0104] It should be noted that Figure 3 The illustrated diagram is only an exemplary architecture diagram of the image processing system provided by the embodiment of the present application. In practical applications, the architecture can be adaptively changed. For example, the image processing scheme provided by the embodiment of the present application can be executed by a terminal. In this case, the aforementioned execution subject computer device is a terminal, and a pre-trained target image generation model is directly deployed in the terminal. In this implementation mode, after obtaining the first image uploaded by the object and the determined style prompt information, the terminal can directly call the target image generation model to perform the specific implementation process of the style conversion for the first image without sending the first image and the style prompt information to the server for related processing.

[0105] It should also be noted that the collection and processing of relevant data in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations. The acquisition of personal information requires the knowledge or consent of the individual subject (or the legal basis for obtaining the information), and the subsequent use and processing of data shall be carried out within the scope of authorization of laws and regulations and the subject of personal information. For example, when the embodiments of this application are applied to specific products or technologies, such as obtaining the first image, it is necessary to obtain the permission or consent of the object holding the first image, and the collection, use and processing of relevant data (such as the collection and release of the bullet screen posted by the object, etc.) need to comply with the relevant laws, regulations and standards of the relevant regions.

[0106] Based on the image processing scheme and system described above, the embodiment of the present application proposes a more detailed image processing method. The image processing method proposed in the embodiment of the present application will be described in detail below in conjunction with the accompanying drawings. It can be seen from the aforementioned related description that the image processing method provided in the embodiment of the present application mainly includes: model pre-training for the initial image generation model, and model inference using the pre-trained target image generation model. For ease of understanding, different embodiments are used in the following to introduce the specific implementation processes of the model pre-training part and the model inference part respectively.

[0107] See also Figure 4 , Figure 4 It is a flowchart of an image processing method provided by an exemplary embodiment of the present application; the flowchart mainly provides a specific implementation process of using a target image processing model to realize image style conversion from the perspective of model inference. Figure 4 The method flow shown can be executed by the aforementioned computer device, such as the aforementioned computer device. Figure 3 The server 302 is shown. The image processing method may include but is not limited to steps S401-S404:

[0108] S401: Acquire a first image to be processed and style prompt information.

[0109] (1) The first image is any image provided by the user that needs to be converted to an image style. It should be understood that the types of image content included in the first image are numerous, including but not limited to at least one of the following: faces, figures (such as half-length or full-length photos, etc.), animals and objects, etc. The embodiment of the present application does not limit the types and quantities of image content included in the first image. For ease of understanding, it is supported to divide the image content into foreground content and background content. For example, the foreground content included in the first image may include a face, and the background content may be blank or non-blank. It is worth noting that, as can be seen from the foregoing, the embodiment of the present application calls a pre-trained target image generation model to achieve image style conversion in the model inference stage; considering that 100,000-level data is used for pre-training in the model pre-training stage, and the image encoding features are directly embedded into the model to learn the content features of the image content, this makes it unnecessary for the embodiment of the present application to limit the image content included in the first image to the image content used in pre-training. That is to say, the embodiment of the present application allows that the image content included in the first image has not been learned in the pre-training stage, that is, even if the image content included in the first image has not been learned in the pre-training stage, better feature learning and style conversion can be achieved for the image content included in the first image, which greatly broadens the image style conversion scenarios or fields to which the present application is applicable.

[0110] The present application embodiment does not limit the method for the computer device to obtain the first image to be processed. Exemplarily, the method for obtaining the first image may include: receiving the first image uploaded by the object through the upload button (or option, component, etc.); or receiving the first image uploaded by the object through a drag operation; etc. Among them: ① The interface diagram for uploading the first image through the upload button can be seen in Figure 5a ,like Figure 5a As shown: the mini program provides the user with an image upload interface 501, and the image upload interface 501 includes an upload button 502; when the user performs a trigger operation on the upload button 502, it indicates that the user has a need to upload the first image to be processed, and a selection interface 503 is displayed, and the selection interface 503 is used to display one or more candidate images that can be uploaded (such as stored locally on the terminal device held by the user, or stored on the Internet, or stored in the cloud, etc.); in this way, when the user can select the first image from the selection interface 503, the mini program or the terminal device determines that the first image to be processed is obtained. Optionally, the selected first image can be displayed in the image upload interface 501, so that the user can adjust the image (such as screenshot or compression) before the first image is style converted, enriching the user's editing ability of the first image and improving the user experience. ② The interface diagram for uploading the first image by dragging and dropping can be seen in Figure 5b ,like Figure 5bAs shown: Assume that the image upload interface 501 includes a drag area 504 and an image display area 505; in response to the user's activation operation (such as a single-click operation) in the drag area 504, indicating the user's specific need to upload the first image to be processed, the user can select the first image that needs style conversion from one or more candidate images displayed in the image display area 505. Specifically, the user can directly perform a drag operation from the image display area 505 to the drag area 504 for any image; when the any image is moved to any display position in the drag area 504 and is released, it is determined that the user has selected the first image (that is, any image), and at this time the applet or terminal device determines that the first image to be processed is obtained. The selection of the first image is achieved by dragging in this way, and to a certain extent, there is no need for interface jumps, which simplifies the user's operation of selecting images and improves the flexibility of image selection.

[0111] It should be noted that the above Figure 5a and Figure 5b These are only two exemplary methods of obtaining the first image provided in the embodiments of the present application, and the embodiments of the present application do not limit the methods of obtaining the first image. For example, the first image can also be obtained by collecting data from a user holding a terminal device in real time (such as taking a photo or recording a video); if a shooting button is included in the image upload interface, in response to the user's selection of the shooting button, a camera deployed in the terminal device can be activated, and the user can be photographed through the camera, and the image obtained by the real-time shooting is used as the first image to be processed.

[0112] (2) Style prompt information is prompt information for indicating style conversion of the first image; when the first image has a first style, the style prompt information can be specifically used to indicate that the first image is converted from the first style to the second style. The first style and the second style are different, and the embodiment of the present application does not limit the style type of the image style (or simply referred to as style) to which the image belongs or has. The image style includes but is not limited to: cyber style, cartoon style, motorcycle style, animation style, realistic style or 3D style, etc. For example, if the first image to be processed is collected by a camera, then the first style of the first image can be a realistic style. At this time, the style prompt information can be used to indicate that the first image with a realistic style is converted into an animation style, and the animation style is the second style.

[0113] The embodiment of the present application does not limit the manner in which the computer device acquires the style prompt information. Exemplarily, the manner in which the style prompt information is acquired includes any of the following: a default setting; or, based on the input operation generated by the object (i.e., the user who has the need to perform style conversion on the first image) for the second style in the style selection interface. In other words, the image generation platform may be provided with a default second style, and no matter what type of the first style the first image has, the first image is converted into a second image with a fixed second style. Alternatively, the second style may also be determined by the user, so that the image generation platform can generate style prompt information for indicating that the first image is converted from the first style to the second style based on the second style selected by the user. Among them, according to the difference in interface elements included in the style selection interface provided by the image generation platform, the input operation of the object for the second style in the style selection interface is different. For example, in the case where the style selection interface includes one or more candidate styles, the input operation is a selection operation of selecting the second style from one or more candidate styles, and the second style is any one of the one or more candidate styles. For another example, in the case where the style selection interface includes a style input area, the input operation is an input operation of inputting the second style in the style input area.

[0114] S402: Perform image coding processing on the first image to obtain image coding features.

[0115] After acquiring the first image to be processed, the computer device can call the pre-trained target image generation model to directly perform image coding processing on the first image to obtain the image coding features of the first image; the image coding features can be used to characterize the content characteristics of the image content included in the first image. For details, see the aforementioned Figure 2b It can be seen from the model structure of the target image generation model shown that the target image generation model includes a first image encoding module, a second image encoding module and a feature fusion module. The image encoding process for the first image is mainly performed by the first image encoding module, the second image encoding module and the feature fusion module. In a specific implementation, the first image encoding module can perform image encoding processing on the first image to obtain the first sub-image feature; similarly, the second image encoding module can perform image encoding processing on the first image to obtain the second sub-image feature; then, the feature fusion module performs feature fusion on the first sub-image feature and the second sub-image feature to obtain the image coding feature corresponding to the first image.

[0116] The first image coding feature, the second image coding feature and the feature fusion model mentioned above are briefly introduced below; among them:

[0117] (1) Both the first image encoding module and the second image encoding module are modules capable of extracting or analyzing image features, such as the first image encoding module is a CLIPImageEncoder module, and the second image encoding module is a VAEEncoder module. That is, the embodiment of the present application supports using CLIPImageEncoder and VAEEncoder to simultaneously perform image encoding processing on the first image, so as to simultaneously encode the picture features of the first image to obtain the picture features of the first image (i.e., image encoding features).

[0118] ①The CLIP model consists of two parts: TextEncoder (which can be expressed as CLIPTextEncoder) and ImageEncoder (which can be expressed as CLIPImageEncoder); CLIPTextEncoder is responsible for text encoding, and CLIPImageEncoder is responsible for image encoding. Specifically, the CLIPTextEncoder in the CLIP model can perform text encoding on the text to extract the semantic features of the text. Similarly, the CLIPImageEncoder in the CLIP model can perform image encoding on the image to extract the image features of the image. Furthermore, the text features of the text and the image features of the image extracted by the two parts of the CLIP model can be expressed in vector form, and there is a semantic mapping relationship between the text features and the image features in vector form; specifically, by using the contrastive learning method, the CLIP model can continuously strengthen the relationship between text and images that express the same meaning / semantics (as reflected in the fact that when the vector distance between the text features of the text and the image features of the image is close, it means that the relationship between the text and the image is close, that is, the meanings expressed by the text and the image are similar), and weaken the relationship between text and images that express different meanings. Considering that the CLIP Image Encoder in the CLIP model has better image encoding capabilities, the embodiment of the present application selects the CLIP Image Encoder in the CLIP model (such as the version may be openai / clip-vit-large-patch14b version) to implement encoded image features; in this way, the output result of the last Transformer layer in the CLIP Image Encoder can be selected as the first sub-image feature obtained for image encoding, which can be defined as C, and the dimension of C is (257, 1024).

[0119] ②The VAE model is a generative model that contains latent variables and can use neural networks to generate data that is not included in the input data. The VAE model mainly consists of two parts: the Encoder (which can be expressed as a VAE Encoder) module and the Decoder (which can be expressed as a VAE Decoder) module, which are responsible for encoding and decoding images respectively; among them, the VAE Encoder module can map the input image I to a latent vector L, and the VAE Decoder module can decode the latent vector L to an image I′, and the latent vector I and the latent vector I′ are highly consistent. Figure 6 As shown, in the diffusion model, the VAEEncoder module maps the image I to a latent vector L. The diffusion model training process includes: adding noise to L to obtain a noise image after adding noise; then, the Unet module can denoise the noise image to obtain a latent vector L′; finally, the VAEDecoder module decodes the latent vector L′ into the image I′; in the above process, the latent vector L and the latent vector L′ are highly consistent. From the above analysis, it can be seen that the latent variable L obtained by encoding the image with VAEEncoder has a strong correlation with the image itself. Therefore, this application supports the selection of the latent variable L of the input image to represent the image feature of the input image, that is, the latent vector L is used as the second sub-image feature of the image encoding (that is, the second sub-image feature can be expressed as L). Among them, the dimension of the image feature output by the VAEDecoder module is (3, image_w / scale, image_h / scale). Among them, "3" is three RGB channels, image_w represents the width of the input image, image_h represents the height of the input image, and scale is generally taken as 8, which means that the dimensionality reduction ratio of the VAEEncoder module is 8. Exemplarily, the embodiment of the present application can unify the width and height of the input image to 1024, so the values ​​of the above image_w and image_h are both 1024.

[0120] (2) Feature fusion module.

[0121] Based on the first image encoding module CLIPImageEncoder mentioned above, the input first image is encoded to obtain the first sub-image feature C, and the second image encoding module VAEEncoder is encoded to obtain the second sub-image feature L. In order to unify the dimensions of the first sub-image feature C and the second sub-image feature L, the embodiment of the present application supports the use of a feature fusion module to achieve feature fusion. Among them, the feature fusion module may refer to a module with feature fusion capabilities. The embodiment of the present application takes the feature fusion module as the TransformLayer module as an example; the flowchart of the TransformLayer module achieving the feature fusion of the first sub-image feature C and the second sub-image feature L corresponding to the first image can be seen in Figure 7 .

[0122] like Figure 7 As shown, the first image feature module and the second image feature module described above are used to perform image coding processing on the first image respectively, and after obtaining the first sub-image feature C and the second sub-image feature L, the first sub-image feature C and the second sub-image feature L are respectively subjected to average operation (i.e., averaging), linear layer conversion, removing dimensions with a dimension of 1, and matrix multiplication, etc., to output the image coding feature of the first image, which can be expressed as ΔW, and the dimension of the image coding feature is (image_emb_dim, image_emb_dim), and image_emb_dim can be defined as 1024. Among them, the above-mentioned average operation can be simply understood as finding the mean of another dimensional matrix based on the original dimension of the sub-image feature (such as the first sub-image feature C), which can effectively achieve dimensionality reduction. The linear layer can unify the dimensions after the average operation, such as unifying them to 1024. Removing the dimension with a dimension of 1 means deleting the dimension with a dimension of 1 in the dimension, and only retaining the dimension that is not 1. Matrix multiplication may refer to multiplying the first sub-image feature C after averaging (i.e. averaging), linear layer conversion, and removal of dimensions with a dimension of 1, and the second sub-image feature L after averaging (i.e. averaging), linear layer conversion, and removal of dimensions with a dimension of 1 to generate an image coding feature ΔW of the first image.

[0123] In summary, on the one hand, the embodiment of the present application supports the method of simultaneously encoding image features using the first image encoding module CLIPImageEncoder and the second image encoding module VAEEncoder, which can rely on the image feature analysis advantages of CLIPImageEncoder and VAEEncoder to fully extract or mine the content characteristics of the image content included in the first image, such as the contour, shape, color, size and texture of the image content, so as to utilize the rich and detailed content characteristics to help improve the image quality of the subsequently generated second image. On the other hand, it also supports the use of the feature fusion module TransformLayer to fuse the sub-image features encoded by the two image encoding modules, which can achieve the unification of feature dimensions while also fusing to obtain richer and more comprehensive image coding features, so that when the second image is generated based on the image coding features, the concreteness of the second image can be enhanced.

[0124] S403: Using an attention algorithm to perform a linear transformation on the first image and the style hint information, and embedding the image encoding features into the linear transformation to generate target semantic features.

[0125] like Figure 2b The target image generation model shown includes a Unet module, which is a U-shaped network structure that can be used in a diffusion model to implement feature extraction and feature fusion to generate new images. Among them, the Unet module includes one or more attention mechanism layers (such as a cross attention mechanism (Cross Attention) layer); the attention mechanism layer is an image and text interaction layer, and the attention algorithm can be used to calculate the encoding result of the image (such as the first image) under the condition of the text (such as the style prompt information involved in the embodiment of the present application). In this way, the attention mechanism layer in the Unet module can use the attention algorithm to perform a linear transformation on the first image and the style prompt information, and the linear transformation mainly realizes the extraction of key information from the first image and the style prompt information. Among them, the formula of the attention algorithm deployed in the attention mechanism layer is as follows:

[0126]

[0127] The calculation process of the attention algorithm involves linear transformation, which includes three sub-linear transformations, namely the first sub-linear transformation Key, the second linear transformation Value, and the third linear transformation Query. K in formula (1) is the matrix output by the first sub-linear transformation Key, V in formula (1) is the matrix output by the second linear transformation, and Q in formula (1) is the matrix output by the third linear transformation Query. K T represents the transpose of the matrix K. d kIt is a value, such as 768, which is mainly used to reduce the dot product range and ensure the stability of the softmax gradient. Softmax() is a normalization function, which is mainly used to ensure the non-negativity of the attention weight and increase nonlinearity.

[0128] It can be seen that the calculation process of the attention algorithm uses three matrices, namely KQV, which can be used to calculate the mutual dependence between different tokens; for text, the token here can be understood as a string (including one or more characters), and for images, the token here can be understood as different parts of the image. In more detail, K and Q can be used to calculate the similarity between the current token and other tokens. This similarity is used as a weight to perform a weighted summation on V, and the result of the weighted summation can be used as the token of the next layer. In the CrossAttention used in the Unet module, KV comes from the text (such as style prompt information) and Q comes from the image (such as the first image). Based on this, the attention mechanism layer in the Unet module allows the model to deeply feel the characteristics represented by the first image and style prompt information, thereby extracting more accurate second style and image characteristics, ensuring that the extracted style semantic characteristics and image characteristics are purer.

[0129] Based on the above introduction to the attention mechanism layer in the Unet model, the embodiment of the present application does not limit the number of attention mechanism layers (or attention algorithm layers) included in the Unet module; however, the specific implementation process of each attention mechanism layer using the attention algorithm to perform linear transformation on the first image and the style prompt information is similar. Exemplarily, any attention mechanism layer in the Unet module can perform linear transformation on the first image and the style prompt information; specifically: the first sub-linear transformation Key and the second sub-linear transformation Value included in the linear transformation can be used to extract text semantics from the style prompt information, so that the execution of the first sub-linear transformation Key and the second sub-linear transformation Value can obtain the style conversion characteristics of the style prompt information; the style conversion characteristics are used to characterize the style parameters required to convert from the first style to the second style, so that the style conversion for the first image can be achieved based on the style parameters. The third sub-linear transformation Query included in the linear transformation is used to extract image features from the first image, so that the execution of the third sub-linear transformation Query can obtain the image characteristics of the first image; the image characteristics are used to characterize the image data of the first image, and the image data can be some basic or conventional data about the first image, and can include at least one of the following: resolution, size data, pixel data, etc.

[0130] It has been verified that the CrossAttention layer in the Unet model (i.e. the attention mechanism layer mentioned above) occupies an important position in the diffusion model, and even with a small amount of parameters, the model can achieve good fitting ability. Based on this, in order to solve the problem of poor text concreteness in the traditional diffusion model that generates images based only on text, the embodiment of the present application makes full use of the strong feature analysis capability of the CrossAttention layer in the Unet model, and designs to directly introduce the image coding feature ΔW obtained by image coding processing for the first image into the CrossAttention layer. This enables the CrossAttention layer to combine the image coding features of the first image for feature analysis in the process of feature analysis of the first image and style hint information, and analyze to obtain more comprehensive, rich and pure semantic features of the first image.

[0131] In detail, the linear transformation performed by the CrossAttention layer on the first image and the style hint information includes a linear projection transformation process. Specifically, the Key, Query and Value in the CrossAttention layer can be regarded as modules respectively; for example, Query is a module, which includes multiple submodules, and the multiple submodules are connected in sequence, so that the output of the previous submodule can be used as the input of the next submodule; wherein the last submodule in the Query module is used to implement the linear projection transformation process, and the output of the last submodule is the final output of the Query module. The embodiment of the present application supports embedding the image coding features obtained by performing image coding processing on the first image into the linear transformation process performed by the CrossAttention layer. Among them, the general logic of embedding can be summarized as follows: after the computer device calls the target image generation model, the attention mechanism layer in the Unet included in the target image generation model can be used to perform a linear transformation on the first image and the style hint information, and obtain the initial semantic features obtained by performing the linear projection transformation process in the linear transformation process; then, the image coding features of the first image obtained by using the first image coding module, the second image coding module and the feature fusion module in the target image generation model are combined with the initial semantic features to generate the combined target semantic features.

[0132] Further, as mentioned above, the linear transformation of the attention mechanism layer includes a first sub-linear transformation Key, a second sub-linear transformation Value and a third sub-linear transformation Query, and each sub-linear transformation (such as the first sub-linear transformation, the second sub-linear transformation and the third sub-linear transformation) includes a linear projection transformation process (i.e., the last linear transformation in each sub-linear transformation). Therefore, the above-described embedding of the image coding feature into the linear transformation process performed by the CrossAttention layer specifically includes embedding the image coding feature into the linear projection part of KQV in the CrossAttention layer. In other words, the above-described logic of combining the image coding feature with the initial semantic feature may specifically include: combining the image coding feature with the initial semantic feature obtained by performing the linear projection transformation process in the first sub-linear transformation Key process to generate the first sub-semantic feature. And, combining the image coding feature with the initial semantic feature obtained by performing the linear projection transformation process in the second sub-linear transformation Value process to generate the second sub-semantic feature. And, combining the image coding feature with the initial semantic feature obtained by performing the linear projection transformation process in the third sub-linear transformation Query process to generate the third sub-semantic feature. Then, a target semantic feature is generated based on the first sub-semantic feature, the second sub-semantic feature and the third sub-semantic feature, that is, the target semantic feature includes the first sub-semantic feature, the second sub-semantic feature and the third sub-semantic feature corresponding to the three sub-linear transformations. According to the different data types (such as text or image) processed by the sub-linear transformations, the initial semantic features obtained by executing the corresponding sub-linear transformations are different; for example, the initial semantic features obtained by executing the linear projection transformation in the first sub-linear transformation process and the linear projection transformation in the second sub-linear transformation process are both style conversion features of the style conversion prompt information; for another example, the initial semantic features obtained by executing the linear projection transformation in the third sub-linear transformation process are image features of the first image.

[0133] Furthermore, taking the embedding of the image coding features of the first image into the third sub-linear transformation Query included in the linear transformation as an example, it is assumed that the initial semantic features obtained by the third sub-linear transformation performing the linear projection transformation are represented as W0x, where W0 is the original parameter of the diffusion model, and x is the output of the previous sub-module adjacent to the last sub-module in the Q module, and the output is used as the input of the last sub-module. Similarly, it is assumed that the image coding features obtained by performing image coding processing on the first image are represented as ΔW. Then the formula for embedding the image coding features ΔW of the first image into the third sub-linear transformation to perform the linear projection transformation processing can be as follows:

[0134] h=W0x+ΔWx (2)

[0135] Among them, h is the third sub-semantic feature obtained by embedding the image encoding feature ΔW into the last sub-module in the Q module.

[0136] It should be understood that the embedding logic of embedding the image coding features of the first image in the first sub-linear transformation Key and the second sub-linear transformation Value is the same as the embedding logic of embedding the image coding features of the first image in the third sub-linear transformation Query described above, and will not be repeated here.

[0137] S404: Generate a second image based on the target semantic feature.

[0138] After obtaining the target semantic features that have both the content characteristics of the image content included in the first image, the image characteristics of the first image, and the style conversion characteristics of the style prompt information based on the above steps, a second image can be generated based on the target semantic features. The second image has the second style indicated by the style prompt information, and the image content included in the second image is the same as the image content included in the first image; taking the face of object A as an example, from the visual effect, it can be observed that the second image with the second style can identify that the image content "face" included in the second image belongs to object A.

[0139] In summary, in the model inference stage of the embodiment of the present application, it is only necessary to load the model parameters of the feature fusion module TransformLayer obtained by the pre-trained initial image generation model to obtain the target image generation model; then, after obtaining the first image and style prompt information, the image coding features of the first image can be obtained and saved, and the image coding features can be inserted into the target image generation model; thereafter, it is only necessary to change the style prompt information (or style prompt words) to obtain the second image that maintains the similarity of the image content and changes the style, without the need for other encoded images and fine-tuning operations. In the above process, the image coding features of the first image are directly extracted and embedded into the model, which not only realizes the introduction of pure image features in the model to make up for the poor concreteness of single text, but also effectively reduces the cost required to generate the second image compared to the need for dozens or even hundreds of images to train the model to learn image features.

[0140] Above Figure 4 The embodiment mainly introduces the image processing method from the perspective of model prediction. Figure 8 The complete image processing method is introduced; among them, Figure 8 The method is mainly explained from the perspective of model pre-training. Figure 8 The method flow shown can be executed by the aforementioned computer device, such as the aforementioned computer device. Figure 3The server 302 is shown. The image processing method may include but is not limited to steps S801-S806:

[0141] S801: Construct a pre-training data set.

[0142] In order to improve the generalization ability of the model after pre-training and realize no training in the model inference stage, the embodiment of the present application proposes to independently construct a pre-training data set of 100,000 levels for pre-training of the initial image generation model. Among them, the construction of the pre-training data set mainly includes two parts, namely, a public data set and a self-collected data set. Among them, the public data set, also known as the public data set, refers to some data sets with a large amount of data that have been made public. The embodiment of the present application does not linearly classify the types of public data sets; the self-collected data set refers to a set of data independently retrieved from the Internet through keyword retrieval and other technologies.

[0143] In a specific implementation, in the process of constructing a pre-training data set, a first number of sample images can be selected from a public data set to form a first data set, and a second number of sample images can be searched from the Internet to form a second data set (i.e., the self-collected data set mentioned above). Exemplarily, taking the public data set as the MS-Celeb-1M data set, the MS-Celeb-1M data set contains at least 100,000 facial images corresponding to celebrities, and the number of facial images corresponding to each celebrity is 100. The embodiment of the present application supports selecting one facial image of each celebrity from the MS-Celeb-1M data set to form the first data set; then it is determined that the first data set includes at least 100,000 (i.e., the first number) sample images, and each sample image includes a face. Considering that the sample images screened from the MS-Celeb-1M data set mainly include faces, in order to increase the diversity of sample data, the embodiment of the present application also supports the use of some search technologies to search for sample images from the Internet to form a second data set.

[0144] Similarly, the example process of searching sample data from the Internet can be found in Fig. 9. First, the images related to the keywords are retrieved by inputting keywords on the Internet, and then some image search technologies or engines are used to download the images related to the keywords. Finally, the basic rules are used to filter out the images of poor quality, and the unfiltered second number of images (such as 10,000) are used to form the second data set; wherein the basic rules may include but are not limited to: the image size of the image is greater than 512*512, the image quality score (aesthetic_score) is higher than 6.5 points, etc. It can be seen that, on the one hand, the sample images are directly selected from the public data set to form the first data set, which effectively improves the collection efficiency of the sample images, and ensures that enough face images of different users can be collected to increase the learning ability of the model; on the other hand, the sample images included in the second data set collected through the Internet are more diverse than the sample images in the public data set, such as the image content can include not only faces, but also sample images of multiple scenes such as full body or half body, thereby ensuring the diversity of sample images in the pre-training data set.

[0145] Furthermore, after the first data set and the second data set are constructed based on the above description, the embodiment of the present application also supports data processing of sample images in the first data set and the second data set, while improving the quality of the sample images, ensuring that the data in the final constructed pre-trained data set meets the data format requirements of the pre-trained model. The data processing involved in this application mainly includes: generating corresponding text descriptions for sample images, and optimizing the image quality of sample images; the following takes any sample image in the first data set or the second data set as an example to represent the target sample image, and introduces the process of data processing of the target sample image, wherein:

[0146] (1) Generate corresponding target text description for the target sample image.

[0147] like Figure 2a In the schematic diagram of the initial image generation model shown, the initial image generation model includes a text description module. The embodiment of the present application supports using the text description module to perform text generation processing on the target sample image to generate a target text description corresponding to the target sample image. The text description module is a module that generates text that can express the image semantics of the image based on the image. In the embodiment of the present application, the text description module is taken as a BLIP model as an example. The BLIP model is a multimodal Transformer model. The model structure diagram of the BLIP model can be found in Fig.10 ,like Fig.10As shown in the figure, the BLIP model is an encoder-decoder hybrid architecture. It mainly consists of four parts: Image Encoder, Text Encoder, Image-grounded Text Encoder and Image-grounded Text Decoder; Image Encoder and Text Encoder are responsible for encoding images and texts respectively; Image-grounded Text Encoder is a binary classification model responsible for judging whether the text and image express the same meaning; Image-grounded Text Decoder is a text generation model responsible for generating corresponding text descriptions based on images. The schematic diagram of using the BLIP model to generate the corresponding target text description based on the target sample image can be seen in Fig.11 ;like Fig.11 As shown in FIG. 1 , after obtaining the target sample image, the Image-grounded Text Decoder module in the BLIP model is used to generate a corresponding target text description for the target sample image.

[0148] (2) Optimize the image quality of the target sample image.

[0149] Taking into account the problems of inconsistent image resolution and size of the sample images included in the first data set and the second data set; in order to further improve the quality and uniformity of the sample images, the embodiment of the present application proposes to optimize the quality of the sample images in the first data set and the second data set to improve the image quality of the sample images included in the final pre-training data set. In a specific implementation, image quality processing is performed on the target sample image to obtain a processed target sample image; wherein the image quality processing includes at least: square mapping processing, image enhancement processing, and size normalization processing, etc. The specific process of the quality optimization process can be found in Fig.12 ,like Fig.12As shown: First, the target sample image is squared, specifically, the rectangular target sample image is converted into a square; in order to retain as much information as possible in the original image, it is supported to convert the rectangle into a square by adding white edges to the sides of the target sample image. Then, the squared target sample image is subjected to image enhancement processing, specifically, the resolution of the target sample image is determined. When it is detected that the resolution of the target sample image is lower than the resolution threshold (such as 1024*1024), the target sample image is subjected to image enhancement processing to enhance the low-resolution target sample image into a high-resolution target sample image, thereby improving the image quality of the target sample image. Among them, the embodiments of the present application do not limit the technical means for achieving image enhancement, such as supporting the use of a CodeFormer model to enhance a low-resolution target sample image. The CodeFormer model supports the organic combination of a variational autoencoder (VQGAN) and a Transformer to repair the input blurred and / or mosaic target sample image and output a target sample image with a higher resolution. Among them, the variational autoencoder VQGAN can be used to encode and decode the target sample image to generate a clear and realistic target sample image, mainly through the quantization method to ensure the high quality of the enhanced target sample image; Transformer uses the self-attention mechanism to judge the dependencies between the pixels of the target sample image, realize global context perception, and generate a coherent and natural restoration result. It can be seen that CodeFormer achieves the best effect of restoring the target sample image by cleverly integrating the advantages of VQGAN and Transformer. Finally, the target sample image after image enhancement is normalized and the image size of the target sample image is adjusted to 1024*1024.

[0150] It should be noted that the text description module mentioned above for generating the corresponding target text description for the target sample image may also be other types of models, which is not limited in the embodiments of the present application. In addition, the specific means and implementation process for the normalization processing of the target sample image are not limited to the image enhancement, square diagram and size unification described above, and may also include means such as cropping, enhancement filtering, etc., which are not limited in the embodiments of the present application.

[0151] Furthermore, in order to ensure the quality of the training data of the pre-training data set, the embodiment of the present application supports performing the above-mentioned data processing on all sample images in the first data set and the second data set, and retains only sample images whose aesthetic scores are greater than a score threshold (such as 6.5 points) from all sample images and adds them to the pre-training data set. Taking the aesthetic score of the target sample image as an example, which is greater than the score threshold, the target sample image and the target text description generated for the target sample image can be combined into an image-text pair and added to the pre-training data set.

[0152] S802: Pre-training the initial image generation model using a pre-training data set to obtain a target image generation model.

[0153] After the 100,000-level pre-training data set is constructed based on the aforementioned step S801, the embodiment of the present application supports the use of the 100,000-level pre-training data set to pre-train the initial image generation model. It is worth noting that when pre-training the initial image generation model, the model pre-training is specifically implemented according to the training parameters; an exemplary training parameter can be seen in Table 1:

[0154] Table 1

[0155] Initial Image Generation Model stablediffusion 1.5 batch_size 64 Learning Rate 1e-5 Number of training rounds 20

[0156] Batch_size: The number of image-text pairs selected for one round of pre-training. The learning rate is one of the important hyperparameters for model training, representing the step size of the gradient moving toward the optimal solution of the loss function in each iteration. The number of training rounds refers to the number of iterations of the initial image generation model on the entire pre-training data set.

[0157] The following is an example of calling the initial image generation model to process an image-text pair in a round of model training to briefly introduce the model pre-training process. The specific implementation process of the pre-training model can be found in Fig.13 , including but not limited to steps s11-s15:

[0158] s11: Get image-text pairs from the pre-training dataset.

[0159] As described above, the pre-training data set is used to pre-train the initial image generation model; the pre-training data set includes a plurality of image-text pairs obtained based on the data screening and data processing described above, and each image-text pair includes a sample image and a text description for describing the sample image. According to the training parameters shown in Table 1, in each round of pre-training, 64 image-text pairs can be selected from the pre-training data set to implement iterative training of the initial image generation model obtained in the previous round of pre-training.

[0160] s12: Use the initial image generation model to matrix the sample images in the image-text pair to obtain the predicted image encoding features.

[0161] Optionally, the matrix processing performed on the sample image in the image-text pair may include image coding processing; in this implementation, the image coding features extracted from the first image may be directly embedded into the model in the model inference stage. Specifically, the initial image generation model includes a first image coding module, a second image coding module, and a feature fusion module; it is worth noting that the model parameters of the first image coding module and the second image coding module are fixed and non-trainable, while the model parameters of the feature fusion module are trainable, that is, the feature fusion module includes the trainable parameters of the initial image generation model; therefore, after the pre-training of the initial image generation model is completed, the model parameters of the feature fusion module can be retained, and the target image generation model can be obtained by loading the model parameters of the feature fusion module in the model inference stage.

[0162] The following is an overview of the process of image encoding processing for sample images in the image-text pair in the pre-training phase: using the first image encoding module (such as CLIP ImageEncoder) in the initial image generation model to perform image encoding processing on the sample images in the image-text pair to obtain first predicted sub-image features; and using the second image encoding module (such as VAEEncoder) in the initial image generation model to perform image encoding processing on the sample images in the image-text pair to obtain second predicted sub-image features; in this way, the feature fusion module in the initial image generation model is used to perform feature fusion on the first predicted sub-image features and the second predicted sub-image features to obtain predicted image coding features corresponding to the image-text pair.

[0163] It should be noted that the specific implementation process of the image encoding processing for the sample image in the pre-training stage described above is similar to the specific implementation process of the image encoding processing for the first image described in the aforementioned step S402, and will not be described in detail here.

[0164] Optionally, the matrix processing performed on the sample images in the image-text pair may include image coding processing and feature dimensionality reduction processing; in this implementation, after the image coding features of the first image are extracted in the model inference stage, the image coding features need to be subjected to dimensionality reduction processing, and the feature matrix after dimensionality reduction is embedded into the model. Among them, the image coding processing is similar to the specific implementation logic of the image coding processing described above, and the feature dimensionality reduction processing is the optimization of the image coding features obtained by the image coding processing on the basis of the image coding processing, aiming to reduce the feature dimension in the form of matrixization and reduce the amount of feature parameters inserted into the model. The embodiment of the present application elaborates on the feature dimensionality reduction processing in the subsequent step S804, which is specifically explained here.

[0165] s13: Use the initial image generation model to perform semantic extraction on the text description in the image-text pair to obtain predicted semantic features.

[0166] Specifically, the initial image generation model includes a text semantic extraction module, so that after receiving the input image text pair, the initial image generation model can call the text semantic extraction module to perform text semantic extraction processing on the text description in the image text pair to obtain a predicted semantic feature; the predicted semantic feature is used to characterize the semantic characteristics of the text description. Figure 2a As shown, the embodiment of the present application supports using the CLIPTextEncoder module in the CLIP module as a text semantic extraction module in the initial image generation model to perform text semantic extraction; for the relevant content of the CLIPTextEncoder module, please refer to the relevant description shown in the aforementioned step S402, which will not be repeated here.

[0167] s14: Embed the predicted image encoding features into the attention algorithm layer, and combine them with the predicted semantic features to generate the predicted image corresponding to the sample image.

[0168] like Figure 2a The initial image generation model shown and Figure 2b As can be seen from the target image generation model shown, the model structures of the initial image generation model and the target image generation model are similar. The process of pre-training the initial image generation model and the process of using the target image generation model to perform style conversion on the first image are similar, and both require the encoding features of the image (such as the sample image in the pre-training stage and the first image in the model inference stage) (such as the predicted image encoding features of the sample image in the pre-training stage and the image encoding features of the first image in the model inference stage) to be embedded into the model (the initial image generation model in the pre-training stage and the target image generation model in the model inference stage); specifically, it is embedded into the linear projection transformation processing in the attention algorithm layer (or attention mechanism layer) in the Unet module included in the model. It is just that the feature fusion module in the initial image generation model needs to be parameter optimized in the pre-training stage, and only the model parameters of the pre-trained feature fusion module need to be loaded in the model inference stage.

[0169] Based on this, the specific embedding logic of embedding the predicted image coding features of the sample image into the attention algorithm layer in the pre-training stage shown in step s14 can refer to the description of the specific embedding logic of embedding the image coding features of the first image into the linear transformation in the aforementioned model inference stage, which will not be repeated here. In this way, the initial image generation model can combine the predicted semantic features of the text description in the image sample pair and the embedded image coding features to generate a predicted image.

[0170] s15: Based on the predicted image and the sample image, pre-train the initial image generation model to obtain a pre-trained target image generation model.

[0171] After the image sample pair is processed by the initial image generation model in the above steps s12-s14 to obtain a predicted image, the loss function of the initial image generation model can be calculated based on the image difference between the predicted image and the sample image in the image sample pair, and the initial image generation model can be iteratively trained according to the loss function; specifically, the model parameters of the feature fusion module included in the initial image generation model are adjusted in the direction of reducing the image difference to obtain a pre-trained target image generation model.

[0172] S803: Acquire a first image to be processed and style prompt information.

[0173] S804: Perform image coding processing on the first image to obtain image coding features.

[0174] S805: Use an attention algorithm to perform a linear transformation on the first image and the style hint information, and embed the image encoding features into the linear transformation to generate target semantic features.

[0175] S806: Generate a second image based on the target semantic feature.

[0176] It should be noted that the specific implementation process of the embodiment shown in steps S803-S806 is the same as the above Figure 4 The specific implementation process shown in steps S401-S404 in the illustrated embodiment is similar, and reference may be made to the relevant description of steps S401-S404, which will not be repeated here.

[0177] It should also be noted that, as mentioned above, the embodiment of the present application is to embed the image coding features of the first image into the model to realize the introduction of the content characteristics of the image content into the model. Among them, the image coding features of the first image are expressed in the form of a matrix. Considering that the matrix may be a sparse matrix containing a large number of 0s, not only the storage volume is large but also the space is wasted. In order to minimize the number of parameters of the embedded model and improve the embedding efficiency, the embodiment of the present application also supports the dimensionality reduction of the image coding features of the first image, aiming to reduce the number of parameters of the embedded model while extracting the main features from the image coding features. In a specific implementation, the initial image generation model includes a matrix conversion module, so that after the initial image generation model performs image coding processing on the first image and obtains the image coding features of the first image, the matrix conversion module can be used to perform feature dimensionality reduction processing on the image coding features of the first image to obtain the feature matrix corresponding to the image coding features; the dimension of the feature matrix is ​​smaller than the dimension of the image features. Then, the feature matrix corresponding to the image coding features is embedded in the linear transformation to generate the target semantic features.

[0178] In detail, the embodiment of the present application supports the use of a singular value decomposition (SVD) algorithm to perform feature dimensionality reduction processing on the image coding features of the first image. Among them, the SVD algorithm is an algorithm widely used in the field of machine learning. It does not need to limit the matrix to be decomposed to a square matrix, that is, it supports feature decomposition of matrices of any dimension. It can be used in recommendation systems and natural language processing and other fields. It is the cornerstone of many machine learning algorithms. In the embodiment of the present application, the singular value decomposition SVD algorithm is mainly used to perform feature decomposition on the image coding features of the first image to achieve the purpose of feature dimensionality reduction.

[0179] In a specific implementation, the specific process of using the SVD algorithm to implement feature dimension reduction processing for the image coding features of the first image may include:

[0180] First, the SVD algorithm is used to perform matrix decomposition of the image coding features according to the row and column information of the image coding features, and two orthogonal matrices and a diagonal matrix corresponding to the image coding features are obtained. Among them, the row and column information of one of the two orthogonal matrices is m×m, and the row and column information of the other orthogonal matrix is ​​n×n. The row and column information of the diagonal matrix is ​​m×n, and the elements of the diagonal matrix except the elements on the main diagonal are 0. The values ​​of the elements on the main diagonal are used to characterize the importance of the content characteristics, and the elements on the main diagonal are arranged from large to small according to the values.

[0181] like Fig.14a As shown, assuming that the image coding feature of the first image is expressed as a matrix ΔW, and the row and column information of the image coding feature is m×n, m and n are positive integers, that is, the matrix ΔW is a matrix with a dimension of m×n, then the SVD formula for defining the image coding feature ΔW according to the SVD algorithm is:

[0182] ΔW=U∑V T (3)

[0183] Among them, U is an m×m orthogonal matrix, V is an n×n orthogonal matrix; both U matrix and V matrix are unitary matrices, that is, U satisfies T U=I,V T V = I. ∑ is an m×n diagonal matrix (or called a singular value matrix), and all elements except the elements on the main diagonal are 0, and each element on the main diagonal is called a singular value.

[0184] Then, r elements are selected from the n elements on the main diagonal of the diagonal matrix in order of the values ​​of the elements from large to small to form the second submatrix, and the first submatrix and the third submatrix are constructed based on the two orthogonal matrices and the second submatrix. Specifically, it can be seen from the above description that the image coding features of the first image can be decomposed into two orthogonal matrices and a diagonal matrix ∑, where the diagonal matrix ∑ is a singular value matrix, which is similar to the eigenvalue in the eigendecomposition. In the singular value matrix, the elements are arranged from large to small, and the singular values ​​decrease very quickly. In many cases, the sum of the first 10% or even 1% of the singular values ​​(i.e., the values ​​of the elements) in the singular value matrix accounts for more than 99% of the sum of all the singular values ​​in the singular value matrix. The larger the value of the singular value, the more important the content characteristics that characterize the image content in the first image. Therefore, if Fig.14b As shown, the embodiment of the present application supports selecting the r singular values ​​with the largest values ​​from the diagonal matrix ∑ to construct the second submatrix ∑ r×r ; and, constructing the first submatrix U based on the left and right singular vectors of the r singular values ​​respectively m×r and the third submatrix That is to say, the matrix ΔW can be represented by three sub-matrices with small parameter amounts, which significantly reduces the number of parameters.

[0185] Finally, the first sub-matrix, the second sub-matrix and the third sub-matrix are multiplied in sequence to obtain the feature matrix corresponding to the image coding feature. In other words, the first sub-matrix U is obtained by decomposing the image coding feature ΔW of the first image. m×r , the second submatrix ∑ r×r and the third submatrix By multiplying, the feature matrix after the feature dimension reduction process can be obtained. Considering that the parameter amount of the feature matrix after the dimension reduction process is small, in the embodiment of the present application, the feature matrix can also be called a lightweight auxiliary matrix.

[0186] Furthermore, the embodiment of the present application uses a singular value decomposition SVD algorithm to decompose the complex image coding feature ΔW of the first image into the product of three sub-matrices with smaller parameters and simpler parameters, that is, these three sub-matrices with small parameters are used to describe the important characteristics of the matrix with large parameters, effectively reducing the number of parameters of the image features inserted into the model and improving the style conversion efficiency. Among them, the formula for inserting the image coding feature of the first image into the model (specifically, into the linear transformation) in the form of a lightweight auxiliary matrix is ​​as follows:

[0187] h=W0x+ΔWx=W0x+BAx (4)

[0188] It can be seen from the formula that the embodiment of the present application decomposes the matrix ΔW into the product of two matrices BA, where:

[0189] B=Um×r ∑ r×r (5)

[0190]

[0191] By decomposing the matrix ΔW into the product of two BA matrices, the m×n dimensional matrix ΔW can be reduced to r×r, where r is much smaller than n. In theory, the smaller the rank r of the multiplication between BA matrices, the smaller the number of parameters inserted into the model.

[0192] In summary, the embodiment of the present application proposes a personalized style transfer method based on the traditional text image diffusion model to generate a lightweight auxiliary matrix based on an active image (such as a first image provided by a user), which can achieve the generation of images that maintain image content similarity (such as face similarity) and change image style without training. Specifically, on the one hand, without manual annotation, a pre-training data set consisting of 100,000 high-quality image sample pairs is constructed, and the image generation model is pre-trained using the pre-training data set to effectively improve the generalization of the model, reduce the risk of overfitting and the training cost of new image content. On the other hand, the image encoding features (or simply referred to as image features) of the image are inserted into the model in the form of a lightweight auxiliary matrix, which makes up for the shortcomings of poor text concreteness in the traditional technology that only relies on the text used in training to generate images, and improves the similarity between the image content included in the generated image and the image content included in the active image. In addition, it has been verified experimentally that the example of the present application can introduce new or specific image content without training on a pre-trained model. In subsequent use, it is only necessary to change the way of style prompt information to obtain multiple style images that maintain the similarity of image content for the new or specific image content, thereby improving the efficiency of image style conversion to a certain extent.

[0193] The method of the embodiment of the present application is described in detail above. In order to facilitate the above-mentioned scheme of the embodiment of the present application to be better implemented, the device of the embodiment of the present application is provided below accordingly. In the embodiment of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of an overall module or unit that includes the module or unit function.

[0194] Fig.15 A schematic diagram of the structure of an image processing device provided by an exemplary embodiment of the present application is shown; the image processing device can be used to perform Figure 4 or Figure 8Some or all of the steps in the method embodiment shown. Fig.15 , the image processing device comprises the following units:

[0195] The acquisition unit 1501 is used to acquire a first image to be processed and style prompt information; the first image has a first style, and the style indication information is used to indicate that the first image is converted from the first style to a second style;

[0196] The processing unit 1502 is configured to perform image coding processing on the first image to obtain image coding features, where the image coding features are used to characterize content characteristics of image content included in the first image;

[0197] The processing unit 1502 is further configured to use an attention algorithm to perform a linear transformation on the first image and the style prompt information, and embed the image coding feature into the linear transformation to generate a target semantic feature; the target semantic feature has content characteristics of the image content included in the first image, image characteristics of the first image, and style conversion characteristics of the style prompt information;

[0198] The processing unit 1502 is further configured to generate a second image based on the target semantic feature; the second image has a second style, and the image content included in the second image is the same as the image content included in the first image.

[0199] In one implementation, the linear transformation includes a linear projection transformation process; the processing unit 1502 is used to embed the image coding feature into the linear transformation to generate the target semantic feature, specifically for:

[0200] Acquire initial semantic features obtained by performing linear projection transformation processing during the linear transformation process; wherein the linear transformation is used to extract key information from the first image and the style hint information;

[0201] The image encoding features and the initial semantic features are combined to generate the target semantic features.

[0202] In one implementation, the linear transformation includes a first sub-linear transformation, a second sub-linear transformation, and a third sub-linear transformation, and the first sub-linear transformation, the second sub-linear transformation, and the third sub-linear transformation all include linear projection transformation processing; the target semantic feature includes a first sub-semantic feature, a second sub-semantic feature, and a third sub-semantic feature;

[0203] The processing unit 1502 is used to combine the image coding feature and the initial semantic feature to generate the target semantic feature, specifically for:

[0204] The image coding feature and the initial semantic feature obtained by performing the linear projection transformation in the first sub-linear transformation process are combined to generate a first sub-semantic feature;

[0205] The image coding feature and the initial semantic feature obtained by performing the linear projection transformation in the second sub-linear transformation process are combined to generate a second sub-semantic feature;

[0206] The image coding feature and the initial semantic feature obtained by performing the linear projection transformation in the third sub-linear transformation process are combined to generate the third sub-semantic feature;

[0207] A target semantic feature is generated based on the first sub-semantic feature, the second sub-semantic feature, and the third sub-semantic feature.

[0208] In one implementation, the first sub-linear transformation and the second sub-linear transformation are used to perform text semantic extraction on the style hint information, and the first sub-linear transformation and the second sub-linear transformation are performed to obtain the style conversion characteristics of the style hint information; the style conversion characteristics of the style conversion hint information are the initial semantic characteristics obtained by performing the linear projection transformation process in the first sub-linear transformation process and the second sub-linear transformation process; the style conversion characteristics are used to characterize the style parameters required for converting from the first style to the second style;

[0209] The third sub-linear transformation is used to extract image features from the first image, and the image characteristics of the first image are obtained by executing the third sub-linear transformation; the image characteristics of the first image are initial semantic features obtained by executing the linear projection transformation in the process of the third sub-linear transformation; the image characteristics are used to characterize the image data of the first image, and the image data includes at least one of the following: resolution, size data and pixel data.

[0210] In one implementation, the processing unit 1502 is further configured to:

[0211] Performing feature dimensionality reduction processing on the image coding features to obtain a feature matrix corresponding to the image coding features; the dimension of the feature matrix is ​​smaller than the dimension of the image coding features;

[0212] The processing unit 1502 is used to embed the image coding feature into the linear transformation to generate the target semantic feature, specifically for:

[0213] The feature matrix corresponding to the image encoding features is embedded into the linear transformation to generate the target semantic features.

[0214] In one implementation, the feature matrix is ​​formed by sequentially multiplying the first submatrix, the second submatrix and the third submatrix; the row and column information of the image coding feature is m×n, where m and n are positive integers; the processing unit 1502 is used to perform feature dimensionality reduction processing on the image coding feature to obtain the feature matrix corresponding to the image coding feature, specifically for:

[0215] According to the row and column information of the image coding features, the image coding features are matrix decomposed to obtain two orthogonal matrices and a diagonal matrix corresponding to the image coding features; wherein, the row and column information of one of the two orthogonal matrices is m×m, and the row and column information of the other orthogonal matrix is ​​n×n; the row and column information of the diagonal matrix is ​​n×n, and the elements other than the main diagonal in the diagonal matrix are 0, the values ​​of the elements on the main diagonal are used to represent the importance of the content characteristics, and the elements on the main diagonal are arranged from large to small according to the values;

[0216] Selecting r elements from the n elements on the main diagonal of the diagonal matrix in descending order of value to form a second submatrix, and constructing the first submatrix and the third submatrix based on the two orthogonal matrices and the second submatrix;

[0217] The first sub-matrix, the second sub-matrix and the third sub-matrix are multiplied in sequence to obtain a feature matrix corresponding to the image coding feature.

[0218] In one implementation, the image processing method is performed by a pre-trained target image generation model, the target image generation model includes an attention algorithm layer; the pre-training process of the target image generation model includes:

[0219] Obtain an image-text pair from a pre-training data set; the image-text pair includes a sample image and a text description for describing the sample image;

[0220] Matrixing the sample images in the image-text pair using the initial image generation model to obtain predicted image coding features; the matrixing includes image coding, or image coding and feature dimensionality reduction; and,

[0221] The initial image generation model is used to perform semantic extraction processing on the text description in the image-text pair to obtain the predicted semantic features;

[0222] The predicted image encoding features are embedded into the attention algorithm layer, and the predicted image corresponding to the sample image is generated by combining the predicted semantic features;

[0223] Based on the predicted image and the sample image, the initial image generation model is pre-trained to obtain a pre-trained target image generation model.

[0224] In one implementation, the image content included in the first image includes at least one of the following: a face, a person, an animal, and an object; the image content has not been learned in the pre-training stage;

[0225] The method of obtaining style prompt information includes any of the following: a default setting; or, generated based on an input operation for a second style in a style selection interface of an object; wherein the style selection interface includes one or more candidate styles, and the input operation is a selection operation of selecting the second style from one or more candidate styles; or, the style selection interface includes a style input area, and the input operation is an input operation of inputting the second style in the style input area.

[0226] In one implementation, the process of constructing a pre-training data set includes:

[0227] Selecting a first number of sample images from a public data set to form a first data set; and,

[0228] Searching a second number of sample images from the Internet to form a second data set; the sum of the first number and the second number is greater than a quantity threshold;

[0229] Data processing is performed on the sample images in the first data set and the second data set respectively to obtain a pre-training data set for pre-training the initial image generation model.

[0230] In one implementation, the initial image generation model includes a text description module; any sample image in the first data set or the second data set is represented as a target sample image; the processing unit 1502 is used to perform data processing on the sample images in the first data set and the second data set respectively, and when a pre-training data set for pre-training the initial image generation model is obtained, it is specifically used to:

[0231] Performing image quality processing on the target sample image to obtain a processed target sample image; the image quality processing at least includes: square mapping processing, image enhancement processing and size normalization processing; and,

[0232] A text description module is used to perform text generation processing on the target sample image to generate a target text description corresponding to the target sample image;

[0233] The target sample image and the target text description corresponding to the target sample image constitute an image-text pair.

[0234] In one implementation, the initial image generation model includes a first image encoding module, a second image encoding module and a feature fusion module; the feature fusion module includes trainable parameters of the initial image generation model; the matrix processing includes image encoding processing; the processing unit 1502 is used to perform matrix processing on the sample image in the image-text pair using the initial image generation model to obtain the predicted image encoding feature, specifically for:

[0235] Using a first image encoding module to perform image encoding processing on a sample image in an image-text pair to obtain a first predicted sub-image feature; and

[0236] Using a second image encoding module to perform image encoding processing on a sample image in the image-text pair to obtain a second predicted sub-image feature;

[0237] A feature fusion module is used to perform feature fusion on the first predicted sub-image feature and the second predicted sub-image feature to obtain a predicted image coding feature corresponding to the image-text pair.

[0238] According to one embodiment of the present application, Fig.15 The various units in the image processing device shown can be individually or completely combined into one or several other units to form, or one (some) of the units can be further divided into multiple functionally smaller units to form, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions. In practical applications, the function of one unit can also be implemented by multiple units, or the functions of multiple units can be implemented by one unit. In other embodiments of the present application, the image processing device may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of multiple units. According to another embodiment of the present application, it can be executed by running on a general computing device such as a computer that includes processing elements and storage elements such as a central processing unit (CPU), a random access storage medium (RAM), and a read-only storage medium (ROM). Figure 4 and Figure 8 A computer program (including program code) for each step involved in the corresponding method shown in FIG. Fig.15 The image processing apparatus shown in the figure can be used to implement the image processing method of the embodiment of the present application. The computer program can be recorded on a computer-readable recording medium, for example, and loaded into the above-mentioned computing device through the computer-readable recording medium and run therein.

[0239] In an embodiment of the present application, after obtaining the first image to be processed and the style hint information for indicating that the first image is to be converted from the first style to the second style, on the one hand, the first image can be directly subjected to image coding processing to obtain the image coding features of the first image, and the image coding features can be used to characterize the content characteristics of the image content included in the first image. On the other hand, an attention algorithm can be used to perform a linear transformation on the first image and the style hint information. Furthermore, the image coding features of the first image are embedded in the linear transformation, so that the target semantic features can be extracted during the linear transformation process, and the target semantic features not only have the content characteristics of the image content included in the first image, but also include the image characteristics of the first image (such as some basic image data of the image, such as resolution, image size, etc.) and the style conversion characteristics of the style hint information (such as the style parameters of the second style, etc.). In this way, a second image can be generated based on target semantic features that include both image features (such as content features and image features of the image content) and text features (such as style conversion features). The second image has a high image quality. The high image quality is specifically reflected in the high similarity between the image content included in the generated second image and the image content included in the first image, and the second image has the second style indicated by the style prompt information, that is, the style conversion is successfully achieved. In the above-mentioned style conversion process, the image coding features of the first image are directly embedded in the linear transformation process. Compared with the need for dozens or even hundreds of images to train the model to learn image features, it not only improves the concreteness of the generated second image with the second style, but also effectively reduces the cost required to generate the second image.

[0240] Fig.16 FIG. 1 shows a schematic diagram of a computer device provided by an exemplary embodiment of the present application. Fig.16 , the computer device includes a processor 1601, a communication interface 1602 and a computer-readable storage medium 1603. The processor 1601, the communication interface 1602 and the computer-readable storage medium 1603 may be connected via a bus or other means. The communication interface 1602 is used to receive and send data. The computer-readable storage medium 1603 may be stored in the memory of the computer device, the computer-readable storage medium 1603 is used to store computer programs, and the processor 1601 is used to execute the computer programs stored in the computer-readable storage medium 1603. The processor 1601 (or CPU (Central Processing Unit)) is the computing core and control core of the computer device, which is suitable for implementing one or more computer programs, and is specifically suitable for loading and executing one or more computer programs to implement corresponding method flows or corresponding functions.

[0241] The embodiment of the present application also provides a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in the computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides a storage space that stores the processing system of the computer device. In addition, one or more computer programs suitable for being loaded and executed by the processor 1601 are also stored in the storage space. It should be noted that the computer-readable storage medium here can be a high-speed RAM memory, or a non-volatile memory, such as at least one disk storage; optionally, it can also be at least one computer-readable storage medium located away from the aforementioned processor.

[0242] In one embodiment, the computer device may be the terminal or server mentioned in the aforementioned embodiment; one or more computer programs are stored in the computer-readable storage medium; the processor 1601 loads and executes the one or more computer programs stored in the computer-readable storage medium to implement the corresponding steps in the above-mentioned display processing method embodiment; in a specific implementation, the one or more computer programs in the computer-readable storage medium are loaded and executed by the processor 1601 to execute the steps of each embodiment of the present application; wherein, the steps of each embodiment of the present application can refer to the relevant description of the aforementioned embodiments, and will not be repeated here.

[0243] Based on the same inventive concept, the principles and beneficial effects of solving the problems provided by the computer device in the embodiment of the present application are similar to the principles and beneficial effects of solving the problems provided by the image processing method in the method embodiment of the present application. Please refer to the principles and beneficial effects of the implementation of the method. For the sake of concise description, they will not be repeated here.

[0244] An embodiment of the present application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the above-mentioned image processing method is implemented.

[0245] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0246] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes a computer program (one or more). When a computer program is loaded and executed on a computer device, the computer program executes the above-mentioned process or function of the embodiment of the present application. The computer device can be a general-purpose computer, a special-purpose computer, a computer network or other programmable devices. The computer program can be stored in a computer-readable storage medium or transmitted by a computer-readable storage medium. The computer program can be transmitted from a website site, a computer device, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer device, server or data center. The computer-readable storage medium can be any available medium that a computer device can access or a data storage device such as a server or a data center that contains one or more available media integrations. Available media can be magnetic media (e.g., floppy disk, hard disk, tape), optical media (e.g., DVD), or semiconductor media (e.g., solid-state drive (Solid State Disk, SSD)) and the like.

[0247] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image processing method, characterized in that: include: Obtaining a first image to be processed and style prompt information; The first image has a first style, and the style prompt information is used to indicate that the first image is converted from the first style to a second style; Performing image coding processing on the first image to obtain image coding features, where the image coding features are used to characterize content characteristics of image content included in the first image; An attention algorithm is used to perform a linear transformation on the first image and the style hint information, and an initial semantic feature obtained by performing a linear projection transformation process in the linear transformation process is obtained; the linear transformation includes a first sub-linear transformation, a second sub-linear transformation, and a third sub-linear transformation; the initial semantic feature obtained by performing the linear projection transformation process in the first sub-linear transformation process and the second sub-linear transformation process is a style conversion feature of the style hint information; the initial semantic feature obtained by performing the linear projection transformation process in the third sub-linear transformation is an image feature of the first image; Combining the image coding feature with the initial semantic feature obtained by performing the linear projection transformation process in the first sub-linear transformation process to generate a first sub-semantic feature; combining the image coding feature with the initial semantic feature obtained by performing the linear projection transformation process in the second sub-linear transformation process to generate a second sub-semantic feature; combining the image coding feature with the initial semantic feature obtained by performing the linear projection transformation process in the third sub-linear transformation process to generate a third sub-semantic feature; Generate a target semantic feature based on the first sub-semantic feature, the second sub-semantic feature and the third sub-semantic feature; the target semantic feature has content characteristics of the image content included in the first image, image characteristics of the first image and style conversion characteristics of the style hint information; A second image is generated based on the target semantic feature; the second image has the second style, and image content included in the second image is the same as image content included in the first image.

2. The method according to claim 1, characterized in that The first sub-linear transformation and the second sub-linear transformation are used to perform text semantic extraction on the style hint information, and the first sub-linear transformation and the second sub-linear transformation are performed to obtain style conversion characteristics of the style hint information; the style conversion characteristics are used to characterize style parameters required for converting from the first style to the second style; The third sub-linear transformation is used to extract image features from the first image, and the image characteristics of the first image are obtained by executing the third sub-linear transformation; the image characteristics are used to characterize image data of the first image, and the image data includes at least one of the following: resolution, size data and pixel data.

3. The method according to any one of claims 1 or 2, characterized in that After performing image coding processing on the first image to obtain image coding features, the method further includes: Performing feature dimensionality reduction processing on the image coding feature to obtain a feature matrix corresponding to the image coding feature; the dimension of the feature matrix is ​​smaller than the dimension of the image coding feature.

4. The method according to claim 3, characterized in that The feature matrix is ​​formed by multiplying the first submatrix, the second submatrix and the third submatrix in sequence; the row and column information of the image coding feature is m×n, where m and n are positive integers; the feature dimension reduction processing is performed on the image coding feature to obtain the feature matrix corresponding to the image coding feature, including: According to the row and column information of the image coding feature, the image coding feature is matrix decomposed to obtain two orthogonal matrices and a diagonal matrix corresponding to the image coding feature; wherein the row and column information of one of the two orthogonal matrices is m×m, and the row and column information of the other orthogonal matrix is ​​n×n; the row and column information of the diagonal matrix is ​​n×n, and the elements other than the main diagonal in the diagonal matrix are 0, the values ​​of the elements on the main diagonal are used to characterize the importance of the content characteristics, and the elements on the main diagonal are arranged from large to small according to the values; Selecting r elements from the n elements on the main diagonal of the diagonal matrix in descending order to form the second sub-matrix, and constructing the first sub-matrix and the third sub-matrix based on the two orthogonal matrices and the second sub-matrix; The first sub-matrix, the second sub-matrix and the third sub-matrix are multiplied in sequence to obtain a feature matrix corresponding to the image coding feature.

5. The method according to claim 1, characterized in that The method is performed by a pre-trained target image generation model, wherein the target image generation model includes an attention algorithm layer; the pre-training process of the target image generation model includes: Obtaining an image-text pair from a pre-training data set; the image-text pair includes a sample image and a text description for describing the sample image; Matrixing the sample images in the image-text pair using the initial image generation model to obtain predicted image coding features; the matrixing includes the image coding process, or the image coding process and feature dimensionality reduction process; and, Using the initial image generation model to perform semantic extraction processing on the text description in the image-text pair to obtain predicted semantic features; Embedding the predicted image encoding features into the attention algorithm layer, and generating a predicted image corresponding to the sample image in combination with the predicted semantic features; Based on the predicted image and the sample image, the initial image generation model is pre-trained to obtain a pre-trained target image generation model.

6. The method according to claim 5, characterized in that The image content included in the first image includes at least one of the following: a face, a person, an animal, and an object; the image content has not been learned in the pre-training stage; The method of obtaining the style prompt information includes any of the following: default setting; Alternatively, it is generated based on an input operation for a second style in a style selection interface of an object; wherein the style selection interface includes one or more candidate styles, and the input operation is a selection operation of selecting the second style from one or more candidate styles; or, the style selection interface includes a style input area, and the input operation is an input operation of inputting the second style in the style input area.

7. The method according to claim 5, characterized in that The construction process of the pre-training data set includes: Selecting a first number of sample images from a public data set to form a first data set; and, Searching a second number of sample images from the Internet to form a second data set; the sum of the first number and the second number is greater than a quantity threshold; Data processing is performed on the sample images in the first data set and the second data set respectively to obtain a pre-training data set for pre-training the initial image generation model.

8. The method according to claim 7, characterized in that The initial image generation model includes a text description module; any sample image in the first data set or the second data set is represented as a target sample image; and the sample images in the first data set and the second data set are respectively processed to obtain a pre-training data set for pre-training the initial image generation model, including: Performing image quality processing on the target sample image to obtain a processed target sample image; the image quality processing at least includes: square mapping processing, image enhancement processing and size normalization processing; and, Using the text description module to perform text generation processing on the target sample image to generate a target text description corresponding to the target sample image; The target sample image and the target text description corresponding to the target sample image form an image-text pair.

9. The method according to claim 5, characterized in that The initial image generation model includes a first image encoding module, a second image encoding module and a feature fusion module; the feature fusion module includes trainable parameters of the initial image generation model; the matrix processing includes the image encoding processing; the matrix processing of the sample image in the image-text pair using the initial image generation model to obtain the predicted image encoding features includes: Using the first image encoding module to perform image encoding processing on the sample image in the image-text pair to obtain a first predicted sub-image feature; and Using the second image encoding module to perform image encoding processing on the sample image in the image-text pair to obtain a second predicted sub-image feature; The feature fusion module is used to perform feature fusion on the first predicted sub-image feature and the second predicted sub-image feature to obtain the predicted image coding feature corresponding to the image-text pair.

10. An image processing device, characterized in that: include: An acquisition unit, used for acquiring a first image to be processed and style prompt information; The first image has a first style, and the style prompt information is used to indicate that the first image is converted from the first style to a second style; a processing unit, configured to perform image coding processing on the first image to obtain image coding features, wherein the image coding features are used to characterize content characteristics of image content included in the first image; The processing unit is further used to perform linear transformation on the first image and the style hint information by using an attention algorithm, and obtain initial semantic features obtained by performing linear projection transformation processing in the linear transformation process; the linear transformation includes a first sub-linear transformation, a second sub-linear transformation, and a third sub-linear transformation, the initial semantic features obtained by performing the linear projection transformation processing in the first sub-linear transformation process and the second sub-linear transformation process are style conversion characteristics of the style hint information, and the initial semantic features obtained by performing the linear projection transformation processing in the third sub-linear transformation are image characteristics of the first image; The processing unit is further used to combine the image coding feature with the initial semantic feature obtained by performing the linear projection transformation process in the first sub-linear transformation process to generate a first sub-semantic feature; combine the image coding feature with the initial semantic feature obtained by performing the linear projection transformation process in the second sub-linear transformation process to generate a second sub-semantic feature; and combine the image coding feature with the initial semantic feature obtained by performing the linear projection transformation process in the third sub-linear transformation process to generate a third sub-semantic feature; The processing unit is further configured to generate a target semantic feature based on the first sub-semantic feature, the second sub-semantic feature, and the third sub-semantic feature; the target semantic feature has content characteristics of the image content included in the first image, image characteristics of the first image, and style conversion characteristics of the style prompt information; The processing unit is further used to generate a second image based on the target semantic feature; the second image has the second style, and the image content included in the second image is the same as the image content included in the first image.

11. A computer device, characterized in that: a processor adapted to execute a computer program; A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by the processor, the image processing method according to any one of claims 1 to 9 is implemented.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor and executing the image processing method according to any one of claims 1 to 9.

13. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the image processing method according to any one of claims 1 to 9 is implemented.

Images