Image style migration method and device, electronic equipment and storage medium

Abstract

The invention provides an image style migration method and device, electronic equipment and a storage medium, and relates to the field of artificial intelligence, in particular to the technical fields of large models, deep learning, computer vision and the like. According to the specific implementation scheme, the method comprises the steps of obtaining a first prompt word of a to-be-processed image displaying a plurality of objects; wherein the first prompt word is used for indicating the overall semantics of the to-be-processed image; performing semantic segmentation on the to-be-processed image to obtain segmented images of the plurality of objects; based on the segmented images of the multiple objects, obtaining second cue words of the multiple objects; wherein the second prompt word is used for indicating local semantics of the object in the to-be-processed image; and performing style migration on the to-be-processed image based on the first cue word and the second cue words of the plurality of objects to obtain a first target image under the target style.

Description

Technical Field

[0001] The present disclosure relates to the field of AI (Artificial Intelligence), specifically to technical fields such as large models, deep learning, and computer vision, and especially to image style transfer methods, devices, electronic devices, and storage media. Background Art

[0002] Image style transfer (or image style conversion) technology is a branch of image processing technology, which aims to convert ordinary photos or other types of original images into images with a target style. Taking the target style as a comic style as an example, the original image can be converted into a comic style artwork. Among them, image style transfer technology can be applied to multiple fields such as digital entertainment, art creation, advertising design, virtual social networking, etc., to enhance the fun, attractiveness and interactive experience of content creation. Summary of the invention

[0003] The present disclosure provides a method, device, electronic device and storage medium for image style transfer.

[0004] According to one aspect of the present disclosure, a method for image style transfer is provided, comprising:

[0005] Acquire a first prompt word of an image to be processed showing multiple objects; wherein the first prompt word is used to indicate the overall semantics of the image to be processed;

[0006] Performing semantic segmentation on the image to be processed to obtain segmented images of the multiple objects;

[0007] Based on the segmented images of the multiple objects, obtaining second prompt words for the multiple objects; wherein the second prompt words are used to indicate the local semantics of the objects in the image to be processed;

[0008] Based on the first prompt word and the second prompt words of the multiple objects, style migration is performed on the image to be processed to obtain a first target image under a target style.

[0009] According to another aspect of the present disclosure, there is provided an image style transfer apparatus, comprising:

[0010] A first acquisition module is used to acquire a first prompt word of an image to be processed showing multiple objects; wherein the first prompt word is used to indicate the overall semantics of the image to be processed;

[0011] A segmentation module, used for performing semantic segmentation on the image to be processed to obtain segmented images of the multiple objects;

[0012] A second acquisition module, used for acquiring second prompt words for the multiple objects based on the segmented images of the multiple objects; wherein the second prompt words are used to indicate the local semantics of the objects in the image to be processed;

[0013] The style transfer module is used to perform style transfer on the image to be processed based on the first prompt word and the second prompt words of the multiple objects to obtain a first target image under a target style.

[0014] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0015] at least one processor; and

[0016] a memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the image style transfer method proposed in the above aspect of the present disclosure.

[0018] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium of computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the image style transfer method proposed in the above aspect of the present disclosure.

[0019] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which, when executed by a processor, implements the image style transfer method proposed in the above aspect of the present disclosure.

[0020] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0022] Figure 1 This is a workflow diagram of the Stable Diffusion model;

[0023] Figure 2 A schematic diagram of the process of the image style transfer method provided in the first embodiment of the present disclosure;

[0024] Figure 3 A schematic diagram of the process of image style transfer method provided in the second embodiment of the present disclosure;

[0025] Figure 4A schematic diagram of the process of image style transfer method provided in the third embodiment of the present disclosure;

[0026] Figure 5 This is the structural diagram of IPAdapter;

[0027] Figure 6 A schematic diagram of the process of image style transfer method provided in the fourth embodiment of the present disclosure;

[0028] Figure 7 A schematic diagram of the process of image style transfer method provided in Embodiment 5 of the present disclosure;

[0029] Figure 8 A schematic diagram of the process of an image style transfer method provided in Embodiment 6 of the present disclosure;

[0030] Fig. 9 It is a schematic diagram of the implementation principle of each embodiment of the present disclosure;

[0031] Fig.10 A schematic diagram of the structure of an image style transfer device provided in Embodiment 7 of the present disclosure;

[0032] Fig.11 A schematic block diagram of an example electronic device that can be used to implement embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0033] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0034] In recent years, with the continuous development of deep learning technology, many image generation models have been widely used for style transfer, for example, converting real images into anime style or comic style. However, when processing multi-person images, existing image generation models often have problems such as inconsistent stylization, distorted details, and confusion of character relationships. Especially in complex scenes, images after style transfer often fail to achieve the expected effect.

[0035] As an example, the image style to be migrated is an anime style or a comic style. Currently, the following solutions are mainly used to convert real images into anime style or comic style:

[0036] The first one is to directly use the pre-trained Stable Diffusion model for image style conversion: this method uses the trained Stable Diffusion model to directly convert the input multi-person image into a comic style.

[0037] Among them, since the pre-trained Stable Diffusion model has learned a variety of image styles and features, it can quickly generate comic-style images (hereinafter referred to as comic images) without further fine-tuning or training the model.

[0038] The second type is image generation based on conditional diffusion: by introducing style labels or other conditions in the image generation process, the generation process of the Stable Diffusion model is controlled.

[0039] This approach allows the Stable Diffusion model to be guided to generate comic images in a specific style by inputting style-related conditions, and provides more customization options for complex generation processes.

[0040] As an example, the workflow of the Stable Diffusion model can be as follows Figure 1 As shown, it is mainly used to convert the original image into a new image with different styles. The following is Figure 1 A detailed explanation of the main components and working principles:

[0041] 1. Pixel Space: The input image x is converted from pixel space to latent space through the encoder E. The output image (x~) is obtained by restoring the features from latent space to pixel space.

[0042] 2. Latent Space: The encoded image representation z undergoes a diffusion process in the latent space, gradually adding noise and eventually becomes z T Among them, the diffusion process is a reverse process, from z T Start to gradually remove the noise and restore the representation z0 of the original image.

[0043] 3. Denoising U-Net (U-Net): Denoising U-Net is a deep neural network responsible for removing noise layer by layer. It consists of multiple modules ( Figure 1Only two modules are used as an example), and each module contains a Query-Key-Value (QKV) attention mechanism. Information is transmitted between modules through skip connections to enhance feature learning capabilities.

[0044] 4. Conditioning: The conditioning part receives the semantic map and text description as input. These conditional information are used to guide the denoising process so that the generated image conforms to the specified image style.

[0045] in, Figure 1 Semantic Map in : It is a map that represents the distribution of object categories in an image. The semantic map divides the image into different regions and assigns a label to each region, indicating which object category the region belongs to (such as sky, tree, road, etc.). This information is very useful for guiding image generation because it tells the model what types of objects should be placed where.

[0046] in, Figure 1 Text description in : refers to the text instructions that describe the image content. It can be a description of the overall style of the image (such as cartoon style, oil painting style), a description of certain details (such as "a cat sitting on a table"), or any other requirements that can be expressed in text form. The text condition allows the model to generate images of corresponding style or content based on the given text description.

[0047] in, Figure 1 Representations in the network refer to high-level abstract features extracted from input data (such as images or text). In the field of machine learning, representation learning is a key technology that aims to automatically discover meaningful patterns and structures in data. For the Stable Diffusion model, representations may include low-level features of the image (such as edges, textures), mid-level features (such as shapes, color combinations), and high-level concepts (such as object categories, scene layouts). These representations will be used to guide the generation process to ensure that the generated images are both qualified and realistic.

[0048] in, Figure 1Reference images in (Images): The reference image itself is also one of the important conditions. In some cases, the user may want the newly generated image to retain some features of the original image, but change other aspects (such as style, perspective). Therefore, in addition to text and other forms of information, directly providing reference images can make it easier for the model to understand what the expected effect is. For example, in the image style transfer task, an original image and a reference image of the target style can be given, and the model's task is to create a new image that has both the original image content and the target style.

[0049] 5. Crossattention: The cross-attention mechanism allows the denoising U-Net to utilize conditional information (such as semantic maps and text) to improve the denoising effect. Among them, the cross-attention selectively focuses on the relevant feature areas by comparing the features and conditional information in the latent space.

[0050] 6. Switch: The switch mechanism determines when and how to use conditional information. This may be based on some rules or dynamically adjusted.

[0051] 7. Concat: Concatenate the conditional information with the features in the latent space to form a richer feature representation, further improving the performance of the model.

[0052] 8. In the Stable Diffusion model, z T-1 The "denoising step" between t and z represents the denoising step. Here T is the time step of the diffusion process, T-1 represents the current time step, and z can refer to an intermediate state or the final state. Specifically, the "denoising step" refers to the process of moving from the noisy state z to the final state z. T-1 Remove some of the noise to make it close to the clean state z. This process is usually done through a trained denoising model (such as the U-Net architecture), which tries to predict and remove the noise components, so that the image gradually becomes clear.

[0053] In summary, the Stable Diffusion model first encodes the input image into the latent space, gradually adding noise in the process. Then, the noise is removed layer by layer through the denoising U-Net, while using conditional information (such as semantic maps and text) to guide the generation process, and finally the style-transferred image is obtained. The whole process combines the powerful expression ability of the diffusion model and the flexibility of conditional generation to achieve high-quality image style transfer.

[0054] The third method is to use text prompts to guide generation: by inputting text prompts to describe the image content, the Stable Diffusion model can perform style transfer on the input image based on the text prompts.

[0055] This method allows users to specify details, style, or poses in an image through a short text description and generate new images that match the description.

[0056] Although the Stable Diffusion model and its variants have made progress in image generation and style transfer, there are still significant shortcomings when dealing with style transfer of multi-person images, especially in preserving the details and similarities of the original images, including the accurate restoration of important visual elements such as faces and clothes.

[0057] That is, the above-mentioned image style transfer scheme (or image style conversion scheme) based on Stable Diffusion, such as directly using pre-trained Stable Diffusion models, conditional diffusion generation, partition processing, etc., although it performs well in single-person image conversion, it is difficult to ensure the restoration of details between characters, style uniformity, and the fusion of characters and background when facing multi-person scenes. The above methods usually result in poor similarity between the style-transferred image and the original image, especially in terms of face, clothing, posture, and interaction details. The style-transferred image lacks realism and cannot accurately reproduce the characteristics of the original image.

[0058] In summary, the image style transfer solutions in related technologies have at least the following problems:

[0059] (1) Loss of details and lack of similarity: Existing pre-trained models often ignore subtle differences between individuals when processing multi-person images. Especially in details such as faces and clothing, the similarity between the style-transferred image and the original image is poor, the facial features of the characters are easily blurred or deformed, and the details of the clothing cannot be accurately presented. This stylization process usually focuses more on the application of the overall style rather than retaining individual features, resulting in the loss of similarity between the style-transferred image and the original image, making it difficult to faithfully present the details.

[0060] (2) Style inconsistency: When processing multi-person images, the above scheme cannot effectively adjust the style of each person individually, resulting in inconsistent styles of different people in the style-transferred image. Some people may be over-stylized, while the styles of other people appear rough and lack a unified image style. In addition, the style fusion of the background and the characters is often not handled naturally enough, resulting in the visual fragmentation of the style-transferred image and the lack of coordination between the characters and the background.

[0061] (3) Distortion of multi-person interactions and actions: For complex multi-person interaction scenes, the existing StableDiffusion model has difficulty preserving the natural interactions and actions between characters. After multi-person images are converted to the target style, the characters’ postures, actions, and interactions often become unnatural, out of proportion, or appear stiff. This causes the style-transferred images to appear stiff when expressing multi-person interactions, and fail to maintain the dynamics and authenticity of the original images.

[0062] (4) Lack of preservation of personalized features: Existing style transfer technologies have difficulty preserving the personalized features of each person in multi-person images, such as facial features, hairstyles, clothing details, etc. Since these technologies usually apply a unified style, the features between characters are easily confused and the uniqueness of individuals cannot be highlighted. As a result, the style-transferred image is difficult to maintain the true appearance and clothing of each person in the original image, causing the style-transferred image to lose its original personality and recognition.

[0063] (5) The balance between efficiency and quality: When processing high-resolution complex multi-person images, the above schemes often face a trade-off between computational efficiency and generated quality. In order to maintain details, the processing speed is slow, while increasing the processing speed may lead to a decrease in the quality of the generated image, with details becoming blurred and lacking depth.

[0064] Therefore, in response to at least one of the above-mentioned problems, the present disclosure proposes an image style transfer method, device, electronic device and storage medium.

[0065] The following describes the image style transfer method, device, electronic device and storage medium of the embodiments of the present disclosure with reference to the accompanying drawings. Before describing the embodiments of the present disclosure in detail, for ease of understanding, common technical terms are first introduced:

[0066] A large model refers to a machine learning model with large-scale parameters and complex computational structures. It is usually built from deep neural networks and has billions or even hundreds of billions of parameters. The purpose of designing a large model is to improve the model's expressiveness and predictive performance, and to be able to handle more complex tasks and data. Large models are widely used in various fields, including natural language processing, computer vision, speech recognition, and recommendation systems.

[0067] Figure 2 A flowchart of the image style transfer method provided in the first embodiment of the present disclosure.

[0068] The embodiment of the present disclosure takes the image style transfer method configured in an image style transfer device as an example. The image style transfer device can be applied to any electronic device so that the electronic device can perform an image processing function.

[0069] Among them, the electronic device can be any device with computing capabilities, such as a personal computer, a mobile terminal, a server, etc. The mobile terminal can be, for example, a mobile phone, a tablet computer, a personal digital assistant, a wearable device, and other hardware devices with various operating systems, touch screens and / or display screens.

[0070] like Figure 2 As shown, the image style transfer method may include the following steps S201 to S204:

[0071] Step S201, obtaining a first prompt word of an image to be processed showing multiple objects; wherein the first prompt word is used to indicate the overall semantics of the image to be processed.

[0072] The objects include but are not limited to: people, pets, animals, etc.

[0073] There is no restriction on the method of acquiring the image to be processed.

[0074] As an example, the image to be processed may be uploaded by a user, that is, in the present disclosure, the uploaded image to be processed may be obtained in response to an upload operation triggered by the user.

[0075] As another example, the image to be processed may be obtained by photographing by a user, that is, in the present disclosure, the photographed image to be processed may be obtained in response to a photographing operation triggered by the user.

[0076] As another example, the image to be processed may be obtained by a user search, that is, in the present disclosure, in response to a search operation triggered by the user, candidate images matching the search keywords input by the user may be obtained, and the user may manually select the image to be processed from the candidate images.

[0077] Among them, the input method of the search keyword may include but is not limited to touch input (such as sliding, clicking, etc.), keyboard input, voice input, etc.

[0078] As another example, the image to be processed may be manually synthesized by a user, that is, in the present disclosure, the image to be processed manually synthesized by the user may be obtained in response to an image synthesis operation triggered by the user.

[0079] As another example, the image to be processed may also be shared by other users, that is, in the present disclosure, the image to be processed sent by other users when a sharing operation is triggered may be received.

[0080] It should be noted that the above-mentioned method of acquiring the image to be processed is only an exemplary description, and the present disclosure is not limited thereto. In actual application, other methods may also be used to acquire the image to be processed.

[0081] In summary, different interactive methods can be used to obtain the image to be processed, which can improve the flexibility and applicability of the method.

[0082] In the embodiment of the present disclosure, after the image to be processed is acquired, a first prompt word (or called an overall prompt word) of the image to be processed may be acquired, wherein the first prompt word is used to indicate the overall semantics of the image to be processed.

[0083] Exemplarily, deep learning technology can be used to perform semantic understanding on the image to be processed to obtain the first prompt word.

[0084] Step S202: semantically segment the image to be processed to obtain segmented images of multiple objects.

[0085] In the embodiments of the present disclosure, semantic segmentation may be performed on the image to be processed based on a semantic segmentation model to obtain segmented images of multiple objects, that is, each segmented image shows only one object.

[0086] Exemplarily, a SegmentAnything (any object segmentation) model may be used to segment objects in an image to be processed to obtain segmented images of multiple objects.

[0087] Step S203, based on the segmented images of the multiple objects, obtaining second prompt words for the multiple objects; wherein the second prompt words are used to indicate the local semantics of the objects in the image to be processed.

[0088] As an example, deep learning technology can be used to perform semantic understanding on the segmented image of each object to obtain a second prompt word (or local prompt word) for each object, where the second prompt word is used to indicate the local semantics of the object in the image to be processed.

[0089] Step S204: performing style migration on the image to be processed based on the first prompt word and the second prompt words of the multiple objects to obtain a first target image in a target style.

[0090] Among them, the target style can be an image style specified by the user, or the target style can also be an image style automatically recommended to the user based on the user's preference, or the target style can also be a default image style, or the target style can also be a popular image style, etc. The embodiments of the present disclosure do not limit this.

[0091] Among them, the target style includes but is not limited to the following image styles: illustration style (such as watercolor, sketch, oil painting, etc.), comic style (such as romantic style, science fiction style, historical style, fantasy style, horror style, healing style, etc.), artistic style (such as impressionism, abstract expressionism, post-impressionism, cubism, etc.), etc.

[0092] As an example, the first prompt word and the second prompt words of multiple objects can be input into the Stable Diffusion model as conditioning information, so that the Stable Diffusion model can perform image style migration or image style conversion on the processed image based on the above conditioning information to obtain a migrated image under the target style (referred to as the first target image in this disclosure).

[0093] As an example, taking the target style as a comic style, the application occasions and application fields of the image processing solution provided by the present disclosure include at least the following:

[0094] (1) Comic creation and adaptation: used to quickly convert multi-person images shot by real people into comic styles, suitable for comic creation, adaptation and production of related works;

[0095] (2) Film and television and animation production: In film and television or animation production, it is used to transform real scenes or multi-person images of actors into comic styles to create special effects or specific styles of works;

[0096] (3) Social media and content creation: Users can quickly transform group photos and other multi-person images into comic styles, making content creation more interesting and attractive;

[0097] (4) Game character and scene design: used to generate comic-style multiplayer character images and scenes, especially suitable for two-dimensional style game design;

[0098] (5) Virtual social and interactive platform: In virtual social or interactive applications, it helps users transform selfies or group photos into comic-style artworks to enhance the interactive experience.

[0099] The image style transfer method of the disclosed embodiment performs style transfer or style conversion on the original image based on a first prompt word for indicating the overall semantics of the original image to be processed (hereinafter referred to as the original image) and a second prompt word for indicating the local semantics of each object (such as a person) in the original image. This can ensure that the first target image after style transfer is highly similar to the original image in overall and detail, so that the generated first target image is more natural and realistic in detail processing, retains the proportions, details and interactions between different objects, and improves the image quality after style transfer in multi-object scenes.

[0100] It should be noted that in the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information are all carried out with the user's consent, and are in compliance with relevant laws and regulations and do not violate public order and good morals.

[0101] In order to clearly illustrate how, in any embodiment of the present disclosure, style transfer is performed on a processed image based on a first prompt word and second prompt words of multiple objects to obtain a first target image under a target style, the present disclosure also proposes an image style transfer method.

[0102] Figure 3 A flowchart of the image style transfer method provided in the second embodiment of the present disclosure.

[0103] like Figure 3 As shown, the image style transfer method may include the following steps S301 to S305:

[0104] Step S301, obtaining a first prompt word of an image to be processed showing multiple objects; wherein the first prompt word is used to indicate the overall semantics of the image to be processed.

[0105] Step S302: semantically segment the image to be processed to obtain segmented images of multiple objects.

[0106] Step S303: acquiring second prompt words for the multiple objects based on the segmented images of the multiple objects; wherein the second prompt words are used to indicate the local semantics of the objects in the image to be processed.

[0107] For explanations of steps S301 to S303, reference may be made to the relevant descriptions in any embodiment of the present disclosure, and will not be repeated here.

[0108] Step S304: generating masks of the multiple objects according to the second prompt words of the multiple objects; wherein the mask is used to indicate the effective range of the second prompt word in the image generation process.

[0109] As an example, a conditioning set mask technique can be used to generate masks of multiple objects according to the second prompt words of multiple objects, wherein the second prompt word of each object only acts within the range specified by the mask of the object to enhance the control of the image.

[0110] Step S305 , performing style migration on the image to be processed according to the second prompt words and masks corresponding to the multiple objects and according to the first prompt word, so as to obtain a first target image under the target style.

[0111] As an example, the first prompt word, the second prompt words corresponding to multiple objects, and the mask can be input into the Stable Diffusion model as conditioning information, so that the Stable Diffusion model can perform image style migration or image style conversion on the processed image based on the above conditioning information to obtain a first target image under the target style.

[0112] As another example, a first prompt word, a second prompt word corresponding to multiple objects, and a mask can be input into a Stable Diffusion model as conditioning information, so that the Stable Diffusion model performs image style transfer or image style transfer on the processed image based on the above conditional information to obtain an initial migration image under the target style, and optimizes or adjusts each facial area in the initial migration image to obtain a first target image.

[0113] The image style transfer method of the disclosed embodiment introduces a mask so that the second prompt word of each object is fixed to be applied only within the scope of action indicated by the mask of the object, thereby isolating the second prompt words of multiple objects, avoiding the situation where image features of multiple objects are intertwined and fused with each other, and further improving the generation quality of the image after style transfer.

[0114] In order to clearly explain how the above embodiment performs style transfer on the image to be processed according to the second prompt words and masks corresponding to multiple objects and according to the first prompt word to obtain a first target image under the target style, the present disclosure also proposes an image style transfer method.

[0115] Figure 4 A flowchart of the image style transfer method provided in Embodiment 3 of the present disclosure.

[0116] like Figure 4 As shown, the image style transfer method may include the following steps S401 to S407:

[0117] Step S401, obtaining a first prompt word of an image to be processed showing multiple objects; wherein the first prompt word is used to indicate the overall semantics of the image to be processed.

[0118] Step S402: semantically segment the image to be processed to obtain segmented images of multiple objects.

[0119] Step S403, based on the segmented images of the multiple objects, obtaining second prompt words for the multiple objects; wherein the second prompt words are used to indicate the local semantics of the objects in the image to be processed.

[0120] Step S404: generating masks of the multiple objects according to the second prompt words of the multiple objects; wherein the mask is used to indicate the effective range of the second prompt word in the image generation process.

[0121] The explanation of steps S401 to S404 can be found in the relevant description in any embodiment of the present disclosure, and will not be repeated here.

[0122] Step S405 , performing style migration on the image to be processed according to the second prompt words and masks corresponding to the multiple objects and according to the first prompt words, to obtain an initial migrated image under the target style.

[0123] As an example, the first prompt word, the second prompt words corresponding to multiple objects, and the mask can be input into the Stable Diffusion model as conditioning information, so that the Stable Diffusion model can perform image style migration or image style conversion on the processed image based on the above conditioning information to obtain the initial migration image under the target style.

[0124] Step S406: extracting facial features of the multiple objects from the segmented images of the multiple objects.

[0125] In the embodiment of the present disclosure, facial features of each object may be extracted from the segmented image of the object based on a facial feature extraction algorithm.

[0126] Exemplarily, taking a person as an example, facial features (ie, facial features) of each object can be extracted from the segmented image of the object based on a facial feature extraction algorithm.

[0127] In any embodiment of the present disclosure, facial features of multiple objects can be extracted by following steps A to C:

[0128] Step A: Using a facial segmentation algorithm, perform facial segmentation on the segmented images of multiple objects to obtain facial images of multiple objects.

[0129] Exemplarily, taking a person as an example, based on a face segmentation algorithm, face segmentation can be performed on segmented images of multiple objects to obtain facial images (or face images) of the multiple objects.

[0130] Step B: Using a coding algorithm, encode the facial images of multiple objects to obtain image features of the multiple objects.

[0131] Step C: Perform linear and normalization processing on the image features of multiple objects respectively to obtain facial features of multiple objects.

[0132] As a possible implementation method, based on IPAdapter-FaceID (network adapter-face ID (Identifier)) technology, face segmentation can be performed on the segmented images of multiple objects to obtain facial images of multiple objects, and the facial images of multiple objects are encoded, linearized and normalized to obtain facial features of multiple objects.

[0133] For example, the structure of IPAdapter can be as follows Figure 5 As shown in the figure, it is mainly used to perform: 1. Image encoding: the input image is passed through the image encoder to obtain image features; 2. Text encoding: the input text description is passed through the text encoder to obtain text features; 3. Feature fusion: through linear transformation and normalization, the image features and text features are fused together; 4. Separate cross attention: using the separate cross attention mechanism, the image features and text features are influenced by each other to achieve cross-modal feature fusion; 5. Denoising generation: the fused feature representation and the noise image x at the current moment are combined t The denoised image x is obtained by denoising U-Net t-1 .

[0134] Among them, the image encoder, text encoder and denoising U-Net are frozen modules. Frozen modules refer to modules whose weights and biases remain unchanged during the training process. The parameters of these modules are fixed and will not be updated through the back-propagation algorithm.

[0135] Among them, the linear layer and the normalization layer are trainable modules. A trainable module refers to a module whose weights and biases are updated according to the gradient of the loss function during the training process.

[0136] In the present disclosure, a facial image can be input into Figure 5 The image encoder, linear layer (Linear) and normalization layer (Layer Normalization, referred to as LN) are connected in series to obtain facial features.

[0137] In this way, the hierarchy and stability of facial feature extraction can be improved, the interpretability of facial features can be improved, and the robustness and generalization ability of the model can be improved.

[0138] Step S407: Based on the facial features of the multiple objects, the initial migration image is subjected to facial adjustment to obtain a first target image.

[0139] In the disclosed embodiment, facial adjustments or optimizations may be performed on the initial migration image based on the facial features of multiple objects to obtain a first target image in a target style.

[0140] In any embodiment of the present disclosure, the first target image can be adjusted by following steps E to F:

[0141] Step E: performing facial redrawing based on facial features of the multiple objects to obtain facial redrawn images of the multiple objects in a target style.

[0142] As an example, the FaceDetailer (facial detail enhancement) technology may be used to perform partial face redrawing based on the facial features of each object, thereby obtaining a face redrawing image of each object in a target style.

[0143] Step F: Based on the image fusion technology and the facial IDs of the multiple objects, the facial redrawn images of the multiple objects and the initial migration images are fused to obtain a first target image in the target style.

[0144] In summary, based on the local redrawing technology, local redrawing is performed according to the facial features of the segmented facial image to obtain a facial redrawn image, and then the facial redrawn image and the initial migration image are fused. This can achieve the effect of repairing the facial area in the image after style transfer, so that the facial features in the repaired image are highly similar to the facial features in the original image to be processed, thereby improving the realism of the image after style transfer.

[0145] The image style transfer method of the disclosed embodiment performs facial adjustments on the initial transfer image based on the facial features of multiple objects, so that the facial features in the final adjusted first target image are highly similar to the facial features in the original image to be processed, thereby retaining the facial details of multiple objects and improving the realism of the image after style transfer.

[0146] In order to clearly explain how to obtain a first prompt word for an image to be processed showing multiple objects in any embodiment of the present disclosure, the present disclosure also proposes an image style transfer method.

[0147] Figure 6 A flowchart of the image style transfer method provided in the fourth embodiment of the present disclosure.

[0148] like Figure 6 As shown, the image style transfer method may include the following steps S601 to S606:

[0149] Step S601 : extracting overall semantic labels associated with multiple objects from an image to be processed showing multiple objects.

[0150] In any embodiment of the present disclosure, the overall semantic label can be used to represent at least one of the following: basic features of multiple objects; action postures of multiple objects; environmental information of multiple objects, etc.

[0151] Among them, basic features include but are not limited to: object identity, such as male, female, child, adult, etc.; object type, such as role, occupation or specific group, etc.; appearance features, such as hairstyle, hair color, facial features (such as eye color, eyebrow shape, etc.), skin color, body shape, clothing style, etc.; object expression; object emotion; object background and environment, etc.

[0152] Among them, action postures include but are not limited to: standing, walking, talking, etc.

[0153] The environmental information includes but is not limited to: indoor, outdoor, urban, rural, weather conditions, etc.

[0154] As an example, a WD14 tagger (WD14 tag generation model, used to automatically generate tags for images) may be used to extract overall semantic tags associated with multiple objects from the image to be processed.

[0155] Step S602: extracting the overall context description associated with the multiple objects from the image to be processed.

[0156] In any one of the embodiments of the present disclosure, the overall context information can be used to characterize at least one of the following: appearance description of multiple objects; emotional state of multiple objects; psychological state of multiple objects; object relationship between multiple objects; interaction mode between multiple objects; scene atmosphere in the image to be processed; emotional color in the image to be processed.

[0157] Among them, appearance description includes but is not limited to: description of clothing, hairstyle, facial features, etc.

[0158] Among them, emotional states include but are not limited to: joy, sadness, anger, etc.

[0159] Among them, psychological state includes but is not limited to: thinking, recalling, tension, etc.

[0160] Among them, object relationships include but are not limited to: intimate relationships, hostile relationships, etc.

[0161] Among them, interactive methods include but are not limited to: physical contact, eye contact, dialogue, etc.

[0162] Among them, emotional colors include but are not limited to: warmth, tension, joy, etc.

[0163] As an example, MoonDream (an open source visual language model) can be used to extract the overall context description associated with multiple objects from the image to be processed.

[0164] Step S603: Generate a first prompt word for the image to be processed according to the overall semantic label and the overall situation description.

[0165] In the disclosed embodiment, the overall semantic label and the overall situation description may be integrated to obtain a first prompt word of the image to be processed, wherein the first prompt word is used to indicate the overall semantics of the image to be processed.

[0166] Step S604: semantically segment the image to be processed to obtain segmented images of multiple objects.

[0167] Step S605 , obtaining second prompt words for the multiple objects based on the segmented images of the multiple objects; wherein the second prompt words are used to indicate the local semantics of the objects in the image to be processed.

[0168] Step S606: Based on the first prompt word and the second prompt words of the multiple objects, style migration is performed on the image to be processed to obtain a first target image in a target style.

[0169] For explanations of steps S604 to S606, reference may be made to the relevant descriptions in any embodiment of the present disclosure, and will not be repeated here.

[0170] In the image style transfer method of the disclosed embodiment, the overall semantic labels associated with multiple objects and the overall context descriptions associated with multiple objects can complement each other, and together provide a comprehensive and in-depth overall semantic description for the image to be processed showing multiple objects, thereby improving the accuracy and richness of the first prompt word generation, and further improving the image quality after style transfer.

[0171] Among them, the overall situation description can provide a comprehensive and in-depth perspective to help understand the objects in the image to be processed and their environment. It not only includes a description of the object's appearance, but also reveals the object's emotional state, psychological state, and their interaction with the environment, etc., thereby constructing a richer and more three-dimensional image story and improving the image quality and authenticity after style transfer.

[0172] Among them, the overall semantic label can provide detailed and vivid descriptive information of the object, which can preserve the proportions, details and interactions between different objects, and ensure the consistency of the image style of the first target image finally generated.

[0173] In order to clearly explain how to obtain second prompt words for multiple objects based on segmented images of multiple objects in any embodiment of the present disclosure, the present disclosure also proposes an image style transfer method.

[0174] Figure 7 A flowchart of the image style transfer method provided in Embodiment 5 of the present disclosure.

[0175] like Figure 7 As shown, the image style transfer method may include the following steps S701 to S706:

[0176] Step S701, obtaining a first prompt word of an image to be processed showing multiple objects; wherein the first prompt word is used to indicate the overall semantics of the image to be processed.

[0177] Step S702: semantically segment the image to be processed to obtain segmented images of multiple objects.

[0178] For explanations of steps S701 to S702, reference may be made to the relevant descriptions in any embodiment of the present disclosure and will not be repeated here.

[0179] Step S703: extracting a local semantic label associated with any object from the segmented image of any object.

[0180] In any embodiment of the present disclosure, local semantic labels can be used to characterize basic features and action postures of any object.

[0181] Among them, basic features include but are not limited to: object identity, such as male, female, child, adult, etc.; object type, such as role, occupation or specific group, etc.; appearance features, such as hairstyle, hair color, facial features (such as eye color, eyebrow shape, etc.), skin color, body shape, clothing style, etc.; object expression; object emotion; object background and environment, etc.

[0182] Among them, action postures include but are not limited to: standing, walking, talking, etc.

[0183] As an example, a WD14 tagger (WD14 tag generation model, used to automatically generate tags for images) can be used to extract local semantic labels from each segmented image.

[0184] Step S704: extracting a local context description associated with any object from the segmented image of any object.

[0185] In any one of the embodiments of the present disclosure, the local context description associated with each object can be used to characterize at least one of the following: the appearance features of the object; the emotional state of the object; the psychological state of the object; the emotional elements in the object's movements and postures; and the interaction information between the object and the background environment.

[0186] Among them, appearance features include but are not limited to: clothing, hairstyle, facial features, etc.

[0187] Among them, emotional states include but are not limited to: joy, sadness, anger, etc.

[0188] Among them, psychological state includes but is not limited to: thinking, recalling, tension, etc.

[0189] Among them, emotional elements include but are not limited to: the subject's confidence when walking, happiness when smiling, etc.

[0190] The interactive information includes, but is not limited to, the position of the character in the environment, the interaction with surrounding objects, etc.

[0191] As an example, MoonDream (an open source visual language model) can be used to extract local context descriptions from segmented images.

[0192] Step S705: Generate a second prompt word for any object according to the local semantic label and the local context description.

[0193] In the disclosed embodiment, the local semantic label and the local context description of the same object may be integrated to obtain a second prompt word of the same object, wherein the second prompt word is used to indicate the local semantics of the same object in the image to be processed.

[0194] Step S706: Based on the first prompt word and the second prompt words of the multiple objects, style migration is performed on the image to be processed to obtain a first target image in a target style.

[0195] For the explanation of step S706, reference may be made to the relevant description in any embodiment of the present disclosure, and no further details will be given here.

[0196] In the image style transfer method of the disclosed embodiment, the local semantic labels and local context descriptions associated with the same object can complement each other and jointly provide a comprehensive and in-depth local semantic description for the segmented image of the same object, which can improve the accuracy and richness of the second prompt word generation, thereby improving the image quality after style transfer.

[0197] Among them, local context description can provide an in-depth perspective to help understand the objects in the segmented image and their environment. It not only includes the appearance characteristics of the object, but also reveals the object's emotional state, psychological state, emotional elements in its movements and postures, and its interaction information with the background environment, thereby improving the image quality and authenticity after style transfer.

[0198] Among them, the local semantic labels can provide detailed and vivid descriptive information of the objects, preserve the proportions and details of each object, and ensure the consistency of the image style of the first target image finally generated.

[0199] In order to clearly illustrate any embodiment of the present disclosure, the present disclosure also proposes an image style transfer method.

[0200] Figure 8 A flowchart of the image style transfer method provided in Embodiment 6 of the present disclosure.

[0201] like Figure 8 As shown, the image style transfer method may include the following steps S801 to S804:

[0202] Step S801, obtaining an image to be processed showing multiple objects.

[0203] Step S802, obtaining a first prompt word of the image to be processed; wherein the first prompt word is used to indicate the overall semantics of the image to be processed.

[0204] For explanations of steps S801 to S802, reference may be made to the relevant descriptions in any embodiment of the present disclosure and will not be repeated here.

[0205] Step S803: determining a reference image according to the image to be processed.

[0206] Exemplarily, the image to be processed may be used as a reference image.

[0207] Exemplarily, the image to be processed may be processed (including but not limited to: denoising processing, enhancement processing, style transfer, etc.) to obtain a reference image.

[0208] Step S804: performing style migration on the image to be processed according to the first prompt word and the reference image to obtain a second target image in a target style.

[0209] The explanation of the target style can refer to the relevant description in the above embodiment, which will not be elaborated here.

[0210] As an example, the InstantID (Instant Identity Recognition) technology may be used to perform style transfer on the image to be processed according to the first prompt word and the reference image to obtain a second target image in the target style.

[0211] Exemplarily, the first prompt word and the reference image can be input into the Stable Diffusion model as conditioning information, so that the Stable Diffusion model performs style migration or style conversion on the image to be processed based on the above conditioning information to obtain a second target image under the target style.

[0212] The image style transfer method of the disclosed embodiment performs style transfer or style conversion on the image to be processed according to the first prompt word and the reference image to obtain a second target image under the target style. The generated second target image can retain the key features and semantic information of the original image to be processed, such as the position, posture and expression of the object, etc., and is more refined and realistic in details, avoiding distortion and blur that may occur in the style transfer process, and improving the image quality after style transfer.

[0213] It should be noted that, taking the target style as a comic style, the existing image style transfer scheme has at least the following technical difficulties in converting multi-person images into comic styles:

[0214] 1. Consistency of facial and body details of multiple people: In the process of converting to comic style, current algorithms have great difficulty in maintaining the consistency of body details, such as clothing and movements, with the original image, and cannot ensure that the face remains highly similar to the original image;

[0215] 2. Stylization errors: When processing multi-person images, the style transfer process can easily lead to inconsistent or distorted comic stylization of the characters’ faces or bodies, affecting the overall style transfer effect;

[0216] 3. The relationship between characters is confusing: The interaction between characters in a multi-person image may be blurred when converted into a comic style, resulting in an imbalance in the proportions between the characters and unnatural movements;

[0217] 4. Processing speed and efficiency issues: When performing comic style transfer on complex multi-person images, the processing speed is slow, especially the control of details may increase the computational complexity.

[0218] In response to at least one of the above-mentioned problems, in any embodiment of the present disclosure, the target style is taken as a comic style as an example, and the comic style transfer of multi-person images is optimized specifically to solve the shortcomings of the current technology in detail expression, character relationship processing, etc.

[0219] As an example, the present disclosure proposes a multi-stage comic style transfer (comic adaptation) algorithm based on the Stable Diffusion model. The implementation principle can be as follows: Fig. 9 As shown, Fig. 9 Only the number of characters in the multi-person image is 2 as an example. The present disclosure solves the technical problems existing in the comic style transfer of multi-person images in the related art by performing specific optimization on the Stable Diffusion model, so that the converted comic image appears more natural in terms of details, style consistency and character interaction. That is, by performing targeted optimization on the Stable Diffusion model, the above-mentioned defects are overcome, and the details of each character (such as facial features, clothing style) can be retained to ensure that the converted comic image is highly similar to the original image in terms of overall and details. In addition, the processing capability of multi-person interaction scenes can be enhanced to ensure that the generated comic image has a natural and consistent comic style when expressing complex postures and character relationships, significantly improving the image quality of comic stylization in multi-person scenes, and is particularly suitable for application scenarios with high requirements for high-quality comic adaptations.

[0220] Compared with the existing solutions, this paper uses semantic segmentation, cue word inference, conditioning set mask and IPAdapter-FaceID technology to ensure the similarity and consistency of character features in multi-person scenarios:

[0221] 1) Semantic segmentation: For example, the SegmentAnything model can be used to segment people in multi-person images. The SegmentAnything model provides a general and fully automatic segmentation solution for segmentation tasks, so that the features of each person are relatively isolated, so that in the process of prompt word reverse deduction, the features of each person can be described separately, and then these feature prompts are fixed only in the area of ​​the person.

[0222] 2) Prompt word inference: For example, WD14 tagger (WD14 tag generation model, used to automatically generate tags for images) and MoonDream (an open source visual language model) can be used to generate overall prompt words for multi-person images (referred to as the first prompt words in this disclosure) and local prompt words for each character (referred to as the second prompt words in this disclosure). They can understand the content in the image and automatically generate usable prompt words (prompt). The difference between the two is that the MoonDream model has a more detailed and vivid understanding of the image, while the WD14 tagger model is more concise and clear. Combining the results of the two, a specific prompt description for each character can be generated, while at the same time having a good understanding of the overall picture.

[0223] 3) Conditioning set mask: In the process of generating images, the conditioning set mask can make the prompt word only work within the range specified by the mask to enhance the control of the image. In the process of multi-person comics, the prompt word of a certain person can be fixed in the mask of this life picture to achieve isolation between prompts and avoid the situation where the features of multiple people are intertwined and fused.

[0224] 4) IPAdapter-FaceID: First, IPAdapter is a text-compatible image prompt adapter for text-to-image diffusion models. The key design is the decoupled cross-attention mechanism, which separates the cross-attention layers of text features and image features. Since IPAdapter uses a frozen pre-trained diffusion model, IPAdapter can not only generalize to other custom models fine-tuned from the same base model, but also achieve controllable generation using existing controllable tools. Taking advantage of the decoupled cross-attention strategy, image prompts can also work well with text prompts to achieve multimodal image generation.

[0225] In addition, in response to the common phenomenon of facial corruption in existing algorithms, the present invention uses FaceDetailer (facial detail enhancement) technology, which is a local redrawing technology that achieves the effect of repairing a corrupted face by segmenting the face and performing local redrawing.

[0226] As a possible implementation method, style transfer can also be performed on multi-person images based on InstantID (Instant Identity Recognition) technology. InstantID technology is a new advanced method that does not require fine-tuning. It only requires a single reference image to generate new comic images while retaining the identity information of the characters in the original multi-person images, supporting various subsequent tasks, such as face recognition and face attribute analysis tasks.

[0227] In summary, the image-to-image technology based on the Stable Diffusion model can be applied to comic adaptation creation, poster production and other fields, so as to achieve the similarity of character features, character postures, clothing colors, etc. in multi-person comic adaptation scenes, and keep the integrity and similarity of multiple faces in multi-person images in the process of converting to comic style. Among them, the Stable Diffusion model adopted in the present disclosure has at least the following advantages:

[0228] 1. Diversity and detail retention: Stable Diffusion gradually generates images through a diffusion process, which can better handle complex multi-person scenes, retain the proportions, details and interactions between characters, while ensuring the consistency of the comic style;

[0229] 2. Better style control: The Stable Diffusion model allows accurate style transfer for each character by adjusting the noise distribution and conditional control during the diffusion process, ensuring that the generated comic images are more natural and realistic in detail processing;

[0230] 3. High processing efficiency: Compared with GAN (Generative Adversarial Networks) and neural style transfer, the Stable Diffusion model has higher computational efficiency while generating high-quality stylized images, and can process large-scale multi-person images more quickly.

[0231] With the above Figures 2 to 8 Corresponding to the image style transfer method provided in the embodiment, the present disclosure also provides an image style transfer device. Figures 2 to 8The image style transfer method provided in the embodiment corresponds to the above, so the implementation of the image style transfer method is also applicable to the image style transfer device provided in the embodiment of the present disclosure, and will not be described in detail in the embodiment of the present disclosure.

[0232] Fig.10 This is a schematic diagram of the structure of the image style transfer device provided in Embodiment 7 of the present disclosure.

[0233] like Fig.10 As shown, the image style transfer apparatus 1000 may include: a first acquisition module 1010 , a segmentation module 1020 , a second acquisition module 1030 and a style transfer module 1040 .

[0234] The first acquisition module 1010 is used to acquire a first prompt word of the image to be processed showing multiple objects; wherein the first prompt word is used to indicate the overall semantics of the image to be processed;

[0235] The segmentation module 1020 is used to perform semantic segmentation on the image to be processed to obtain segmented images of multiple objects;

[0236] A second acquisition module 1030 is used to acquire second prompt words of the multiple objects based on the segmented images of the multiple objects; wherein the second prompt words are used to indicate the local semantics of the objects in the image to be processed;

[0237] The style transfer module 1040 is used to perform style transfer on the image to be processed based on the first prompt word and the second prompt words of multiple objects to obtain a first target image under the target style.

[0238] In a possible implementation of the embodiments of the present disclosure, the style transfer module 1040 is used to: generate masks of multiple objects based on second prompt words of multiple objects; wherein the mask is used to indicate the scope of action of the second prompt word in the image generation process; perform style transfer on the image to be processed based on the second prompt words and masks corresponding to the multiple objects, and based on the first prompt word, to obtain a first target image under the target style.

[0239] In a possible implementation of the embodiments of the present disclosure, the style transfer module 1040 is used to: perform style transfer on the image to be processed according to the second prompt words and masks corresponding to the multiple objects and according to the first prompt words to obtain an initial transfer image under the target style; extract facial features of the multiple objects from the segmented images of the multiple objects; and perform facial adjustments on the initial transfer image based on the facial features of the multiple objects to obtain a first target image.

[0240] In a possible implementation of the embodiments of the present disclosure, the style transfer module 1040 is used to: perform facial redrawing based on facial features of multiple objects to obtain facial redrawn images of the multiple objects under the target style; and fuse the facial redrawn images of the multiple objects with the initial transfer image to obtain a first target image.

[0241] In a possible implementation of the embodiments of the present disclosure, the style transfer module 1040 is used to: perform facial segmentation on the segmented images of multiple objects to obtain facial images of multiple objects; encode the facial images of multiple objects to obtain image features of multiple objects; and perform linear and normalization processing on the image features of multiple objects respectively to obtain facial features of multiple objects.

[0242] In a possible implementation of the embodiment of the present disclosure, the first acquisition module 1010 is used to: extract an overall semantic label associated with multiple objects from the image to be processed; extract an overall situation description associated with multiple objects from the image to be processed; and generate a first prompt word based on the overall semantic label and the overall situation description.

[0243] In a possible implementation of the embodiment of the present disclosure, the overall semantic label is used to represent at least one of the following: basic features of multiple objects; action postures of multiple objects; environmental information of multiple objects;

[0244] The overall situation description is used to characterize at least one of the following: appearance description of multiple objects; emotional states of multiple objects; psychological states of multiple objects; object relationships between multiple objects; interaction methods between multiple objects; scene atmosphere in the image to be processed; emotional color in the image to be processed.

[0245] In a possible implementation of the embodiments of the present disclosure, the second acquisition module 1030 is used to: extract a local semantic label associated with any object from a segmented image of any object; extract a local context description associated with any object from a segmented image of any object; and generate a second prompt word for any object based on the local semantic label and the local context description.

[0246] In a possible implementation of the embodiment of the present disclosure, a local semantic label is used to characterize the basic features and action posture of any object;

[0247] The local context description is used to characterize at least one of the following: the appearance features of any object; the emotional state of any object; the psychological state of any object; the emotional elements in the movements and postures of any object; and the interaction information between any object and the background environment.

[0248] In a possible implementation of the embodiment of the present disclosure, the image style transfer apparatus 1000 may further include:

[0249] A determination module, used for determining a reference image according to the image to be processed;

[0250] The style transfer module 1040 is further used to perform style transfer on the image to be processed according to the first prompt word and the reference image to obtain a second target image under the target style.

[0251] In a possible implementation of the embodiment of the present disclosure, the image style transfer apparatus 1000 further includes:

[0252] The third acquisition module is used to execute any one of the following to acquire the image to be processed: in response to an upload operation, acquire the uploaded image to be processed; in response to a shooting operation, acquire the captured image to be processed; in response to a search operation, search for candidate images matching the search keywords, and determine the image to be processed from the candidate images; receive the image to be processed sent in response to a share operation; in response to an image synthesis operation, acquire the synthesized image to be processed.

[0253] The image style transfer device of the disclosed embodiment performs style transfer or style conversion on the original image based on a first prompt word for indicating the overall semantics of the original image to be processed (hereinafter referred to as the original image) and a second prompt word for indicating the local semantics of each object (such as a person) in the original image. This can ensure that the first target image after style transfer is highly similar to the original image in overall and detail, so that the generated first target image is more natural and realistic in detail processing, retains the proportions, details and interactions between different objects, and improves the image quality after style transfer in multi-object scenes.

[0254] In order to implement the above embodiments, the present disclosure also provides an electronic device, which may include at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the image style transfer method proposed in any of the above embodiments of the present disclosure.

[0255] In order to implement the above embodiments, the present disclosure further provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the image style transfer method proposed in any of the above embodiments of the present disclosure.

[0256] In order to implement the above embodiments, the present disclosure further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the image style transfer method proposed in any of the above embodiments of the present disclosure.

[0257] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0258] Fig.11 A schematic block diagram of an example electronic device that can be used to implement embodiments of the present disclosure is shown.

[0259] Among them, the electronic device may include the server and the client in the above-mentioned embodiment. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are only examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0260] like Fig.11 As shown, the electronic device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in a ROM (Read-Only Memory) 1102 or a computer program loaded from a storage unit 1107 to a RAM (Random Access Memory) 1103. In the RAM 1103, various programs and data required for the operation of the device 1100 can also be stored. The computing unit 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. An I / O (Input / Output) interface 1105 is also connected to the bus 1104.

[0261] A number of components in the device 1100 are connected to the I / O interface 1105, including: an input unit 1106, such as a keyboard, a mouse, etc.; an output unit 1107, such as various types of displays, speakers, etc.; a storage unit 1108, such as a disk, an optical disk, etc.; and a communication unit 1109, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1109 allows the device 1100 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0262] The computing unit 1101 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a CPU (Central Processing Unit), a GPU (Graphic Processing Units), various dedicated AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, a DSP (Digital Signal Processor), and any appropriate processor, controller, microcontroller, etc. The computing unit 1101 performs the various methods and processes described above, such as the above-described image style transfer method. For example, in some embodiments, the above-described image style transfer method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1108. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1100 via the ROM 1102 and / or the communication unit 1109. When the computer program is loaded into the RAM 1103 and executed by the computing unit 1101, one or more steps of the image style transfer method described above may be performed. Alternatively, in other embodiments, the computing unit 1101 may be configured to execute the above-mentioned image style transfer method in any other appropriate manner (for example, by means of firmware).

[0263] Various embodiments of the systems and techniques described above herein may be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application Specific Standard Products), SOCs (System On Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that may be executed and / or interpreted on a programmable system including at least one programmable processor that may be a dedicated or general-purpose programmable processor that may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0264] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0265] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include electrical connections based on one or more lines, portable computer disks, hard disks, RAM, ROM, EPROM (Electrically Programmable Read-Only-Memory) or flash memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0266] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0267] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: LAN (Local Area Network), WAN (Wide Area Network), the Internet, and blockchain networks.

[0268] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services (Virtual Private Server). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0269] It should be noted that artificial intelligence is a discipline that studies how computers can simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.), and includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, as well as machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.

[0270] According to the technical solution of the embodiment of the present disclosure, style migration or style conversion is performed on the original image based on the first prompt word used to indicate the overall semantics of the original image to be processed (hereinafter referred to as the original image) and the second prompt word used to indicate the local semantics of each object (such as a person) in the original image. This can ensure that the first target image after style migration is highly similar to the original image in terms of both overall and details, so that the generated first target image is more natural and realistic in detail processing, retains the proportions, details and interactions between different objects, and improves the image quality after style migration in multi-object scenes.

[0271] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0272] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for image style transfer, comprising: Acquire a first prompt word of an image to be processed showing multiple objects; wherein the first prompt word is used to indicate the overall semantics of the image to be processed; Performing semantic segmentation on the image to be processed to obtain segmented images of the multiple objects; Based on the segmented images of the multiple objects, obtaining second prompt words for the multiple objects; wherein the second prompt words are used to indicate the local semantics of the objects in the image to be processed; Based on the first prompt word and the second prompt words of the multiple objects, style migration is performed on the image to be processed to obtain a first target image under a target style.

2. The method according to claim 1, wherein: The performing style migration on the image to be processed based on the first prompt word and the second prompt words of the multiple objects to obtain a first target image under a target style includes: Generate masks of the multiple objects according to the second prompt words of the multiple objects; wherein the mask is used to indicate the scope of action of the second prompt word in the image generation process; According to the second prompt words and masks corresponding to the multiple objects and according to the first prompt word, style migration is performed on the image to be processed to obtain a first target image under the target style.

3. The method according to claim 2, wherein: The performing style migration on the image to be processed according to the second prompt words and masks corresponding to the multiple objects and according to the first prompt word to obtain a first target image under the target style includes: Performing style migration on the image to be processed according to the second prompt words and masks corresponding to the multiple objects and according to the first prompt word to obtain an initial migration image under the target style; extracting facial features of the plurality of objects from the segmented images of the plurality of objects; Based on the facial features of the multiple objects, facial adjustments are performed on the initial migration image to obtain the first target image.

4. The method according to claim 3, wherein: The step of performing facial adjustment on the initial migration image based on the facial features of the multiple objects to obtain the first target image comprises: Performing facial redrawing based on facial features of the multiple objects to obtain facial redrawn images of the multiple objects in the target style; The facial redrawn images of the multiple objects and the initial migration image are fused to obtain the first target image.

5. The method according to claim 3, wherein: Extracting facial features of the multiple objects from the segmented images of the multiple objects includes: Performing facial segmentation on the segmented images of the multiple objects to obtain facial images of the multiple objects; Encoding the facial images of the multiple objects to obtain image features of the multiple objects; Linear and normalized processing is performed on the image features of the multiple objects respectively to obtain facial features of the multiple objects.

6. The method according to claim 1, wherein: The step of obtaining a first prompt word of an image to be processed showing multiple objects includes: Extracting overall semantic labels associated with the plurality of objects from the image to be processed; Extracting, from the image to be processed, an overall situation description associated with the plurality of objects; The first prompt word is generated according to the overall semantic tag and the overall situation description.

7. The method according to claim 6, wherein: The overall semantic label is used to represent at least one of the following: basic features of the multiple objects; action postures of the multiple objects; and environmental information of the multiple objects. The overall situation description is used to characterize at least one of the following: the appearance description of the multiple objects; the emotional state of the multiple objects; the psychological state of the multiple objects; the object relationship between the multiple objects; the interaction mode between the multiple objects; the scene atmosphere in the image to be processed; The emotional color in the image to be processed.

8. The method according to claim 1, wherein: The acquiring second prompt words for the multiple objects based on the segmented images of the multiple objects includes: Extracting a local semantic label associated with any object from a segmented image of the object; Extracting a local context description associated with any one of the objects from the segmented image of any one of the objects; A second prompt word for any one of the objects is generated according to the local semantic label and the local context description.

9. The method according to claim 8, wherein: The local semantic label is used to characterize the basic features and action posture of any object; The local situation description is used to characterize at least one of the following: the appearance features of any object; the emotional state of any object; the psychological state of any object; the emotional elements in the action posture of any object; and the interaction information between any object and the background environment.

10. The method according to any one of claims 1 to 9, wherein: The method further comprises: Determining a reference image according to the image to be processed; According to the first prompt word and the reference image, style migration is performed on the image to be processed to obtain a second target image under the target style.

11. The method according to any one of claims 1 to 9, wherein: The image to be processed is obtained by using any of the following methods: In response to the uploading operation, acquiring the uploaded image to be processed; In response to a shooting operation, acquiring the shot image to be processed; In response to a search operation, searching for candidate images matching the search keyword, and determining the image to be processed from the candidate images; receiving the image to be processed sent in response to a sharing operation; In response to the image synthesis operation, the synthesized image to be processed is obtained.

12. An image style transfer device, comprising: A first acquisition module is used to acquire a first prompt word of an image to be processed showing multiple objects; wherein the first prompt word is used to indicate the overall semantics of the image to be processed; A segmentation module, used for performing semantic segmentation on the image to be processed to obtain segmented images of the multiple objects; A second acquisition module, used for acquiring second prompt words for the multiple objects based on the segmented images of the multiple objects; wherein the second prompt words are used to indicate the local semantics of the objects in the image to be processed; The style transfer module is used to perform style transfer on the image to be processed based on the first prompt word and the second prompt words of the multiple objects to obtain a first target image under a target style.

13. The device according to claim 12, wherein: The style transfer module is used to: Generate masks of the multiple objects according to the second prompt words of the multiple objects; wherein the mask is used to indicate the scope of action of the second prompt word in the image generation process; According to the second prompt words and masks corresponding to the multiple objects and according to the first prompt word, style migration is performed on the image to be processed to obtain a first target image under the target style.

14. The device according to claim 13, wherein: The style transfer module is used to: Performing style migration on the image to be processed according to the second prompt words and masks corresponding to the multiple objects and according to the first prompt word to obtain an initial migration image under the target style; extracting facial features of the plurality of objects from the segmented images of the plurality of objects; Based on the facial features of the multiple objects, facial adjustments are performed on the initial migration image to obtain the first target image.

15. The device according to claim 14, wherein: The style transfer module is used to: Performing facial redrawing based on facial features of the multiple objects to obtain facial redrawn images of the multiple objects in the target style; The facial redrawn images of the multiple objects and the initial migration image are fused to obtain the first target image.

16. The device according to claim 14, wherein: The style transfer module is used to: Performing facial segmentation on the segmented images of the multiple objects to obtain facial images of the multiple objects; Encoding the facial images of the multiple objects to obtain image features of the multiple objects; Linear and normalized processing is performed on the image features of the multiple objects respectively to obtain facial features of the multiple objects.

17. The device according to claim 12, wherein: The first acquisition module is used to: Extracting overall semantic labels associated with the plurality of objects from the image to be processed; Extracting, from the image to be processed, an overall situation description associated with the plurality of objects; The first prompt word is generated according to the overall semantic tag and the overall situation description.

18. The device according to claim 17, wherein: The overall semantic label is used to represent at least one of the following: basic features of the multiple objects; action postures of the multiple objects; and environmental information of the multiple objects. The overall situation description is used to characterize at least one of the following: the appearance description of the multiple objects; the emotional state of the multiple objects; the psychological state of the multiple objects; the object relationship between the multiple objects; the interaction mode between the multiple objects; the scene atmosphere in the image to be processed; The emotional color in the image to be processed.

19. The device according to claim 12, wherein: The second acquisition module is used to: Extracting a local semantic label associated with any object from a segmented image of the object; Extracting a local context description associated with any one of the objects from the segmented image of any one of the objects; A second prompt word for any one of the objects is generated according to the local semantic label and the local context description.

20. The device according to claim 19, wherein The local semantic label is used to characterize the basic features and action posture of any object; The local context description is used to characterize at least one of the following: the appearance features of any object; the emotional state of any object; the psychological state of any object; the emotional elements in the action posture of any object; and the interaction information between any object and the background environment.

21. The device according to any one of claims 12 to 20, wherein: The device also includes: A determination module, used to determine a reference image according to the image to be processed; The style transfer module is further used to: perform style transfer on the image to be processed according to the first prompt word and the reference image to obtain a second target image under the target style.

22. The device according to any one of claims 12 to 20, wherein: The device also includes: The third acquisition module is used to perform any of the following to acquire the image to be processed: in response to an upload operation, acquire the uploaded image to be processed; in response to a shooting operation, acquire the captured image to be processed; in response to a search operation, search for candidate images matching the search keywords, and determine the image to be processed from the candidate images; receive the image to be processed sent in response to a share operation; in response to an image synthesis operation, acquire the synthesized image to be processed.

23. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the image style transfer method according to any one of claims 1 to 11.

24. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable the computer to execute the image style transfer method according to any one of claims 1-11.

25. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the steps of the image style transfer method according to any one of claims 1 to 11.

Images