Image style transfer processing method and device, electronic device and storage medium

By acquiring the depth features and element information of the image, generating a large model and generating a target image, the problem of image style fusion in the prior art is solved, and the personalized and customized effect of image style transfer is achieved.

CN118570053BActive Publication Date: 2024-11-26HAIMA CLOUD TIANJIN INFORMATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411057367.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-02
Publication Date
2024-11-26
Estimated Expiration
2044-08-02

AI Technical Summary

Technical Problem

The prior art is difficult to effectively integrate the style of reference images while maintaining the integrity of the image content, and cannot meet users' needs for personalized styles and customized effects.

Method used

By obtaining the depth characteristics and element information of the pending image and reference image, a large model with reference image elements is generated, and the target image is generated based on the prompt words, depth characteristics and large model, the style of the reference image is successfully applied, while retaining the pending image. Features of the image.

Benefits of technology

It realizes that while maintaining the integrity of the image content, it effectively integrates the reference image style to meet users' needs for personalized style and customized effects.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118570053B_ABST
    Figure CN118570053B_ABST
Patent Text Reader

Abstract

The present application provides an image style transfer processing method and device, an electronic device and a storage medium, the method comprising: obtaining an image to be processed and a reference image, generating prompt words for the image to be processed and the reference image, and extracting depth features of the image to be processed; extracting element information of the reference image, and fusing the extracted element information with a large model to obtain a large model with element information of the reference image; generating a first target image based on the prompt words, the depth features and the large model with element information of the reference image, so as to achieve the purpose of effectively fusing the style of the reference image while maintaining the integrity of the specified image content, and meet the user's needs for personalized style and customized effects.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computers, and in particular to an image style migration processing method and device, an electronic device and a storage medium. Background Art

[0002] With the development of digital image processing technology, image style transfer and redrawing have become areas of widespread concern to users. Existing methods mainly focus on processing images through algorithms and filters. When these methods apply the style to the target image, they cannot completely retain the characteristics of the original image, resulting in the final redrawing effect not meeting the user's expectations. Users may need to customize the redrawn image with a specific style according to their needs, but existing technologies usually lack a good style customization and adjustment mechanism and cannot meet users' needs for personalized style and customized effects.

[0003] Therefore, how to provide a solution to transfer the style of a reference image to a specified image, so as to effectively integrate the style of the reference image while maintaining the integrity of the specified image content and meet the user's needs for personalized style and customized effects, has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] In view of the technical problems existing in the prior art, the embodiments of the present application provide an image style transfer processing method and device, an electronic device and a storage medium.

[0005] In a first aspect, an embodiment of the present application provides an image style transfer processing method, comprising:

[0006] Acquire an image to be processed and a reference image, generate prompt words for the image to be processed and the reference image, and extract depth features of the image to be processed;

[0007] Extracting element information of the reference image, and fusing the extracted element information with the large model to obtain a large model with the element information of the reference image;

[0008] A first target image is generated based on the prompt word, the depth feature and the large model with reference image element information.

[0009] In a second aspect, the embodiment of the present application further provides an image style transfer processing device, comprising:

[0010] An extraction unit, used to obtain an image to be processed and a reference image, generate prompt words for the image to be processed and the reference image, and extract a depth feature of the image to be processed;

[0011] A fusion unit, used to extract element information of the reference image, and fuse the extracted element information with the large model to obtain a large model with the element information of the reference image;

[0012] A processing unit is used to generate a first target image based on the prompt word, the depth feature and the large model with reference image element information.

[0013] In a third aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the image style transfer processing method described in the first aspect are executed.

[0014] In a fourth aspect, an embodiment of the present application further provides an electronic device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium through the bus, and the processor executes the machine-readable instructions to perform the steps of the image style transfer processing method as described in the first aspect.

[0015] In the above scheme, the large model with reference image elements integrates the style of the reference image. In this way, the first target image generated based on the prompt words of the image to be processed and the reference image, the deep features of the image to be processed and the large model with reference image elements successfully applies the style of the reference image and completely retains the features of the image to be processed. At the same time, the user can specify an image with a certain style as the reference image. Therefore, it is possible to achieve the purpose of effectively integrating the style of the reference image while maintaining the integrity of the specified image content and meet the user's needs for personalized style and customized effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A schematic diagram of a flow chart of an embodiment of an image style transfer processing method provided in an embodiment of the present application;

[0017] Figure 2 A schematic diagram of the structure of an embodiment of an image style transfer processing device provided in an embodiment of the present application;

[0018] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of order, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the present application, or remove one or more operations from the flowchart.

[0020] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0021] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0022] Reference Figure 1 FIG. 1 is a flow chart of an image style transfer processing method provided by an embodiment of the present application, the method comprising:

[0023] S10, acquiring an image to be processed and a reference image, generating prompt words for the image to be processed and the reference image, and extracting depth features of the image to be processed;

[0024] In this implementation, it should be noted that both the image to be processed and the reference image can be images uploaded by the user, or images specified by the user from existing images. Before generating the prompt word, the larger values ​​of the width and height of the image to be processed and the reference image can be adjusted to 1024, and the smaller values ​​can be scaled according to the corresponding proportion, so that the ratio of the width to height of the image to be processed and the reference image after scaling remains unchanged, and the larger values ​​of the width and height are both 1024. The prompt word is used to describe the image content. For example, if the image to be processed is a photo of a girl, the prompt word of the image to be processed can be girl, long hair, blue sweater, etc. The existing prompt word generation model can be used to generate prompt words for the image to be processed and the reference image. When extracting deep features, the depth preprocessor of the ControlNet component can be used to process the image to be processed to obtain a depth map, and the prompt word and the depth map are input into the depth model corresponding to the depth preprocessor. The depth model extracts the depth features from the depth map and outputs the fusion data of the prompt word and the depth feature (ControlNet image feature).

[0025] S11, extracting element information of the reference image, and fusing the extracted element information with the large model to obtain a large model with the element information of the reference image;

[0026] In this implementation, it should be noted that the large model (Checkpoint) may include at least one of a 3D realistic large model, a 2.5D realistic large model, and a 2D comic large model. The element information of the reference image refers to the information of the elements presented in the reference image. For example, if the reference image contains elements such as a rainbow and grass, the element information of the reference image may include the color distribution of the rainbow and grass. The existing fusion technology may be used to fuse the extracted element information with the large model, which will not be described in detail here.

[0027] S12: Generate a first target image based on the prompt word, the depth feature and the large model with reference image element information.

[0028] In this implementation, it should be noted that when generating the first target image, the fusion data of the prompt word and the depth feature can be input into a large model with reference image elements to obtain the first target image. For example, if the image to be processed and the reference image are both portrait images, the first target image finally generated is a portrait image with the style of the reference image, and this portrait image retains the features of the image to be processed, such as the posture and appearance of the characters in the two images are consistent. For each large model, a corresponding first target image can be generated.

[0029] The image style transfer processing method provided in the embodiment of the present application is a large model with reference image elements that integrates the style of the reference image. In this way, the first target image generated based on the prompt words of the image to be processed and the reference image, the deep features of the image to be processed and the large model with reference image elements successfully applies the style of the reference image and completely retains the features of the image to be processed. At the same time, the user can specify an image with a certain style as the reference image. Therefore, it is possible to achieve the purpose of effectively integrating the style of the reference image while maintaining the integrity of the specified image content and meet the user's needs for personalized style and customized effects.

[0030] Based on the aforementioned method embodiment, the generating of prompt words for the image to be processed and the reference image may include:

[0031] Using a prompt word generation model, respectively generating image prompt words based on the image to be processed and the reference image to obtain a first image prompt word and a second image prompt word;

[0032] The positive prompt words and negative prompt words of the image to be processed and the reference image are specified.

[0033] In this embodiment, it should be noted that in addition to generating image prompt words based on the image to be processed and the reference image, positive prompt words and / or negative prompt words for the image to be processed and the reference image can also be specified. Positive prompt words can be words such as high quality, ultra-clear, 32K, etc. that can reflect high-quality image quality; while negative prompt words are exactly the opposite of positive prompt words, such as words such as worst quality, monochrome, etc. that represent non-high-quality image quality. Both positive prompt words and negative prompt words need to be input by the user. The positive prompt words of the image to be processed and the reference image can be the same or different. The negative prompt words of the image to be processed and the reference image can be the same or different.

[0034] Based on the foregoing method embodiment, extracting the depth feature of the image to be processed may include:

[0035] If the image to be processed is a person image, generating a depth map and a skeleton map of the image to be processed, and extracting features of the depth map and the skeleton map; or

[0036] If the image to be processed is a scene image, a depth map and a line drawing image of the image to be processed are generated, and features of the depth map and the line drawing image are extracted.

[0037] In this implementation, it should be noted that if the image to be processed is a person image, the depth preprocessor of the ControlNet component can be used to process the image to be processed to obtain a depth map, and the prompt word and the depth map are input into the depth model corresponding to the depth preprocessor. The depth model extracts the depth features from the depth map and outputs the fusion data of the prompt word and the depth features. The posture (OpenPose) preprocessor of the ControlNet component is used to process the image to be processed to obtain the skeleton map, the fusion data of the prompt word and the depth features, and the skeleton map is input into the posture model corresponding to the posture preprocessor. The posture model extracts the skeleton features from the skeleton map, and outputs the fusion data of the prompt word, the depth features and the skeleton features (the depth features and the skeleton features are the ControlNet image features).

[0038] If the image to be processed is a scene image, the depth preprocessor of the ControlNet component can be used to process the image to be processed to obtain a depth map, and the prompt word and the depth map are input into the depth model corresponding to the depth preprocessor. The depth model extracts the depth features from the depth map and outputs the fusion data of the prompt word and the depth features. The lineart preprocessor of the ControlNet component is used to process the image to be processed to obtain the lineart map, the fusion data of the prompt word and the depth features, and the lineart model corresponding to the lineart preprocessor. The lineart model extracts the lineart features from the lineart map and outputs the fusion data of the prompt word, the depth features and the lineart features (the depth features and the lineart features are the ControlNet image features).

[0039] Based on the foregoing method embodiment, the method may further include:

[0040] If the image to be processed is a person image, performing face-changing processing on the first target image using the image to be processed to obtain a face-changing image;

[0041] Perform facial cutout on the face-changing image to obtain a facial image, and input the specified positive prompt words, negative prompt words, and the facial image into a large model to generate a second target image.

[0042] In this implementation, it should be noted that if the image to be processed is a person, the image to be processed can be used to perform face-changing processing on the first target image corresponding to the 3D realistic large model and / or the first target image corresponding to the 2.5D realistic large model to obtain a face-changing image, and the facial area in the face-changing image is the facial area in the image to be processed. Afterwards, the face-changing image can be face-repaired so that the facial edges of the face-changing image are softer, and the light and shadow and tone are more natural. Specifically, during face repair, the width and height of the face-changing image can be enlarged by 2 times, and the enlarged image can be face-cut to obtain a facial image, and then the specified positive prompt word (input by the user, which can be different from the positive prompt word in the aforementioned embodiment), the negative prompt word (which can be the same as the negative prompt word in the aforementioned embodiment), and the facial image can be input into the large model in step S11, and the output of the large model is the repaired image, that is, the second target image, wherein the large model can be the large model in step S11, or a model after the large model with reference image elements and the lora model are fused. It should be noted that the facial region in the second target image finally obtained in this step is consistent with the facial region of the image to be processed, while the styles of other elements are the same as those of the first target image.

[0043] Reference Figure 2 FIG. 1 is a schematic diagram of a structure of an image style transfer processing device provided in an embodiment of the present application, the device comprising:

[0044] The extraction unit 20 is used to obtain the image to be processed and the reference image, generate prompt words for the image to be processed and the reference image, and extract the depth feature of the image to be processed;

[0045] A fusion unit 21 is used to extract element information of the reference image and fuse the extracted element information with the large model to obtain a large model with the element information of the reference image;

[0046] The processing unit 22 is used to generate a first target image based on the prompt word, the depth feature and the large model with reference image element information.

[0047] The image style transfer processing device provided in the embodiment of the present application has a large model with reference image elements that integrates the style of the reference image. In this way, the first target image generated based on the prompt words of the image to be processed and the reference image, the deep features of the image to be processed and the large model with reference image elements successfully applies the style of the reference image and completely retains the features of the image to be processed. At the same time, the user can specify an image with a certain style as the reference image. Therefore, it is possible to achieve the purpose of effectively integrating the style of the reference image while maintaining the integrity of the specified image content and meet the user's needs for personalized style and customized effects.

[0048] Based on the above device embodiment, the extraction unit can be used for:

[0049] Using a prompt word generation model, respectively generating image prompt words based on the image to be processed and the reference image to obtain a first image prompt word and a second image prompt word;

[0050] The positive prompt words and negative prompt words of the image to be processed and the reference image are specified.

[0051] Based on the above device embodiment, the extraction unit can be used for:

[0052] If the image to be processed is a person image, generating a depth map and a skeleton map of the image to be processed, and extracting features of the depth map and the skeleton map; or

[0053] If the image to be processed is a scene image, a depth map and a line drawing image of the image to be processed are generated, and features of the depth map and the line drawing image are extracted.

[0054] Based on the above device embodiment, the processing unit may also be used for:

[0055] If the image to be processed is a person image, performing face-changing processing on the first target image using the image to be processed to obtain a face-changing image;

[0056] Perform facial cutout on the face-changing image to obtain a facial image, and input the specified positive prompt words, negative prompt words, and the facial image into a large model to generate a second target image.

[0057] The image style transfer processing device provided in the embodiment of the present application has an implementation process that is consistent with the image style transfer processing method provided in the embodiment of the present application, and the effect that can be achieved is also the same as the image style transfer processing method provided in the embodiment of the present application, and will not be repeated here.

[0058] like Figure 3 As shown, an electronic device provided by an embodiment of the present application includes: a processor 30, a memory 31 and a bus 32, wherein the memory 31 stores machine-readable instructions executable by the processor 30, and when the electronic device is running, the processor 30 communicates with the memory 31 through the bus 32, and the processor 30 executes the machine-readable instructions to perform the steps of the above-mentioned image style transfer processing method.

[0059] Specifically, the memory 31 and the processor 30 can be general-purpose memories and processors, which are not specifically limited here. When the processor 30 runs the computer program stored in the memory 31, the image style transfer processing method can be executed.

[0060] Corresponding to the above-mentioned image style transfer processing method, an embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned image style transfer processing method are executed.

[0061] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for image style transfer, characterized in that: include: Acquire an image to be processed and a reference image, generate prompt words for the image to be processed and the reference image, and extract deep features of the image to be processed, wherein the prompt words are used to describe the image content; Extracting element information of the reference image, and fusing the extracted element information with the large model to obtain a large model with element information of the reference image, where the element information of the reference image refers to information of elements presented in the reference image; Generate a first target image based on the prompt word, the depth feature and the large model with reference image element information; Also includes: If the image to be processed is a person image, performing face-changing processing on the first target image using the image to be processed to obtain a face-changing image; Perform facial cutout on the face-changing image to obtain a facial image, and input the specified positive prompt words, negative prompt words, and the facial image into a large model to generate a second target image.

2. The method according to claim 1, characterized in that The step of generating prompt words for the image to be processed and the reference image includes: Using a prompt word generation model, respectively generating image prompt words based on the image to be processed and the reference image to obtain a first image prompt word and a second image prompt word; The positive prompt words and negative prompt words of the image to be processed and the reference image are specified.

3. The method according to claim 1 or 2, characterized in that The extracting the depth feature of the image to be processed comprises: Generate a depth map and a skeleton map of the image to be processed, and extract features of the depth map and the skeleton map.

4. An image style transfer processing device, characterized in that: include: An extraction unit, used to obtain an image to be processed and a reference image, generate prompt words for the image to be processed and the reference image, and extract deep features of the image to be processed, wherein the prompt words are used to describe the image content; A fusion unit, used to extract element information of the reference image, and fuse the extracted element information with the large model to obtain a large model with element information of the reference image, where the element information of the reference image refers to information of elements presented in the reference image; A processing unit, configured to generate a first target image based on the prompt word, the depth feature and the large model with reference image element information; The processing unit is further used for: If the image to be processed is a person image, performing face-changing processing on the first target image using the image to be processed to obtain a face-changing image; Perform facial cutout on the face-changing image to obtain a facial image, and input the specified positive prompt words, negative prompt words, and the facial image into a large model to generate a second target image.

5. The device according to claim 4, characterized in that The extraction unit is used for: Using a prompt word generation model, respectively generating image prompt words based on the image to be processed and the reference image to obtain a first image prompt word and a second image prompt word; The positive prompt words and negative prompt words of the image to be processed and the reference image are specified.

6. The device according to claim 4 or 5, characterized in that The extraction unit is used for: Generate a depth map and a skeleton map of the image to be processed, and extract features of the depth map and the skeleton map.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the image style transfer processing method according to any one of claims 1 to 3 are executed.

8. An electronic device, characterized in that: include: A processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of the image style transfer processing method as described in any one of claims 1 to 3.