Image processing method and device

Abstract

The invention provides an image processing method and device, and belongs to the technical field of computer vision. And obtaining a to-be-redrawn element and a target stylization generation model in the original image, guiding the target stylization generation model to redraw the to-be-redrawn element by adopting the guiding information so as to obtain a redrawn element of a target style, and generating a target image according to the redrawn element and the original image. The target stylization generation model is obtained based on training of a sample image marked with a target style. The guide information comprises control information of the element to be redrawn. The control information comprises imaging characteristics of the element to be redrawn in the original image. According to the method, the target stylized generation model is guided by the guide information to redraw the to-be-redrawn elements in the original image, so that when the to-be-redrawn elements are converted into the redrawn elements of the target style by the target stylized generation model, the imaging features of the elements before and after redrawing are kept consistent, and the quality and stability of the generated stylized image are improved.

Description

(19) State Intellectual Property Office (12) Invention Patent Application (10) Application Publication Number (43) Application Publication Date (21) Application Number 202411174001.8 (22) Application Date 2024.08.26 (66) Domestic Priority Data 202410800094.4 2024.06.19 CN (71) Applicant Huawei Cloud Computing Technology Co., Ltd. Address 550025, Guizhou Province, Guiyang City, Gui'an New District, Qianzhong Avenue, Xinggong Road, Huawei Cloud Data Center (72) Inventors Li Likun, Jia Haozhe, Liu Letao, Xu Di, Zhou Xuan (74) Patent Agency Beijing Sangao Yongxin Intellectual Property Agency Co., Ltd. 11138 Patent Attorney Yan Jing (51) Int.Cl. G06T 11 / 00 (2006.01) G06N 3 / 0475 (2023.01) G06N 3 / 084 (2023.01) (54) Invention Title: Image Processing Method and Apparatus (57) Abstract: This application provides an image processing method and apparatus, belonging to the field of computer vision technology. The method involves acquiring the elements to be redrawn and a target stylization generation model from the original image, using guiding information to guide the target stylization generation model to redraw the elements to be redrawn, thereby obtaining redrawn elements of the target style, and generating a target image based on the redrawn elements and the original image. The target stylization generation model is trained based on sample images labeled with the target style. The guiding information includes control information for the elements to be redrawn. The control information includes the imaging features of the elements to be redrawn in the original image. This application uses guiding information to guide the target stylization generation model to redraw the elements to be redrawn in the original image, ensuring that the target stylization generation model maintains consistent imaging features before and after redrawing when converting the elements to be redrawn into redrawn elements of the target style, thus improving the quality and stability of the generated stylized image. Claims 4 pages, Description 16 pages, Drawings 6 pages, CN 121170044 A 2025.12.19 CN 1 21 17 00 44 A 1. An image processing method, characterized in that the method comprises: acquiring elements to be redrawn in an original image, the elements to be redrawn including foreground imaging and / or background imaging; acquiring a target stylization generation model, the target stylization generation model being trained based on sample images labeled with a target style; guiding the target stylization generation model to redraw the elements to be redrawn using guiding information to obtain redrawn elements of the target style, the guiding information including control information of the elements to be redrawn, the control information including imaging features of the elements to be redrawn in the original image; generating a target image based on the redrawn elements and the original image. 2. The method according to claim 1, characterized in that the control information includes the imaging features of the elements to be redrawn in the original image.3. The method according to claim 1 or 2, wherein the method further comprises: extracting imaging features of the element to be redrawn from the original image to obtain the control information. 4. The method according to any one of claims 1 to 3, wherein the imaging features of the element to be redrawn in the original image include one or more of the space, position, color, outline, or texture of the element to be redrawn in the original image. 5. The method according to any one of claims 1 to 4, wherein the guidance information further comprises image content description information corresponding to the original image, the image content description information being used to describe the semantics expressed by the original image. 6. The method according to claim 5, wherein the original image is from a video, and the method further comprises: performing scene segmentation on the video to obtain multiple video segments; for each of the multiple video segments, performing video content recognition on the video segment to obtain video content description information corresponding to the video segment, and using the video content description information as the image content description information corresponding to each frame of the video segment. 7. The method according to any one of claims 1 to 6, wherein obtaining the target stylization generation model comprises: training an initial stylization generation model using a training sample set to obtain the target stylization generation model, wherein the training sample set includes one or more sample images labeled with the target style. 8. The method according to any one of claims 1 to 6, wherein the method is applied to an image processing device, the image processing device having multiple preset initial stylization generation models, and obtaining the target stylization generation model comprises: training the multiple initial stylization generation models respectively using a training sample set to obtain multiple candidate stylization generation models, wherein the training sample set includes one or more sample images labeled with the target style; obtaining the target stylization generation model from the multiple candidate stylization generation models. 9. The method according to claim 8, wherein obtaining the target stylization generation model from the plurality of candidate stylization generation models comprises: inputting test images to the plurality of candidate stylization generation models respectively to obtain a plurality of predicted images output by the plurality of candidate stylization generation models; performing image quality evaluation on the plurality of predicted images respectively using an aesthetic scoring algorithm; outputting one or more candidate images with the best image quality among the plurality of predicted images; and, in response to receiving a selection instruction for any candidate image among the one or more candidate images, using the candidate stylization generation model used to generate the candidate image indicated by the selection instruction as the target stylization generation model.10. The method according to any one of claims 7 to 9, characterized in that the initial stylization generation model includes a generative model and a stylization model, the stylization model being used to stylize the output of the generative model to change the style of the output of the generative model; training the initial stylization generation model using the training sample set includes: performing a training process on the stylization model using the training sample set, wherein the training process includes: inputting the sample image to the generative model and the stylization model respectively to obtain a stylized image output by the initial stylization generation model, the stylized image being generated based on a base image and style data, the base image being output by the generative model according to the input sample image, and the style data being output by the stylization model according to the input sample image; if the probability that the image style of the stylized image is the target style is less than a probability threshold, adjusting the model parameters of the stylization model using backpropagation based on the error loss between the stylized image and the sample image, and using the stylization model after model parameter adjustment as a new stylization model, and performing the training process again; If the probability that the image style of the stylized image is the target style is greater than or equal to the probability threshold, the training process ends. 11. The method according to claim 10, wherein the generative model is a diffusion model, an autoregressive model, or a generative adversarial network. 12. The method according to any one of claims 1 to 6, wherein the method is applied to an image processing device, the image processing device has preset stylized generative models corresponding to multiple styles, the multiple styles including the target style; the step of obtaining the target stylized generative model includes: outputting multiple style options corresponding to the multiple styles; in response to receiving a selection instruction for the style option corresponding to the target style, using the stylized generative model corresponding to the target style as the target stylized generative model. 13. The method according to any one of claims 1 to 12, wherein the method further includes: adjusting the size of the redrawing element so that the size of the redrawing element is consistent with the size of the element to be redrawn; the step of generating a target image based on the redrawing element and the original image includes: replacing the element to be redrawn in the original image with the redrawing element to obtain the target image. 14. An image processing device, characterized in that the image processing device comprises: a first acquisition module, configured to acquire elements to be redrawn in an original image, the elements to be redrawn including foreground imaging and / or background imaging; and a second acquisition module, configured to acquire a target stylization generation model, the target stylization generation model being trained based on sample images labeled with a target style.A redrawing module is used to guide the target stylization generation model to redraw the element to be redrawn using guiding information to obtain a redrawn element of the target style. The guiding information includes control information of the element to be redrawn, and the control information includes the imaging features of the element to be redrawn in the original image. A generation module is used to generate a target image based on the redrawn element and the original image. 15. The image processing apparatus according to claim 14, wherein the control information includes one or more of the line drawing, depth, or normal of the element to be redrawn in the original image. 16. The image processing apparatus according to claim 14 or 15, wherein the image processing apparatus further includes a feature extraction module; the feature extraction module is used to extract the imaging features of the element to be redrawn from the original image to obtain the control information. 17. The image processing apparatus according to any one of claims 14 to 16, wherein the imaging features of the element to be redrawn in the original image include one or more of the space, position, color, outline, or texture of the element to be redrawn in the original image. 18. The image processing apparatus according to any one of claims 14 to 17, wherein the guiding information further includes image content description information corresponding to the original image, the image content description information being used to describe the semantics expressed by the original image. 19. The image processing apparatus according to claim 18, wherein the original image comes from a video, and the image processing apparatus further includes a scene processing module and a content recognition module; the scene processing module is used to perform scene processing on the video to obtain multiple video segments; the content recognition module is used to perform video content recognition on each of the multiple video segments to obtain video content description information corresponding to the video segment, and to use the video content description information as image content description information corresponding to each frame image in the video segment. 20. The image processing apparatus according to any one of claims 14 to 19, wherein the second acquisition module is used to: train an initial stylization generation model using a training sample set to obtain the target stylization generation model, the training sample set including one or more sample images labeled with the target style. 21. The image processing device according to any one of claims 14 to 19, characterized in that the image processing device is pre-configured with a plurality of initial stylization generation models, and the second acquisition module is configured to: train the plurality of initial stylization generation models respectively using a training sample set to obtain a plurality of candidate stylization generation models, wherein the training sample set includes one or more sample images labeled with the target style;22. The image processing apparatus according to claim 21, wherein the second acquisition module is configured to: input test images to the plurality of candidate stylization generation models respectively to obtain a plurality of predicted images output by the plurality of candidate stylization generation models; evaluate the image quality of the plurality of predicted images respectively using an aesthetic scoring algorithm; output one or more candidate images with the best image quality among the plurality of predicted images; and, in response to receiving a selection instruction for any candidate image among the one or more candidate images, use the candidate stylization generation model used to generate the candidate image indicated by the selection instruction as the target stylization generation model. 23. The image processing apparatus according to any one of claims 20 to 22, characterized in that the initial stylization generation model includes a generative model and a stylization model, the stylization model being used to stylize the output result of the generative model to change the style of the output result of the generative model; the second acquisition module is used to: perform a training process on the stylization model using the training sample set, wherein the training process includes: inputting the sample image to the generative model and the stylization model respectively to obtain a stylized image output by the initial stylization generation model, the stylized image being generated based on a base image and style data, the base image being output by the generative model according to the input sample image, and the style data being output by the stylization model according to the input sample image; If the probability that the image style of the stylized image is the target style is less than a probability threshold, the stylized model is adjusted by backpropagation based on the error loss between the stylized image and the sample image, and the stylized model with adjusted model parameters is used as a new stylized model, and the training process is executed again; if the probability that the image style of the stylized image is the target style is greater than or equal to the probability threshold, the training process ends. 24. The image processing device according to claim 23, wherein the generative model is a diffusion model, an autoregressive model, or a generative adversarial network. 25. The image processing device according to any one of claims 14 to 19, wherein the image processing device has preset stylized generative models corresponding to multiple styles, the multiple styles including the target style; the second acquisition module is used to: output multiple style options corresponding to the multiple styles; and, in response to receiving a selection instruction for the style option corresponding to the target style, use the stylized generative model corresponding to the target style as the target stylized generative model.26. The image processing apparatus according to any one of claims 14 to 25, characterized in that the image processing apparatus further comprises an adjustment module; the adjustment module is configured to adjust the size of the redrawing element so that the size of the redrawing element is consistent with the size of the element to be redrawn; the generation module is configured to replace the element to be redrawn in the original image with the redrawing element to obtain the target image. 27. A computing device cluster, characterized in that it comprises at least one computing device, each computing device comprising a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the computing device cluster to perform the method according to any one of claims 1 to 13. 28. A computer program product comprising instructions, characterized in that, when the instructions are executed by the computing device cluster, the computing device cluster causes the computing device cluster to perform the method according to any one of claims 1 to 13. 29. A computer-readable storage medium, characterized in that it comprises computer program instructions, which, when executed by the computing device cluster, cause the computing device cluster to perform the method according to any one of claims 1 to 13. Claims 4 / 4 Page 5 CN 121170044 A Image Processing Method and Apparatus

[0001] This application claims priority to Chinese Patent Application No. 202410800094.4, filed on June 19, 2024, entitled "A Custom Style Generative Video Styling Method and Apparatus", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of computer vision technology, and particularly to an image processing method and apparatus. Background Art

[0003] Artificial intelligence (AI) is the science of using machines to extend human intelligence. AI stylization is a recently emerging artificial intelligence generative content (AIGC) technology. AI stylization refers to using AI technology to convert images or videos into images or videos of another style, that is, using AI technology to stylize images or videos. Among them, stylizing video means stylizing each frame of the video separately. Currently, when using AI stylization technology to stylize images, the generated stylized images often suffer from poor quality.

[0004] This application provides an image processing method and apparatus.

[0005] In a first aspect, an image processing method is provided. The method includes: acquiring elements to be redrawn in an original image, the elements to be redrawn including foreground imaging and / or background imaging; acquiring a target stylization generation model; and generating a target stylization model.The model is trained based on sample images labeled with the target style. Guiding information is used to guide the target stylization generation model to redraw the elements to be redrawn, resulting in redrawn elements in the target style. This guiding information includes control information for the elements to be redrawn, which includes the imaging features of the elements to be redrawn in the original image. A target image is generated based on the redrawn elements and the original image.

[0006] This application uses guiding information to guide the target stylization generation model to redraw the elements to be redrawn in the original image. Since the guiding information includes control information describing the imaging features of the elements to be redrawn in the original image, the target stylization generation model maintains consistent imaging features before and after redrawing when converting the elements to be redrawn into redrawn elements of the target style. Therefore, it avoids distortion of the redrawn elements in the image and changes in image content, thereby improving the quality and stability of the generated stylized image. Furthermore, this application can perform stylization processing on foreground and / or background imaging in the image, meaning it can choose to redraw part or all of the content in the image, offering high flexibility in stylization processing.

[0007] Optionally, the control information includes, but is not limited to, one or more of the line drawing, depth, or normal of the element to be redrawn in the original image.

[0008] Optionally, the above method further includes: extracting the imaging features of the element to be redrawn from the original image to obtain the control information.

[0009] Optionally, the imaging features of the element to be redrawn in the original image include, but are not limited to, one or more of the space, position, color, or texture of the element to be redrawn in the original image. Specification 1 / 16 pages 6 CN 121170044 A

[0010] Optionally, the guidance information also includes image content description information corresponding to the original image, which is used to describe the semantics expressed by the original image.

[0011] This application guides the target stylization generation model to redraw the element to be redrawn in the original image by using image content description information as guidance information, so that when the target stylization generation model converts the element to be redrawn into a redrawn element of the target style, it maintains the consistency of the image content before and after redrawing, thereby enabling the redrawn image to maintain the original picture characteristics and further improving the quality and stability of the generated stylized image.

[0012] Optionally, the original image comes from a video, and the above method further includes: performing scene segmentation on the video to obtain multiple video segments. For each video segment among the multiple video segments, video content recognition is performed on the video segment to obtain video content description information corresponding to the video segment, and the video content description information is used as the image content description information corresponding to each frame image in the video segment.

[0013] This application, by performing scene segmentation on the video and video content recognition, enables each frame image in the same video segment to...The corresponding image content description information is the same. Therefore, after stylizing each frame in the same video segment after the split-scene, the image content of each frame in the same video segment can be kept consistent, thus solving the problem of poor inter-frame continuity and ensuring video smoothness and consistency of style before and after the video.

[0014] Optionally, one implementation of obtaining the target stylization generation model includes: training an initial stylization generation model using a training sample set to obtain the target stylization generation model. The training sample set includes one or more sample images labeled with the target style.

[0015] Optionally, the above method is applied to an image processing device, which has multiple initial stylization generation models pre-set. Another implementation of obtaining the target stylization generation model includes: training multiple initial stylization generation models using a training sample set to obtain multiple candidate stylization generation models. The training sample set includes one or more sample images labeled with the target style; obtaining the target stylization generation model from the multiple candidate stylization generation models.

[0016] Optionally, the implementation method of obtaining the target stylization generation model from multiple candidate stylization generation models includes: inputting test images into multiple candidate stylization generation models respectively to obtain multiple predicted images output by multiple candidate stylization generation models. Using an aesthetic scoring algorithm to evaluate the image quality of the multiple predicted images respectively. Outputting one or more candidate images with the best image quality among the multiple predicted images. In response to receiving a selection instruction for any candidate image among the one or more candidate images, using the candidate stylization generation model used to generate the candidate image indicated by the selection instruction as the target stylization generation model.

[0017] In this application, the image processing device can use an aesthetic scoring algorithm to evaluate the quality of images generated by different stylization generation models, and combined with user preferences, provide a high-quality stylization generation model for stylization processing of images.

[0018] Optionally, the initial stylization generation model includes a generative model and a stylization model. The stylization model is used to stylize the output result of the generative model to change the style of the output result of the generative model. The implementation method for training the initial stylization generation model using a training sample set includes: performing a training process on the stylization model using the training sample set. The training process includes: inputting sample images into both the generative model and the stylization model to obtain the stylized image output by the initial stylization generation model. The stylized image is generated based on the base image and style data. The base image is output by the generative model based on the input sample images, and the style data is output by the stylization model based on the input sample images. If the probability of determining that the stylized image's style is the target style is less than a probability threshold, the model parameters of the stylization model are adjusted using backpropagation based on the error loss between the stylized image and the sample images. This is explained in detail.Page 2 / 16, CN 121170044 A. The stylized model with adjusted model parameters is used as a new stylized model, and the training process is executed again. If the probability that the image style of the stylized image is the target style is greater than or equal to the probability threshold, the training process ends.

[0019] In this implementation, only the model parameters of the stylized model need to be adjusted to change the style of the output image of the stylized generation model, without adjusting the model parameters of the large generative model. The computational resources required during model training are less, and the model training efficiency can be improved.

[0020] Optionally, the generative model is a diffusion model, an autoregressive model, or a generative adversarial network.

[0021] Optionally, the above method is applied to an image processing device, which has multiple stylized generation models corresponding to various styles, including the target style. Another implementation of obtaining the target stylized generation model includes: outputting multiple style options corresponding to various styles. In response to receiving a selection instruction for the style option corresponding to the target style, the stylized generation model corresponding to the target style is used as the target stylized generation model.

[0022] In this application, the user can select an existing corresponding preset style according to stylization requirements.

[0023] Optionally, the above method further includes: adjusting the size of the redrawing element so that the size of the redrawing element is consistent with the size of the element to be redrawn. Accordingly, one implementation of generating a target image based on the redrawing element and the original image includes: replacing the element to be redrawn in the original image with the redrawing element to obtain the target image.

[0024] Since some generative models, such as diffusion models, are very sensitive to the proportion of the redrawing object in the image, in this application, the image processing device can first scale the element to be redrawn to meet the requirements of the generative model, then independently redraw the element to be redrawn through a stylized generative model containing the generative model, and scale the obtained redrawing element to the size in the original image, and finally replace it with the overlay, which can enhance the stability of the redrawing element.

[0025] In a second aspect, an image processing device is provided. The image processing device includes multiple functional modules, which interact with each other to implement the methods of the first aspect and its various embodiments. The plurality of functional modules can be implemented based on software, hardware, or a combination of software and hardware, and the plurality of functional modules can be arbitrarily combined or divided based on specific implementations.

[0026] In a third aspect, a computing device cluster is provided, including at least one computing device, each computing device including a processor and a memory. The processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the methods described in the first aspect and its various embodiments.

[0027] Fourthly, a computer program product comprising instructions is provided, which, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the methods described in the first aspect and its embodiments.

[0028] Fifthly, a computer-readable storage medium is provided, comprising computer program instructions, which, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the methods described in the first aspect and its embodiments.

[0029] Sixthly, a chip is provided, comprising programmable logic circuitry and / or program instructions, which, when executed, implement the methods described in the first aspect and its embodiments.

[0030] Figure 1 is a flowchart illustrating an image processing method provided in an embodiment of this application;

[0031] Figure 2 is a diagram illustrating person detection provided in an embodiment of this application;

[0032] Figure 3 is a diagram illustrating portrait segmentation provided in an embodiment of this application;

[0033] Figure 4 is a structural diagram illustrating a stylized generation model provided in an embodiment of this application;

[0034] Figure 5 is a diagram illustrating control information provided in an embodiment of this application; Specification 3 / 16 pages 8 CN 121170044 A

[0035] Figure 6 is a diagram illustrating image redrawing based on line drawing signals provided in an embodiment of this application;

[0036] Figure 7 is a flowchart illustrating the implementation of stylized processing of video provided in an embodiment of this application;

[0037] Figure 8 is a structural diagram illustrating an image processing device provided in an embodiment of this application;

[0038] Figure 9 is a structural diagram illustrating a computing device provided in an embodiment of this application;

[0039] Figure 10 is a structural diagram illustrating a computing device cluster provided in an embodiment of this application;

[0040] Figure 11 is a schematic diagram of another computing device cluster structure provided in an embodiment of this application. Detailed Description

[0041] To make the purpose, technical solution and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0042] AIGC refers to media content generated by artificial intelligence algorithms according to user instructions, such as text, voice, images or videos. AI stylization is mainly divided into image stylization and video stylization. Image stylization refers to stylizing existing images. Video stylization refers to stylizing existing videos. Among them, stylizing a video means stylizing each frame of the video separately. Therefore, image stylization is the foundation and key to realizing video stylization.

[0043] Currently, when using AI stylization technology to stylize images, it is impossible to stylize the image stably while maintaining the original image characteristics. For example, after stylizing an image, the image may show...Deformation (such as the easy deformation of foreground images like people in an image), and changes in image content (such as the generation of additional objects) can also occur. This results in poor quality of the generated stylized image.

[0044] Based on this, this application provides a technical solution that uses a stylized generation model trained on sample images of a specified style to convert the original image into an image of that specified style. This allows users to simply provide an image and convert it into the specified style, achieving automatic and efficient stylized processing of the image. In addition, by guiding the stylized generation model to redraw the elements in the image according to the imaging characteristics of the elements in the original image, the redrawn elements are made consistent with the imaging characteristics of the elements before redrawing. This ensures that the redrawn image retains the original image characteristics and avoids image distortion and changes in image content, thereby improving the quality and stability of the generated stylized image. The technical solution provided in this application is applied to an image processing device, which can be any computing device with computing power. The specific implementation of this technical solution is as follows: First, the elements to be redrawn and the target stylized generation model in the original image are obtained. The elements to be redrawn include foreground imaging and / or background imaging. The target stylization generation model is trained on sample images labeled with the target style. Then, guiding information is used to guide the target stylization generation model to redraw the element to be redrawn, resulting in a redrawn element in the target style. This guiding information includes control information for the element to be redrawn, which includes the imaging features of the element in the original image. Finally, a target image is generated based on the redrawn element and the original image. This application uses guiding information to guide the target stylization generation model to redraw the element to be redrawn in the original image. Since the guiding information includes control information describing the imaging features of the element to be redrawn in the original image, the target stylization generation model maintains consistent imaging features before and after redrawing the element when converting it into a redrawn element in the target style. This avoids distortion of the redrawn element and changes in image content, thus improving the quality and stability of the generated stylized image. Furthermore, this application can perform stylization processing on foreground and / or background imaging in the image, meaning it can choose to redraw part or all of the content in the image, offering high flexibility in stylization processing.

[0045] Optionally, in some embodiments, the guidance information further includes image content description information corresponding to the original image. This image content description information, as described on page 4 / 16 of this specification (CN 121170044 A), is used to describe the semantics expressed by the original image. When the element to be redrawn includes foreground imaging, the image content description information can be used to describe the semantics expressed by the foreground imaging. For example, if the foreground imaging is a person, the image content description information includes, but is not limited to, the person's action, gender, hairstyle, and height. In the element to be redrawn...In the case of background imaging, image content description information can be used to describe the semantics expressed by the background imaging, including but not limited to the objects and background colors contained in the background imaging. This application guides the target stylization generation model to redraw the elements to be redrawn in the original image by using image content description information as guiding information. This allows the target stylization generation model to maintain the consistency of image content before and after redrawing when converting the elements to be redrawn into redrawing elements of the target style. As a result, the redrawn image can maintain the original picture characteristics, further improving the quality and stability of the generated stylized image.

[0046] Optionally, in the video stylization scenario, the technical solution provided in this application can be used to perform image stylization processing on each frame of the video separately. When the original image is from a video, the image processing device can perform segmentation processing on the video to obtain multiple video segments. Then, for each of the multiple video segments, video content recognition is performed on the video segment to obtain the video content description information corresponding to the video segment, and the video content description information is used as the image content description information corresponding to each frame of the video segment. This application performs scene segmentation and video content recognition on the video to ensure that the image content description information corresponding to each frame in the same video segment is the same. In this way, after stylizing each frame in the same video segment after scene segmentation, the image content of each frame in the same video segment can be kept consistent, thereby solving the problem of poor inter-frame continuity and ensuring the smoothness of the video and the consistency of the style before and after the video.

[0047] The method flow of the embodiment of this application is illustrated below.

[0048] For example, FIG1 is a schematic flowchart of an image processing method 100 provided by an embodiment of this application. As shown in FIG1, the method 100 includes but is not limited to the following steps 101 to 104.

[0049] Step 101: The image processing device acquires the elements to be redrawn in the original image, which includes foreground imaging and / or background imaging.

[0050] Optionally, the image processing device first acquires the original image, which can be a single image or a frame in the video. For example, an image processing device can acquire a video uploaded by a user and execute the image processing method provided in this application embodiment on each frame of the video to achieve video stylization. The video can be a real-shot video, or it can be an animation, a movie, or other forms and styles of video.

[0051] Optionally, the original image includes only background imaging, or the original image includes both background imaging and foreground imaging. That is, this application embodiment does not limit whether the image to be stylized contains foreground imaging; pure background images (such as pure landscape images) can also be stylized using the image processing method provided in this application embodiment.

[0052] Optionally, when the original image is a pure background image, the elements to be redrawn in the original image only include background imaging. When the original image includes background imaging and foreground imaging, the elements to be redrawn in the original image may only include foreground imaging, or only include background imaging, or may include both foreground and background imaging.

[0053] This application embodiment uses an original image including background imaging and foreground imaging as an example for explanation. After acquiring the original image, the image processing device can separate the foreground and background of the original image to obtain separate foreground and background imaging. Then, the foreground imaging is redrawn, or the background imaging is redrawn, or the foreground and background imaging are redrawn separately to improve the redrawing quality. For example, the image processing device can use an image matting algorithm to remove the background in the original image to obtain foreground imaging, or remove the foreground in the original image to obtain background imaging. This application embodiment does not limit the specific type of image matting algorithm used. Instruction manual, page 5 / 16, 10 CN 121170044 A

[0054] For example, the elements to be redrawn in the original image include human figures (foreground figures). The image processing device can first use a human detection algorithm to detect the human region in the original image, then use a portrait segmentation algorithm to extract the binary image of the human figure, and then use the binary image of the human figure to mask the human region to obtain the human figure. For example, Figure 2 is a schematic diagram of human detection provided by an embodiment of this application. As shown in Figure 2, the area within the rectangular detection box is the detected human region. As another example, Figure 3 is a schematic diagram of portrait segmentation provided by an embodiment of this application. As shown in Figure 3, the upper left image is the human region detected in Figure 2, and the upper right image is the binary image of the human figure. The white area of ​​the binary image is the human figure, and the black area is the background image. By using the binary image of the human figure (upper right image) to mask the human region (upper left image), the human figure (lower image) can be obtained. Optionally, the person detection algorithm used in this application embodiment can be any object detection algorithm implemented based on deep learning algorithms, including but not limited to the YoLo (you only look once) algorithm or MediaPipe (an open-source multimedia machine learning model application framework). The portrait segmentation algorithm used in this application embodiment includes but is not limited to the portrait segmentation algorithm based on fully convolutional networks (FCN) (abbreviated as: PortraitFCN algorithm), principal component analysis (PCA) algorithm, or MediaPipe. This application embodiment does not limit the person detection algorithm and portrait segmentation algorithm used.

[0055] Step 102: The image processing device obtains the target stylization generation model.

[0056] The target stylization generation model is trained based on sample images labeled with the target style. Optionally, the target stylization generation model is implemented based on a generative model. A generative model is a model that can generate observation data. The output of a generative model includes, but is not limited to, text, images, or videos.

[0057] Optionally, the styles in this application embodiment can be divided according to different dimensions. For example, different painting styles such as animation, oil painting, ink painting, or simple line drawing can be regarded as different styles. For another example, different photography styles such as documentary style, natural style, black and white style, or portrait style can be regarded as different styles. For another example, different color systems can be regarded as different styles. This application embodiment does not limit the definition and division of styles.

[0058] Optionally, in the above step 102, there are multiple ways for the image processing device to obtain the target stylization generation model. This application embodiment mainly illustrates the following two implementation methods.

[0059] In the first implementation method of the above step 102, the image processing device obtains the target stylization generation model by training the target stylization generation model using a training sample set. The training sample set includes one or more sample images labeled with the target style. It is worth noting that in most cases, the training sample set includes multiple sample images labeled with the target style. Using a large number of samples for model training can improve the model accuracy.

[0060] In conjunction with the first case of the first implementation method described above, the image processing device has a pre-set initial stylization generation model. The image processing device uses the training sample set to train the initial stylization generation model to obtain the target stylization generation model. Optionally, the initial stylization generation model includes a generative model, that is, in this embodiment, the generative model can be used as a basis to train a stylization generation model that outputs a specific style image.

[0061] Optionally, the initial stylization generation model includes a generative model and a stylization model. The stylization model is used to stylize the output result of the generative model to change the style of the output result of the generative model. The specific implementation method of the image processing device using the training sample set to train the initial stylization generation model includes: the image processing device using the training sample set to perform a training process on the stylization model. The training process includes: inputting sample images into the generative model and the stylization model respectively to obtain a stylized image output by the initial stylization generative model. This stylized image is generated based on the base image and style data. The base image is output by the generative model based on the input sample images, and the style data is output by the stylization model based on the input sample images. If the probability of determining the style of the stylized image as the target style is less than a probability threshold, a reverse engineering method is used based on the error loss between the stylized image and the sample images. (See page 6 / 16, CN)121170044 A The stylization model is adjusted by propagation, and the stylization model after the model parameter adjustment is used as the new stylization model, and the training process is executed again. If the probability that the image style of the stylized image is the target style is greater than or equal to the probability threshold, the training process ends.

[0062] Optionally, after obtaining the stylized image output by the initial stylization generation model, an aesthetic scoring algorithm can be used to evaluate the image quality of the stylized image, and the probability that the image style of the stylized image is the target style is determined according to the image quality evaluation result. The aesthetic scoring algorithm includes, but is not limited to, algorithms that evaluate image quality such as the contrastive language image pre-training (CLIP) algorithm. For example, image quality can be represented by CLIP scoring. The higher the CLIP score, the better the image quality. The CLIP algorithm scores by learning to match images and text. The CLIP scoring mechanism involves the similarity calculation of images and text, and gives a score by calculating the similarity between the input image and text. For example, the CLIP algorithm is used to score the stylized image as the target style. If the CLIP score is lower than a preset threshold, the probability that the stylized image is the target style is less than the probability threshold. If the CLIP score is higher than or equal to the preset threshold, the probability that the stylized image is the target style is greater than or equal to the probability threshold.

[0063] For example, Figure 4 is a schematic diagram of the structure of a stylized generative model provided in an embodiment of this application. As shown in Figure 4, the left side is the generative model and the right side is the stylized model. During training, the model parameters of the generative model are fixed, and only the model parameters of the stylized model are changed, that is, only the stylized model is trained. During prediction, the input data is simultaneously input into the generative model and the stylized model, and during output, the base image output by the generative model and the style data output by the stylized model are matrix-added to inject the style data into the base image to obtain the output data, thereby realizing the stylization processing of the image.

[0064] Among them, the generative model is a large model with the characteristics of complex model structure and large model parameter scale. Generative models, for example, include but are not limited to diffusion models, autoregressive models, or generative adversarial networks (GANs). Stylized models are fine-tuned models used in conjunction with larger models, characterized by simple model structure and small parameter size. Stylized models, for example, include but are not limited to low-rank adapters (LoRA) or textural inversion models. In this implementation, only...Adjusting the model parameters of the stylization model can change the style of the output image of the stylization generation model without adjusting the model parameters of the large generative model. This requires less computational resources during model training and can improve model training efficiency. Of course, in this embodiment, sample images can also be used to directly fine-tune the generative model, that is, directly adjust a small number of model parameters of the generative model to obtain the target stylization generation model. Fine-tuning training refers to further training on a specific task based on an existing model to adapt to the specific data and requirements of the task. This embodiment does not limit the structure and training method of the stylization generation model.

[0065] Optionally, the sample images in the training sample set can be uploaded by the user to the image processing device. For example, the user can upload a group of sample images with similar styles, such as anime or landscape paintings with the same style, statues of the same shape, or natural landscape photos with the same landform. The sample images uploaded by the user can be labeled with style tags. Alternatively, the sample images uploaded by the user can also be unlabeled with style tags. After the image processing device obtains the sample images, it performs style recognition and style tag labeling on the sample images through a large language model.

[0066] In this embodiment, the user only needs to upload a set of sample images with similar styles to generate a corresponding stylization generation model. Then, the user only needs to provide the original image whose style needs to be changed, and the stylization generation model can be used to redraw the original image to obtain a target image with a style similar to the sample image. That is, the user only needs to provide the desired style image to achieve efficient and high-quality style conversion of existing images, making it convenient for the user to perform custom stylization processing on images. Specification 7 / 16 pages 12 CN 121170044 A

[0067] In conjunction with the second case of the first implementation method described above, the image processing device has multiple initial stylization generation models pre-set. The image processing device uses a training sample set to train the multiple initial stylization generation models respectively to obtain multiple candidate stylization generation models, and then obtains the target stylization generation model from the multiple candidate stylization generation models. Here, the training method of the image processing device for each initial stylization generation model can refer to the training method of the image processing device for the initial stylization generation model in the first case described above, which will not be repeated here in this embodiment.

[0068] Optionally, the image processing device has multiple pre-set initial stylization generation models, or multiple pre-set generative models. Different generative models can be used as the basis to train stylization generation models that output images of a specific style. For example, each initial stylization generation model includes a generative model and a stylization model. Training multiple initial stylization generation models using a training sample set is equivalent to using the training sample set to divide...The stylization model is trained with multiple different generative models to select a suitable generative model as the generative model in the target stylization generative model.

[0069] Optionally, the image processing device obtains the implementation method of the target stylization generative model from multiple candidate stylization generative models, including the following steps A1 to A4.

[0070] In step A1, the image processing device inputs test images to multiple candidate stylization generative models respectively to obtain multiple predicted images output by the multiple candidate stylization generative models.

[0071] Optionally, the test images are labeled with style tags.

[0072] In step A2, the image processing device uses an aesthetic scoring algorithm to evaluate the image quality of the multiple predicted images respectively.

[0073] Optionally, the aesthetic scoring algorithm includes, but is not limited to, the CLIP algorithm and other algorithms for evaluating image quality. When the test images are labeled with style tags, the image quality can be represented by a CLIP score. The higher the CLIP score, the better the image quality.

[0074] In step A3, the image processing device outputs one or more candidate images with the best image quality among the multiple predicted images.

[0075] For example, the image processing device may display one or more candidate images, or the image processing device may send one or more candidate images to a connected display device for user selection.

[0076] In step A4, in response to receiving a selection instruction for any candidate image among the one or more candidate images, the image processing device uses the candidate stylization generation model used to generate the candidate image indicated by the selection instruction as the target stylization generation model.

[0077] In this embodiment of the application, the image processing device can use an aesthetic scoring algorithm to evaluate the quality of images generated by different stylization generation models, and combine user preferences to provide a high-quality stylization generation model for stylization processing of images. Of course, the image processing device can also directly use the candidate stylization generation model used to generate the predicted image with the best image quality among the multiple predicted images as the target stylization generation model, that is, automatically match the stylization generation model that can provide the best quality style image through aesthetic scoring.

[0078] In the second implementation of step 102 above, the image processing device has pre-set stylization generation models corresponding to multiple styles, including a target style. The image processing device obtains the target stylization generation model by outputting multiple style options corresponding to the multiple styles. In response to receiving a selection instruction for the style option corresponding to the target style, the image processing device uses the stylization generation model corresponding to the target style as the target stylization generation model. Optionally, the pre-set stylization generation model in the image processing device can be trained by the image processing device, or it can be trained by other devices and then sent to the image processing device. (Page 13, 8 / 16 of the specification)CN 121170044 A

[0079] In this embodiment, the user can select an existing preset style according to stylization requirements.

[0080] Step 103: The image processing device uses guidance information to guide the target stylization generation model to redraw the element to be redrawn, so as to obtain the redrawn element of the target style.

[0081] Optionally, the guidance information can be extracted from the original image by the image processing device, or the guidance information can be uploaded to the image processing device by the user, or the image processing device can provide guidance information for the user to select. This embodiment does not limit the way the image processing device obtains the guidance information.

[0082] Wherein, the guidance information includes control information of the element to be redrawn, which includes, but is not limited to, the imaging features of the element to be redrawn in the original image. The imaging features of the element to be redrawn in the original image included in the control information include, but are not limited to, one or more of the space, position, color, outline or texture of the element to be redrawn in the original image. When the original image comes from video, the imaging features of the element to be redrawn in the original image included in the control information may also include optical flow. It is worth noting that if it is necessary to change a certain imaging feature of the element to be redrawn, the control information does not include that imaging feature of the element to be redrawn in the original image. For example, if it is necessary to change the color of the element to be redrawn, the control information does not include the color of the element to be redrawn in the original image. Conversely, the control information can carry the target color that the element to be redrawn needs to become, so that after the image processing device redraws the element to be redrawn, it obtains a redrawn element with the target color. The target color can be input or selected by the user.

[0083] Optionally, the image processing device extracts the imaging features of the element to be redrawn from the original image to obtain the control information of the element to be redrawn. Optionally, the control information includes one or more of the line drawing, depth, or normal of the element to be redrawn in the original image. Among them, the line drawing can reflect the outline of the element to be redrawn. The image processing device can use algorithms such as sk_model to estimate the line drawing signal of the element to be redrawn in the original image. Depth and normal can reflect the texture of the element to be redrawn. The image processing device can use algorithms such as Midas or Zoe to estimate the depth signal of the element to be redrawn in the original image. Image processing devices can use algorithms such as point cloud library (PCL) to estimate the normal signals of the elements to be redrawn in the original image. For example, Figure 5 is a schematic diagram of control information provided in an embodiment of this application. In Figure 5, the left image is a line drawing signal image, the middle image is a depth signal image, and the right image is a normal signal image. Using control information to guide the target stylization generation model to redraw the elements to be redrawn can reduce the randomness of the redrawing process, ensuring that the redrawn elements generated by the target stylization generation model retain the imaging characteristics before redrawing. For example, Figure 6 is a schematic diagram of a control information based on...A schematic diagram of image redrawing using line drawing signals. In Figure 6, the upper left image is the original image, the upper right image is the line drawing signal of the turtle in the original image, and the lower left and lower right images are the redrawn images generated based on this line drawing signal. As shown in Figure 6, since the line drawing signal describes the turtle's outline, when redrawing the original image based on this line drawing signal, the outline of the turtle in the generated redrawn image can be kept consistent with the outline of the turtle in the original image, thus avoiding turtle deformation.

[0084] Optionally, the control algorithm guiding the target stylization generation model to redraw the elements to be redrawn includes, but is not limited to, redrawing range shape control algorithms such as ControlNet. Correspondingly, the control information includes, but is not limited to, all control signals involved in ControlNet, and the method of obtaining the control information can refer to the method of obtaining control signals in ControlNet.

[0085] In this embodiment of the application, by using control information describing the imaging features of the element to be redrawn in the original image as guiding information, the target stylization generation model is guided to redraw the element to be redrawn in the original image. This ensures that when the target stylization generation model converts the element to be redrawn into a redrawn element of the target style, the imaging features of the element before and after redrawing are consistent. Therefore, it can avoid the deformation of the redrawn elements in the image and the increase or decrease of image content, thereby improving the quality and stability of the generated stylized image.

[0086] Optionally, the guiding information also includes image content description information corresponding to the original image. This image content description information is used to describe the semantics expressed by the original image. When the element to be redrawn includes foreground imaging, the image content description information can be used to describe the semantics expressed by the foreground imaging. For example, if the foreground imaging is a person imaging, the image content description information includes, but is not limited to, the person's action, gender, hairstyle, and height. When the element to be redrawn includes background imaging, the image content description information can be used to describe the semantics expressed by the background imaging, including but not limited to the objects and background colors contained in the background imaging.

[0087] In this embodiment of the application, by using the image content description information as guiding information to guide the target stylization generation model to redraw the element to be redrawn in the original image, the target stylization generation model maintains the consistency of the image content before and after redrawing when converting the element to be redrawn into a target style redrawing element, thereby enabling the redrawn image to maintain the original picture characteristics, further improving the quality and stability of the generated stylized image.

[0088] Optionally, when the above-mentioned original image comes from video, the image processing device can perform segmentation processing on the video to obtain multiple video segments, and then for each of the multiple video segments, process the video segment.Video content recognition is performed to obtain video content description information corresponding to the video segment, and this video content description information is used as the image content description information corresponding to each frame in the video segment.

[0089] Here, video segmentation processing refers to using a deep learning algorithm to segment an entire video, wherein the multiple video segments obtained after segmentation are segments shot from different shots or different angles. Optionally, the video segmentation algorithm used in the embodiments of this application includes, but is not limited to, video frame similarity analysis algorithm, structural similarity index (SSIM) algorithm, and Fréchet inception distance (FID) value approximation. The video content recognition algorithm used in the embodiments of this application includes, but is not limited to, video content extraction algorithms such as VidArts.

[0090] In this embodiment of the application, by performing scene segmentation processing and video content recognition on the video, the image content description information corresponding to each frame image in the same video segment is the same. In this way, after stylizing each frame image in the same video segment after scene segmentation, the screen content of each frame image in the same video segment can be kept consistent, thereby solving the problem of poor inter-frame continuity and ensuring video smoothness and consistency of style before and after the video.

[0091] Step 104: The image processing device generates a target image based on the redrawing element and the original image.

[0092] Optionally, the image processing device can first adjust the size of the redrawing element so that the size of the redrawing element is consistent with the size of the element to be redrawn, and then use the redrawing element to cover and replace the element to be redrawn in the original image to obtain the target image.

[0093] Since some generative models, such as diffusion models, are very sensitive to the proportion of the redrawn object in the image, in this embodiment, the image processing device can first scale the element to be redrawn obtained in step 101 to meet the requirements of the generative model, then independently redraw the element to be redrawn using a stylized generative model containing the generative model, and scale the obtained redrawn element to the size in the original image, and finally perform overlay replacement, which can enhance the stability of the redrawn element.

[0094] Optionally, when the element to be redrawn is a foreground image, after obtaining the redrawn foreground image corresponding to the foreground image, the image processing device can use the redrawn foreground image to overlay and replace the foreground image in the original image to obtain the target image. When the element to be redrawn is a background image, after obtaining the redrawn background image corresponding to the background image, the image processing device can add the foreground image to the corresponding position in the redrawn background image according to the imaging position of the foreground image in the original image to obtain the target image. When the element to be redrawn includes a foreground imageIn the case of image and background imaging, after the image processing device obtains the redrawn foreground imaging corresponding to the foreground imaging and the redrawn background imaging corresponding to the background imaging respectively, it can add the redrawn foreground imaging to the corresponding position in the redrawn background imaging according to the imaging position of the foreground imaging in the original image (see page 10 / 16 of CN 121170044 A) to obtain the target image.

[0095] In the image processing method provided in the embodiments of this application, the target stylization generation model is guided to redraw the elements to be redrawn in the original image by using guidance information. Since the guidance information includes control information for describing the imaging characteristics of the elements to be redrawn in the original image, the target stylization generation model maintains the consistency of the imaging characteristics of the elements before and after redrawing when converting the elements to be redrawn into redrawn elements of the target style. Therefore, it can avoid the deformation of the redrawn elements in the image and the increase or decrease of image content, thereby improving the quality and stability of the generated stylized image. Furthermore, image content description information can be used as guiding information to guide the target stylization generation model to redraw the elements to be redrawn in the original image. This ensures that the target stylization generation model maintains consistency in the image content before and after redrawing when converting the elements to be redrawn into redrawn elements of the target style. As a result, the redrawn image can retain the original image characteristics, further improving the quality and stability of the generated stylized image. In addition, the embodiments of this application can perform stylization processing on foreground imaging and / or background imaging in the image, that is, it can select to redraw part or all of the content in the image, which provides high flexibility in stylization processing.

[0096] Optionally, in a video stylization scenario, after the image processing device completes image stylization processing on the video frames in the original video, it can perform consistency post-processing on the redrawn video frames to smooth the video, improve the stability and quality of the redrawn video, and output the final video. Among them, the consistency post-processing algorithm includes, but is not limited to, Fastblend (an algorithm to enhance video coherence, stability and consistency) or optical flow related algorithms. For example, the implementation process of image processing device performing consistency post-processing on redrawn video frames is as follows: First, the optical flow information between two adjacent video frames in the original video is obtained, and the optical flow information is transferred to the redrawn video frame corresponding to the two adjacent video frames. Then, the blending algorithm is used to mix the information of the original video and the redrawn video, making the result of the redrawn video smoother and reducing flicker.

[0097] For example, Figure 7 is a schematic diagram of the implementation process of stylizing video provided by an embodiment of this application. As shown in Figure 7, the original video is first obtained, and the foreground imaging of the original video is human imaging. Then, the foreground imaging and background imaging in the original video are detected and separated to obtain a human sequence composed of human imaging and a background sequence composed of background imaging.For example, taking the need to redraw the image of a person in the original video as an example, the control information and image content description information corresponding to the image of a person in each frame of the image are extracted, and the size of the image of the person is adjusted to control the proportion of the person. The image of the person is input into the stylization generation model, the proportion of the person is controlled, and the stylization generation model is guided to redraw the image of each frame of the person using the corresponding control information and image content description information, and the redrawn image of the person is output. Then, a redrawn video is generated based on the redrawn image of the person and the background sequence, and the redrawn video is post-processed to obtain the target video. The stylization generation model can be trained based on the stylized image provided by the user, or it can be a model with an existing style selected by the user.

[0098] The order of the steps of the image processing method provided in the embodiments of this application can be adjusted appropriately. For example, the order of the above steps 101 and 102 can be swapped, or they can be executed simultaneously. The steps can also be added or removed according to the situation. Any method that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application.

[0099] The following describes the virtual device of the embodiments of this application.

[0100] For example, FIG8 is a schematic diagram of the structure of an image processing device 800 provided in an embodiment of the present application. As shown in FIG8, the image processing device 800 includes, but is not limited to, a first acquisition module 801, a second acquisition module 802, a redrawing module 803, and a generation module 804. Optionally, referring to FIG8, the image processing device 800 also includes one or more of a feature extraction module 805, a storyboard processing module 806, a content recognition module 807, or an adjustment module 808.

[0101] The first acquisition module 801 is used to acquire the elements to be redrawn in the original image, including the foreground imaging specification 11 / 16 pages 16 CN 121170044 A and / or the background imaging.

[0102] The second acquisition module 802 is used to acquire a target stylization generation model, which is trained based on sample images labeled with the target style.

[0103] The redrawing module 803 is used to guide the target stylization generation model to redraw the elements to be redrawn using guiding information, so as to obtain redrawing elements of the target style. The guiding information includes control information of the elements to be redrawn, and the control information includes the imaging features of the elements to be redrawn in the original image.

[0104] The generation module 804 is used to generate a target image based on the redrawing elements and the original image.

[0105] Optionally, the control information includes one or more of the line drawing, depth, or normal of the elements to be redrawn in the original image.

[0106] Optionally, the feature extraction module 805 is used to extract the imaging features of the elements to be redrawn from the original image to obtain the control information.

[0107] Optionally, the imaging features of the element to be redrawn in the original image include one or more of the space, position, color, outline, or texture of the element to be redrawn in the original image.

[0108] Optionally, the guidance information also includes image content description information corresponding to the original image, which is used to describe the semantics expressed by the original image.

[0109] Optionally, the original image comes from a video. The storyboard processing module 806 is used to perform storyboard processing on the video to obtain multiple video segments. The content recognition module 807 is used to perform video content recognition on each video segment among the multiple video segments to obtain the video content description information corresponding to the video segment, and use the video content description information as the image content description information corresponding to each frame image in the video segment.

[0110] Optionally, the second acquisition module 802 is used to: train the initial stylization generation model using a training sample set to obtain the target stylization generation model, wherein the training sample set includes one or more sample images labeled with the target style.

[0111] Alternatively, the image processing device has multiple initial stylization generation models pre-set, and the second acquisition module 802 is used to: train the multiple initial stylization generation models using a training sample set to obtain multiple candidate stylization generation models, wherein the training sample set includes one or more sample images labeled with the target style; and acquire the target stylization generation model from the multiple candidate stylization generation models.

[0112] Optionally, the second acquisition module 802 is specifically used to: input test images to the multiple candidate stylization generation models to obtain multiple predicted images output by the multiple candidate stylization generation models; use an aesthetic scoring algorithm to evaluate the image quality of the multiple predicted images; output one or more candidate images with the best image quality among the multiple predicted images; and in response to receiving a selection instruction for any candidate image among the one or more candidate images, use the candidate stylization generation model used to generate the candidate image indicated by the selection instruction as the target stylization generation model.

[0113] Optionally, the initial stylization generation model includes a generative model and a stylization model. The stylization model is used to stylize the output of the generative model to change the style of the output of the generative model. The second acquisition module 802 is specifically used to: perform a training process on the stylization model using a training sample set, wherein the training process includes: inputting sample images to the generative model and the stylization model respectively to obtain a stylized image output by the initial stylization generation model. The stylized image is generated based on the base image and style data. The base image is output by the generative model according to the input sample image, and the style data is output by the stylization model according to the input sample image. If the probability that the image style of the stylized image is the target style is less than the probability threshold, an inverse method is used based on the error loss between the stylized image and the sample image.The stylization model is adjusted by propagation, and the stylization model after parameter adjustment is used as a new stylization model. The training process is executed again. If the probability that the image style of the stylized image is the target style is greater than or equal to the probability threshold, the training process ends.

[0114] Optionally, the generative model is a diffusion model, an autoregressive model, or a generative adversarial network.

[0115] Alternatively, the image processing device has multiple stylization generative models corresponding to multiple styles, including the target style. The second acquisition module 802 is used to: output multiple style options corresponding to multiple styles; in response to receiving the selection instruction for the style option corresponding to the target style, the stylization generative model corresponding to the target style is used as the target stylization generative model.

[0116] Optionally, the adjustment module 808 is used to adjust the size of the redrawing element so that the size of the redrawing element is consistent with the size of the element to be redrawn. The generation module 804 is used to replace the element to be redrawn in the original image with the redrawing element to obtain the target image.

[0117] The first acquisition module 801, the second acquisition module 802, the redrawing module 803, the generation module 804, the feature extraction module 805, the storyboard processing module 806, the content recognition module 807, and the adjustment module 808 can all be implemented in software or in hardware. For example, the implementation of the first acquisition module 801 will be described below. Similarly, the implementation of the second acquisition module 802, the redrawing module 803, the generation module 804, the feature extraction module 805, the storyboard processing module 806, the content recognition module 807, and the adjustment module 808 can refer to the implementation of the first acquisition module 801.

[0118] As an example of a software functional unit, the first acquisition module 801 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the computing instance may be one or more. For example, the first acquisition module 801 may include code running on multiple hosts / virtual machines / containers. It should be noted that multiple hosts / virtual machines / containers used to run this code can be distributed in the same region or in different regions. Furthermore, multiple hosts / virtual machines / containers used to run this code can be distributed in the same availability zone (AZ) or in different AZs, each AZ comprising one data center or multiple geographically proximate data centers. Typically, a region can include multiple AZs.

[0119] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Typically, one VPC is set up within one region. Cross-region communication between two VPCs within the same region, and between VPCs in different regions, requires a communication gateway to be set up within each VPC to achieve interconnection between VPCs.

[0120] As an example of a hardware functional unit, the first acquisition module 801 may include at least one computing device, such as a server. Alternatively, the first acquisition module 801 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0121] The multiple computing devices included in the first acquisition module 801 can be distributed in the same region or in different regions. The multiple computing devices included in the first acquisition module 801 can be distributed in the same Availability Zone (AZ) or in different AZs. Similarly, the multiple computing devices included in the first acquisition module 801 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0122] It should be noted that in other embodiments, the first acquisition module 801 can be used to execute any step of the image processing method described on pages 13 / 16 of the specification 18 CN 121170044 A, the second acquisition module 802 can be used to execute any step of the image processing method, the redrawing module 803 can be used to execute any step of the image processing method, the generation module 804 can be used to execute any step of the image processing method, the feature extraction module 805 can be used to execute any step of the image processing method, the storyboard processing module 806 can be used to execute any step of the image processing method, the content recognition module 807 can be used to execute any step of the image processing method, and the adjustment module 808 can be used to execute any step of the image processing method.

[0123] The steps implemented by the first acquisition module 801, the second acquisition module 802, the redrawing module 803, the generation module 804, the feature extraction module 805, the storyboard processing module 806, the content recognition module 807, and the adjustment module 808 can be specified as needed. The first acquisition module 801, the second acquisition module 802, the redrawing module 803, the generation module 804, the feature extraction module 805, the storyboard processing module 806, the content recognition module 807, and the adjustment module 808 respectively implement different steps in the image processing method to achieve all the functions of the image processing device.

[0124] The hardware device of the embodiments of this application is illustrated below.

[0125] For example, FIG9 is a schematic diagram of the structure of a computing device 900 provided in an embodiment of this application. As shown in FIG9, the computing device 900 includes: a bus 902, a processor 904, a memory 906, and a communication interface 908. The processor 904, the memory 906, and the communication interface 908 communicate through the bus 902. The computing device 900 can be a server or a terminal device. It should be understood that the embodiments of this application do not limit the number of processors and memories in the computing device 900.

[0126] The bus 902 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus may be divided into address bus, data bus, control bus, etc. For ease of representation, only one line is used in FIG9, but it does not mean that there is only one bus or one type of bus. The bus 902 may include a path for transmitting information between various components of the computing device 900 (e.g., memory 906, processor 904, communication interface 908).

[0127] The processor 904 may include any one or more of processors such as a central processing unit (CPU), graphics processing unit (GPU), microprocessor (MP), or digital signal processor (DSP).

[0128] The memory 906 may include volatile memory, such as random access memory (RAM). The memory 906 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD).

[0129] The memory 906 stores executable program code, and the processor 904 executes the executable program code to implement the functions of the aforementioned first acquisition module, second acquisition module, redrawing module and generation module respectively, thereby implementing the above-mentioned image processing method, such as the above-mentioned method 100. That is, the memory 906 stores instructions for executing the image processing method.

[0130] Alternatively, the memory 906 stores executable code, and the processor 904 executes the executable code to implement the functions of the aforementioned image processing device 800, thereby implementing the image processing method. That is, the memory 906 stores instructions for executing the image processing method.

[0131] The communication interface 908 uses a transceiver module such as, but not limited to, a network interface card, a transceiver, etc., to realize communication between the computing device 900 and other devices or communication networks.

[0132] This application embodiment also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device may be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device may also be a desktop computer, laptop computer, or smartphone, etc. (Pages 14 / 16, CN 121170044 A).

[0133] For example, FIG10 is a schematic diagram of a computing device cluster provided in an embodiment of the present application. As shown in FIG10, the computing device cluster includes at least one computing device 900. The memory 906 of one or more computing devices 900 in the computing device cluster may store the same instructions for executing image processing methods.

[0134] In some possible implementations, the memory 906 of one or more computing devices 900 in the computing device cluster may also store partial instructions for executing image processing methods. In other words, a combination of one or more computing devices 900 can jointly execute instructions for executing image processing methods.

[0135] It should be noted that the memory 906 of different computing devices 900 in the computing device cluster may store different instructions, which are used to execute partial functions of the image processing device. That is, the instructions stored in the memory 906 of different computing devices 900 can implement the functions of one or more of the aforementioned modules, including but not limited to the first acquisition module, the second acquisition module, the redrawing module, and the generation module.

[0136] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. The network can be a wide area network (WAN) or a local area network (LAN), etc. For example, Figure 11 is a schematic diagram of another computing device cluster structure provided in an embodiment of this application. As shown in Figure 11, two computing devices 900A and 900B communicate via a network.Connection. Specifically, the connection is made with the network through the communication interface in each computing device. In this possible implementation, the memory 906 in computing device 900A stores instructions for executing the functions of the first acquisition module and the second acquisition module. At the same time, the memory 906 in computing device 900B stores instructions for executing the functions of the redrawing module and the generation module.

[0137] The connection method between the computing device clusters shown in FIG11 can be considered that the image processing method provided in the embodiments of this application needs to transmit data and calculate data, so the functions implemented by the redrawing module 803 and the generation module 804 are considered to be executed by computing device 900B.

[0138] It should be understood that the functions of computing device 900A shown in FIG11 can also be completed by multiple computing devices 900. Similarly, the functions of computing device 900B can also be completed by multiple computing devices 900.

[0139] The embodiments of this application also provide another computing device cluster. The connection relationship between the computing devices in this computing device cluster can be similarly referred to the connection method of the computing device clusters shown in FIG10 and FIG11. The difference is that the memory 906 of one or more computing devices 900 in the computing device cluster may store the same instructions for executing the image processing method.

[0140] In some possible implementations, the memory 906 of one or more computing devices 900 in the computing device cluster may also store partial instructions for executing the image processing method. In other words, a combination of one or more computing devices 900 can jointly execute the instructions for executing the image processing method.

[0141] It should be noted that the memory 906 of different computing devices 900 in the computing device cluster may store different instructions for executing partial functions of the image processing device. That is, the instructions stored in the memory 906 of different computing devices 900 can implement the functions of the image processing device.

[0142] This application embodiment also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions that can run on a computing device or be stored in any available medium. When the computer program product is run on at least one computing device, it causes at least one computing device to execute the image processing method.

[0143] This application embodiment also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives). The computer-readable storage medium includes instructions that instruct the computing device...The image processing method is prepared for execution.

[0144] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and not to limit them; although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent substitutions for some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application. Instruction Manual 16 / 16 Page 21 CN 121170044 A Figure 1 Figure 2 Instruction Manual Appendix 1 / 6 Page 22 CN 121170044 A Figure 3 Figure 4 Instruction Manual Appendix 2 / 6 Page 23 CN 121170044 A Figure 5 Figure 6 Instruction Manual Appendix 3 / 6 Page 24 CN 121170044 A Figure 7 Figure 8 Instruction Manual Appendix 4 / 6 Page 25 CN 121170044 A Figure 9 Figure 10 Instruction Manual Appendix 5 / 6 Page 26 CN 121170044 A Figure 11 Instruction Manual Appendix 6 / 6 Page 27 CN 121170044 A Our Ref.: C065 / P1032 / HK IMAGE PROCESSING METHOD AND DEVICE Abstract The present application provides an image processing method and device, belonging to the field of computer vision technology. An element to be redrawn from an original image and a target stylization generation model are acquired; guidance information is used to guide the target stylization generation model to redraw the element to be redrawn, so as to obtain a redrawn element of a target style; and a target image is generated on the basis of the redrawn element and the originalimage, where the target stylization generation model is obtained by means of performing training on the basis of a sample image labeled with the target style, the guidance information includes control information of the element to be redrawn, and the control information includes imaging features, in the original image, of the element to be redrawn. In the present application, guidance information is used to guide a target stylization generation model to redraw an element to be redrawn which is in an original image, so that when the target stylization generation model converts the element to be redrawn into a redrawn element of a target style, imaging features of the element before and after redrawing are maintained the same, thereby improving the quality and stability of a generated stylized image.

Claims

1. An image processing method, characterized in that, The method includes: Obtain the elements to be redrawn in the original image, including foreground imaging and / or background imaging; A target stylization generation model is obtained, which is trained based on sample images labeled with the target style; The target stylization generation model is guided by guidance information to redraw the element to be redrawn in order to obtain the redrawn element of the target style. The guidance information includes the control information of the element to be redrawn, and the control information includes the imaging features of the element to be redrawn in the original image. Generate a target image based on the redrawn elements and the original image.

2. The method according to claim 1, characterized in that, The control information includes one or more of the line art, depth, or normal of the element to be redrawn in the original image.

3. The method according to claim 1 or 2, characterized in that, The method further includes: The imaging features of the element to be redrawn are extracted from the original image to obtain the control information.

4. The method according to any one of claims 1 to 3, characterized in that, The imaging features of the element to be redrawn in the original image include one or more of the space, position, color, outline, or texture of the element to be redrawn in the original image.

5. The method according to any one of claims 1 to 4, characterized in that, The guidance information also includes image content description information corresponding to the original image, which is used to describe the semantics expressed by the original image.

6. The method according to claim 5, characterized in that, The original image is from a video, and the method further includes: The video is split into multiple segments; For each of the plurality of video segments, video content recognition is performed on the video segment to obtain video content description information corresponding to the video segment, and the video content description information is used as image content description information corresponding to each frame of the video segment.

7. The method according to any one of claims 1 to 6, characterized in that, The process of obtaining the target stylization generation model includes: The initial stylization generation model is trained using a training sample set to obtain the target stylization generation model. The training sample set includes one or more sample images labeled with the target style.

8. The method according to any one of claims 1 to 6, characterized in that, The method is applied to an image processing device, which has multiple pre-set initial stylization generation models. The step of obtaining the target stylization generation model includes: The multiple initial stylization generation models are trained using a training sample set to obtain multiple candidate stylization generation models. The training sample set includes one or more sample images labeled with the target style. The target stylization generation model is obtained from the plurality of candidate stylization generation models.

9. The method according to claim 8, characterized in that, The step of obtaining the target stylization generation model from the plurality of candidate stylization generation models includes: The test image is input into the multiple candidate stylization generation models respectively to obtain multiple predicted images output by the multiple candidate stylization generation models; An aesthetic scoring algorithm is used to evaluate the image quality of each of the multiple predicted images. Output the one or more candidate images with the best image quality among the multiple predicted images; In response to receiving a selection instruction for any of the one or more candidate images, the candidate stylization generation model used to generate the candidate image indicated by the selection instruction is taken as the target stylization generation model.

10. The method according to any one of claims 7 to 9, characterized in that, The initial stylization generation model includes a generative model and a stylization model. The stylization model is used to stylize the output of the generative model in order to change the style of the output of the generative model. Training the initial stylization generation model using the training sample set includes: The stylization model is trained using the training sample set, wherein the training process includes: The sample images are input into the generative model and the stylization model respectively to obtain the stylized image output by the initial stylization generative model. The stylized image is generated based on the base image and style data. The base image is output by the generative model based on the input sample images, and the style data is output by the stylization model based on the input sample images. If the probability that the image style of the stylized image is the target style is less than the probability threshold, the stylized model is adjusted by backpropagation based on the error loss between the stylized image and the sample image, and the stylized model after the model parameter adjustment is used as the new stylized model, and the training process is executed again. If the probability that the stylized image style is the target style is greater than or equal to the probability threshold, the training process ends.

11. The method according to claim 10, characterized in that, The generative model is a diffusion model, an autoregressive model, or a generative adversarial network.

12. The method according to any one of claims 1 to 6, characterized in that, The method is applied to an image processing device, which has pre-set stylization generation models corresponding to multiple styles, including the target style; obtaining the target stylization generation model includes: Output multiple style options corresponding to the various styles; In response to receiving a selection instruction for a style option corresponding to the target style, the stylization generation model corresponding to the target style is used as the target stylization generation model.

13. The method according to any one of claims 1 to 12, characterized in that, The method further includes: Adjust the size of the redrawn element so that the size of the redrawn element is the same as the size of the element to be redrawn; The step of generating a target image based on the redrawn elements and the original image includes: The redrawing element is used to cover and replace the element to be redrawn in the original image to obtain the target image.

14. An image processing device, characterized in that, The image processing device includes: The first acquisition module is used to acquire elements to be redrawn in the original image, the elements to be redrawn include foreground imaging and / or background imaging; The second acquisition module is used to acquire the target stylization generation model, which is trained based on sample images labeled with the target style; The redrawing module is used to guide the target stylization generation model to redraw the element to be redrawn using guidance information, so as to obtain the redrawing element of the target style. The guidance information includes the control information of the element to be redrawn, and the control information includes the imaging features of the element to be redrawn in the original image. A generation module is used to generate a target image based on the redrawn elements and the original image.

15. The image processing apparatus according to claim 14, characterized in that, The control information includes one or more of the line art, depth, or normal of the element to be redrawn in the original image.

16. The image processing apparatus according to claim 14 or 15, characterized in that, The image processing device also includes a feature extraction module; The feature extraction module is used to extract the imaging features of the element to be redrawn from the original image to obtain the control information.

17. The image processing apparatus according to any one of claims 14 to 16, characterized in that, The imaging features of the element to be redrawn in the original image include one or more of the space, position, color, outline, or texture of the element to be redrawn in the original image.

18. The image processing apparatus according to any one of claims 14 to 17, characterized in that, The guidance information also includes image content description information corresponding to the original image, which is used to describe the semantics expressed by the original image.

19. The image processing apparatus according to claim 18, characterized in that, The original image comes from video, and the image processing device further includes a scene processing module and a content recognition module; The segmentation processing module is used to perform segmentation processing on the video to obtain multiple video segments; The content recognition module is used to perform video content recognition on each of the plurality of video segments to obtain video content description information corresponding to the video segment, and to use the video content description information as image content description information corresponding to each frame of the video segment.

20. The image processing apparatus according to any one of claims 14 to 19, characterized in that, The second acquisition module is used for: The initial stylization generation model is trained using a training sample set to obtain the target stylization generation model. The training sample set includes one or more sample images labeled with the target style.

21. The image processing apparatus according to any one of claims 14 to 19, characterized in that, The image processing device has multiple pre-set initial stylization generation models, and the second acquisition module is used for: The multiple initial stylization generation models are trained using a training sample set to obtain multiple candidate stylization generation models. The training sample set includes one or more sample images labeled with the target style. The target stylization generation model is obtained from the plurality of candidate stylization generation models.

22. The image processing apparatus according to claim 21, characterized in that, The second acquisition module is used for: The test image is input into the multiple candidate stylization generation models respectively to obtain multiple predicted images output by the multiple candidate stylization generation models; An aesthetic scoring algorithm is used to evaluate the image quality of each of the multiple predicted images. Output the one or more candidate images with the best image quality among the multiple predicted images; In response to receiving a selection instruction for any of the one or more candidate images, the candidate stylization generation model used to generate the candidate image indicated by the selection instruction is taken as the target stylization generation model.

23. The image processing apparatus according to any one of claims 20 to 22, characterized in that, The initial stylization generation model includes a generative model and a stylization model. The stylization model is used to stylize the output of the generative model in order to change the style of the output of the generative model. The second acquisition module is used for: The stylization model is trained using the training sample set, wherein the training process includes: The sample images are input into the generative model and the stylization model respectively to obtain the stylized image output by the initial stylization generative model. The stylized image is generated based on the base image and style data. The base image is output by the generative model based on the input sample images, and the style data is output by the stylization model based on the input sample images. If the probability that the image style of the stylized image is the target style is less than the probability threshold, the stylized model is adjusted by backpropagation based on the error loss between the stylized image and the sample image, and the stylized model after the model parameter adjustment is used as the new stylized model, and the training process is executed again. If the probability that the stylized image style is the target style is greater than or equal to the probability threshold, the training process ends.

24. The image processing apparatus according to claim 23, characterized in that, The generative model is a diffusion model, an autoregressive model, or a generative adversarial network.

25. The image processing apparatus according to any one of claims 14 to 19, characterized in that, The image processing device has pre-set stylization generation models corresponding to multiple styles, including the target style; the second acquisition module is used for: Output multiple style options corresponding to the various styles; In response to receiving a selection instruction for a style option corresponding to the target style, the stylization generation model corresponding to the target style is used as the target stylization generation model.

26. The image processing apparatus according to any one of claims 14 to 25, characterized in that, The image processing device also includes an adjustment module; The adjustment module is used to adjust the size of the redrawn element so that the size of the redrawn element is consistent with the size of the element to be redrawn; The generation module is used to replace the element to be redrawn in the original image with the redrawn element to obtain the target image.

27. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1 to 13.

28. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device cluster, the computing device cluster causes the computing device cluster to perform the method as described in any one of claims 1 to 13.

29. A computer-readable storage medium, characterized in that, Includes computer program instructions, which, when executed by a cluster of computing devices, perform the method as described in any one of claims 1 to 13.

Images