Image processing method and device

Abstract

The present application belongs to the technical field of computer vision. Provided are an image processing method and device. The method comprises: acquiring from an original image an element to be redrawn, and a target stylization generation model; using guidance information to guide the target stylization generation model to redraw the element to be redrawn, so as to obtain a redrawn element of a target style; and generating a target image on the basis of the redrawn element and the original image, wherein the target stylization generation model is obtained by means of performing training on the basis of a sample image labeled with the target style, the guidance information comprises control information of the element to be redrawn, and the control information comprises imaging features, in the original image, of the element to be redrawn. In the present application, guidance information is used to guide a target stylization generation model to redraw an element to be redrawn which is in an original image, such that when the target stylization generation model converts the element to be redrawn into a redrawn element of a target style, imaging features of the element before and after redrawing are maintained the same, thereby improving the quality and stability of a generated stylized image.

Description

Image processing methods and devices

[0001] This application claims priority to Chinese Patent Application No. 202410800094.4, filed on June 19, 2024, entitled "A Generative Video Stylization Method and Apparatus with Custom Styles", and Chinese Patent Application No. 202411174001.8, filed on August 26, 2024, entitled "Image Processing Method and Apparatus", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of computer vision technology, and in particular to an image processing method and device. Background Technology

[0003] Artificial intelligence (AI) is the science of extending human intelligence using machines. AI stylization is a recently emerging AI-generated content (AIGC) technology. AI stylization refers to using AI technology to transform images or videos into another style; that is, using AI technology to stylize images or videos. Specifically, stylizing video involves stylizing each frame of the video individually. Currently, when using AI stylization technology to stylize images, the resulting stylized images often suffer from poor quality. Summary of the Invention

[0004] This application provides an image processing method and apparatus.

[0005] Firstly, an image processing method is provided. This method includes: acquiring elements to be redrawn in an original image, the elements to be redrawn including foreground and / or background imaging; acquiring a target stylization generation model, the target stylization generation model being trained based on sample images labeled with a target style; using guiding information to guide the target stylization generation model to redraw the elements to be redrawn, to obtain redrawn elements of the target style. The guiding information includes control information for the elements to be redrawn, the control information including the imaging features of the elements to be redrawn in the original image; and generating a target image based on the redrawn elements and the original image.

[0006] This application employs guiding information to guide the target stylization generation model in redrawing elements to be redrawn in the original image. Since the guiding information includes control information describing the imaging characteristics of the elements to be redrawn in the original image, the target stylization generation model maintains consistent imaging characteristics before and after redrawing when transforming the elements into redrawn elements of the target style. Therefore, it avoids distortion of redrawn elements and changes in image content, thereby improving the quality and stability of the generated stylized image. Furthermore, this application can perform stylization processing on foreground and / or background imaging in the image, meaning it can selectively redraw part or all of the image content, offering high flexibility in stylization processing.

[0007] Optionally, the control information includes, but is not limited to, one or more of the line art, depth, or normal of the element to be redrawn in the original image.

[0008] Optionally, the above method further includes: extracting imaging features of the elements to be redrawn from the original image to obtain control information.

[0009] Optionally, the imaging features of the element to be redrawn in the original image include, but are not limited to, one or more of the space, position, color, or texture of the element to be redrawn in the original image.

[0010] Optionally, the guidance information may also include image content description information corresponding to the original image, which is used to describe the semantics expressed by the original image.

[0011] This application guides the target stylization generation model to redraw the elements to be redrawn in the original image by using image content description information as guiding information. This ensures that the target stylization generation model maintains the consistency of image content before and after redrawing when converting the elements to be redrawn into redrawing elements of the target style. As a result, the redrawn image can retain the original image characteristics, further improving the quality and stability of the generated stylized image.

[0012] Optionally, the original image comes from a video, and the method further includes: performing scene segmentation on the video to obtain multiple video segments. For each of the multiple video segments, video content recognition is performed on the video segment to obtain video content description information corresponding to the video segment, and this video content description information is used as the image content description information corresponding to each frame in that video segment.

[0013] This application achieves the same image content description information for each frame in the same video segment by performing segmentation processing and video content recognition. After stylizing each frame in the same video segment after segmentation, the image content of each frame in the same video segment can be kept consistent, thereby solving the problem of poor inter-frame continuity and ensuring video smoothness and consistency of style before and after the video.

[0014] Optionally, one way to obtain the target stylization generative model includes: training an initial stylization generative model using a training sample set to obtain the target stylization generative model, wherein the training sample set includes one or more sample images labeled with the target style.

[0015] Optionally, the above method is applied to an image processing device, which has multiple initial stylization generation models pre-installed. Another way to obtain the target stylization generation model includes: training the multiple initial stylization generation models using a training sample set to obtain multiple candidate stylization generation models, wherein the training sample set includes one or more sample images labeled with the target style; and obtaining the target stylization generation model from the multiple candidate stylization generation models.

[0016] Optionally, the implementation of obtaining the target stylization generation model from multiple candidate stylization generation models includes: inputting test images into multiple candidate stylization generation models respectively to obtain multiple predicted images output by the multiple candidate stylization generation models; using an aesthetic scoring algorithm to evaluate the image quality of the multiple predicted images respectively; outputting one or more candidate images with the best image quality among the multiple predicted images; and, in response to receiving a selection instruction for any candidate image among the one or more candidate images, using the candidate stylization generation model used to generate the candidate image indicated by the selection instruction as the target stylization generation model.

[0017] In this application, the image processing device can use an aesthetic scoring algorithm to evaluate the quality of images generated by different stylization generation models, and combine user preferences to provide a high-quality stylization generation model for stylization processing of images.

[0018] Optionally, the initial stylization generation model includes a generative model and a stylization model. The stylization model is used to stylize the output of the generative model to change its style. The implementation of training the initial stylization generation model using a training sample set includes: performing a training process on the stylization model using the training sample set. The training process includes: inputting sample images to both the generative and stylization models to obtain a stylized image output by the initial stylization generation model. The stylized image is generated based on a base image and style data. The base image is output by the generative model based on the input sample images, and the style data is output by the stylization model based on the input sample images. If the probability that the stylized image's style is the target style is less than a probability threshold, the model parameters of the stylization model are adjusted using backpropagation based on the error loss between the stylized image and the sample images. The stylized model with adjusted parameters is then used as a new stylization model, and the training process is executed again. If the probability that the stylized image's style is the target style is greater than or equal to the probability threshold, the training process ends.

[0019] In this implementation, the style of the output image of the stylization generation model can be changed simply by adjusting the model parameters of the stylization model, without having to adjust the model parameters of the large generative model. This reduces the computational resources required during model training and improves model training efficiency.

[0020] Alternatively, the generative model can be a diffusion model, an autoregressive model, or a generative adversarial network.

[0021] Optionally, the above method is applied to an image processing device, which has pre-set stylization generation models corresponding to multiple styles, including a target style. Another implementation of obtaining the target stylization generation model includes: outputting multiple style options corresponding to the multiple styles. In response to receiving a selection instruction for a style option corresponding to the target style, the stylization generation model corresponding to the target style is used as the target stylization generation model.

[0022] In this application, users can select existing preset styles according to their stylistic needs.

[0023] Optionally, the above method further includes: adjusting the size of the redrawn elements to make the size of the redrawn elements consistent with the size of the elements to be redrawn. Accordingly, one implementation of generating a target image based on the redrawn elements and the original image includes: replacing the elements to be redrawn in the original image with the redrawn elements to obtain the target image.

[0024] Since some generative models, such as diffusion models, are very sensitive to the proportion of the redrawn object in the image, in this application, the image processing device can first scale the element to be redrawn to meet the requirements of the generative model, then independently redraw the element to be redrawn through a stylized generative model containing the generative model, scale the obtained redrawn element to the size in the original image, and finally perform overlay replacement, which can enhance the stability of the redrawn element.

[0025] Secondly, an image processing device is provided. The image processing device includes multiple functional modules that interact to implement the methods described in the first aspect and its various embodiments. The multiple functional modules can be implemented based on software, hardware, or a combination of both, and can be arbitrarily combined or divided based on specific implementations.

[0026] Thirdly, a computing device cluster is provided, including at least one computing device, each computing device including a processor and a memory. The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the methods described in the first aspect and its various embodiments.

[0027] Fourthly, a computer program product containing instructions is provided, which, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the methods described in the first aspect and its various embodiments.

[0028] Fifthly, a computer-readable storage medium is provided, including computer program instructions that, when executed by a cluster of computing devices, perform the methods described in the first aspect and its various embodiments.

[0029] In a sixth aspect, a chip is provided, including programmable logic circuitry and / or program instructions, which, when the chip is running, implement the methods described in the first aspect and its various embodiments.

[0030] It should be understood that the beneficial effects of the technical solutions and corresponding possible implementations of the second to sixth aspects of this application can be found in the above description of the technical effects of the first aspect and its corresponding possible implementations, and will not be repeated here. Attached Figure Description

[0031] Figure 1 is a schematic flowchart of an image processing method provided in an embodiment of this application;

[0032] Figure 2 is a schematic diagram of a person detection method provided in an embodiment of this application;

[0033] Figure 3 is a schematic diagram of portrait segmentation provided in an embodiment of this application;

[0034] Figure 4 is a schematic diagram of the structure of a stylized generative model provided in an embodiment of this application;

[0035] Figure 5 is a schematic diagram of control information provided in an embodiment of this application;

[0036] Figure 6 is a schematic diagram of image redrawing based on line drawing signals provided in an embodiment of this application;

[0037] Figure 7 is a schematic diagram of the implementation process of stylizing video according to an embodiment of this application;

[0038] Figure 8 is a schematic diagram of the structure of an image processing device provided in an embodiment of this application;

[0039] Figure 9 is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0040] Figure 10 is a schematic diagram of the structure of a computing device cluster provided in an embodiment of this application;

[0041] Figure 11 is a schematic diagram of another computing device cluster provided in an embodiment of this application. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0043] AIGC refers to media content generated by artificial intelligence algorithms based on user instructions, such as text, audio, images, or video. AI stylization is mainly divided into image stylization and video stylization. Image stylization refers to stylizing existing images. Video stylization refers to stylizing existing videos. Specifically, stylizing video involves stylizing each frame of the video individually. Therefore, image stylization is the foundation and key to achieving video stylization.

[0044] Currently, when using AI stylization technology to stylize images, it is impossible to stylize them stably while preserving the original characteristics of the image. For example, after stylization, the image may be distorted (e.g., foreground elements such as people in the image are easily distorted), and the image content may also change (e.g., additional objects are generated). This results in poor quality stylized images.

[0045] Based on this, this application provides a technical solution that converts an original image into an image of the specified style using a stylized generation model trained on sample images of a specified style. This allows users to convert an image into the specified style simply by providing the image, achieving automatic and efficient stylized image processing. Furthermore, by guiding the stylized generation model to redraw elements in the image according to their imaging characteristics in the original image, the redrawn elements maintain the same imaging characteristics as the original elements, thus preserving the original image characteristics and avoiding image distortion or changes in image content, thereby improving the quality and stability of the generated stylized image. The technical solution provided in this application is applied to an image processing device, which can be any computing device with computing power. The specific implementation of this technical solution is as follows: First, the elements to be redrawn and the target stylized generation model in the original image are obtained. The elements to be redrawn include foreground imaging and / or background imaging. The target stylized generation model is trained based on sample images labeled with the target style. Then, guiding information is used to guide the target stylized generation model to redraw the elements to be redrawn, resulting in redrawn elements of the target style. The guidance information includes control information for the element to be redrawn, which includes the imaging features of the element in the original image. Finally, a target image is generated based on the redrawn element and the original image. This application uses guidance information to guide a target stylization generation model to redraw the element to be redrawn in the original image. Because the guidance information includes control information describing the imaging features of the element to be redrawn in the original image, the target stylization generation model maintains consistent imaging features before and after redrawing the element when converting it into a redrawn element of the target style. This avoids distortion of the redrawn element and changes in image content, thus improving the quality and stability of the generated stylized image. Furthermore, this application can perform stylization processing on foreground and / or background imaging in the image, meaning it can choose to redraw part or all of the content in the image, offering high flexibility in stylization processing.

[0046] Optionally, in some embodiments, the guiding information further includes image content description information corresponding to the original image, which is used to describe the semantics expressed by the original image. When the element to be redrawn includes foreground imaging, the image content description information can be used to describe the semantics expressed by the foreground imaging. For example, if the foreground imaging is a person, the image content description information includes, but is not limited to, the person's action, gender, hairstyle, and height. When the element to be redrawn includes background imaging, the image content description information can be used to describe the semantics expressed by the background imaging, including, but not limited to, objects contained in the background imaging and background color. This application guides the target stylization generation model to redraw the elements to be redrawn in the original image by using image content description information as guiding information. This ensures that the target stylization generation model maintains consistency in image content before and after redrawing when converting the elements to be redrawn into redrawn elements of the target style, thereby enabling the redrawn image to retain the original image characteristics and further improving the quality and stability of the generated stylized image.

[0047] Optionally, in video stylization scenarios, the technical solution provided in this application can be used to perform image stylization processing on each frame of the video separately. When the original image is from a video, the image processing device can perform scene segmentation on the video to obtain multiple video segments. Then, for each of these video segments, video content recognition is performed to obtain the corresponding video content description information, which is then used as the image content description information for each frame of that video segment. This application, by performing scene segmentation and video content recognition on the video, ensures that the image content description information corresponding to each frame of the same video segment is the same. Thus, after stylizing each frame of the same video segment after scene segmentation, the consistency of the image content of each frame within the same video segment can be maintained, thereby solving the problem of poor inter-frame continuity and ensuring video smoothness and consistency of style throughout the video.

[0048] The method flow of the embodiments of this application is illustrated below.

[0049] For example, Figure 1 is a schematic flowchart of an image processing method 100 provided in an embodiment of this application. As shown in Figure 1, the method 100 includes, but is not limited to, the following steps 101 to 104.

[0050] Step 101: The image processing device acquires the elements to be redrawn in the original image, including foreground imaging and / or background imaging.

[0051] Optionally, the image processing device first acquires an original image, which can be a single image or a frame from a video. For example, the image processing device can acquire a video uploaded by a user and execute the image processing method provided in this application embodiment on each frame of the video to achieve video stylization. The video can be a live-action video, or it can be an animation, a movie, or other forms and styles of video.

[0052] Optionally, the original image may only include background imaging, or the original image may include both background imaging and foreground imaging. That is, the embodiments of this application do not limit whether the image to be stylized contains foreground imaging; pure background images (such as pure landscape images) can also be stylized using the image processing method provided in the embodiments of this application.

[0053] Optionally, if the original image is a pure background image, the elements to be redrawn in the original image include only the background image. If the original image includes both background and foreground images, the elements to be redrawn in the original image may include only the foreground image, or only the background image, or both the foreground and background images.

[0054] This application embodiment uses an original image including background and foreground images as an example for illustration. After acquiring the original image, the image processing device can separate the foreground and background to obtain separate foreground and background images. Then, the foreground image can be redrawn, or the background image can be redrawn, or both the foreground and background images can be redrawn separately to improve the redrawing quality. For example, the image processing device can use an image matting algorithm to remove the background from the original image to obtain the foreground image, or remove the foreground from the original image to obtain the background image. This application embodiment does not limit the specific type of image matting algorithm used.

[0055] For example, if the elements to be redrawn in the original image include a person image (foreground image), the image processing device can first use a person detection algorithm to detect the person region in the original image, then use a portrait segmentation algorithm to extract the binary image of the person, and then use the binary image of the person to mask the person region to obtain the person image. For example, Figure 2 is a schematic diagram of person detection provided by an embodiment of this application. As shown in Figure 2, the area within the rectangular detection box is the detected person region. As another example, Figure 3 is a schematic diagram of portrait segmentation provided by an embodiment of this application. As shown in Figure 3, the upper left image is the person region detected in Figure 2, and the upper right image is the binary image of the person, where the white area of ​​the binary image is the person image and the black area is the background image. By using the binary image of the person (upper right image) to mask the person region (upper left image), the person image (lower image) can be obtained. Optionally, the person detection algorithm used in the embodiments of this application can be any object detection algorithm based on deep learning algorithms, including but not limited to the YoLo (you only look once) algorithm or MediaPipe (an open-source multimedia machine learning model application framework). The portrait segmentation algorithms used in this application include, but are not limited to, portrait segmentation algorithms based on fully convolutional networks (FCN) (hereinafter referred to as PortraitFCN algorithm), principal component analysis (PCA) algorithms, or MediaPipe. This application does not limit the person detection algorithm or portrait segmentation algorithm used.

[0056] Step 102: The image processing device acquires the target stylization generation model.

[0057] The target styleization generation model is trained on sample images labeled with the target style. Optionally, the target styleization generation model is implemented based on a generative model. A generative model is a model capable of generating observed data. The output of a generative model includes, but is not limited to, text, images, or videos.

[0058] Optionally, the styles in this application embodiment can be categorized according to different dimensions. For example, different painting styles such as animation, oil painting, ink painting, or line drawing can be considered as different styles. Similarly, different photographic styles such as documentary style, natural style, black and white style, or portrait style can be considered as different styles. Furthermore, different color schemes can be considered as different styles. This application embodiment does not limit the definition and categorization of styles.

[0059] Optionally, in step 102 above, there are multiple ways for the image processing device to obtain the target stylization generation model. This application embodiment mainly illustrates the following two implementation methods.

[0060] In the first implementation of step 102 above, the image processing device acquires the target stylization generation model, which can be achieved by training the model using a training sample set. The training sample set includes one or more sample images labeled with the target style. It is worth noting that, in most cases, the training sample set includes multiple sample images labeled with the target style; using a large number of samples for model training can improve model accuracy.

[0061] In conjunction with the first scenario of the first implementation described above, the image processing device has a pre-built initial stylization generation model. The image processing device trains the initial stylization generation model using a training sample set to obtain a target stylization generation model. Optionally, the initial stylization generation model includes a generative model; that is, in this embodiment, the generative model can be used as a basis to train a stylization generation model that outputs an image with a specific style.

[0062] Optionally, the initial stylization generation model includes a generative model and a stylization model. The stylization model is used to stylize the output of the generative model to change its style. The specific implementation of the image processing device training the initial stylization generation model using a training sample set includes: the image processing device executing a training process on the stylization model using the training sample set. The training process includes: inputting sample images to the generative model and the stylization model respectively to obtain a stylized image output by the initial stylization generation model. This stylized image is generated based on a base image and style data. The base image is output by the generative model based on the input sample image, and the style data is output by the stylization model based on the input sample image. If the probability that the stylized image's style is the target style is less than a probability threshold, the stylization model's parameters are adjusted using backpropagation based on the error loss between the stylized image and the sample image. The stylized model with adjusted parameters is then used as a new stylization model, and the training process is executed again. If the probability that the stylized image's style is the target style is greater than or equal to the probability threshold, the training process ends.

[0063] Optionally, after obtaining the stylized image output by the initial stylization generation model, an aesthetic scoring algorithm can be used to evaluate the image quality of the stylized image. Based on the image quality evaluation result, the probability that the stylized image's style corresponds to the target style can be determined. Aesthetic scoring algorithms include, but are not limited to, algorithms that evaluate image quality, such as the contrastive language image pre-training (CLIP) algorithm. For example, image quality can be represented by a CLIP score; a higher CLIP score indicates better image quality. The CLIP algorithm scores images by learning to match them with text. The CLIP scoring mechanism involves calculating the similarity between the image and the text, providing a score by calculating the similarity between the input image and the text. For example, when using the CLIP algorithm to score a stylized image for the target style, if the CLIP score is below a preset threshold, the probability that the stylized image's style corresponds to the target style is less than the threshold; if the CLIP score is higher than or equal to the preset threshold, the probability that the stylized image's style corresponds to the target style is greater than or equal to the threshold.

[0064] For example, Figure 4 is a schematic diagram of the structure of a stylized generative model provided in an embodiment of this application. As shown in Figure 4, the left side is the generative model, and the right side is the stylized model. During training, the model parameters of the generative model remain fixed, and only the model parameters of the stylized model are changed, i.e., only the stylized model is trained. During prediction, the input data is simultaneously input into the generative model and the stylized model, and during output, the base image output by the generative model and the style data output by the stylized model are matrix-added to inject style data into the base image to obtain the output data, thereby achieving stylized processing of the image.

[0065] Generative models are large models characterized by complex structures and large parameter sizes. Examples include, but are not limited to, diffusion models, autoregressive models, or generative adversarial networks (GANs). Stylization models, on the other hand, are fine-tuning models used in conjunction with large models, characterized by simple structures and small parameter sizes. Examples include, but are not limited to, low-rank adapters (LoRA) or textural inversion models. In this approach, only the parameters of the stylization model need to be adjusted to change the style of the output image of the stylization generation model, without needing to adjust the parameters of the large generative model itself. This reduces the computational resources required during model training and improves training efficiency. Alternatively, in this embodiment, sample images can be used to directly fine-tune the generative model, adjusting only a small number of parameters to obtain the target stylization generation model. Fine-tuning training refers to further training an existing model on a specific task to adapt it to the specific data and requirements of that task. This embodiment does not limit the structure or training method of the stylization generation model.

[0066] Optionally, sample images in the training sample set can be uploaded by users to the image processing device. For example, a user can upload a group of sample images with similar styles, such as anime or landscape paintings with the same style, statues of the same form, or natural scenery photos with the same terrain. The user-uploaded sample images can be labeled with style tags. Alternatively, the user-uploaded sample images may not be labeled with style tags. After acquiring the sample images, the image processing device performs style recognition and style tagging on the sample images using a large language model.

[0067] In this embodiment, users only need to upload a set of sample images with similar styles to generate a corresponding stylization generation model. Then, by providing the original image whose style needs to be changed, the stylization generation model can redraw the original image, resulting in a target image with a style similar to the sample images. In other words, users only need to provide an image with the desired style to achieve efficient and high-quality style conversion of existing images, facilitating custom stylization processing of images.

[0068] In conjunction with the second scenario of the first implementation described above, the image processing device pre-configures multiple initial stylization generation models. The image processing device trains each of these initial stylization generation models using a training sample set to obtain multiple candidate stylization generation models, and then selects the target stylization generation model from these candidate models. Here, the training method for each initial stylization generation model can refer to the training method for the initial stylization generation model in the first scenario described above; this embodiment will not be repeated here.

[0069] Optionally, the image processing device may have multiple pre-built initial stylization generation models, or multiple pre-built generative models. Different generative models can be used as the basis for training stylization generation models that output images with a specific style. For example, each initial stylization generation model includes a generative model and a stylization model. Training multiple initial stylization generation models using training sample sets is equivalent to training stylization models with multiple different generative models using training sample sets, so as to select a suitable generative model as the generative model in the target stylization generation model.

[0070] Optionally, the image processing device obtains the implementation of the target stylization generation model from multiple candidate stylization generation models, including the following steps A1 to A4.

[0071] In step A1, the image processing device inputs the test image into multiple candidate stylization generation models to obtain multiple predicted images output by the multiple candidate stylization generation models.

[0072] Optionally, the test images are labeled with style tags.

[0073] In step A2, the image processing device uses an aesthetic scoring algorithm to evaluate the image quality of the multiple predicted images.

[0074] Optionally, the aesthetic scoring algorithm includes, but is not limited to, algorithms such as the CLIP algorithm for judging image quality. When the test image is labeled with a style tag, the image quality can be represented by a CLIP score; the higher the CLIP score, the better the image quality.

[0075] In step A3, the image processing device outputs one or more candidate images with the best image quality among the multiple predicted images.

[0076] For example, the image processing device may display one or more alternative images, or the image processing device may send one or more alternative images to a connected display device for the user to select.

[0077] In step A4, in response to receiving a selection instruction for any of the one or more candidate images, the image processing device uses the candidate stylization generation model used to generate the candidate image indicated by the selection instruction as the target stylization generation model.

[0078] In this embodiment, the image processing device can use an aesthetic scoring algorithm to evaluate the quality of images generated by different stylization generation models, and, combined with user preferences, provide a high-quality stylization generation model for stylization processing of the image. Alternatively, the image processing device can directly use the candidate stylization generation model that generates the best-quality predicted image among the multiple predicted images as the target stylization generation model; that is, it can automatically match the stylization generation model that provides the best-quality stylized image through aesthetic scoring.

[0079] In the second implementation of step 102 above, the image processing device has pre-set stylization generation models corresponding to multiple styles, including a target style. The image processing device acquires the target stylization generation model by outputting multiple style options corresponding to the multiple styles. In response to receiving a selection instruction for a style option corresponding to the target style, the image processing device uses the stylization generation model corresponding to the target style as the target stylization generation model. Optionally, the pre-set stylization generation model in the image processing device can be trained by the image processing device itself, or it can be trained by other devices and then sent to the image processing device.

[0080] In this embodiment of the application, the user can select an existing preset style according to stylistic requirements.

[0081] Step 103: The image processing device uses guiding information to guide the target stylization generation model to redraw the element to be redrawn, so as to obtain the redrawn element of the target style.

[0082] Optionally, the guidance information can be extracted from the original image by the image processing device, or the guidance information can be uploaded to the image processing device by the user, or the image processing device can provide guidance information for the user to choose from. This application embodiment does not limit the method by which the image processing device obtains the guidance information.

[0083] The guidance information includes control information for the element to be redrawn, which includes, but is not limited to, the imaging features of the element in the original image. These imaging features, as included in the control information, include, but are not limited to, one or more of the following: spatial characteristics, position, color, contour, or texture. When the original image is from video, the imaging features may also include optical flow. It is worth noting that if a certain imaging feature of the element to be redrawn needs to be changed, this feature will not be included in the control information. For example, if the color of the element needs to be changed, the color will not be included in the control information. Conversely, the control information may carry the target color that the element needs to become, so that after the image processing device redraws the element, it obtains a redrawn element with the target color, which can be input or selected by the user.

[0084] Optionally, the image processing device extracts imaging features of the element to be redrawn from the original image to obtain control information of the element to be redrawn. Optionally, the control information includes one or more of the line drawing, depth, or normal of the element to be redrawn in the original image. The line drawing can reflect the outline of the element to be redrawn. The image processing device can use algorithms such as sk_model to estimate the line drawing signal of the element to be redrawn in the original image. Depth and normal can reflect the texture of the element to be redrawn. The image processing device can use algorithms such as Midas or Zoe to estimate the depth signal of the element to be redrawn in the original image. The image processing device can use algorithms such as point cloud library (PCL) to estimate the normal signal of the element to be redrawn in the original image. For example, Figure 5 is a schematic diagram of control information provided in an embodiment of this application. In Figure 5, the left image is a line drawing signal diagram, the middle image is a depth signal diagram, and the right image is a normal signal diagram. Using control information to guide the target stylization generation model in redrawing the elements to be redrawn can reduce the randomness of the redrawing process and ensure that the redrawn elements generated by the target stylization generation model retain the imaging characteristics before redrawing. For example, Figure 6 is a schematic diagram of image redrawing based on line drawing signals provided in an embodiment of this application. In Figure 6, the upper left image is the original image, the upper right image is the line drawing signal map of the turtle in the original image, and the lower left and lower right images are the redrawn images generated based on the line drawing signal map, respectively. As shown in Figure 6, since the line drawing signal map describes the outline of the turtle, when redrawing the original image based on the line drawing signal map, the outline of the turtle in the generated redrawn image can be kept consistent with the outline of the turtle in the original image, thereby avoiding distortion of the turtle.

[0085] Optionally, the control algorithm guiding the target stylization generation model to redraw the elements to be redrawn includes, but is not limited to, redraw range shape control algorithms such as ControlNet. Correspondingly, the control information includes, but is not limited to, all control signals involved in ControlNet, and the method for obtaining the control information can refer to the method for obtaining control signals in ControlNet.

[0086] In this embodiment, the control information used to describe the imaging features of the element to be redrawn in the original image is used as guiding information to guide the target stylization generation model to redraw the element to be redrawn in the original image. This ensures that the target stylization generation model maintains consistent imaging features of the element before and after redrawing when converting the element to be redrawn into a redrawn element of the target style. Therefore, it can avoid situations such as deformation of the redrawn element in the image and changes in the amount or amount of image content, thereby improving the quality and stability of the generated stylized image.

[0087] Optionally, the guidance information also includes image content description information corresponding to the original image, which is used to describe the semantics expressed by the original image. When the element to be redrawn includes a foreground image, the image content description information can be used to describe the semantics expressed by the foreground image. For example, if the foreground image is a person, the image content description information includes, but is not limited to, the person's action, gender, hairstyle, and height. When the element to be redrawn includes a background image, the image content description information can be used to describe the semantics expressed by the background image, including, but not limited to, the objects contained in the background image and the background color.

[0088] In this embodiment, image content description information is used as guiding information to guide the target stylization generation model to redraw the elements to be redrawn in the original image. This ensures that the target stylization generation model maintains consistency between the image content before and after redrawing when converting the elements to be redrawn into redrawn elements of the target style. As a result, the redrawn image can retain the original image characteristics, further improving the quality and stability of the generated stylized image.

[0089] Optionally, when the original image is from a video, the image processing device can perform scene segmentation on the video to obtain multiple video segments. Then, for each of the multiple video segments, video content recognition is performed on the video segment to obtain the video content description information corresponding to the video segment, and the video content description information is used as the image content description information corresponding to each frame in the video segment.

[0090] The video segmentation process refers to using deep learning algorithms to divide a complete video segment into multiple video segments, each shot from a different camera angle. Optionally, the video segmentation algorithms used in this application include, but are not limited to, video frame similarity analysis algorithms, structural similarity index (SSIM) algorithms, and Fréchet inception distance (FID) approximation. The video content recognition algorithms used in this application include, but are not limited to, video content extraction algorithms such as VidArts.

[0091] In this embodiment of the application, by performing scene segmentation and video content recognition on the video, the image content description information corresponding to each frame in the same video segment is the same. In this way, after stylizing each frame in the same video segment after scene segmentation, the image content of each frame in the same video segment can be kept consistent, thereby solving the problem of poor inter-frame continuity and ensuring video smoothness and consistency of style before and after the video.

[0092] Step 104: The image processing device generates a target image based on the redrawn element and the original image.

[0093] Optionally, the image processing device may first adjust the size of the redrawing element to make the size of the redrawing element consistent with the size of the element to be redrawn, and then use the redrawing element to cover and replace the element to be redrawn in the original image to obtain the target image.

[0094] Since some generative models, such as diffusion models, are very sensitive to the proportion of the redrawn object in the image, in this embodiment, the image processing device can first scale the element to be redrawn obtained in step 101 to meet the requirements of the generative model, then independently redraw the element to be redrawn using a stylized generative model that includes the generative model, scale the obtained redrawn element to the size in the original image, and finally perform overlay replacement. This can enhance the stability of the redrawn element.

[0095] Optionally, when the element to be redrawn is a foreground image, after acquiring the redrawn foreground image corresponding to the foreground image, the image processing device can use the redrawn foreground image to cover and replace the foreground image in the original image to obtain the target image. When the element to be redrawn is a background image, after acquiring the redrawn background image corresponding to the background image, the image processing device can add the foreground image to the corresponding position in the redrawn background image according to the imaging position of the foreground image in the original image to obtain the target image. When the element to be redrawn includes both foreground and background images, after acquiring the redrawn foreground image corresponding to the foreground image and the redrawn background image corresponding to the background image respectively, the image processing device can add the redrawn foreground image to the corresponding position in the redrawn background image according to the imaging position of the foreground image in the original image to obtain the target image.

[0096] In the image processing method provided in this application embodiment, guiding information is used to guide the target stylization generation model to redraw the elements to be redrawn in the original image. Since the guiding information includes control information describing the imaging characteristics of the elements to be redrawn in the original image, the target stylization generation model maintains consistent imaging characteristics before and after redrawing when converting the elements to be redrawn into redrawn elements of the target style. Therefore, it avoids distortion of the redrawn elements in the image and situations where the image content increases or decreases, thereby improving the quality and stability of the generated stylized image. Furthermore, image content description information can also be used as guiding information to guide the target stylization generation model to redraw the elements to be redrawn in the original image. This ensures that the target stylization generation model maintains consistent image content before and after redrawing when converting the elements to be redrawn into redrawn elements of the target style, thus allowing the redrawn image to retain its original image characteristics and further improving the quality and stability of the generated stylized image. In addition, this application embodiment can perform stylization processing on foreground and / or background imaging in the image, that is, it can select to redraw part or all of the content in the image, providing high flexibility in stylization processing.

[0097] Optionally, in video stylization scenarios, after performing image stylization processing on the video frames of the original video, the image processing device can perform consistency post-processing on the redrawn video frames to smooth the video, improve the stability and quality of the redrawn video, and output the final video. The consistency post-processing algorithm includes, but is not limited to, Fastblend (an algorithm to enhance video coherence, stability, and consistency) or optical flow-related algorithms. For example, the implementation process of consistency post-processing for redrawn video frames by the image processing device is as follows: First, the optical flow information between two adjacent video frames in the original video is obtained, and this optical flow information is transferred to the corresponding redrawn video frames. Then, the blending algorithm is used to mix the information from the original video and the redrawn video, making the redrawn video smoother and reducing flicker.

[0098] For example, Figure 7 is a schematic diagram of the implementation process of stylizing video according to an embodiment of this application. As shown in Figure 7, the original video is first acquired, and the foreground image of the original video is a person image. Then, the foreground image and background image in the original video are detected and separated to obtain a person sequence composed of person images and a background sequence composed of background images. Taking the need to redraw the person images in the original video as an example, the control information and image content description information corresponding to each frame of the person image in the person sequence are extracted, and the size of the person image is adjusted to control the proportion of the person. The person sequence is input into the stylization generation model, the proportion of the person is controlled, and the stylization generation model is guided to redraw each frame of the person image using the corresponding control information and image content description information, and the redrawn person sequence is output. Then, a redrawn video is generated based on the redrawn person sequence and background sequence, and the redrawn video is post-processed for consistency to obtain the target video. The stylization generation model can be trained based on a stylized image provided by the user, or it can be a model with an existing style selected by the user.

[0099] The order of steps in the image processing method provided in this application can be adjusted appropriately. For example, the order of steps 101 and 102 can be interchanged, or they can be performed simultaneously. Steps can also be added or removed as needed. Any variations that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application.

[0100] The following describes an example of a virtual device in an embodiment of this application.

[0101] For example, Figure 8 is a schematic diagram of the structure of an image processing device 800 provided in an embodiment of this application. As shown in Figure 8, the image processing device 800 includes, but is not limited to, a first acquisition module 801, a second acquisition module 802, a redrawing module 803, and a generation module 804. Optionally, referring to Figure 8, the image processing device 800 also includes one or more of a feature extraction module 805, a scene processing module 806, a content recognition module 807, or an adjustment module 808.

[0102] The first acquisition module 801 is used to acquire the elements to be redrawn in the original image, including foreground imaging and / or background imaging.

[0103] The second acquisition module 802 is used to acquire the target stylization generation model, which is trained based on sample images labeled with the target style.

[0104] The redrawing module 803 is used to guide the target stylization generation model to redraw the elements to be redrawn using guiding information, so as to obtain redrawing elements of the target style. The guiding information includes control information of the elements to be redrawn, and the control information includes the imaging features of the elements to be redrawn in the original image.

[0105] The generation module 804 is used to generate a target image based on the redrawn elements and the original image.

[0106] Optionally, the control information includes one or more of the line art, depth, or normals of the element to be redrawn in the original image.

[0107] Optionally, the feature extraction module 805 is used to extract imaging features of the elements to be redrawn from the original image to obtain control information.

[0108] Optionally, the imaging features of the element to be redrawn in the original image include one or more of the following: space, position, color, outline, or texture of the element to be redrawn in the original image.

[0109] Optionally, the guidance information may also include image content description information corresponding to the original image, which is used to describe the semantics expressed by the original image.

[0110] Optionally, the original image comes from a video. The storyboard processing module 806 is used to perform storyboard processing on the video to obtain multiple video segments. The content recognition module 807 is used to perform video content recognition on each of the multiple video segments to obtain video content description information corresponding to the video segment, and uses the video content description information as the image content description information corresponding to each frame in the video segment.

[0111] Optionally, the second acquisition module 802 is used to: train the initial stylization generation model using a training sample set to obtain the target stylization generation model, wherein the training sample set includes one or more sample images labeled with the target style.

[0112] Alternatively, the image processing device may have multiple initial stylization generation models pre-installed. The second acquisition module 802 is used to: train the multiple initial stylization generation models using a training sample set to obtain multiple candidate stylization generation models, wherein the training sample set includes one or more sample images labeled with the target style; and acquire the target stylization generation model from the multiple candidate stylization generation models.

[0113] Optionally, the second acquisition module 802 is specifically used for: inputting test images into multiple candidate stylization generation models respectively to obtain multiple predicted images output by multiple candidate stylization generation models; using an aesthetic scoring algorithm to evaluate the image quality of the multiple predicted images respectively; outputting one or more candidate images with the best image quality among the multiple predicted images; and in response to receiving a selection instruction for any candidate image among the one or more candidate images, using the candidate stylization generation model used to generate the candidate image indicated by the selection instruction as the target stylization generation model.

[0114] Optionally, the initial stylization generation model includes a generative model and a stylization model. The stylization model is used to stylize the output of the generative model to change the style of its output. The second acquisition module 802 is specifically used to: perform a training process on the stylization model using a training sample set. The training process includes: inputting sample images to the generative model and the stylization model respectively to obtain a stylized image output by the initial stylization generation model. The stylized image is generated based on a base image and style data. The base image is output by the generative model based on the input sample image, and the style data is output by the stylization model based on the input sample image. If the probability that the stylized image's style is the target style is less than a probability threshold, the model parameters of the stylization model are adjusted using backpropagation based on the error loss between the stylized image and the sample image. The stylized model with adjusted model parameters is then used as a new stylization model, and the training process is executed again. If the probability that the stylized image's style is the target style is greater than or equal to the probability threshold, the training process ends.

[0115] Alternatively, the generative model can be a diffusion model, an autoregressive model, or a generative adversarial network.

[0116] Alternatively, the image processing device may have pre-set stylization generation models corresponding to multiple styles, including the target style; the second acquisition module 802 is used to: output multiple style options corresponding to multiple styles; and in response to receiving a selection instruction for the style option corresponding to the target style, use the stylization generation model corresponding to the target style as the target stylization generation model.

[0117] Optionally, the adjustment module 808 is used to adjust the size of the redrawn element so that the size of the redrawn element is consistent with the size of the element to be redrawn. The generation module 804 is used to replace the element to be redrawn in the original image with the redrawn element to obtain the target image.

[0118] The first acquisition module 801, the second acquisition module 802, the redrawing module 803, the generation module 804, the feature extraction module 805, the storyboard processing module 806, the content recognition module 807, and the adjustment module 808 can all be implemented in software or in hardware. For example, the implementation of the first acquisition module 801 will be described below. Similarly, the implementation of the second acquisition module 802, the redrawing module 803, the generation module 804, the feature extraction module 805, the storyboard processing module 806, the content recognition module 807, and the adjustment module 808 can refer to the implementation of the first acquisition module 801.

[0119] As an example of a software functional unit, the first acquisition module 801 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the aforementioned computing instance may be one or more. For example, the first acquisition module 801 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0120] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0121] As an example of a hardware functional unit, the first acquisition module 801 may include at least one computing device, such as a server. Alternatively, the first acquisition module 801 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0122] The multiple computing devices included in the first acquisition module 801 can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in the first acquisition module 801 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the first acquisition module 801 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0123] It should be noted that, in other embodiments, the first acquisition module 801 can be used to execute any step in the image processing method, the second acquisition module 802 can be used to execute any step in the image processing method, the redrawing module 803 can be used to execute any step in the image processing method, the generation module 804 can be used to execute any step in the image processing method, the feature extraction module 805 can be used to execute any step in the image processing method, the storyboard processing module 806 can be used to execute any step in the image processing method, the content recognition module 807 can be used to execute any step in the image processing method, and the adjustment module 808 can be used to execute any step in the image processing method.

[0124] The steps implemented by the first acquisition module 801, the second acquisition module 802, the redrawing module 803, the generation module 804, the feature extraction module 805, the storyboard processing module 806, the content recognition module 807, and the adjustment module 808 can be specified as needed. The first acquisition module 801, the second acquisition module 802, the redrawing module 803, the generation module 804, the feature extraction module 805, the storyboard processing module 806, the content recognition module 807, and the adjustment module 808 respectively implement different steps in the image processing method to realize all the functions of the image processing device.

[0125] The hardware devices of the embodiments of this application are illustrated below.

[0126] For example, Figure 9 is a schematic diagram of a computing device 900 provided in an embodiment of this application. As shown in Figure 9, the computing device 900 includes: a bus 902, a processor 904, a memory 906, and a communication interface 908. The processor 904, the memory 906, and the communication interface 908 communicate with each other via the bus 902. The computing device 900 can be a server or a terminal device. It should be understood that the embodiments of this application do not limit the number of processors and memories in the computing device 900.

[0127] Bus 902 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one line is used in Figure 9, but this does not imply that there is only one bus or one type of bus. Bus 902 can include pathways for transmitting information between various components of computing device 900 (e.g., memory 906, processor 904, communication interface 908).

[0128] Processor 904 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0129] Memory 906 may include volatile memory, such as random access memory (RAM). Memory 906 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0130] The memory 906 stores executable program code, and the processor 904 executes the executable program code to implement the functions of the aforementioned first acquisition module, second acquisition module, redrawing module, and generation module, respectively, thereby implementing the image processing method described above, such as method 100. That is, the memory 906 stores instructions for executing the image processing method.

[0131] Alternatively, the memory 906 stores executable code, which the processor 904 executes to implement the functions of the aforementioned image processing device 800, thereby implementing the image processing method. That is, the memory 906 stores instructions for executing the image processing method.

[0132] The communication interface 908 uses transceiver modules, such as, but not limited to, network interface cards and transceivers, to enable communication between the computing device 900 and other devices or communication networks.

[0133] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0134] For example, Figure 10 is a schematic diagram of a computing device cluster provided in an embodiment of this application. As shown in Figure 10, the computing device cluster includes at least one computing device 900. The memory 906 in one or more computing devices 900 in the computing device cluster may store the same instructions for executing image processing methods.

[0135] In some possible implementations, the memory 906 of one or more computing devices 900 in the computing device cluster may also store partial instructions for executing the image processing method. In other words, a combination of one or more computing devices 900 can jointly execute the instructions for executing the image processing method.

[0136] It should be noted that the memory 906 in different computing devices 900 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the image processing device. That is, the instructions stored in the memory 906 of different computing devices 900 can implement the functions of one or more of the aforementioned modules, including but not limited to the first acquisition module, the second acquisition module, the redrawing module, and the generation module.

[0137] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. For example, Figure 11 is a schematic diagram of another computing device cluster structure provided in an embodiment of this application. As shown in Figure 11, two computing devices 900A and 900B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this type of possible implementation, the memory 906 in computing device 900A stores instructions for executing the functions of a first acquisition module and a second acquisition module. Simultaneously, the memory 906 in computing device 900B stores instructions for executing the functions of a redrawing module and a generation module.

[0138] The connection method between the computing device clusters shown in Figure 11 can be considered in the case that the image processing method provided in this application embodiment needs to transmit and calculate data, so the functions implemented by the redrawing module 803 and the generation module 804 are to be executed by the computing device 900B.

[0139] It should be understood that the functions of computing device 900A shown in Figure 11 can also be performed by multiple computing devices 900. Similarly, the functions of computing device 900B can also be performed by multiple computing devices 900.

[0140] This application also provides another computing device cluster. The connection relationship between the computing devices in this computing device cluster can be similarly referred to the connection method of the computing device cluster shown in Figures 10 and 11. The difference is that the memory 906 of one or more computing devices 900 in this computing device cluster can store the same instructions for executing image processing methods.

[0141] In some possible implementations, the memory 906 of one or more computing devices 900 in the computing device cluster may also store partial instructions for executing the image processing method. In other words, a combination of one or more computing devices 900 can jointly execute the instructions for executing the image processing method.

[0142] It should be noted that the memory 906 in different computing devices 900 within the computing device cluster can store different instructions for executing some functions of the image processing device. That is, the instructions stored in the memory 906 of different computing devices 900 can implement the functions of the image processing device.

[0143] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform an image processing method.

[0144] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of being stored by a computing device, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to perform an image processing method.

[0145] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

Claims

1. An image processing method, characterized by, The method comprises: obtaining an element to be redrawn in an original image, the element to be redrawn comprising foreground imaging and / or background imaging; obtaining a target stylization generation model, the target stylization generation model being trained based on sample images labeled with a target style; using guide information to guide the target stylization generation model to redraw the element to be redrawn to obtain a redrawn element of the target style, the guide information comprising control information of the element to be redrawn, the control information comprising imaging features of the element to be redrawn in the original image; generating a target image according to the redrawn element and the original image.

2. The method of claim 1, wherein, The control information comprises one or more of a line drawing, a depth, or a normal of the element to be redrawn in the original image.

3. The method according to claim 1 or 2, characterized in that, The method further comprises: extracting imaging features of the element to be redrawn from the original image to obtain the control information.

4. The method according to any one of claims 1 to 3, characterized in that, The imaging features of the element to be redrawn in the original image comprise one or more of a space, a position, a color, a contour, or a texture of the element to be redrawn in the original image.

5. The method according to any one of claims 1 to 4, characterized in that, The guide information further comprises image content description information corresponding to the original image, the image content description information being used to describe semantics expressed by the original image.

6. The method of claim 5, wherein, The original image is from a video, and the method further comprises: performing a shot processing on the video to obtain a plurality of video segments; for each video segment in the plurality of video segments, performing video content recognition on the video segment to obtain video content description information corresponding to the video segment, and taking the video content description information as image content description information corresponding to each frame of image in the video segment.

7. The method according to any one of claims 1 to 6, characterized in that, The method comprises: training an initial stylization generation model using a training sample set to obtain the target stylization generation model, the training sample set comprising one or more sample images labeled with the target style.

8. The method according to any one of claims 1 to 6, characterized in that, The method is applied to an image processing device, and a plurality of initial stylization generation models are pre-stored in the image processing device, and the obtaining of the target stylization generation model comprises: training the plurality of initial stylization generation models respectively using a training sample set to obtain a plurality of candidate stylization generation models, the training sample set comprising one or more sample images labeled with the target style; obtaining the target stylization generation model from the plurality of candidate stylization generation models.

9. The method of claim 8, wherein, The method comprises: inputting a test image into each of the plurality of candidate stylization generation models to obtain a plurality of predicted images output by the plurality of candidate stylization generation models; judging image quality of the plurality of predicted images respectively using an aesthetic score algorithm; outputting one or more selected images with the best image quality from the plurality of predicted images; in response to receiving a selection instruction for any of the one or more selected images, taking a candidate stylization generation model used to generate the selected image indicated by the selection instruction as the target stylization generation model.

10. The method according to any one of claims 7 to 9, characterized in that, The initial stylization generation model comprises a generative model and a stylization model, and the stylization model is configured to stylize an output result of the generative model to change the style of the output result of the generative model. The initial stylization generation model is trained by using the training sample set, comprising: The training process is performed on the stylization model by using the training sample set, wherein the training process comprises: The sample image is input into the generative model and the stylization model respectively to obtain a stylized image output by the initial stylization generation model, the stylized image being generated based on a base image and style data, the base image being output by the generative model according to the input sample image, and the style data being output by the stylization model according to the input sample image; If it is determined that the probability that the image style of the stylized image is the target style is less than a probability threshold, the model parameter of the stylization model is adjusted in a back propagation manner according to an error loss between the stylized image and the sample image, and the stylization model after the model parameter adjustment is taken as a new stylization model, and the training process is performed again; If it is determined that the probability that the image style of the stylized image is the target style is greater than or equal to the probability threshold, the training process is ended.

11. The method of claim 10, wherein, The generative model is a diffusion model, an autoregressive model or a generative adversarial network.

12. The method according to any one of claims 1 to 6, characterized in that, The method is applied to an image processing device, and a plurality of stylization generation models corresponding to a plurality of styles are pre-stored in the image processing device, the plurality of styles including the target style; and the target stylization generation model is obtained by: outputting a plurality of style options corresponding to the plurality of styles; in response to receiving a selection instruction for a style option corresponding to the target style, taking the stylization generation model corresponding to the target style as the target stylization generation model.

13. The method of any one of claims 1 to 12, wherein, The method further comprises: adjusting the size of the redrawn element to make the size of the redrawn element consistent with the size of the element to be redrawn; generating a target image according to the redrawn element and the original image, comprising: covering and replacing the element to be redrawn in the original image with the redrawn element to obtain the target image.

14. An image processing apparatus characterized by comprising: The image processing device comprises: a first obtaining module configured to obtain an element to be redrawn in an original image, the element to be redrawn comprising foreground imaging and / or background imaging; a second obtaining module configured to obtain a target stylization generation model, the target stylization generation model being trained based on a sample image labeled with a target style; a redrawing module configured to guide the target stylization generation model to redraw the element to be redrawn by using guide information to obtain a redrawn element of the target style, the guide information comprising control information of the element to be redrawn, and the control information comprising one or more of the following: a sketch, a depth or a normal of the element to be redrawn in the original image; a generating module configured to generate a target image according to the redrawn element and the original image.

15. The image processing device according to claim 14, characterized by The control information comprises one or more of the following: a sketch, a depth or a normal of the element to be redrawn in the original image.

16. The image processing device according to claim 14 or 15, characterized by The image processing device also includes a feature extraction module; The feature extraction module is used to extract the imaging features of the element to be redrawn from the original image to obtain the control information.

17. The image processing device according to any one of claims 14 to 16, characterized by, The imaging features of the element to be redrawn in the original image include one or more of the space, position, color, outline, or texture of the element to be redrawn in the original image.

18. The image processing device according to any one of claims 14 to 17, characterized by, The guidance information also includes image content description information corresponding to the original image, which is used to describe the semantics expressed by the original image.

19. The image processing device according to claim 18, characterized by The original image comes from video, and the image processing device further includes a scene processing module and a content recognition module; The segmentation processing module is used to perform segmentation processing on the video to obtain multiple video segments; The content recognition module is used to perform video content recognition on each of the plurality of video segments to obtain video content description information corresponding to the video segment, and to use the video content description information as image content description information corresponding to each frame of the video segment.

20. The image processing device according to any one of claims 14 to 19, characterized by, The second acquisition module is used for: The initial stylization generation model is trained using a training sample set to obtain the target stylization generation model. The training sample set includes one or more sample images labeled with the target style.

21. The image processing device according to any one of claims 14 to 19, characterized by, The image processing device has multiple pre-set initial stylization generation models, and the second acquisition module is used for: The multiple initial stylization generation models are trained using a training sample set to obtain multiple candidate stylization generation models. The training sample set includes one or more sample images labeled with the target style. The target stylization generation model is obtained from the plurality of candidate stylization generation models.

22. The image processing device according to claim 21, characterized by The second acquisition module is used for: The test image is input into the multiple candidate stylization generation models respectively to obtain multiple predicted images output by the multiple candidate stylization generation models; An aesthetic scoring algorithm is used to evaluate the image quality of each of the multiple predicted images. Output the one or more candidate images with the best image quality among the multiple predicted images; In response to receiving a selection instruction for any of the one or more candidate images, the candidate stylization generation model used to generate the candidate image indicated by the selection instruction is taken as the target stylization generation model.

23. The image processing device according to any one of claims 20 to 22, characterized by The initial stylization generation model includes a generative model and a stylization model. The stylization model is used to stylize the output of the generative model in order to change the style of the output of the generative model. The second acquisition module is used for: The stylization model is trained using the training sample set, wherein the training process includes: The sample images are input into the generative model and the stylization model respectively to obtain the stylized image output by the initial stylization generative model. The stylized image is generated based on the base image and style data. The base image is output by the generative model based on the input sample images, and the style data is output by the stylization model based on the input sample images. If the probability that the image style of the stylized image is the target style is less than the probability threshold, the stylized model is adjusted by backpropagation based on the error loss between the stylized image and the sample image, and the stylized model after the model parameter adjustment is used as the new stylized model, and the training process is executed again. If the probability that the stylized image style is the target style is greater than or equal to the probability threshold, the training process ends.

24. The image processing device according to claim 23, characterized by The generative model is a diffusion model, an autoregressive model, or a generative adversarial network.

25. The image processing apparatus according to any one of claims 14 to 19, characterized by The image processing device has pre-set stylization generation models corresponding to multiple styles, including the target style; the second acquisition module is used for: Output multiple style options corresponding to the various styles; In response to receiving a selection instruction for a style option corresponding to the target style, the stylization generation model corresponding to the target style is used as the target stylization generation model.

26. The image processing device according to any one of claims 14 to 25, characterized by, The image processing device also includes an adjustment module; The adjustment module is used to adjust the size of the redrawn element so that the size of the redrawn element is consistent with the size of the element to be redrawn; The generation module is used to replace the element to be redrawn in the original image with the redrawn element to obtain the target image.

27. A cluster of computing devices, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1 to 13.

28. A computer program product comprising instructions, wherein: When the instruction is executed by the computing device cluster, the computing device cluster causes the computing device cluster to perform the method as described in any one of claims 1 to 13.

29. A computer-readable storage medium, characterized in that, Includes computer program instructions, which, when executed by a cluster of computing devices, perform the method as described in any one of claims 1 to 13.

Images