Image processing model training method, image processing method, video processing model training method and video processing method

The image processing model is trained through the posture characteristics of multiple reference images, and the problem of poor details and quality when generating portrait videos in a single reference image is solved, and accurate video generation is achieved from multiple perspectives.

CN118982592APending Publication Date: 2024-11-19ALIBABA (CHINA) CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202410891514.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-03
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

When the prior art uses a single reference image to generate portrait videos, the video details are poor and the quality is poor. Especially when generating multiple perspectives, it is necessary to randomly generate or speculate that content without viewing angles does not exist, resulting in a decline in video quality.

Method used

By determining multiple reference images from different perspectives of the target object, the reference pose images and the target pose images of each reference image are obtained, and the image processing model is trained using reference network features and target pose features to generate accurate predicted images to avoid invisible areas random generation.

Benefits of technology

Improve the details and quality of video generation, ensure accurate restoration of every detail of the target object in the predicted image, and improve the accuracy of generation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118982592A_ABST
    Figure CN118982592A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides an image processing model training method, an image processing method, a video processing model training method and a video processing method. The image processing model training method comprises the following steps: determining a plurality of reference images of a target object under different visual angles, a reference attitude image of the target object in each reference image, a target image of the target object, a target attitude image of the target object in the target image, and a noise image corresponding to the target image; obtaining reference image features of the reference images, reference attitude features of the reference attitude images and target attitude features of the target attitude image; obtaining a reference network feature according to the reference image feature and the reference attitude feature, and obtaining a target prediction feature according to the reference network feature, the target attitude feature and the noise image; obtaining a prediction image according to the target prediction feature; according to the prediction image and the target image, training an image processing model to obtain a trained image processing model; and the image generation quality is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of this specification relate to the field of computer technology, and in particular to an image processing model training method, an image processing method, a video processing model training method, and a video processing method. Background Art

[0002] When using a machine learning model to generate portrait videos, it usually only supports the input of a single reference image. However, when using a single reference image to generate portrait videos, only a single-perspective video can be generated. Or when generating multi-perspective videos, for perspectives that do not exist in the reference image, a machine learning model is required to randomly generate or infer the image content corresponding to the perspective.

[0003] As a result, the generated videos usually have poor details and overall poor video quality. Summary of the invention

[0004] In view of this, an embodiment of this specification provides an image processing model training method. One or more embodiments of this specification also relate to another image processing model training method, an image processing method, a video processing model training method, another video processing model training method, a video processing method, a computing device, a computer-readable storage medium, and a computer program product to solve the technical defects of poor video details and poor video quality existing in the prior art.

[0005] According to a first aspect of an embodiment of this specification, there is provided an image processing model training method, comprising:

[0006] Determine a plurality of reference images of a target object at different viewing angles, a reference pose image of the target object in each reference image, a target image of the target object, a target pose image of the target object in the target image, and a noise image corresponding to the target image, wherein the pose of the target object in each reference image is different from that in the target image;

[0007] Acquire reference image features of each reference image, reference posture features of each reference posture image, and target posture features of the target posture image;

[0008] Obtaining reference network features according to the reference image features and the reference posture features, and obtaining target prediction features according to the reference network features, the target posture features and the noise image;

[0009] Obtaining a predicted image according to the target prediction feature, wherein the posture of the target object in the predicted image is determined by referring to the posture of the target object in the target posture image;

[0010] The image processing model is trained according to the predicted image and the target image to obtain a trained image processing model.

[0011] According to a second aspect of an embodiment of this specification, there is provided an image processing model training device, comprising:

[0012] an image determination module, configured to determine a plurality of reference images of a target object at different viewing angles, a reference posture image of the target object in each reference image, a target image of the target object, a target posture image of the target object in the target image, and a noise image corresponding to the target image, wherein the posture of the target object in each reference image is different from that in the target image;

[0013] A feature acquisition module, configured to acquire reference image features of each reference image, reference posture features of each reference posture image, and target posture features of the target posture image;

[0014] A feature acquisition module, configured to obtain a reference network feature according to the reference image feature and the reference posture feature, and to obtain a target prediction feature according to the reference network feature, the target posture feature and the noise image;

[0015] an image acquisition module, configured to acquire a predicted image according to the target prediction feature, wherein the posture of the target object in the predicted image is determined by referring to the posture of the target object in the target posture image;

[0016] The model training module is configured to train the image processing model according to the predicted image and the target image to obtain a trained image processing model.

[0017] According to a third aspect of an embodiment of this specification, another image processing model training method is provided, including:

[0018] Determine a plurality of reference images of a target object at different viewing angles, a reference posture image of the target object in each reference image, a target image of the target object, a target posture image of the target object in the target image, and a noise image corresponding to the target image, and input the plurality of reference images, each reference posture image, the target posture image, and the noise image into an image processing model, wherein the posture of the target object in each reference image is different from that in the target image, and the image processing model includes a feature extraction unit, a reference network unit, a denoising network unit, and a prediction unit;

[0019] Utilizing the feature extraction unit, obtaining reference image features of each reference image, reference posture features of each reference posture image, and target posture features of the target posture image;

[0020] Using the reference network unit, according to the reference image feature and the reference posture feature, a reference network feature is obtained, and using the denoising network unit, according to the reference network feature, the target posture feature and the noise image, a target prediction feature is obtained;

[0021] Using the prediction unit, obtaining a prediction image according to the target prediction feature, wherein the posture of the target object in the prediction image is determined by referring to the posture of the target object in the target posture image;

[0022] The image processing model is trained according to the predicted image and the target image to obtain a trained image processing model.

[0023] According to a fourth aspect of the embodiments of this specification, a video processing model training method is provided, including:

[0024] Determine a plurality of reference images of a target object at different viewing angles, a reference pose image of the target object in each reference image, a target video of the target object, a target pose image sequence of the target object in the target video, and a noise image sequence corresponding to the target video;

[0025] Acquire reference image features of each reference image, reference posture features of each reference posture image, and a target posture feature sequence of the target posture image sequence;

[0026] Obtaining reference network features according to the reference image features and the reference posture features, obtaining a candidate prediction feature sequence according to the reference network features, the target posture feature sequence and the noise image sequence, and obtaining a target prediction feature sequence according to the candidate prediction feature sequence and the temporal relationship between the candidate prediction features;

[0027] Obtaining a predicted video according to the target prediction feature sequence, wherein the posture of the target object in the predicted video is determined by referring to the posture of the target object in the target video;

[0028] The video processing model is trained according to the predicted video and the target video to obtain a trained video processing model, wherein the video processing model is determined based on the trained image processing model in the above-mentioned image processing model training method.

[0029] According to a fifth aspect of the embodiments of this specification, a video processing model training device is provided, including:

[0030] An image determination module is configured to determine a plurality of reference images of a target object at different viewing angles, a reference posture image of the target object in each reference image, a target video of the target object, a target posture image sequence of the target object in the target video, and a noise image sequence corresponding to the target video;

[0031] A feature acquisition module, configured to acquire reference image features of each reference image, reference posture features of each reference posture image, and a target posture feature sequence of the target posture image sequence;

[0032] A feature acquisition module is configured to obtain a reference network feature according to the reference image feature and the reference posture feature, obtain a candidate prediction feature sequence according to the reference network feature, the target posture feature sequence and the noise image sequence, and obtain a target prediction feature sequence according to the candidate prediction feature sequence and the temporal relationship between the candidate prediction features;

[0033] A video acquisition module is configured to acquire a predicted video according to the target prediction feature sequence, wherein the posture of the target object in the predicted video is determined by referring to the posture of the target object in the target video;

[0034] The model training module is configured to train the video processing model according to the predicted video and the target video to obtain a trained video processing model, wherein the video processing model is determined based on the trained image processing model in the above-mentioned image processing model training method.

[0035] According to a sixth aspect of the embodiments of this specification, another video processing model training method is provided, including:

[0036] Determine multiple reference images of the target object under different viewing angles, reference posture images of the target object in each reference image, a target video of the target object, a target posture image sequence of the target object in the target video, and a noise image sequence corresponding to the target video, and input the multiple reference images, each reference posture image, the target posture image sequence, and the noise image sequence into a video processing model, wherein the video processing model is determined based on the trained image processing model in the above-mentioned image processing model training method, and the video processing model includes a feature extraction unit, a reference network unit, a denoising network unit, and a prediction unit;

[0037] Utilizing the feature extraction unit, obtaining reference image features of each reference image, reference posture features of each reference posture image, and a target posture feature sequence of the target posture image sequence;

[0038] Using the reference network unit, performing attention processing on the reference image features and the reference posture features to obtain reference network features, and using the denoising network unit, performing spatial attention and cross attention processing on the reference network features, the target posture feature sequence and the noise image sequence to obtain a candidate prediction feature sequence, and using the denoising network unit, performing temporal attention processing on the candidate prediction feature sequence to obtain a target prediction feature sequence;

[0039] Using the prediction unit, according to the target prediction feature sequence, a predicted video is obtained, wherein the posture of the target object in the predicted video is determined by referring to the posture of the target object in the target video;

[0040] The video processing model is obtained by training according to the predicted video and the target video.

[0041] According to a seventh aspect of the embodiments of this specification, there is provided an image processing method, including:

[0042] Determine a reference image and a target image, obtain a reference pose image of a reference object in the reference image based on the reference image, and obtain a target pose image of a target object in the target image and a noise image corresponding to the target image based on the target image;

[0043] According to the reference image, the reference posture image, the target posture image and the noise image, an image processing model is used to generate an image to obtain a generated image, wherein the image processing model is trained by the above-mentioned image processing model training method, the appearance of the object in the generated image is determined by the appearance of the reference object, and the action posture of the object in the generated image is determined by the action posture of the target object.

[0044] According to an eighth aspect of the embodiments of this specification, a video processing method is provided, including:

[0045] Determine a reference image and a target video, obtain a reference pose image of a reference object in the reference image based on the reference image, and obtain a target pose sequence image of a target object in the target video and a noise sequence image corresponding to the target video based on the target video;

[0046] According to the reference image, the reference posture image, the target posture sequence image and the noise sequence image, a video processing model is used to generate a video to obtain a generated video, wherein the video processing model is trained by the above-mentioned video processing model training method, the appearance of the object in the generated video is determined by the appearance of the reference object, and the action posture of the object in the generated video is determined by the action posture of the target object.

[0047] According to a ninth aspect of the embodiments of this specification, a computing device is provided, including:

[0048] Memory and processor;

[0049] The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor, the steps of the above-mentioned image processing model training method, video processing model training method, image processing method, and video processing method are implemented.

[0050] According to the tenth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores a computer program / instruction, and when the computer program / instruction is executed by a processor, the steps of the above-mentioned image processing model training method, video processing model training method, image processing method, and video processing method are implemented.

[0051] According to the eleventh aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned image processing model training method, video processing model training method, image processing method, and video processing method.

[0052] An image processing model training method provided by an embodiment of the present specification can accurately capture the image of the target object from multiple perspectives by using multiple reference images at different perspectives, avoiding the random generation of invisible areas, and obtain reference network features related to the reference image by obtaining reference image features of each reference image and reference posture features of each reference posture image. The reference network features and target posture features of the target posture image can be used to guide the process of restoring an image from a noise image in the image processing model, thereby obtaining accurate target prediction features, that is, the target prediction features include features related to the reference image in the reference image features and target posture features of the target posture image. On the basis of obtaining the predicted image based on the target prediction features, the accuracy of the generated predicted image is improved, and by accurately capturing the image of the target object from multiple perspectives, each detail of the target object can be restored in the predicted image with reference to the image of the target object at each perspective, thereby ensuring the generation quality of the target object in the predicted image. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is a scene schematic diagram of an image processing model training method provided by an embodiment of this specification;

[0054] Figure 2 is a flowchart of an image processing model training method provided by an embodiment of this specification;

[0055] Figure 3 is a flowchart of another image processing model training method provided by an embodiment of this specification;

[0056] Figure 4 It is a network framework diagram of an image processing model provided by an embodiment of this specification;

[0057] Figure 5 is a training schematic diagram of an image processing model training method provided by an embodiment of this specification;

[0058] Figure 6 is a flow chart of an image processing method provided by an embodiment of this specification;

[0059] Figure 7 is a flow chart of a video processing model training method provided by an embodiment of this specification;

[0060] Figure 8 is a flowchart of another video processing model training method provided by an embodiment of this specification;

[0061] Fig. 9 is a network framework diagram of a video processing model provided by an embodiment of this specification;

[0062] Fig.10 is a training schematic diagram of a video processing model training method provided by an embodiment of this specification;

[0063] Fig.11 is a flow chart of a video processing method provided by an embodiment of this specification;

[0064] Fig.12 It is a structural schematic diagram of an image processing model training device provided by an embodiment of this specification;

[0065] Fig.13 It is a structural schematic diagram of a video processing model training device provided by an embodiment of this specification;

[0066] Fig.14 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0067] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0068] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0069] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0070] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0071] First, the terms involved in one or more embodiments of this specification are explained.

[0072] Diffus ion model: a generative model based on the diffusion process.

[0073] Unet network structure: A symmetrical U-shaped network structure, commonly used in deep learning.

[0074] CLIP (Contrastive Language-Image Pre-Training) model: a pre-trained neural network model, specifically a picture-text correlation matching model.

[0075] Among the existing methods, some researchers have tried to use the idea of ​​3D reconstruction from multi-view images to complete portrait video generation. However, this requires precise camera parameters and a large number of multi-view images (more than 5). In addition, the overall realism of the rendered output image is poor when using the idea of ​​3D reconstruction and then neural rendering.

[0076] In this specification, an image processing model training method is provided. This specification also involves another image processing model training method, an image processing method, a video processing model training method, another video processing model training method, a video processing method, a computing device, a computer-readable storage medium and a computer program product, which are described in detail one by one in the following embodiments.

[0077] See also Figure 1 , Figure 1 A schematic diagram of a scenario of an image processing model training method provided according to an embodiment of the present specification is shown.

[0078] Specifically, the image processing model training method is implemented by applying the terminal device 102 and the server 104. The terminal device 102 is used to send multiple reference images of the target object at different viewing angles and the target image of the target object to the server 104. Of course, when a posture estimation model and an image denoising model are deployed on the client, the posture estimation model is used to obtain the reference posture image of the target object in each reference image and the target posture image of the target object in the target image, and the image denoising model is used to obtain the noise image corresponding to the target image. Thus, the terminal device 102 can be used to send multiple reference images of the target object at different viewing angles, multiple reference posture images, target posture images and noise images to the server 104. This specification does not impose any restrictions on this.

[0079] An image processing model is trained in the server 104, and the image processing model includes an image feature extraction unit, a posture feature extraction unit, a reference network unit, and a denoising network unit; when the server 104 receives multiple reference images, multiple reference posture images, a target posture image, and a noise image sent by the terminal device 102, the multiple reference images are input into the image feature extraction unit to obtain reference image features of each reference image, the reference posture features of each reference posture image and the target posture features of the target posture image are obtained by using the posture feature extraction unit, the reference image features and the reference posture features are processed by the reference network unit to obtain reference network features, and the reference network features, the target posture features, and the noise image are processed by the denoising network unit to obtain target prediction features; a prediction image is obtained according to the target prediction features, and the image processing model is trained according to the prediction image and the target image. When the trained image processing model is obtained, the trained image processing model can be deployed to the terminal device 102, or the model interface information corresponding to the trained image processing model is returned to the terminal device 102, so that the terminal device 102 can call the trained image processing model through the model interface information.

[0080] The end-side device 102 may include a browser, an APP (Application), or a web application such as an H5 (Hyper Text Markup Language 5, Hypertext Markup Language Version 5) application, or a light application (also known as a mini-program, a lightweight application) or a cloud application, etc. The end-side device may be based on the software development kit (SDK, Software Development Kit) of the corresponding service provided by the server, such as based on the real-time communication (RTC, Real Time Communication) SDK development and acquisition, etc. The end-side device may be deployed in an electronic device and needs to rely on the device to run or some APPs in the device to run, etc. The electronic device may have a display screen and support information browsing, etc., such as a personal mobile terminal such as a mobile phone, a tablet computer, a personal computer, etc. Various other types of applications may also be configured in the electronic device, such as human-computer dialogue applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0081] Server 104 can be understood as a server that provides various services, including physical servers and cloud servers, such as a server that provides communication services to multiple clients, a server for background training that provides support for models used on clients, and a server that processes data sent by clients. It should be noted that server 104 can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. Server 104 can also be a server for a distributed system, or a server combined with a blockchain. Server 104 can also be a cloud server for basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (CDN, Content De l ivery Network), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0082] It is worth noting that the image processing model training method provided in the embodiments of this specification can be executed by the server 104. In other embodiments of this specification, the image processing model can be deployed in the terminal device 102, so that the terminal device 102 can also have similar functions as the server 104, thereby executing the image processing model training method provided in the embodiments of this specification; in other embodiments, the image processing model training method provided in the embodiments of this specification can also be executed jointly by the terminal device 102 and the server 104.

[0083] An image processing model training method provided by an embodiment of the present specification can accurately capture the image of the target object from multiple perspectives by using multiple reference images at different perspectives, avoiding the random generation of invisible areas, and obtain reference network features related to the reference image by obtaining reference image features of each reference image and reference posture features of each reference posture image. The reference network features and target posture features of the target posture image can be used to guide the process of restoring an image from a noise image in the image processing model, thereby obtaining accurate target prediction features, that is, the target prediction features include features related to the reference image in the reference image features and target posture features of the target posture image. On the basis of obtaining the predicted image based on the target prediction features, the accuracy of the generated predicted image is improved, and by accurately capturing the image of the target object from multiple perspectives, each detail of the target object can be restored in the predicted image with reference to the image of the target object at each perspective, thereby ensuring the generation quality of the target object in the predicted image.

[0084] See also Figure 2 , Figure 2A flowchart of an image processing model training method provided by an embodiment of the present specification is shown, which specifically includes the following steps.

[0085] Step 202: Determine multiple reference images of the target object at different viewing angles, reference posture images of the target object in each reference image, a target image of the target object, a target posture image of the target object in the target image, and a noise image corresponding to the target image, wherein the posture of the target object in each reference image is different from that of the target object in the target image.

[0086] Among them, the target object can be understood as an object that can display a variety of postures or gestures, such as a human or a fictional animated character; multiple perspectives can be understood as the sight angles of viewing the target object from multiple different specific positions or directions, and the multiple perspectives include but are not limited to the front perspective, the left perspective, the right perspective, and the back perspective; the reference image can be understood as an image used as a reference in the image generation process, and the reference image includes but is not limited to providing an image background reference, providing a target object's appearance reference, etc. The reference image in the embodiments of this specification is used to provide a target object's appearance reference in image generation.

[0087] The target image can be understood as the generation target of the image processing model. The target image is the label image corresponding to the predicted image output by the image processing model, and the target image provides a posture reference for the target object during the image generation process. For example, if the posture of the target object in the target image is "hands on hips", then the posture of the target object in the generated predicted image should also be "hands on hips".

[0088] The reference posture image and the target posture image can be understood as posture images obtained by performing posture detection on the target objects in the reference image and the target image, which contain the action posture of the target object but not the appearance of the target object; and the target posture image can be used as a control signal in the image generation process to make the posture of the target object in the generated predicted image consistent with the posture in the target posture image.

[0089] The noisy image can be understood as an image obtained by adding noise to a target image. For example, a noisy image is obtained by adding random Gaussian noise to a target image using a DDIM model (Denoising Diffusion Implicit Models, an algorithm model for generating high-fidelity images).

[0090] Specifically, multiple reference images of the target object at different viewing angles are determined to obtain the appearance of the target object from multiple perspectives, and a label image corresponding to the generated image is obtained by determining the target image. In addition, although the target object in the reference image and the target image has implicit posture, the posture and the appearance of the target object are coupled here. The purpose of obtaining the reference posture image and the target posture image separately is to decouple the appearance of the target object from the action posture, which helps to enhance the consistency between the generated result and the input posture.

[0091] The purpose of obtaining a noise image corresponding to a target image is to use the noise image as the input of an image processing model, perform denoising on the noise image, and combine the appearance of the target object in the reference image and the action posture of the target object in the target image during the denoising process to generate an image of the appearance of the target object in the reference image and the action posture of the target object in the target image.

[0092] In one or more embodiments of the present specification, a reference image and a target image may be first determined, and on this basis, a reference pose image, a target pose image, and a noise image may be accurately and efficiently obtained using a machine learning model, and the specific implementation method is as follows:

[0093] The step of determining a plurality of reference images of the target object at different viewing angles, a reference posture image of the target object in each reference image, a target image of the target object, a target posture image of the target object in the target image, and a noise image corresponding to the target image includes:

[0094] Determine a plurality of reference images of the target object at different viewing angles and a target image of the target object;

[0095] Inputting the plurality of reference images and the target image into a pose estimation model respectively to obtain a reference pose image of the target object in each of the reference images and a target pose image of the target object in the target image;

[0096] The target image is input into an image denoising model to obtain a noise image corresponding to the target image, wherein the posture estimation model and the image denoising model are machine learning models.

[0097] The posture estimation model is used to identify and extract the action posture of the object in the input image, thereby generating a posture image that does not contain the appearance features of the object but contains the action posture of the object. The image noise addition model is used to add noise to the input image, thereby obtaining a noise image corresponding to the input image, such as by adding random Gaussian noise to the input image to obtain a noise image corresponding to the input image.

[0098] For example, when the target object is a human, a posture estimation model is used, such as ViTPose (Vision Transformer Pose, a human posture estimation method), to perform human key point detection on the person in the reference image, and mark the positions of the person's head, shoulders, hands, knees, feet, etc., so as to obtain the corresponding reference posture image. Similarly, a target posture image corresponding to the target image is obtained, and an image denoising model is used, such as the DDIM model, to perform a denoising process on the target image, and random Gaussian noise is added to obtain a noise image corresponding to the target image.

[0099] In practical applications, the posture estimation model can realize the recognition and extraction of the action posture of the object in the input image through different types of posture detectors. For example, the posture detector can be a two-dimensional skeleton key point detector, which is used to detect the skeleton key points and obtain the posture image of the object by connecting the skeleton key points; or the posture detector can also be a three-dimensional action detector, which can fit the three-dimensional posture of the object in the input image, and then obtain the posture image of the object by rendering.

[0100] It should be noted that the purpose of obtaining the reference posture image and the target posture image in the embodiments of this specification is to obtain the posture information of the target object (the posture information describes the movement posture of the target object in space, that is, how the target object locates its body parts, joints, and overall shape and direction information), and obtaining the reference posture image and the target posture image is actually a specific way to obtain the posture information of the target object. With the development of technology, it is not limited to using posture images to obtain the posture information of the target object, and other technical means can be used to obtain the posture information of the target object.

[0101] The image processing model training method provided in the embodiments of this specification can quickly and accurately obtain a reference pose avatar, a target pose image, and a noise image based on the determination of a reference image and a target image by using a pose estimation model and an image denoising model.

[0102] In one or more embodiments of the present specification, reference images at different viewing angles may be obtained in a variety of implementation methods, such as from a reference video of the target object, or directly by photographing the target object at different viewing angles. The specific implementation methods are as follows:

[0103] The step of determining a plurality of reference images of the target object at different viewing angles includes:

[0104] Determine a reference video of the target object, and obtain, from the reference video, a plurality of video frames of the target object at different viewing angles as reference images; or

[0105] A plurality of reference images of the target object at different viewing angles are captured.

[0106] Specifically, multiple reference images of the target object at different perspectives can be obtained through a variety of implementation methods, for example, from a reference video of the target object, multiple video frames of the target object at different perspectives are obtained, and the multiple video frames are determined as multiple reference images, or multiple reference images of the target object at different perspectives are obtained by directly shooting.

[0107] When the target object is a fictional animated character, multiple reference images of the target object from different perspectives can be obtained through the design drawing of the animated character. The design drawing of an animated character usually includes a front view of the character (showing the character's facial expression, hairstyle, clothing details and body proportions), a side view of the character (emphasizing the head outline, the height of the nose, the thickness of the body and the side layering of the clothing), and a back view of the character (showing the character's back design, including the back style of the hairstyle and the details of the back of the clothing). Therefore, multiple reference images from different perspectives can be obtained based on the design drawing of the animated character.

[0108] The image processing model training method provided in the embodiments of this specification can obtain multiple reference images of the target object through a variety of implementation methods, so that multiple reference images of the target object can be reasonably obtained in a suitable manner according to actual application requirements.

[0109] Step 204: Acquire reference image features of each reference image, reference posture features of each reference posture image, and target posture features of the target posture image.

[0110] On the basis of the image processing model to be trained, reference image features corresponding to the reference image, reference posture features of the reference posture image, and target posture features of the target posture image are obtained, so that processing is performed on the features in the image processing model.

[0111] In one or more embodiments of the present specification, by inputting the determined multiple reference images, reference posture images, and target posture images into the image processing model, image features corresponding to each image are obtained in the image processing model, and the specific implementation method is as follows:

[0112] The acquiring of the reference image features of each reference image, the reference posture features of each reference posture image, and the target posture features of the target posture image includes:

[0113] Inputting the plurality of reference images, each reference pose image and the target pose image into the image processing model;

[0114] In the image processing model, feature extraction is performed on each reference image, each reference posture image and the target posture image to obtain reference image features of each reference image, reference posture features of each reference posture image and target posture features of the target posture image.

[0115] Specifically, by inputting a reference image, a reference posture image, and a target posture image into an image processing model, feature extraction can be performed on these images in the image processing model, thereby obtaining reference image features of each reference image, reference posture features of each reference posture image, and target posture features of the target posture image, wherein the reference image features include image information corresponding to the reference image that can characterize image content, structure, etc., the reference posture features are used to characterize the posture information of the target object in the reference image, and the target posture features are used to characterize the posture information of the target object in the target image.

[0116] The image processing model training method provided in the embodiments of this specification can extract features of reference images, reference posture images and target posture images in the image processing model by inputting reference images, reference posture images and target posture images into the image processing model, so that the image processing model can perform in-depth analysis and processing based on the corresponding features, thereby improving the processing speed of the image processing model.

[0117] In one or more embodiments of the present specification, the image processing model includes an image feature extraction unit and a posture feature extraction unit; the image feature extraction unit performs image feature extraction on the image, and the posture feature extraction unit performs posture feature extraction on the image, and the specific implementation method is as follows:

[0118] The step of extracting features from each reference image, each reference posture image, and the target posture image in the image processing model to obtain reference image features of each reference image, reference posture features of each reference posture image, and target posture features of the target posture image includes:

[0119] Utilizing the image feature extraction unit to extract image features from each reference image, and obtaining reference image features of each reference image;

[0120] The posture feature extraction unit is used to extract posture features from each reference posture image and the target posture image, respectively, to obtain reference posture features of each reference posture image and target posture features of the target posture image.

[0121] Among them, the image feature extraction unit can be understood as a network unit that focuses on extracting image information from an image. The image feature extraction unit can be a specific layer or a group of layers in a deep learning model, or it can exist as an independent model. When the image feature extraction unit is an independent model, the image feature extraction unit can be a pre-trained feature extraction model, which is trained separately to extract image features and can be embedded in the image processing model as a fixed feature extractor.

[0122] The posture feature extraction unit can be understood as a network unit that focuses on extracting posture information from an image containing posture. For example, the posture feature extraction unit is a 4-layer convolutional network, and the convolution kernel size of each layer is 4x4, the step size is 2x2, and the corresponding number of channels is 16, 32, 64, and 128.

[0123] Specifically, multiple reference images are input into an image feature extraction unit, and each reference image is encoded by the image feature extraction unit to extract reference image features corresponding to each reference image. In the embodiment of this specification, the reference image features mainly include appearance features of the target object in the reference image; the reference posture image and the target posture image are input into a posture feature extraction unit, and the posture feature extraction unit extracts posture features from the reference posture image to obtain reference posture features of each reference posture image, and the posture feature extraction unit extracts posture features from the target posture image to obtain target posture features of the target posture image.

[0124] The image processing model training method provided in the embodiments of this specification performs image feature extraction on a reference image based on an image feature extraction unit, and performs posture feature extraction on a reference posture image and a target posture image based on a posture feature extraction unit. That is, for different types of images, the key features corresponding to such images can be obtained more accurately based on appropriate feature extraction units.

[0125] Step 206: Obtain reference network features based on the reference image features and the reference posture features, and obtain target prediction features based on the reference network features, the target posture features and the noise image.

[0126] Specifically, the reference network feature is used to enhance the realism, detail richness and similarity of the generated image with the reference image during the image generation process. When the reference network feature is obtained based on the reference image feature and the reference posture feature, the reference network feature contains rich image information of the reference image and posture information of the target object in the reference image. The image information includes but is not limited to the color distribution, texture, object shape and other information of the reference image, and the posture information of the target object includes but is not limited to the position, direction and other information of the target object.

[0127] By combining the reference network features with the target posture features and the noise image, the obtained target prediction features contain the image information of the reference image in the reference network features and the posture information in the target posture features.

[0128] In one or more embodiments of the present specification, the image processing model further includes a reference network unit and a denoising network unit, wherein the reference network unit and the denoising network unit have the same network structure and shared initialization weights, and both the reference network unit and the denoising network unit include a spatial attention layer;

[0129] The step of obtaining a reference network feature according to the reference image feature and the reference posture feature, and obtaining a target prediction feature according to the reference network feature, the target posture feature and the noise image includes:

[0130] Using the spatial attention layer of the reference network unit, performing spatial attention processing on the reference image features and the reference posture features to obtain the reference network features;

[0131] The spatial attention layer of the denoising network unit is used to perform spatial attention processing on the reference network features, the target posture features and the noise image to obtain the target prediction features.

[0132] Among them, the reference network unit is used to process the reference image to obtain relevant information of the reference image, and the relevant information of the reference image can be passed to the denoising network unit; the denoising network unit is the main network for image generation, which is used to denoise the noisy image to generate a predicted image.

[0133] By utilizing the spatial attention mechanism of the spatial attention layer, different weights can be assigned to different positions of the input data, thereby emphasizing areas that are more important to the image generation task. For example, when generating a predicted image of a target object from a frontal perspective, the spatial attention layer based on the denoising network unit can increase the weight of reference images of the target object from a frontal perspective among multiple reference images; and since the spatial attention layer can dynamically adjust its attention focus according to different inputs, the image processing model can automatically adapt to changes in key features in different scenarios. The spatial attention layer can achieve more efficient and accurate information processing by dynamically adjusting the degree of attention to different parts of the input data.

[0134] In practical applications, the reference network unit and the denoising network unit can both be Unet network structures. On the basis that the reference network unit and the denoising network unit have the same network structure and shared initialization weights, the reference network features of the reference network unit can be integrated into the denoising network unit through spatial attention. When the reference network features are obtained through reference image features corresponding to multiple reference images and reference posture features corresponding to multiple reference posture images, the denoising network unit can refer to the appearance of the target object in the reference images from multiple perspectives during the denoising process, thereby avoiding the mental generation of invisible areas when using a reference image as a reference, thereby reducing the accuracy of the generated image.

[0135] Specifically, when processing multiple reference images, the reference posture features corresponding to the reference images are also input into the reference network unit. After spatial attention processing is performed by the spatial attention layer of the reference network unit, the reference image features and reference posture features contained in each reference network feature are focused on more important parts.

[0136] When the reference network features are obtained, the reference network features are input into the denoising network unit, the target posture features and the noise image are added to obtain the denoising network features, and the reference network features and the denoising network features are fused. For example, in the width dimension, the reference network features and the denoising network features are spliced, and the spliced ​​features are input into the spatial attention layer of the denoising network unit. When the denoising network features also contain posture information, based on the spatial attention layer of the denoising network unit, the denoising network unit can adaptively assign different weights to multiple reference images according to the correlation between postures and the correlation between images, thereby obtaining suitable target prediction features.

[0137] For example, multiple reference images include a reference image of a target object facing forward with hands on hips, a reference image of the target object facing forward with one hand on hips, and a reference image of the target object facing sideways with both hands on hips; on the basis of generating a predicted image of a target object facing sideways with one hand on hips, the reference image of the target object facing forward may be assigned a lower weight, while the reference image of the target object facing sideways may be assigned a higher weight. At the same time, the reference image of the target object with one hand on hips may also be weighted higher than the reference image of the target object with both hands on hips.

[0138] The feature maps of each reference image are fused according to the assigned weights to obtain the target prediction features, which will contain information from all reference images but focus on features that are more relevant and important to the predicted image.

[0139] The image processing model training method provided in the embodiments of the present specification integrates the reference network features of the reference network unit into the denoising network unit, so that the denoising network unit can adaptively assign weights to the reference network features of each reference image according to the spatial attention mechanism of the spatial attention layer, thereby obtaining more accurate target prediction features.

[0140] In one or more embodiments of the present specification, the image feature extraction unit includes a first image feature extraction layer and a second image feature extraction layer; different image feature extraction units are used to extract image features for each reference image, so as to extract features of the reference image from multiple angles, so that the generated image can better retain the details of the reference image. The specific implementation method is as follows:

[0141] The step of extracting image features from each reference image using the image feature extraction unit to obtain reference image features of each reference image includes:

[0142] Using the first image feature extraction layer to extract image features from each of the reference images, to obtain first reference image features of each of the reference images;

[0143] The second image feature extraction layer is used to extract image features from each reference image to obtain second reference image features of each reference image, wherein the second reference image features include semantic features of the reference image.

[0144] Among them, the first image feature extraction layer can be understood as an encoder layer, which is used to extract image features of the reference image; and the second image feature extraction layer can be understood as a CLIP feature extraction layer, that is, the second image feature extraction layer realizes feature extraction through the CLIP model, and the extracted second reference image features can be understood as CLIP features, and the second reference image features include semantic features of the reference image.

[0145] Specifically, when multiple reference images are input into an image processing model and image feature extraction is performed using an image feature extraction unit, image feature extraction can be performed on the reference images through multiple image feature extraction layers, so that feature representations at different levels of the image can be learned when different image feature extraction layers are trained using different training methods.

[0146] The image processing model training method provided in the embodiments of this specification can extract image features of the reference image from multiple angles and obtain reference image features at different levels by using different image feature extraction layers to extract image features of the reference image, so that the generated predicted image can better retain the details of the reference image from multiple dimensions and improve the accuracy of the predicted image.

[0147] In one or more embodiments of the present specification, the image processing model further includes a reference network unit and a denoising network unit, wherein the reference network unit and the denoising network unit have the same network structure and shared initialization weights, and both the reference network unit and the denoising network unit include a spatial attention layer and a cross attention layer; in the cross attention layer, the features output by the spatial attention layer of the denoising network unit and the features of the second reference image are cross-attended, and the specific implementation method is as follows:

[0148] The step of obtaining a reference network feature according to the reference image feature and the reference posture feature, and obtaining a target prediction feature according to the reference network feature, the target posture feature and the noise image includes:

[0149] Using the spatial attention layer of the reference network unit, performing spatial attention processing on the first reference image feature and the reference posture feature to obtain the reference network feature;

[0150] Using the spatial attention layer of the denoising network unit, spatial attention processing is performed on the reference network features, the target posture features and the noise image to obtain target key features;

[0151] The cross-attention layer of the denoising network unit is used to perform cross-attention processing on the second reference image features and the target key features to obtain the target prediction features.

[0152] The target key feature can be understood as the feature output by the spatial attention layer of the denoising network unit. The target key feature is a feature that selects the key part of the reference network feature adaptively through the spatial attention mechanism of the spatial attention layer. The cross attention mechanism is a bidirectional attention mechanism that obtains better representation by establishing associations between two input sequences.

[0153] Specifically, the first reference image feature is input into the spatial attention layer of the reference network unit, so as to perform spatial attention processing on the first reference image feature and the reference posture feature through the spatial attention layer of the reference network unit to obtain the reference network feature, and the reference network feature is integrated into the denoising network unit. In the spatial attention layer of the denoising network unit, spatial attention processing is performed on the reference network feature, the target posture feature and the noise image to obtain the target key feature. The processing performed in the spatial attention layer of the denoising network unit here is similar to that in the above embodiment and will not be repeated here.

[0154] The target key features output by the spatial attention layer in the denoising network unit and the second reference image features are input into the cross attention layer of the denoising network unit, so that the second reference image features and the target key features are cross-attended through the cross attention layer of the denoising network unit to obtain target prediction features, which contain richer image features of the reference image.

[0155] The image processing model training method provided in the embodiments of the present specification can make the obtained target prediction features contain richer image features in the reference image by performing cross-attention processing on the second reference image features and the target key features output by the spatial attention layer in the cross-attention layer of the denoising network unit, so that the generated prediction image can better retain the details of the reference image from multiple dimensions, thereby improving the accuracy of the predicted image.

[0156] In one or more embodiments of the present specification, the reference network unit includes multiple spatial attention layers and multiple cross attention layers; a spatial attention layer and a cross attention layer can be regarded as a group of layers, a reference network feature is obtained in the spatial attention layer of each reference network unit, and a cross attention process is performed on the second reference image feature and the reference network feature in each cross attention layer, and the specific implementation method is as follows:

[0157] The using the spatial attention layer of the reference network unit to perform spatial attention processing on the first reference image feature and the reference posture feature to obtain the reference network feature includes:

[0158] Performing feature fusion on the first reference image feature and the reference posture feature to obtain a reference fused feature, and performing spatial attention processing on the reference fused feature at each spatial attention layer of the reference network unit to obtain the reference network feature;

[0159] After performing spatial attention processing on the first reference image feature and the reference posture feature using the spatial attention layer of the reference network unit to obtain the reference network feature, the method further includes:

[0160] At each cross-attention layer of the reference network unit, performing cross-attention processing on the second reference image feature and the reference network feature to obtain a reference cross-feature;

[0161] The reference cross feature is determined as the reference fusion feature, and the step of performing spatial attention processing on the reference fusion feature in each spatial attention layer of the reference network unit to obtain the reference network feature is continued.

[0162] The denoising network unit includes multiple spatial attention layers and multiple cross attention layers;

[0163] The method of using the spatial attention layer of the denoising network unit to perform spatial attention processing on the reference network features, the target posture features and the noise image to obtain the target key features includes:

[0164] The target posture feature and the noise image are subjected to feature fusion to obtain denoising fusion features, and in each spatial attention layer of the denoising network unit, the reference network feature and the denoising fusion feature are subjected to spatial attention processing to obtain the target key features;

[0165] The method of using the cross attention layer of the denoising network unit to perform cross attention processing on the second reference image feature and the target key feature to obtain the target prediction feature includes:

[0166] In each cross-attention layer of the denoising network unit, cross-attention processing is performed on the second reference image features and the target key features to obtain the target prediction features, and the target prediction features are determined as the denoising fusion features, and the step of continuing to perform spatial attention processing on the reference network features and the denoising fusion features in each spatial attention layer of the denoising network unit to obtain the target key features.

[0167] In practical applications, the first reference image feature and the reference posture feature are feature fused, so that the fused reference fusion feature is input into the first spatial attention layer of the reference network unit, and the reference fusion feature is processed by spatial attention in the first spatial attention layer of the reference network unit to obtain the reference network feature; and the target posture feature and the noise image are feature fused to obtain the denoising fusion feature, and the reference network feature obtained by using the first spatial attention layer of the reference network unit and the denoising fusion feature are feature spliced ​​in the width dimension (specifically, when the reference network unit and the denoising network unit are both Unet structures, the features are transmitted in the reference network unit and the denoising network unit in the form of feature maps, and the feature maps have three different dimensions: height, width and channel (also called depth). By performing feature splicing in the width dimension, it is ensured that the spliced ​​feature map remains consistent in the height dimension, and the width dimension is the sum of the widths of all input feature maps), and the spliced ​​features are input into the first spatial attention layer of the denoising network unit, and the spliced ​​features are processed by the spatial attention layer to obtain the target key features.

[0168] The second reference image feature and the target key feature output by the first spatial attention layer in the denoising network unit are input into the first cross attention layer in the reference network unit. The first cross attention layer of the denoising network unit is used to perform cross attention processing on the second reference image feature and the target key feature to obtain the target prediction feature. To ensure the consistency between the reference network unit and the denoising network unit, the second reference image feature and the reference network feature output by the first spatial attention layer in the reference network unit are input into the first cross attention layer in the reference network unit. The reference cross feature output by the first cross attention layer in the reference network unit will be used as the input feature of the second spatial attention layer in the reference network unit, that is, the reference fusion feature. Similarly, in each spatial attention layer of the reference network unit, the reference fusion feature is spatially attended to obtain the reference network feature. In each cross attention layer of the reference network unit, the second reference image feature and the reference network feature are cross-attended to obtain the reference cross feature.

[0169] Similarly, in the denoising network unit, the target prediction features output by the current cross-attention layer are determined as denoising fusion features, which will be used as the input of the next spatial attention layer. In each spatial attention layer in the denoising network unit, spatial attention processing is performed on the reference network features and the denoising fusion features to obtain the target key features, and in each cross-attention layer of the denoising network unit, cross-attention processing is performed on the second reference image features and the target key features to obtain the target prediction features.

[0170] It should be noted that although the spatial attention layers of the reference network units process the reference fusion features, and the spatial attention layers of the denoising network units process the concatenated features of the reference network features and the denoising fusion features, since the reference network units and the denoising network units are Unet network structures, the features will be upsampled and downsampled in Unet. Therefore, the feature sizes of the reference fusion features processed in the spatial attention layer of each reference network unit are different. Similarly, the feature sizes of the concatenated features processed in the spatial attention layer of each denoising network unit are different.

[0171] The image processing model training method provided in the embodiments of this specification integrates the reference network features of each spatial attention layer in the reference network unit into each spatial attention layer corresponding to the denoising network unit, which can improve the accuracy of the target prediction features.

[0172] Step 208: Obtain a predicted image based on the target prediction feature, wherein the posture of the target object in the predicted image is determined by referring to the posture of the target object in the target posture image.

[0173] Among them, the predicted image can be understood as an image generated by an image processing model, in which the posture of the target object in the predicted image tends to the posture of the target object in the target image, and the appearance of the target object in the predicted image tends to the appearance of the target object in the reference image, and the background and other areas of the predicted image tend to the corresponding areas of the reference image.

[0174] In one or more embodiments of the present specification, the image processing model further includes a prediction unit; the prediction unit is used to decode the target prediction feature to generate a predicted image, and the specific implementation method is as follows:

[0175] The step of obtaining a predicted image according to the target prediction feature comprises:

[0176] The prediction unit is used to decode the target prediction feature to obtain a predicted image.

[0177] Specifically, since the input to the reference network unit and the denoising network unit are features after feature extraction of the image or posture image using the image feature extraction unit and the posture feature extraction unit, after being processed by the reference network unit and the denoising network unit, the output is the target prediction feature of the same dimension as these features. In order to obtain the predicted image through the target prediction feature, the target prediction feature needs to be decoded to obtain the predicted image.

[0178] The image processing model training method provided in the embodiments of this specification is to reconstruct a predicted image with the same dimension as the target image through target prediction features, decode the target prediction features, and thus reasonably obtain the predicted image.

[0179] Step 210: Train the image processing model according to the predicted image and the target image to obtain a trained image processing model.

[0180] In one or more embodiments of the present specification, the step of training the image processing model according to the predicted image and the target image to obtain the trained image processing model includes:

[0181] Determine a loss function between the predicted image and the target image, and train an image processing model according to the loss function to obtain a trained image processing model.

[0182] Specifically, when a predicted image is obtained, the image processing model is trained by the predicted image and the target image to obtain a trained image processing model. For example, the image processing model is trained by calculating the loss function (mean square error, cross entropy loss, etc.) of the predicted image and the target image. It should be noted that the training of the image processing model is mainly to adjust the network parameters of the posture feature extraction unit, the reference network unit and the denoising network unit of the image processing model.

[0183] In practical applications, after the image processing model is trained, the image processing model can be deployed on the client so that the client can use the trained image processing model to generate images locally on the client, or when the model interface information corresponding to the image processing model is sent to the client, the client's computing resources can be saved. The client interacts with the image processing model deployed on the server through the model interface information. After generating the image, the image processing model can send the generated image to the client.

[0184] An image processing model training method provided by an embodiment of the present specification can accurately capture the image of the target object from multiple reference images by determining multiple reference images of the target object under different viewing angles, thereby avoiding the random generation of invisible areas. By obtaining the reference image features of each reference image, the reference posture features of each reference posture image, and the target posture features of the target posture image, these features can be adaptively selected in the image processing model to obtain accurate target prediction features. On the basis of obtaining the predicted image based on the target prediction features, the accuracy of the generated predicted image is improved, and every detail of the target object is restored in the predicted image.

[0185] See also Figure 3 , Figure 3 A flowchart of another image processing model training method provided by an embodiment of the present specification is shown, which specifically includes the following steps.

[0186] Step 302: Determine multiple reference images of the target object at different viewing angles, a reference posture image of the target object in each reference image, a target image of the target object, a target posture image of the target object in the target image, and a noise image corresponding to the target image, and input the multiple reference images, each reference posture image, the target posture image, and the noise image into an image processing model.

[0187] Among them, the posture of the target object in each reference image is different from that in the target image, and the image processing model includes a feature extraction unit, a reference network unit, a denoising network unit, and a prediction unit.

[0188] Step 304: Utilizing the feature extraction unit, obtain reference image features of each reference image, reference posture features of each reference posture image, and target posture features of the target posture image.

[0189] Step 306: Using the reference network unit, according to the reference image features and the reference posture features, obtain reference network features, and using the denoising network unit, according to the reference network features, the target posture features and the noise image, obtain target prediction features.

[0190] Step 308: Utilizing the prediction unit, according to the target prediction feature, a predicted image is obtained, wherein the posture of the target object in the predicted image is determined by referring to the posture of the target object in the target posture image.

[0191] Step 310: Train the image processing model according to the predicted image and the target image to obtain a trained image processing model.

[0192] For the specific implementation, please refer to the above embodiments, which will not be described in detail here.

[0193] An image processing model training method provided by an embodiment of the present specification can accurately capture the image of the target object from multiple reference images by determining multiple reference images of the target object under different viewing angles, thereby avoiding the random generation of invisible areas. By obtaining the reference image features of each reference image, the reference posture features of each reference posture image, and the target posture features of the target posture image, these features can be adaptively selected in the image processing model to obtain accurate target prediction features. On the basis of obtaining the predicted image based on the target prediction features, the accuracy of the generated predicted image is improved, and every detail of the target object is restored in the predicted image.

[0194] See also Figure 4 , Figure 4 A network framework diagram of an image processing model provided by an embodiment of the present specification is shown. Based on the network framework diagram, training of the image processing model is achieved through the following contents.

[0195] Specifically, multiple reference images of the target object at different viewing angles and a target image of the target object are determined. When the target object in each reference image has a different object posture from that in the target image, the target object is made to perform actions with the action posture in the target image, and the appearance of the target object is based on the appearance images in the multiple reference images, thereby generating a predicted image in which the appearance image of the target object is the appearance image in the reference image and the action posture is the action posture of the target object in the reference image.

[0196] In practical applications, after determining the target image, the image denoising model can be used to denoise the target image to obtain the denoised image (i.e. Figure 4 Noise in the image), for example, using the DDIM model (Denoising Diffusion Implicit Models) to add noise to the target image, a noise image corresponding to the target image can be obtained by adding random Gaussian noise.

[0197] The image processing model includes a posture feature extraction layer, which takes a posture image as input and outputs posture features corresponding to the posture image. Therefore, before using the posture feature extraction layer for processing, a reference posture image corresponding to the reference image and a target posture image corresponding to the target image must be obtained; that is, the image processing model must include a posture estimation unit to perform posture detection on the object in the input image and construct a corresponding posture image.

[0198] That is, when the input of the image processing model is a target image, a noisy image of the target image, and multiple reference images, the target image and the multiple reference images are respectively input into the posture estimation unit of the image processing model, and multiple reference posture images corresponding to the multiple reference images and a target posture image corresponding to the target image (that is, a posture graph corresponding to the target graph) are constructed, and the multiple reference posture images and the target posture image are respectively input into the posture feature extraction layer to obtain multiple reference posture features corresponding to the multiple reference posture images and target posture features corresponding to the target posture images.

[0199] Of course, in practical applications, the posture estimation unit can also be used as an independent posture estimation model, independent of the image processing model; in this case, multiple reference images and the target image are input into the posture estimation model, and multiple reference posture images corresponding to the multiple reference images and the target posture image corresponding to the target image are constructed, and the target posture image, the noise image of the target image, the multiple reference images and the multiple reference posture images are input into the image processing model, and the posture feature extraction layer of the image processing model is used to obtain multiple reference posture features corresponding to the multiple reference posture images and the target posture features corresponding to the target posture image.

[0200] Specifically, the image processing model includes two image processing units, one is a reference image processing unit (i.e., the reference network unit in the above embodiment), and the other is a denoising image processing unit (i.e., the denoising network unit in the above embodiment). The denoising image processing unit and the reference image processing unit are two Unet networks with similar structures, sharing initialization weights, and both the denoising image processing unit and the reference image processing unit include a spatial attention layer and a cross attention layer; therefore, the denoising image processing unit can learn features associated with the same feature space from the reference image processing unit.

[0201] Using the first image feature extraction layer (i.e. Figure 4 The encoder in the image processing unit) extracts image features from multiple reference images to obtain multiple reference image features, which contain appearance information of the target object; the multiple reference image features are fused with the multiple reference posture features extracted above (such as feature addition), the fused reference fused features are input into the spatial attention layer of the reference image processing unit, and the spatial attention layer of the reference image processing unit is used to obtain multiple reference network features.

[0202] The target posture feature is fused with the noisy image to obtain a denoising fusion feature. The denoising fusion feature is fused with the multiple reference network features obtained above (for example, in the width dimension, multiple reference network features and denoising fusion features are spliced), and then input into the spatial attention layer of the denoising image processing unit. The spatial attention layer can provide an adaptive mechanism that can automatically adjust the focus according to the specific content of each sample, and obtain the target key features that assign different weights to each reference network feature and denoising network feature.

[0203] In an embodiment of the present specification, when both the reference network features and the denoising network features contain posture information, the spatial attention layer of the denoising image processing unit can adaptively assign different weights to the reference network features and the denoising network features according to the correlation of the posture information between the two features and the image correlation between the reference image and the target image, so that the image processing model can focus on specific key areas; for example, when the target image is an image of user a with his hands on his hips and tilted 45 degrees to the right, the weights of the features corresponding to the front perspective reference image, the features corresponding to the right perspective reference image, and the features corresponding to the rear perspective reference image among multiple reference image processing features can be increased, while the weights of the features corresponding to the left perspective reference image can be reduced.

[0204] It should be noted that in order to extract image features of multiple reference images from multiple directions so that the generated predicted image can better retain the details of the reference image, the image processing model can include a second image feature extraction layer (i.e. Figure 4The CLIP feature extraction layer in the image processing example is used to extract CLIP features of multiple reference images (i.e., the second reference image features in the above embodiment). Different image feature extraction layers capture and express image features in different ways. By extracting features from reference images through different image feature extraction layers, the diversity of image features can be increased, which helps the image processing model capture information in more dimensions.

[0205] The extracted CLIP features and the target key features output by the spatial attention layer in the denoising image processing unit are input into the cross attention layer of the denoising image processing unit to obtain the target prediction features, and the target prediction features are input into the next spatial attention layer; the specific training process can be found in Figure 5 , Figure 5 A training schematic diagram of an image processing model training method provided by an embodiment of the present specification is shown, wherein, when there are 5 reference images, there are 4 reference network features, that is, N is 4, w represents the width of the feature, and h represents the height of the feature.

[0206] When the Unet structure includes multiple spatial attention layers (spatial attention modules) and cross attention layers (cross attention modules), in each spatial attention layer, multiple reference network features of the reference image processing unit are integrated into the spatial attention layer of the denoising image processing unit; and in each cross attention layer, the CLIP feature is integrated into the cross attention layer of the denoising image processing unit.

[0207] In practical applications, in order to ensure the consistency of features between the reference image processing unit and the denoising image processing unit, the features of the reference image processing unit can be applied to the denoising image processing unit. Similarly, in each cross-attention layer of the reference image processing unit, the network features are input into the cross-attention layer of the reference image processing unit.

[0208] The target prediction features output by the denoising image processing unit are input into the decoder (i.e., the prediction unit of the above-mentioned embodiment), and the decoder is used to output the predicted image, so as to obtain the image processing model through training based on the predicted image and the target image. In the embodiment of this specification, the posture feature extraction layer, the reference image processing unit and the denoising image processing unit of the image processing model are trained in particular.

[0209] The image processing model training method provided in the embodiments of this specification can use the image processing model to obtain multiple features corresponding to the multiple reference images by determining multiple reference images of the target object under multiple perspectives, thereby accurately capturing the image of the target object from multiple perspectives, avoiding the random generation of invisible areas, and restoring every detail of the target object in the generated predicted image.

[0210] See also Figure 6, Figure 6 A flowchart of an image processing method provided by an embodiment of the present specification is shown, which specifically includes the following steps.

[0211] Step 602: Determine a reference image and a target image, obtain a reference pose image of a reference object in the reference image based on the reference image, and obtain a target pose image of a target object in the target image and a noise image corresponding to the target image based on the target image.

[0212] In one or more embodiments of the present specification, the user interacts with the user interface of the client to enable the client to send the reference image and the target image to the server. The specific implementation method is as follows:

[0213] Determine the reference image and the target image, including:

[0214] A reference image and a target image sent by a client are received, wherein the reference image and the target image are sent when a specific operation is triggered on a user interaction interface of the client.

[0215] The user can pre-set several target images in the user interaction interface of the client. After the user uploads any reference image or obtains the reference image by shooting, the user can select a target image, so that the client sends the reference image and the target image to the server.

[0216] Step 604: Based on the reference image, the reference posture image, the target posture image and the noise image, an image is generated using an image processing model to obtain a generated image, wherein the image processing model is trained by the above-mentioned image processing model training method, the appearance of the object in the generated image is determined by the appearance of the reference object, and the motion posture of the object in the generated image is determined by the motion posture of the target object.

[0217] Specifically, taking the application of image processing methods in the field of animation as an example, the user can provide any images of an animated character from different perspectives. These images provided by the user are reference images, and a target image is selected to enable the animated character to display action postures using the posture of the object in the target image. The multiple reference images provided by the user and the selected target image are input into the image processing model, and the image processing model is used to generate an image in which the animated character displays action postures using the posture of the object in the target image.

[0218] The image processing method provided in the embodiments of this specification, based on the pre-trained weights of the reference network unit and the denoising network unit, can provide reasonable supplements and prior knowledge for image generation, and by introducing a technology that can process any number of reference image inputs, it can adaptively extract key information from these reference images, accurately capture the user's image from multiple perspectives, avoid the mental generation of invisible areas, and significantly enhance the accuracy of image reproduction.

[0219] See also Figure 7 , Figure 7 A flowchart of a video processing model training method provided by an embodiment of the present specification is shown, which specifically includes the following steps.

[0220] Step 702: Determine multiple reference images of the target object at different viewing angles, reference pose images of the target object in each reference image, a target video of the target object, a target pose image sequence of the target object in the target video, and a noise image sequence corresponding to the target video.

[0221] Step 704: Acquire reference image features of each reference image, reference posture features of each reference posture image, and a target posture feature sequence of the target posture image sequence.

[0222] In one or more embodiments of the present specification, the step of acquiring reference image features of each reference image, reference posture features of each reference posture image, and a target posture feature sequence of the target posture image sequence includes:

[0223] Inputting the plurality of reference images, the reference pose images and the target pose image sequence into the image processing model;

[0224] In the image processing model, feature extraction is performed on the multiple reference images, the reference posture images, and the target posture image sequence to obtain reference image features of the reference images, reference posture features of the reference posture images, and a target posture feature sequence of the target posture image sequence.

[0225] Step 706: Obtain reference network features based on the reference image features and the reference posture features, obtain candidate prediction feature sequences based on the reference network features, the target posture feature sequence and the noise image sequence, and obtain a target prediction feature sequence based on the candidate prediction feature sequence and the temporal relationship between the candidate prediction features.

[0226] In one or more embodiments of the present specification, obtaining a reference network feature according to the reference image feature and the reference posture feature, and obtaining a candidate prediction feature sequence according to the reference network feature, the target posture feature sequence and the noise image sequence, includes:

[0227] The image processing model is used to obtain reference network features according to the reference image features and the reference posture features, and a candidate prediction feature sequence is obtained according to the reference network features, the target posture feature sequence and the noise image sequence.

[0228] Among them, the candidate prediction feature sequence can be understood as the features output by the cross-attention layer of the denoising network unit.

[0229] Specifically, the video processing model uses the image processing model obtained through the above training as a base model, trains the video processing model based on the above image processing model, and obtains a target posture image sequence of the target object in the target video based on the determined target video, and then obtains a target posture feature sequence of the target posture image sequence, wherein the target posture feature sequence includes time information. For the specific implementation, please refer to the above embodiment, which will not be repeated here.

[0230] In one or more embodiments of the present specification, a temporal attention layer is set in the denoising network unit of the image processing model; and obtaining a target prediction feature sequence according to the candidate prediction feature sequence and the temporal relationship between the candidate prediction features includes:

[0231] The temporal attention layer is used to perform temporal attention processing on the candidate prediction features in the candidate prediction feature sequence and the temporal relationship between the candidate prediction features to obtain a target prediction feature sequence.

[0232] Specifically, the temporal attention layer is introduced into the image processing model trained in the previous stage, and the temporal attention layer is set behind the cross attention layer in each denoising network unit, and the spatial attention layer and the cross attention layer in the image processing model are kept unchanged.

[0233] The video processing model still selects multiple reference images, takes multiple frames of noisy images as input, selects multiple frames of continuous target images and continuous control signals (target posture image sequence). In practical applications, in order to align with the multi-frame denoising network features of the denoising network unit, the reference network features are copied in the time dimension, and then spliced ​​with the multi-frame denoising network features for subsequent training; in the temporal attention layer, the attention operation is performed on the candidate prediction feature sequence output by the cross-attention layer in the denoising network unit in the time dimension, that is, the candidate prediction features in the candidate prediction feature sequence and the temporal relationship between the candidate prediction features are subjected to temporal attention processing, so as to learn the correlation between different frames and obtain the target prediction feature sequence.

[0234] The video processing model training method provided in the embodiments of this specification performs an attention operation on the candidate prediction feature sequence output by the cross-attention layer in the denoising network unit through the temporal attention layer, and learns the correlation between different frames, thereby enabling the video processing model to have the ability to generate continuous images and the generated video is smoother.

[0235] Step 708: Obtain a predicted video according to the target prediction feature sequence, wherein the posture of the target object in the predicted video is determined by referring to the posture of the target object in the target video.

[0236] Step 710: Train the video processing model according to the predicted video and the target video to obtain a trained video processing model, wherein the video processing model is determined based on the trained image processing model in the above-mentioned image processing model training method.

[0237] In one or more embodiments of the present specification, the step of training a video processing model according to the predicted video and the target video to obtain a trained video processing model includes:

[0238] According to the predicted video and the target video, the network parameters of the temporal attention layer in the denoising network unit of the image processing model are adjusted to obtain a trained video processing model.

[0239] In practical applications, since the video processing model uses the image processing model as the base model, training is performed on the newly set temporal attention layer without changing the network parameters of other network units.

[0240] For the specific implementation, please refer to the above embodiments, which will not be described in detail here.

[0241] The video processing model training method provided in the embodiments of this specification sets a video-related time attention layer on the basis of the image processing model obtained through training, and uses the time attention layer to learn the correlation between different frames in the target video, so that the video processing model has the ability to generate continuous pictures, and the generated video is smoother. In addition, based on the input of multiple reference images, it can accurately capture the user's image from multiple perspectives and restore every detail of the user in the generated video.

[0242] See also Figure 8 , Figure 8 A flowchart of another video processing model training method provided by an embodiment of the present specification is shown, which specifically includes the following steps.

[0243] Step 802: Determine multiple reference images of the target object under different viewing angles, reference posture images of the target object in each reference image, a target video of the target object, a target posture image sequence of the target object in the target video, and a noise image sequence corresponding to the target video, and input the multiple reference images, each reference posture image, the target posture image sequence, and the noise image sequence into a video processing model.

[0244] Among them, the video processing model is determined based on the trained image processing model in the above-mentioned image processing model training method, and the video processing model includes a feature extraction unit, a reference network unit, a denoising network unit, and a prediction unit.

[0245] Step 804: Utilizing the feature extraction unit, obtain reference image features of each reference image, reference posture features of each reference posture image, and a target posture feature sequence of the target posture image sequence.

[0246] Step 806: Using the reference network unit, obtain reference network features according to the reference image features and the reference posture features, and using the denoising network unit, obtain candidate prediction feature sequences according to the reference network features, the target posture feature sequence and the noise image sequence, and obtain a target prediction feature sequence according to the candidate prediction feature sequences.

[0247] Step 808: Utilizing the prediction unit, according to the target prediction feature sequence, a predicted video is obtained, wherein the posture of the target object in the predicted video is determined by referring to the posture of the target object in the target video.

[0248] Step 810: Train a video processing model according to the predicted video and the target video to obtain a trained video processing model.

[0249] For the specific implementation, please refer to the above embodiments, which will not be described in detail here.

[0250] The video processing model training method provided in the embodiments of this specification sets a video-related time attention layer on the basis of the image processing model obtained through training, and uses the time attention layer to learn the correlation between different frames in the target video, so that the video processing model has the ability to generate continuous pictures, and the generated video is smoother. In addition, based on the input of multiple reference images, it can accurately capture the user's image from multiple perspectives and restore every detail of the user in the generated video.

[0251] See also Fig. 9 , Fig. 9 A network framework diagram of a video processing model provided by an embodiment of the present specification is shown. Based on the network framework diagram, training of the video processing model is achieved through the following contents.

[0252] The video processing model is determined on the basis of the above-mentioned image processing model obtained through training; specifically, a temporal attention layer is introduced into the denoising image processing unit in the above-mentioned image processing model, and the spatial attention layer and the cross attention layer are kept unchanged to obtain the video processing model.

[0253] In practical applications, the video processing model needs to determine multiple reference images of the target object and a target video of the target object, and obtain multiple frames of continuous target images from the target video.

[0254] In practical applications, on the basis of determining the target image, the image denoising model can be used to add noise to multiple frames of target images to obtain a noisy image after adding noise. For example, the DDIM model (Denoising Diffus ion Implicit Models) can be used to perform the denoising process on multiple frames of target images. By adding random Gaussian noise, multiple frames of noise images corresponding to the multiple frames of target images (i.e., the noise image sequence in the above embodiment) can be obtained.

[0255] When the input of the image processing model is multiple-frame target images, multiple-frame noise images of multiple-frame target images, and multiple reference images, the multiple-frame target images and the multiple reference images are input into the posture estimation network layer of the image processing model, and multiple reference posture images corresponding to the multiple reference images and a target posture image sequence corresponding to the multiple-frame target images (i.e., the posture sequence in the figure) are respectively constructed, and the multiple reference posture images and the target posture image sequence are input into the posture feature extraction layer to obtain multiple reference posture features corresponding to the multiple reference posture images and a target posture feature sequence corresponding to the target posture image sequence, wherein the target posture feature sequence contains time information.

[0256] Of course, in practical applications, the posture estimation network layer can also be used as an independent posture estimation model, independent of the image processing model; in this case, multiple reference images and multiple frames of target images are input into the posture estimation model to construct multiple reference posture images corresponding to the multiple reference images and a target posture image sequence corresponding to the multiple frames of target images, and the target posture image sequence, the multi-frame noise images of the multiple frames of target images, the multiple reference images and the multiple reference posture images are input into the image processing model, and the posture feature extraction layer of the image processing model is used to obtain multiple reference posture features corresponding to the multiple reference posture images and a target posture feature sequence corresponding to the target posture image sequence.

[0257] Specifically, the video processing model includes two video processing units, one is a reference video processing unit (that is, the reference image processing unit in the above embodiment, Fig. 9 ), a denoising video processing unit (a temporal attention layer is introduced in the denoising image processing unit in the above embodiment, Fig. 9 denoising network in .

[0258] Using steps similar to the above embodiment, multiple reference network features are obtained by using the spatial attention layer of the reference image processing unit.

[0259] The target posture feature sequence is fused with the multi-frame noise image to obtain the target video processing feature (i.e., the multi-frame denoising network feature in the above embodiment). On the basis that the target posture feature sequence contains time information, the target video processing feature also contains time information. In order to align the reference network feature with the target video processing feature, multiple reference network features are copied in the time dimension, and then fused with the target video processing feature. The copy is input into the spatial attention layer of the denoising video processing unit to obtain the target reference feature that assigns different weights to each reference network feature and target image processing feature.

[0260] Similar to the above embodiment, CLIP image features corresponding to multiple reference images are obtained and integrated into the cross-attention layer of the denoising video processing unit. At each spatial attention layer, multiple reference network features of the reference video processing unit are integrated into the spatial attention layer of the denoising video processing unit; and at each cross-attention layer, the CLIP image features are integrated into the cross-attention layer of the denoising image processing unit.

[0261] A temporal attention layer is introduced after each cross-attention layer of the denoising video processing unit. The features output by the cross-attention layer are input into the temporal attention layer. The temporal attention layer performs attention operation in the time dimension to obtain the correlation information between multiple frames of target images in the target posture image sequence according to the time information contained in the features, and outputs a candidate prediction feature sequence containing time sequence information. For the specific training process, see Fig.10 , Fig.10 A training schematic diagram of a video processing model training method provided by an embodiment of the present specification is shown, wherein t in the denoising network feature represents the time dimension.

[0262] A temporal attention module (i.e., the temporal attention layer in the above embodiment) is introduced after the cross-attention module to perform temporal attention processing to obtain the correlation between frames of the target video.

[0263] The features output by the denoising image processing unit are input into the decoder, and the decoder is used to output a predicted video, so as to train the video processing model based on the predicted image and the target video. In the embodiment of this specification, the temporal attention layer of the video processing model is trained to enable the video processing model to have the ability to generate continuous images, and the generated video is smoother.

[0264] The video processing model training method provided in the embodiments of this specification introduces a technology that can process any number of reference image inputs to generate high-quality videos. By adaptively extracting key information from these reference images, it not only significantly enhances the accuracy of image reproduction, but also improves the expressiveness of video generation.

[0265] See also Fig.11 , Fig.11 A flowchart of a video processing method provided by an embodiment of the present specification is shown, which specifically includes the following steps.

[0266] Step 1102: Determine a reference image and a target video, obtain a reference posture image of a reference object in the reference image based on the reference image, and obtain a target posture sequence image of a target object in the target video and a noise sequence image corresponding to the target video based on the target video.

[0267] In one or more embodiments of the present specification, determining the reference image and the target video includes:

[0268] A reference image and a target video sent by a client are received, wherein the reference image and the target video are sent when a specific operation is triggered on a user interaction interface of the client.

[0269] Step 1104: Based on the reference image, the reference posture image, the target posture sequence image and the noise sequence image, a video processing model is used to generate a video to obtain a generated video, wherein the video processing model is trained by the above-mentioned video processing model training method, the appearance of the object in the generated video is determined by the appearance of the reference object, and the action posture of the object in the generated video is determined by the action posture of the target object.

[0270] Specifically, taking the application of the video processing method to a short video platform as an example, the user can provide multiple images of himself / herself under different perspectives. These images provided by the user are reference images, and a short video is selected as the target video, so that the user can use the posture of the object in the target video to display the action posture. The multiple images provided by the user and the selected target video are input into the video processing model, and the video processing model is used to generate a video in which the user displays the action posture with the posture of the object in the target video.

[0271] Moreover, when the user provides multiple images from different perspectives, even if the object in the target video selected by the user changes from a front perspective to a back perspective, the video processing model can avoid the mental generation of invisible areas, but adaptively increase the weights of the corresponding features of the image and the back image provided by the user, thereby ensuring that the video processing model can still generate smooth, high-quality videos.

[0272] For another example, when the video processing method is applied to a sports scene, the video processing method can provide participants with an immersive experience, making the participants feel as if they are really doing sports. The video processing method can not only capture every subtle movement of the characters in the sports activities, but also vividly display these details when making cool sports videos.

[0273] The video processing method provided in the embodiments of this specification can support any number of reference images, greatly improve the flexibility of use, and open up a broad imagination space for future product design.

[0274] Corresponding to the above method embodiment, this specification also provides an image processing model training device embodiment, Fig.12 FIG. 1 is a schematic diagram showing the structure of an image processing model training device provided by an embodiment of the present specification. Fig.12 As shown, the device comprises:

[0275] The image determination module 1202 is configured to determine a plurality of reference images of a target object at different viewing angles, a reference pose image of the target object in each reference image, a target image of the target object, a target pose image of the target object in the target image, and a noise image corresponding to the target image, wherein the pose of the target object in each reference image is different from that in the target image;

[0276] A feature acquisition module 1204 is configured to acquire reference image features of each reference image, reference posture features of each reference posture image, and target posture features of the target posture image;

[0277] The feature acquisition module 1206 is configured to obtain a reference network feature according to the reference image feature and the reference posture feature, and obtain a target prediction feature according to the reference network feature, the target posture feature and the noise image;

[0278] An image acquisition module 1208 is configured to acquire a predicted image according to the target prediction feature, wherein the posture of the target object in the predicted image is determined by referring to the posture of the target object in the target posture image;

[0279] The model training module 1210 is configured to train the image processing model according to the predicted image and the target image to obtain a trained image processing model.

[0280] Optionally, the image determination module 1202 is further configured to:

[0281] Determine a plurality of reference images of the target object at different viewing angles and a target image of the target object;

[0282] Inputting the plurality of reference images and the target image into a pose estimation model respectively to obtain a reference pose image of the target object in each of the reference images and a target pose image of the target object in the target image;

[0283] The target image is input into an image denoising model to obtain a noise image corresponding to the target image, wherein the posture estimation model and the image denoising model are machine learning models.

[0284] Optionally, the image determination module 1202 is further configured to:

[0285] Determine a reference video of the target object, and obtain, from the reference video, a plurality of video frames of the target object at different viewing angles as reference images; or

[0286] A plurality of reference images of the target object at different viewing angles are captured.

[0287] Optionally, the feature acquisition module 1204 is further configured to:

[0288] Inputting the plurality of reference images, each reference pose image and the target pose image into the image processing model;

[0289] In the image processing model, feature extraction is performed on each reference image, each reference posture image and the target posture image to obtain reference image features of each reference image, reference posture features of each reference posture image and target posture features of the target posture image.

[0290] Optionally, the feature acquisition module 1204 is further configured to:

[0291] Utilizing the image feature extraction unit to extract image features from each reference image, and obtaining reference image features of each reference image;

[0292] The posture feature extraction unit is used to extract posture features from each reference posture image and the target posture image, respectively, to obtain reference posture features of each reference posture image and target posture features of the target posture image.

[0293] Optionally, the feature acquisition module 1206 is further configured to:

[0294] Using the spatial attention layer of the reference network unit, performing spatial attention processing on the reference image features and the reference posture features to obtain the reference network features;

[0295] The spatial attention layer of the denoising network unit is used to perform spatial attention processing on the reference network features, the target posture features and the noise image to obtain the target prediction features.

[0296] Optionally, the feature acquisition module 1204 is further configured to:

[0297] Using the first image feature extraction layer to extract image features from each of the reference images, to obtain first reference image features of each of the reference images;

[0298] The second image feature extraction layer is used to extract image features from each reference image to obtain second reference image features of each reference image, wherein the second reference image features include semantic features of the reference image.

[0299] Optionally, the feature acquisition module 1206 is further configured to:

[0300] Using the spatial attention layer of the reference network unit, performing spatial attention processing on the first reference image feature and the reference posture feature to obtain the reference network feature;

[0301] Using the spatial attention layer of the denoising network unit, spatial attention processing is performed on the reference network features, the target posture features and the noise image to obtain target key features;

[0302] The cross-attention layer of the denoising network unit is used to perform cross-attention processing on the second reference image features and the target key features to obtain the target prediction features.

[0303] Optionally, the feature acquisition module 1206 is further configured to:

[0304] Performing feature fusion on the first reference image feature and the reference posture feature to obtain a reference fused feature, and performing spatial attention processing on the reference fused feature at each spatial attention layer of the reference network unit to obtain the reference network feature;

[0305] At each cross-attention layer of the reference network unit, performing cross-attention processing on the second reference image feature and the reference network feature to obtain a reference cross-feature;

[0306] The reference cross feature is determined as the reference fusion feature, and the step of performing spatial attention processing on the reference fusion feature in each spatial attention layer of the reference network unit to obtain the reference network feature is continued.

[0307] Optionally, the feature acquisition module 1206 is further configured to:

[0308] The target posture feature and the noise image are subjected to feature fusion to obtain denoising fusion features, and in each spatial attention layer of the denoising network unit, the reference network feature and the denoising fusion feature are subjected to spatial attention processing to obtain the target key features;

[0309] In each cross-attention layer of the denoising network unit, cross-attention processing is performed on the second reference image features and the target key features to obtain the target prediction features, and the target prediction features are determined as the denoising fusion features, and the step of continuing to perform spatial attention processing on the reference network features and the denoising fusion features in each spatial attention layer of the denoising network unit to obtain the target key features.

[0310] Optionally, the image acquisition module 1208 is further configured to:

[0311] The prediction unit is used to decode the target prediction feature to obtain a predicted image.

[0312] Optionally, the model training module 1210 is further configured to:

[0313] Determine a loss function between the predicted image and the target image, and train an image processing model according to the loss function to obtain a trained image processing model.

[0314] An image processing model training device provided by an embodiment of the present specification can use multiple reference images at different viewing angles to accurately capture the image of the target object from multiple viewing angles by determining multiple reference images of the target object at different viewing angles, thereby avoiding the random generation of invisible areas, and obtain reference network features related to the reference image by obtaining reference image features of each reference image and reference posture features of each reference posture image. The reference network features and target posture features of the target posture image can be used to guide the process of restoring an image from a noise image in the image processing model, thereby obtaining accurate target prediction features, that is, the target prediction features include features related to the reference image in the reference image features and target posture features of the target posture image. On the basis of obtaining a predicted image based on the target prediction features, the accuracy of the generated predicted image is improved, and by accurately capturing the image of the target object from multiple viewing angles, each detail of the target object can be restored in the predicted image with reference to the image of the target object at each viewing angle, thereby ensuring the generation quality of the target object in the predicted image.

[0315] The above is a schematic scheme of an image processing model training device of this embodiment. It should be noted that the technical scheme of the image processing model training device and the technical scheme of the above-mentioned image processing model training method belong to the same concept, and the details not described in detail in the technical scheme of the image processing model training device can be found in the description of the technical scheme of the above-mentioned image processing model training method.

[0316] Corresponding to the above method embodiment, this specification also provides a video processing model training device embodiment, Fig.13 FIG. 1 is a schematic diagram showing the structure of a video processing model training device provided by an embodiment of the present specification. Fig.13 As shown, the device comprises:

[0317] An image determination module 1302 is configured to determine a plurality of reference images of a target object at different viewing angles, a reference pose image of the target object in each reference image, a target video of the target object, a target pose image sequence of the target object in the target video, and a noise image sequence corresponding to the target video;

[0318] A feature acquisition module 1304 is configured to acquire reference image features of each reference image, reference posture features of each reference posture image, and a target posture feature sequence of the target posture image sequence;

[0319] The feature acquisition module 1306 is configured to obtain a reference network feature according to the reference image feature and the reference posture feature, obtain a candidate prediction feature sequence according to the reference network feature, the target posture feature sequence and the noise image sequence, and obtain a target prediction feature sequence according to the candidate prediction feature sequence and the temporal relationship between the candidate prediction features;

[0320] The video acquisition module 1308 is configured to acquire a predicted video according to the target prediction feature sequence, wherein the posture of the target object in the predicted video is determined by referring to the posture of the target object in the target video;

[0321] The model training module 1310 is configured to train the video processing model according to the predicted video and the target video to obtain a trained video processing model, wherein the video processing model is determined based on the trained image processing model in the above-mentioned image processing model training method.

[0322] Optionally, the feature acquisition module 1304 is further configured to:

[0323] Inputting the plurality of reference images, the reference pose images and the target pose image sequence into the image processing model;

[0324] In the image processing model, feature extraction is performed on the multiple reference images, the reference posture images, and the target posture image sequence to obtain reference image features of the reference images, reference posture features of the reference posture images, and a target posture feature sequence of the target posture image sequence.

[0325] Optionally, the feature acquisition module 1306 is further configured to:

[0326] The image processing model is used to obtain reference network features according to the reference image features and the reference posture features, and a candidate prediction feature sequence is obtained according to the reference network features, the target posture feature sequence and the noise image sequence.

[0327] Optionally, the feature acquisition module 1306 is further configured to:

[0328] The temporal attention layer is used to perform temporal attention processing on the candidate prediction features in the candidate prediction feature sequence and the temporal relationship between the candidate prediction features to obtain a target prediction feature sequence.

[0329] Optionally, the model training module 1310 is further configured to:

[0330] According to the predicted video and the target video, the network parameters of the temporal attention layer in the denoising network unit of the image processing model are adjusted to obtain the video processing model.

[0331] The video processing model training device provided in the embodiment of this specification performs an attention operation on the candidate prediction feature sequence output by the cross attention layer in the denoising network unit through the temporal attention layer, and learns the correlation between different frames, thereby enabling the video processing model to have the ability to generate continuous images and the generated video is smoother.

[0332] The above is a schematic scheme of a video processing model training device of this embodiment. It should be noted that the technical scheme of the video processing model training device and the technical scheme of the video processing model training method described above belong to the same concept, and the details not described in detail in the technical scheme of the video processing model training device can be found in the description of the technical scheme of the video processing model training method described above.

[0333] Fig.14 The block diagram of a computing device 1400 according to one embodiment of the present specification is shown. The components of the computing device 1400 include but are not limited to a memory 1410 and a processor 1420. The processor 1420 is connected to the memory 1410 via a bus 1430, and the database 1450 is used to store data.

[0334] The computing device 1400 also includes an access device 1440 that enables the computing device 1400 to communicate via one or more networks 1460. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 1440 may include one or more of any type of network interface, wired or wireless (e.g., a network interface card (NIC)), such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a World Wide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, and a near field communication (NFC).

[0335] In one embodiment of the present specification, the above components of the computing device 1400 and Fig.14 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Fig.14 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0336] The computing device 1400 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 1400 may also be a mobile or stationary server.

[0337] Among them, the processor 1420 is used to execute the following computer program / instructions, which, when executed by the processor, implement the steps of the above-mentioned image processing model training method, video processing model training method, image processing method, and video processing method.

[0338] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the computing device embodiment, since it is basically similar to the image processing model training method, video processing model training method, image processing method, and video processing method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the image processing model training method, video processing model training method, image processing method, and video processing method embodiments.

[0339] An embodiment of the present specification also provides a computer-readable storage medium, which stores a computer program / instruction, and when the computer program / instruction is executed by a processor, the steps of the above-mentioned image processing model training method, video processing model training method, image processing method, and video processing method are implemented.

[0340] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the computer-readable storage medium embodiment, since it is basically similar to the image processing model training method, video processing model training method, image processing method, and video processing method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the image processing model training method, video processing model training method, image processing method, and video processing method embodiments.

[0341] An embodiment of the present specification also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned image processing model training method, video processing model training method, image processing method, and video processing method.

[0342] The above is a schematic scheme of a computer program product of this embodiment. It should be noted that the technical scheme of the computer program product and the technical schemes of the above-mentioned image processing model training method, video processing model training method, image processing method, and video processing method belong to the same concept, and the details not described in detail in the technical scheme of the computer program product can be referred to the description of the technical schemes of the above-mentioned image processing model training method, video processing model training method, image processing method, and video processing method.

[0343] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0344] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0345] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0346] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0347] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. A method for training an image processing model, include: Determine a plurality of reference images of a target object at different viewing angles, a reference pose image of the target object in each reference image, a target image of the target object, a target pose image of the target object in the target image, and a noise image corresponding to the target image, wherein the pose of the target object in each reference image is different from that in the target image; Acquire reference image features of each reference image, reference posture features of each reference posture image, and target posture features of the target posture image; Obtaining reference network features according to the reference image features and the reference posture features, and obtaining target prediction features according to the reference network features, the target posture features and the noise image; Obtaining a predicted image according to the target prediction feature, wherein the posture of the target object in the predicted image is determined by referring to the posture of the target object in the target posture image; The image processing model is trained according to the predicted image and the target image to obtain a trained image processing model.

2. The image processing model training method according to claim 1, wherein the determining includes a plurality of reference images of the target object at different viewing angles, a reference posture image of the target object in each reference image, a target image of the target object, a target posture image of the target object in the target image, and a noise image corresponding to the target image. include: Determine a plurality of reference images of the target object at different viewing angles and a target image of the target object; Inputting the plurality of reference images and the target image into a pose estimation model respectively to obtain a reference pose image of the target object in each of the reference images and a target pose image of the target object in the target image; The target image is input into an image denoising model to obtain a noise image corresponding to the target image, wherein the posture estimation model and the image denoising model are machine learning models.

3. The image processing model training method according to claim 1 or 2, wherein the determining of multiple reference images of the target object at different viewing angles is performed. include: Determine a reference video of the target object, and obtain, from the reference video, a plurality of video frames of the target object at different viewing angles as reference images; or A plurality of reference images of the target object at different viewing angles are captured.

4. The image processing model training method according to claim 1, wherein the reference image features of each reference image, the reference posture features of each reference posture image, and the target posture features of the target posture image are obtained. include: Inputting the plurality of reference images, each reference pose image and the target pose image into the image processing model; In the image processing model, feature extraction is performed on each reference image, each reference posture image and the target posture image to obtain reference image features of each reference image, reference posture features of each reference posture image and target posture features of the target posture image.

5. According to the image processing model training method of claim 4, the image processing model comprises an image feature extraction unit and a posture feature extraction unit; The feature extraction is performed on each reference image, each reference posture image and the target posture image in the image processing model to obtain reference image features of each reference image, reference posture features of each reference posture image and target posture features of the target posture image. include: Utilizing the image feature extraction unit to extract image features from each reference image, and obtaining reference image features of each reference image; The posture feature extraction unit is used to extract posture features from each reference posture image and the target posture image, respectively, to obtain reference posture features of each reference posture image and target posture features of the target posture image.

6. The image processing model training method according to claim 1 or 5, wherein the image processing model further comprises a reference network unit and a denoising network unit. in, The reference network unit and the denoising network unit have the same network structure and shared initialization weights, and both the reference network unit and the denoising network unit include a spatial attention layer; The step of obtaining a reference network feature according to the reference image feature and the reference posture feature, and obtaining a target prediction feature according to the reference network feature, the target posture feature and the noise image includes: Using the spatial attention layer of the reference network unit, performing spatial attention processing on the reference image features and the reference posture features to obtain the reference network features; The spatial attention layer of the denoising network unit is used to perform spatial attention processing on the reference network features, the target posture features and the noise image to obtain the target prediction features.

7. According to the image processing model training method of claim 5, the image feature extraction unit comprises a first image feature extraction layer and a second image feature extraction layer; The step of extracting image features from each reference image using the image feature extraction unit to obtain reference image features of each reference image includes: Using the first image feature extraction layer to extract image features from each of the reference images, to obtain first reference image features of each of the reference images; The second image feature extraction layer is used to extract image features from each reference image to obtain second reference image features of each reference image, wherein the second reference image features include semantic features of the reference image.

8. The image processing model training method according to claim 7, wherein the image processing model further comprises a reference network unit and a denoising network unit. in, The reference network unit and the denoising network unit have the same network structure and shared initialization weights, and both the reference network unit and the denoising network unit include a spatial attention layer and a cross attention layer; The step of obtaining a reference network feature according to the reference image feature and the reference posture feature, and obtaining a target prediction feature according to the reference network feature, the target posture feature and the noise image includes: Using the spatial attention layer of the reference network unit, performing spatial attention processing on the first reference image feature and the reference posture feature to obtain the reference network feature; Using the spatial attention layer of the denoising network unit, spatial attention processing is performed on the reference network features, the target posture features and the noise image to obtain target key features; The cross-attention layer of the denoising network unit is used to perform cross-attention processing on the second reference image features and the target key features to obtain the target prediction features.

9. The image processing model training method according to claim 8, wherein the reference network unit comprises a plurality of spatial attention layers and a plurality of cross attention layers; The spatial attention layer of the reference network unit is used to perform spatial attention processing on the first reference image feature and the reference posture feature to obtain the reference network feature, include: Performing feature fusion on the first reference image feature and the reference posture feature, Obtaining a reference fusion feature, and performing spatial attention processing on the reference fusion feature in each spatial attention layer of the reference network unit to obtain the reference network feature; After performing spatial attention processing on the first reference image feature and the reference posture feature using the spatial attention layer of the reference network unit to obtain the reference network feature, the method further includes: At each cross-attention layer of the reference network unit, performing cross-attention processing on the second reference image feature and the reference network feature to obtain a reference cross-feature; The reference cross feature is determined as the reference fusion feature, and the step of performing spatial attention processing on the reference fusion feature in each spatial attention layer of the reference network unit to obtain the reference network feature is continued.

10. The image processing model training method according to claim 8, wherein the denoising network unit comprises a plurality of spatial attention layers and a plurality of cross attention layers; The spatial attention layer of the denoising network unit is used to perform spatial attention processing on the reference network features, the target posture features and the noise image to obtain the target key features, include: The target posture feature and the noise image are subjected to feature fusion to obtain denoising fusion features, and in each spatial attention layer of the denoising network unit, the reference network feature and the denoising fusion feature are subjected to spatial attention processing to obtain the target key features; The method of using the cross attention layer of the denoising network unit to perform cross attention processing on the second reference image feature and the target key feature to obtain the target prediction feature includes: In each cross-attention layer of the denoising network unit, cross-attention processing is performed on the second reference image features and the target key features to obtain the target prediction features, and the target prediction features are determined as the denoising fusion features, and the step of continuing to perform spatial attention processing on the reference network features and the denoising fusion features in each spatial attention layer of the denoising network unit to obtain the target key features.

11. The image processing model training method according to claim 1, wherein the image processing model comprises a prediction unit; According to the target prediction feature, a prediction image is obtained, include: The prediction unit is used to decode the target prediction feature to obtain a predicted image.

12. The image processing model training method according to claim 1, wherein the image processing model is trained according to the predicted image and the target image to obtain a trained image processing model, include: Determine a loss function between the predicted image and the target image, and train an image processing model according to the loss function to obtain a trained image processing model.

13. A method for training an image processing model, include: Determine a plurality of reference images of a target object at different viewing angles, a reference posture image of the target object in each reference image, a target image of the target object, a target posture image of the target object in the target image, and a noise image corresponding to the target image, and input the plurality of reference images, each reference posture image, the target posture image, and the noise image into an image processing model, wherein the posture of the target object in each reference image is different from that in the target image, and the image processing model includes a feature extraction unit, a reference network unit, a denoising network unit, and a prediction unit; Utilizing the feature extraction unit, obtaining reference image features of each reference image, reference posture features of each reference posture image, and target posture features of the target posture image; Using the reference network unit, according to the reference image feature and the reference posture feature, a reference network feature is obtained, and using the denoising network unit, according to the reference network feature, the target posture feature and the noise image, a target prediction feature is obtained; Using the prediction unit, obtaining a prediction image according to the target prediction feature, wherein the posture of the target object in the prediction image is determined by referring to the posture of the target object in the target posture image; The image processing model is trained according to the predicted image and the target image to obtain a trained image processing model.

14. A video processing model training method, include: Determine a plurality of reference images of a target object at different viewing angles, a reference pose image of the target object in each reference image, a target video of the target object, a target pose image sequence of the target object in the target video, and a noise image sequence corresponding to the target video; Acquire reference image features of each reference image, reference posture features of each reference posture image, and a target posture feature sequence of the target posture image sequence; Obtaining reference network features according to the reference image features and the reference posture features, obtaining a candidate prediction feature sequence according to the reference network features, the target posture feature sequence and the noise image sequence, and obtaining a target prediction feature sequence according to the candidate prediction feature sequence and the temporal relationship between the candidate prediction features; Obtaining a predicted video according to the target prediction feature sequence, wherein the posture of the target object in the predicted video is determined by referring to the posture of the target object in the target video; According to the predicted video and the target video, the video processing model is trained to obtain a trained video processing model, wherein the video processing model is determined based on the trained image processing model in the image processing model training method according to any one of claims 1 to 12.

15. The video processing model training method according to claim 14, wherein the reference image features of each reference image, the reference posture features of each reference posture image, and the target posture feature sequence of the target posture image sequence are obtained. include: Inputting the plurality of reference images, the reference pose images and the target pose image sequence into the image processing model; In the image processing model, feature extraction is performed on the multiple reference images, the reference posture images, and the target posture image sequence to obtain reference image features of the reference images, reference posture features of the reference posture images, and a target posture feature sequence of the target posture image sequence.

16. The video processing model training method according to claim 15, wherein the reference network features are obtained according to the reference image features and the reference posture features, and the candidate prediction feature sequence is obtained according to the reference network features, the target posture feature sequence and the noise image sequence, include: The image processing model is used to obtain reference network features according to the reference image features and the reference posture features, and a candidate prediction feature sequence is obtained according to the reference network features, the target posture feature sequence and the noise image sequence.

17. The video processing model training method according to claim 16, wherein a temporal attention layer is set in the denoising network unit of the image processing model; According to the candidate prediction feature sequence and the temporal relationship between the candidate prediction features, a target prediction feature sequence is obtained. include: The temporal attention layer is used to perform temporal attention processing on the candidate prediction features in the candidate prediction feature sequence and the temporal relationship between the candidate prediction features to obtain a target prediction feature sequence.

18. The video processing model training method according to claim 17, wherein the video processing model is trained according to the predicted video and the target video to obtain a trained video processing model, include: According to the predicted video and the target video, the network parameters of the temporal attention layer in the denoising network unit of the image processing model are adjusted to obtain a trained video processing model.

19. A video processing model training method, include: Determine multiple reference images of the target object at different viewing angles, reference posture images of the target object in each reference image, a target video of the target object, a target posture image sequence of the target object in the target video, and a noise image sequence corresponding to the target video, and input the multiple reference images, each reference posture image, the target posture image sequence, and the noise image sequence into a video processing model, wherein the video processing model is determined based on the trained image processing model in the image processing model training method according to any one of claims 1 to 12, and the video processing model includes a feature extraction unit, a reference network unit, a denoising network unit, and a prediction unit; Utilizing the feature extraction unit, obtaining reference image features of each reference image, reference posture features of each reference posture image, and a target posture feature sequence of the target posture image sequence; Using the reference network unit, according to the reference image feature and the reference posture feature, a reference network feature is obtained; using the denoising network unit, according to the reference network feature, the target posture feature sequence and the noise image sequence, a candidate prediction feature sequence is obtained; and according to the candidate prediction feature sequence, a target prediction feature sequence is obtained; Using the prediction unit, according to the target prediction feature sequence, a predicted video is obtained, wherein the posture of the target object in the predicted video is determined by referring to the posture of the target object in the target video; The video processing model is obtained by training according to the predicted video and the target video.

20. An image processing method, include: Determine a reference image and a target image, obtain a reference pose image of a reference object in the reference image based on the reference image, and obtain a target pose image of a target object in the target image and a noise image corresponding to the target image based on the target image; According to the reference image, the reference posture image, the target posture image and the noise image, an image is generated using an image processing model to obtain a generated image, wherein the image processing model is trained by any one of the image processing model training methods of claims 1-12, the appearance of the object in the generated image is determined by the appearance of the reference object, and the motion posture of the object in the generated image is determined by the motion posture of the target object.

21. The image processing method according to claim 20, wherein the determining of the reference image and the target image comprises: include: A reference image and a target image sent by a client are received, wherein the reference image and the target image are sent when a specific operation is triggered on a user interaction interface of the client.

22. A video processing method, include: Determine a reference image and a target video, obtain a reference pose image of a reference object in the reference image based on the reference image, and obtain a target pose sequence image of a target object in the target video and a noise sequence image corresponding to the target video based on the target video; According to the reference image, the reference posture image, the target posture sequence image and the noise sequence image, a video processing model is used to generate a video to obtain a generated video, wherein the video processing model is trained by any one of the video processing model training methods of claims 14-18, the appearance of the object in the generated video is determined by the appearance of the reference object, and the action posture of the object in the generated video is determined by the action posture of the target object.

23. The video processing method according to claim 22, wherein the determining of the reference image and the target video, include: A reference image and a target video sent by a client are received, wherein the reference image and the target video are sent when a specific operation is triggered on a user interaction interface of the client.

24. A computing device, include: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer program / instructions are executed by the processor, the steps of the method described in any one of claims 1 to 23 are implemented.

25. A computer-readable storage medium storing a computer program / instruction, which implements the steps of the method according to any one of claims 1 to 23 when executed by a processor.

26. A computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 23.

Citation Information

Cited By

  • Image processing model training method, image processing method, video processing model training method, and video processing method

    WO2026007667A1