Object pose transfer method and device, equipment and storage medium

Abstract

The application provides an object posture migration method and device, equipment and computer readable storage medium, and relates to the technical field of computer vision. The method comprises the following steps: acquiring posture data for representing limb actions of a source object in a video frame of the source object; the posture data comprises skeleton features representing an object skeleton posture and model features representing an object model posture; based on the skeleton features and the model features, dynamic texture data and texture coordinate data of a target object are respectively generated; according to the dynamic texture data and the texture coordinate data, a foreground migration image for displaying the target object is generated; the limb actions of the target object in the foreground migration image are the same as the limb actions of the source object in the video frame; and based on the foreground migration image, a migration video frame is generated. Through the object posture migration method provided by the application, a migration video frame with rich details can be generated.

Description

Technical Field

[0001] This application relates to the field of machine vision technology, and in particular to an object pose transfer method, apparatus, device, and computer-readable storage medium. Background Technology

[0002] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. Computer vision (CV) is a crucial branch of AI. In practical applications of computer vision, object pose transfer technology has a wide range of applications, such as the synthesis of virtual anchors or virtual teachers.

[0003] Object pose transfer technology can generate images / videos of a target object performing the same actions as the source object, given both a source object image / video and a target object image / video. However, traditional object pose transfer techniques often produce images / videos with insufficient detail and low realism. Summary of the Invention

[0004] This application provides an object pose transfer method, apparatus, device, and computer-readable storage medium, which can generate richly detailed transfer video frames.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides an object pose transfer method, comprising: acquiring pose data from a video frame of a source object to represent the limb movements of the source object; the pose data includes skeletal features representing the skeletal pose of the object and model features representing the pose of the object model; generating dynamic texture data and texture coordinate data of the target object based on the skeletal features and model features respectively; generating a foreground transfer image for displaying the target object based on the dynamic texture data and texture coordinate data; the limb movements of the target object in the foreground transfer image are the same as the limb movements of the source object in the video frame; and generating a transfer video frame based on the foreground transfer image.

[0007] In some embodiments of this application, obtaining skeleton features in a video frame of the source object to characterize the skeleton pose of the object includes: obtaining a set of skeleton key points of the source object in the video frame; and obtaining skeleton features based on the set of skeleton key points.

[0008] In some embodiments of this application, obtaining model features from video frames of the source object to characterize the pose of the object model includes: constructing a three-dimensional model of the source object based on the video frames; the three-dimensional model includes multiple model vertices; constructing a Laplacian matrix based on the multiple model vertices; determining multiple eigenvalues ​​corresponding to the Laplacian matrix and eigenvectors corresponding to each eigenvalue; determining N target eigenvalues ​​that meet preset requirements among the multiple eigenvalues; determining the vertex features of each model vertex based on the N eigenvectors corresponding to the N target eigenvalues; and projecting the vertex features of each model vertex onto a two-dimensional plane to obtain model features.

[0009] In some embodiments of this application, the texture coordinate data includes migration coordinate information corresponding to each migrated pixel in the foreground migration image; the dynamic texture data includes a migration texture map corresponding to each texture region; generating dynamic texture data and texture coordinate data based on skeleton features and model features includes: generating migration coordinate information corresponding to each migrated pixel according to skeleton features and model features; the migration coordinate information includes the coordinate information and confidence information of the migrated pixel in each texture region; generating a migration texture map corresponding to each texture region according to skeleton features and model features; generating a foreground migration image based on dynamic texture data and texture coordinate data includes: generating a foreground migration image based on the migration texture map corresponding to each texture region, the coordinate information and confidence information of each foreground pixel in each texture region.

[0010] In some embodiments of this application, generating a migrated video frame based on a foreground migration image includes: processing an initial background image according to skeleton features and model features to obtain a background migration image; and fusing the foreground migration image and the background migration image to obtain a migrated video frame.

[0011] In some embodiments of this application, fusing the foreground migration image and the background migration image to obtain a migrated video frame includes: determining the probability that a target pixel in the migrated video frame corresponding to each migrated pixel belongs to the background migration image based on the confidence information of each migrated pixel being located in each texture region; and determining the migrated video frame based on the probability that each target pixel belongs to the background migration image, the foreground migration image, and the background migration image.

[0012] In some embodiments of this application, the above-described object pose transfer method is implemented by a trained pose transfer model; the training process of the pose transfer model includes: acquiring original video frames from a target object video; extracting original skeleton features and original model features from the original video frames; inputting the original skeleton features and original model features into the pose transfer model to be trained to obtain predicted video frames; wherein, the pose prediction network is used to generate predicted texture data and predicted coordinate data based on the original skeleton features and original model features, generate a foreground prediction image based on the predicted texture data and predicted coordinate data, and generate predicted video frames based on the foreground prediction image; and adjusting the parameters of the pose transfer model to be trained based on the original video frames and predicted video frames to obtain the trained pose transfer model.

[0013] In some embodiments of this application, the pose transfer model includes a foreground generation network and a background generation network; the step of inputting the original skeleton features and original model features into the pose transfer model to be trained to obtain a predicted video frame includes: inputting the original skeleton features and original model features into the foreground generation network to obtain a foreground prediction image; inputting the original skeleton features, original model features, and an initial background image into the background generation network to obtain a background prediction image; and fusing the foreground prediction image and the background prediction image to obtain a predicted video frame.

[0014] In some embodiments of this application, the foreground generation network includes a texture image generation sub-network and a texture coordinate generation sub-network. The step of inputting the original skeleton features and original model features into the foreground generation network to obtain a foreground prediction image includes: inputting the original skeleton features and original model features into a pre-trained texture coordinate generation sub-network to obtain predicted coordinate data; the predicted coordinate data includes predicted coordinate information corresponding to each foreground prediction pixel in the foreground prediction image; the predicted coordinate information includes the coordinate information and confidence information of the foreground prediction pixel at each texture location; inputting the original skeleton features and original model features into the texture image generation sub-network to obtain the predicted texture data, the predicted texture data including a predicted texture map corresponding to each texture location; and generating a foreground prediction image based on the predicted texture map corresponding to each texture location, the coordinate information of each foreground prediction pixel at each texture location, and the confidence information.

[0015] In some embodiments of this application, the texture image generation subnetwork includes an encoding layer and a decoding layer; the step of inputting the original skeleton features and the original model features into the texture image generation subnetwork to obtain a predicted texture map corresponding to each texture part includes: projecting the original model features onto the texture space to obtain input texture features; inputting the input texture features, the original skeleton features, and the original model features into the encoding layer to obtain encoded features; and inputting the encoded features into the decoding layer to obtain a predicted texture map corresponding to each texture part.

[0016] In some embodiments of this application, the decoding layer includes a local decoder corresponding to each texture region; the step of inputting the encoded features into the decoding layer to obtain the texture prediction map corresponding to each texture region includes: inputting the encoded features into the local decoder corresponding to each texture region respectively; and obtaining the texture prediction map corresponding to each texture region output by the local decoder corresponding to each texture region.

[0017] In some embodiments of this application, fusing the foreground prediction image and the background prediction image to obtain a predicted video frame includes: determining the probability that the output pixel corresponding to each foreground prediction pixel in the predicted video frame belongs to the background prediction image based on the confidence information of each foreground prediction pixel being located in each texture region; and determining the predicted video frame based on the probability that each output pixel belongs to the background prediction image, the foreground prediction image, and the background prediction image.

[0018] In some embodiments of this application, the method further includes: acquiring multiple video frames to be processed in the target video; performing background segmentation on the multiple video frames to be processed using a segmentation network to obtain multiple first background frames; processing the multiple first background frames using an image completion algorithm to obtain multiple second background frames; and performing average fusion processing on the multiple second background frames to obtain an initial background image.

[0019] In some embodiments of this application, the pre-training process of the texture coordinate generation sub-network includes: obtaining a texture coordinate training set; the texture coordinate training set includes multiple sample pose information and the real coordinate information corresponding to each sample pose information; inputting each sample pose information into the texture coordinate generation sub-network to be trained to obtain the predicted coordinate information corresponding to each sample pose information; inputting the predicted coordinate information and the real coordinate information corresponding to each sample pose information into a preset coordinate loss function to determine the coordinate loss value; using the coordinate loss value to train the texture coordinate generation sub-network until the coordinate loss value converges, and outputting the pre-trained texture coordinate generation sub-network.

[0020] In some embodiments of this application, adjusting the parameters of the pose transfer model to be trained based on the original video frames and predicted video frames to obtain a trained pose transfer model includes: calculating the value of a first loss function based on pose information, the original video frames, and the predicted video frames; calculating the value of a second loss function based on the original video frames and the predicted video frames; calculating a total loss value based on the values ​​of the first loss function and the second loss function; and adjusting the parameters of the pose transfer model to be trained based on the total loss value until a preset convergence condition is reached, and taking the pose transfer model to be trained when the preset convergence condition is reached as the trained pose transfer model.

[0021] In some embodiments of this application, the method further includes: acquiring neighboring predicted video frames of the predicted video frame; the neighboring predicted video frames are predicted images of the previous frame of the original video frame; inputting the neighboring predicted video frames into a preset optical flow prediction model to obtain the optical flow prediction image of the current frame; calculating the value of a third loss function based on the optical flow prediction image and the predicted video frame input; the calculation of the total loss value based on the values ​​of the first loss function and the second loss function includes: calculating the total loss value based on the values ​​of the first loss function, the second loss function, and the third loss function.

[0022] This application provides an object pose transfer device, the device comprising:

[0023] The acquisition module is used to acquire pose data from the video frames of the source object that represent the limb movements of the source object; the pose data includes skeletal features representing the skeletal pose of the object and model features representing the pose of the object model.

[0024] The first generation module is used to generate dynamic texture data and texture coordinate data of the target object based on skeleton features and model features, respectively.

[0025] The second generation module is used to generate a foreground migration image for displaying the target object based on dynamic texture data and texture coordinate data; the limb movements of the target object in the foreground migration image are the same as the limb movements of the source object in the video frame.

[0026] The third generation module is used to generate migration video frames based on the foreground migration image.

[0027] This application provides an object pose transfer device, including:

[0028] Memory, used to store executable instructions;

[0029] The processor, when executing executable instructions stored in the memory, implements the object pose transfer method provided in the embodiments of this application.

[0030] This application provides a computer-readable storage medium storing executable instructions for inducing a processor to execute and implement the object pose transfer method provided in this application.

[0031] The embodiments of this application have the following beneficial effects:

[0032] This application embodiment obtains skeleton features representing the object's skeleton posture and model features representing the object's model posture from video frames. It can accurately obtain the source object's limb movements in the current video frame from both the skeleton and object model dimensions. Since the feature information of the source object's limb movements is extracted from two dimensions, "from the inside out," the accuracy of the source object's limb movements to the target object's limb movements can be improved during the object posture transfer process, exhibiting strong robustness. Furthermore, since dynamic texture data and texture coordinate data are generated based on the skeleton features and model features, respectively, the generated dynamic texture data and texture coordinate data can change with different limb movements, improving the restoration of texture details during the object posture transfer process. The generated transfer video frames are richer in detail and have higher realism. Attached Figure Description

[0033] Figure 1 This is an optional architecture diagram of the object pose transfer system provided in the embodiments of this application;

[0034] Figure 2 This is a schematic diagram of the structure of the object pose transfer device provided in the embodiments of this application;

[0035] Figure 3 This is an optional flowchart illustrating the object pose transfer method provided in an embodiment of this application;

[0036] Figure 4 This is an optional flowchart illustrating the object pose transfer method provided in an embodiment of this application;

[0037] Figure 5 This is an optional flowchart illustrating the object pose transfer method provided in an embodiment of this application;

[0038] Figure 6 This is an optional flowchart illustrating the object pose transfer method provided in an embodiment of this application;

[0039] Figure 7A This is an optional flowchart illustrating the object pose transfer method provided in an embodiment of this application;

[0040] Figure 7B This is an optional architecture diagram of the attitude transfer model provided in the embodiments of this application;

[0041] Figure 8A This is an optional flowchart illustrating the object pose transfer method provided in an embodiment of this application;

[0042] Figure 8B This is a schematic diagram of an optional architecture of the foreground generation network provided in an embodiment of this application;

[0043] Figure 9 This is an optional flowchart illustrating the object pose transfer method provided in an embodiment of this application;

[0044] Figure 10 This is an optional schematic diagram of the attitude transfer process provided in an embodiment of this application;

[0045] Figure 11 This is an optional schematic diagram of the texture image generation process provided in the embodiments of this application. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0047] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0048] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0049] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0050] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described in this embodiment can be implemented in an order other than that illustrated or described in this embodiment.

[0051] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0052] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0053] (1) Image translation: Transforming the image representation of an object into another image representation of the object, that is, finding a function that can map an image in domain A to domain B.

[0054] (2) Screen space: refers to the space where the rendering result is located, that is, the space defined on the object image that the user directly sees.

[0055] (3) Texture space: refers to the space where the texture map is located, also called UV space, which is the space defined on the texture map. In the traditional graphics rendering pipeline, points on the texture space (texture map) can be mapped onto the surface of the 3D model after UV mapping, and then rendered onto the 2D image.

[0056] See Figure 1 , Figure 1 This is an optional architecture diagram of the object pose transfer system 100 provided in the embodiments of this application. In order to support an object pose transfer application, the terminal (terminal 400-1 and terminal 400-2 are shown as examples) connects to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two. Figure 1 It is also shown that server 200 can be a server cluster, which includes servers 200-1 to 200-3. Similarly, servers 200-1 to 200-3 can be physical machines or virtual machines built using virtualization technologies (such as container technology and virtual machine technology). This application embodiment does not limit this; of course, in this embodiment, a single server can also be used to provide services. The object can be an object with multiple postures, such as a human body, an animal, or a robot.

[0057] This application pertains to Computer Vision (CV). Computer vision is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in tasks such as target recognition, tracking, and measurement, and further performs image processing to create images more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision researches related theories and technologies, attempting to establish artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0058] See Figure 2 , Figure 2 This is a schematic diagram of the structure of the object pose transfer device 500 provided in the embodiments of this application. Figure 2 The illustrated object pose transfer device 500 includes at least one processor 510, a memory 550, at least one network interface 520, and a user interface 530. The various components in the object pose transfer device 500 are coupled together via a bus system 540. It is understood that the bus system 540 is used to implement communication between these components. In addition to a data bus, the bus system 540 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2 The general labeled all buses as Bus System 540.

[0059] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0060] User interface 530 includes one or more output devices 531 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 530 also includes one or more input devices 532, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0061] Memory 550 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 550 described in this application embodiment is intended to include any suitable type of memory. Memory 550 may optionally include one or more storage devices physically located away from processor 510.

[0062] In some embodiments of this application, memory 550 is capable of storing data to support various operations. Examples of such data include programs, modules, and data structures, or subsets or supersets thereof, as illustrated below.

[0063] Operating system 551 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0064] The network communication module 552 is used to reach other computing devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0065] Display module 553 is configured to enable the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 531 (e.g., a display screen, a speaker, etc.) associated with user interface 530;

[0066] The input processing module 554 is used to detect and translate one or more user inputs or interactions from one or more input devices 532.

[0067] In some embodiments of this application, the object pose transfer device provided in the embodiments of this application can be implemented in a combination of hardware and software. As an example, the object pose transfer device provided in the embodiments of this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the object pose transfer method provided in the embodiments of this application.

[0068] In some embodiments of this application, the object pose transfer device provided in this application can be implemented in software. Figure 2An object pose transfer device 555 stored in memory 550 is shown. It can be software in the form of programs and plug-ins, including the following software modules: acquisition module 5551, first generation module 5552, second generation module 5553 and third generation module 5554. These modules are logical and can therefore be arbitrarily combined or further split according to the functions implemented.

[0069] The functions of each module will be explained below.

[0070] In other embodiments, the apparatus provided in this application can be implemented in hardware. As an example, the apparatus provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the object pose transfer method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0071] This application will describe the object pose transfer method provided by this application, taking the terminal as the execution subject, in conjunction with exemplary applications and implementations of the terminal provided in the embodiments of this application. The following description uses a person as an example to illustrate the object pose transfer method of this application.

[0072] See Figure 3 , Figure 3 This is an optional flowchart illustrating an object pose transfer method provided in an embodiment of this application, which will be combined with... Figure 3 The steps shown are explained.

[0073] In step 301, posture data representing the limb movements of the source person is obtained from the video frames of the source person; the posture data includes skeletal features representing the posture of the person's skeleton and model features representing the posture of the person's model.

[0074] In some embodiments of this application, in order to achieve the task of object pose transfer, that is, to transfer the action pose of the source person to the target person and obtain an image of the target person with the same limb movements as the source person, it is necessary to extract pose data to represent the limb movements of the source person in step 301.

[0075] The video frame is extracted from the motion video corresponding to the source person. The motion video contains multiple consecutive video frames representing the source person's continuous limb movements. In this embodiment, pose transfer can be performed on each video frame in the motion video, or a portion of the video frames can be extracted for pose transfer, or only any single video frame can be subjected to pose transfer; this application does not impose any limitations on this.

[0076] In some embodiments of this application, the posture data includes skeletal features characterizing the posture of a human skeleton and model features characterizing the posture of a human model.

[0077] In some embodiments of this application, skeletal features representing the skeletal posture of a person can be obtained from a video frame containing the source person's skeleton by: obtaining a set of key points of the source person's skeleton in the video frame; and obtaining the skeletal features based on the set of key points. The above steps can be implemented using a preset skeleton recognition model: the video frame is input into the skeleton recognition model to obtain the skeletal features corresponding to the video frame; the skeletal features include, but are not limited to, the key points corresponding to the source person's skeleton, key point coordinates, key point order, and key point connection relationships. The skeleton recognition model can be a model built based on pose recognition algorithms such as AlphaPose or Openpose. For example, the skeleton recognition model can be used to detect human joints as key points and connect multiple key points to obtain skeletal features representing the skeletal posture of the person.

[0078] In some embodiments of this application, the model features used to characterize the pose of a person in a video frame can be obtained by the following method: constructing a three-dimensional model of the source person based on the video frame; the three-dimensional model includes multiple model vertices; constructing a Laplacian matrix based on the multiple model vertices; determining multiple eigenvalues ​​corresponding to the Laplacian matrix and eigenvectors corresponding to each eigenvalue; determining N target eigenvalues ​​that meet preset requirements among the multiple eigenvalues; determining the vertex features of each model vertex based on the N eigenvectors corresponding to the N target eigenvalues; and projecting the vertex features of each model vertex onto a two-dimensional plane to obtain the model features.

[0079] Among them, the obtained feature values ​​can be sorted according to their numerical values, and the N feature values ​​with smaller values ​​can be determined as the N target feature values ​​that meet the preset requirements.

[0080] For example, if the 3D model corresponding to the video frame includes M model vertices, a Laplacian matrix of size M*M can be obtained based on the vertex coordinates of these M model vertices. Correspondingly, multiple eigenvalues ​​and an eigenvector corresponding to each eigenvalue can be obtained from this Laplacian matrix. This eigenvector is an M-dimensional vector. According to the magnitude relationship of each eigenvalue, the N eigenvalues ​​with the smallest eigenvalues ​​are taken as target eigenvalues. And based on the target eigenvectors corresponding to the N target eigenvalues, the vertex features of the M model vertices are obtained. The dimension of these vertex features is N. In the process of projecting the vertex features of each model vertex into the screen space, a rasterization algorithm in computer graphics can be used to project the vertex features of each model vertex in 3D space into the screen space (2D plane) to obtain the model features.

[0081] For example, if there are 4 vertices, a 4x4 Laplacian matrix can be obtained. Based on this Laplacian matrix, we can obtain the eigenvalue P1 and its corresponding eigenvector O1 (O11, O12, O13, O14); the eigenvalue P2 and its corresponding eigenvector O2 (O21, O22, O23, O24); the eigenvalue P3 and its corresponding eigenvector O3 (O31, O32, O33, O34); and the eigenvalue P4 and its corresponding eigenvector O4 (O41, O42, O43, O44). Taking the three smallest eigenvalues ​​P1, P2, and P3, based on the corresponding O1, O2, and O3, we can obtain the vertex features of each vertex as (O11, O21, O31), (O21, O22, O32), and (O31, O32, O33), respectively. The dimension of each vertex feature is 3.

[0082] In step 302, dynamic texture data and texture coordinate data of the target character are generated based on skeleton features and model features, respectively.

[0083] In some embodiments of this application, step 302 may include generating dynamic texture data of the target character based on the skeletal features and model features of the source character; and generating texture coordinate data of the target character based on the skeletal features and model features of the source character, for a total of two subtasks. The two subtasks can be executed in parallel or in an alternating order.

[0084] In some embodiments of this application, the dynamic texture data of the target person is used to characterize the dynamic texture data corresponding to the target person performing the same limb movements as the source person. This dynamic texture data is a two-dimensional planar image in texture space. After obtaining the dynamic texture data of the target person in texture space, this dynamic texture data in texture space can be mapped to screen space to obtain a foreground transition image in screen space. The texture coordinate data includes the mapping relationship between each pixel of the foreground transition image in screen space and each pixel of the dynamic texture data in texture space.

[0085] In step 303, a foreground migration image for displaying the target person is generated based on the dynamic texture data and texture coordinate data; the limb movements of the target person in the foreground migration image are the same as the limb movements of the source person in the video frame.

[0086] In some embodiments of this application, after obtaining the dynamic texture data in the texture space, the pixels in the dynamic texture data can be mapped to the screen space according to the mapping relationship between each pixel of the foreground migration image in the screen space and each pixel of the dynamic texture data in the texture space contained in the texture coordinate data, so as to obtain the foreground migration image in the screen space. The foreground migration image includes a target character with the same limb movements as the source character.

[0087] In step 304, a migration video frame is generated based on the foreground migration image.

[0088] In some embodiments of this application, the above-mentioned generation of a migration video frame based on the foreground migration image can be achieved by: obtaining a preset background image, fusing the foreground migration image and the background image, and generating the migration video frame. The preset background image can be a background image determined from a plurality of preset selectable background images based on user configuration operations; the background image can be a classroom background, street background, nature background, etc.; the preset background image can also be extracted from video data containing the target person.

[0089] In some embodiments of this application, during the process of fusing the foreground migration image and a preset background image to generate the migration video frame, a light and shadow image of the target person in the background image can be generated based on the pose data. The migration video frame is obtained by fusing the light and shadow image, the background image, and the foreground migration image. The migration video frame can be generated using layer rendering. For example, the foreground migration image can be used as the first layer, the light and shadow image as the second layer, and the background image as the third layer. The images corresponding to the three images are fused according to their priority from the first layer to the third layer to obtain the migration video frame. Alternatively, the migration video frame can be generated using pixel filling. For example, for each pixel in the migration video frame, the probability of each pixel belonging to each image is determined. Combining the probability of the pixel belonging to each image with the pixel value corresponding to the pixel in each image, a target pixel value is obtained. This target pixel value is then used to fill the pixel to obtain the migration video frame.

[0090] Through the embodiments of this application, for Figure 3 As can be seen from the above exemplary implementation, the embodiments of this application can accurately obtain the limb movements of the source person in the current video frame from both the skeleton dimension and the person model dimension by obtaining the skeleton features representing the posture of the person's skeleton and the model features representing the posture of the person model in the video frame. Since the feature information of the source person's limb movements is extracted from the two dimensions of "from the inside out", the accuracy of the limb movements of the source person to the limb movements of the target person can be improved during the object posture transfer process, and it has strong robustness. Furthermore, since dynamic texture data and texture coordinate data are generated based on the skeleton features and model features respectively, the generated dynamic texture data and texture coordinate data can change with different limb movements, which improves the restoration of texture details during the object posture transfer process, and the generated transfer video frame has richer details and higher realism.

[0091] See Figure 4 , Figure 4 This is an optional flowchart illustrating an object pose transfer method provided in an embodiment of this application, based on... Figure 3 , Figure 3 The shown step 302 can be updated to step 401, and step 303 can be updated to step 402, which will be combined Figure 4 The steps shown are explained.

[0092] In step 401, dynamic texture data and texture coordinate data of the target person are generated based on skeleton features and model features, respectively; the texture coordinate data includes the migration coordinate information corresponding to each migration pixel in the foreground migration image; the dynamic texture data includes the migration texture map corresponding to each texture part.

[0093] In this foreground transfer image, the transfer pixels used to represent the human body's movements are all pixels on the human body. In this embodiment, the human body is divided into multiple texture regions according to body parts, and each texture region corresponds to a transfer texture map. The texture region may include body parts such as the head, face, torso, arms, legs, and feet.

[0094] In some embodiments of this application, step 4011 can be used to generate texture coordinate data of the target character based on skeleton features and model features.

[0095] In step 4011, migration coordinate information corresponding to each migrated pixel is generated based on the skeleton features and model features; the migration coordinate information includes the coordinate information and confidence information of the migrated pixel in each texture location.

[0096] The migration coordinate information of the migrated pixel includes the confidence information of the migrated pixel in each of the texture regions. For example, if the confidence of the migrated pixel for each texture region is 0, it means that the migrated pixel in the foreground migration image does not belong to the limbs of the target person. If the confidence of the migrated pixel for the torso is 90%, the confidence for the arm is 10%, and the confidence for the other texture regions is 0, it means that the migrated pixel in the foreground migration image may belong to the torso or the arm, and the probability of it belonging to the torso is relatively high. The corresponding posture in this case may be that the person is facing away from the camera and the arm is covered by the torso.

[0097] The migration coordinate information of the migrated pixel also includes the coordinate information of the migrated pixel in each of the aforementioned texture regions. Based on the above example, if the confidence level of the migrated pixel for the torso is 90% and the confidence level for the arm is 10%, the migration coordinate information also includes the relative coordinates of the migrated pixel in the migration texture map corresponding to the torso, and the relative coordinates of the migrated pixel in the migration texture map corresponding to the arm. These relative coordinates are used to characterize the mapping relationship between pixels in the migration texture map corresponding to each texture region in texture space and migrated pixels in the foreground migration image in screen space. It should be noted that if the confidence level of the migrated pixel for some texture regions is 0, then it is not necessary to obtain the coordinate information (relative coordinates) corresponding to these texture regions.

[0098] In some embodiments of this application, step 4012 can be used to generate dynamic texture data of the target character based on skeleton features and model features.

[0099] In step 4012, a migration texture map corresponding to each texture part is generated based on the skeleton features and model features.

[0100] In this process, considering that the texture data of the character object changes with different postures and actions, for example, for the transfer texture map of the arm, when the skeletal and model features indicate that the arm is in a horizontally extended state, the corresponding transfer texture map is in a flat, unfolded state; when the skeletal and model features indicate that the arm is in a curled state, the corresponding transfer texture map is in a compressed, wrinkled state. In step 4012, transfer texture maps of each texture part matching the source character's limb movements are generated based on the skeletal and model features.

[0101] In step 402, a foreground migration image is generated based on the migration texture map corresponding to each texture region, the coordinate information and confidence information of each foreground pixel in each texture region.

[0102] In some embodiments of this application, the foreground migration image can be generated based on the migration texture map corresponding to each texture region, the coordinate information and confidence information of each foreground pixel in each texture region, in the following way: For each foreground pixel, the confidence of the foreground pixel in each texture region is determined, the texture region with the highest confidence is taken as the texture region to be mapped, the pixel to be migrated is determined in the migration texture map corresponding to the texture region to be mapped according to the coordinate information, the pixel value of the pixel to be migrated is taken as the pixel value of the foreground pixel, and so on, until the pixel value mapping of each foreground pixel is completed, and the foreground migration image is obtained.

[0103] In some embodiments of this application, the foreground migration image can also be generated based on the migration texture map corresponding to each texture region, the coordinate information and confidence information of each foreground pixel in each texture region, in the following manner: For each foreground pixel, determine the confidence of the foreground pixel in each texture region, and determine the pixel value of the foreground pixel in each texture region based on the coordinate information of the foreground pixel in each texture region and the migration texture map corresponding to each texture region; use the confidence of the foreground pixel in each texture region to perform a weighted summation of the pixel value of the foreground pixel in each region to obtain the pixel value of the foreground pixel in the foreground migration image, and so on, until the pixel value mapping of each foreground pixel is completed, and the foreground migration image is obtained. Based on the above embodiments, if the confidence level of the migrated pixel is 90% for the torso area, 10% for the arm area, and 0% for all other texture areas, then the pixel value of the migrated pixel in the torso area is 200, and the pixel value in the arm area is 30. Therefore, the pixel value of the migrated pixel in the migrated foreground image is 183. The above example uses the grayscale channel of the migrated pixel as an example; this application can also use the same method to determine the pixel values ​​of the migrated pixel in other channels such as R (red), G (green), and B (blue).

[0104] In some embodiments of this application, the foreground migration image can be generated using formula (1-1):

[0105]

[0106] Among them, I fg T represents the foreground transfer image. i Let φ represent the texture map of the i-th part, and φ is the mapping function that maps the texture map to screen space according to UV coordinates.

[0107] Through the embodiments of this application, for Figure 4 As can be seen from the above exemplary implementation, the embodiments of this application generate corresponding migration texture maps for different texture parts of the human body based on the skeleton features and the model features. In this way, not only can the generated migration texture map change dynamically with the different limb movements of the source person, but also dynamic texture maps that fit more closely to different texture parts can be generated, thereby enriching the details of the texture map and improving the accuracy of the object posture transfer process.

[0108] See Figure 5 , Figure 5 This is an optional flowchart illustrating an object pose transfer method provided in an embodiment of this application, based on... Figure 3 or Figure 4 ,by Figure 4 For example, Figure 4 Step 304 shown can be implemented through steps 501 and 502, combining Figure 5 The steps shown are explained.

[0109] In step 501, the initial background image is processed based on the skeleton features and model features to obtain the background migration image.

[0110] In some embodiments of this application, the initial background image can be processed with light and shadow matching the limb movements of the source character based on the skeleton features and the model features, so that the processed background migration image contains the corresponding light and shadow information.

[0111] In step 502, the foreground migration image and the background migration image are fused to obtain the migrated video frame.

[0112] In some embodiments of this application, the above-mentioned fusion of the foreground migration image and the background migration image can be achieved through steps 5021 and 5022 to obtain the migration video frame.

[0113] In step 5021, based on the confidence information of each migrated pixel being located in each texture region, the probability that the target pixel corresponding to each migrated pixel in the migrated video frame belongs to the background migrated image is determined.

[0114] Specifically, for each migrated pixel, the sum of confidence information of the migrated pixel being located in each of the texture regions is determined. This sum is the probability that the migrated pixel belongs to the foreground migrated image. Correspondingly, 1 minus the probability of belonging to the foreground migrated image yields the probability that the foreground pixel belongs to the background migrated image.

[0115] In step 5022, the migrated video frame is determined based on the probability that each target pixel belongs to the background migration image, the foreground migration image, and the background migration image.

[0116] The above step 5022 can be implemented in the following way: obtain a preset background probability threshold; when the probability that the migrated pixel belongs to the background migration image is greater than or equal to the background probability threshold, fill the pixel of the migrated pixel in the migration video frame with the pixel value corresponding to the migrated pixel in the background migration image; when the probability that the migrated pixel belongs to the background migration image is less than the background probability threshold, fill the pixel of the migrated pixel in the migration video frame with the pixel value corresponding to the migrated pixel in the foreground migration image.

[0117] Step 5022 above can also be achieved using the following formula (1-2):

[0118] I syn =Ifg ⊙(1-P0)+I bg ⊙P0 formula (1-2);

[0119] Among them, I syn Indicates the migration of video frames, I fg Represents a foreground transfer image, I bg Let P0 represent the probability that a pixel belongs to the background-transferred image, and ⊙ (element-wise production) represent the multiplication operation performed on the pixels in the image. For example, when calculating the pixel value of the first pixel, I can be used... fg The pixel value of the first pixel is multiplied by the probability that the first pixel belongs to the background migration image.

[0120] Through the embodiments of this application, for Figure 5 As can be seen from the above exemplary implementation, the embodiments of this application process the initial background image according to the skeleton features and model features to obtain a background migration image containing light and shadow information, which can improve the realism of the migration video frame; and since the light and shadow information is obtained based on the skeleton features and model features, the light and shadow information can be closer to the light and shadow state in the real scene, thus further improving the realism of the migration video frame; at the same time, based on the confidence information of each migration pixel point in each texture part, this application determines the probability that the target pixel point corresponding to each migration pixel point in the migration video frame belongs to the background migration image, which can improve the accuracy of fusion in the process of fusing the foreground migration image and the background migration image, thereby improving the accuracy of the object pose migration process.

[0121] In some embodiments of this application, Figure 3 The object pose transfer method in [the document] can be implemented using a trained pose transfer model; see [link / reference]. Figure 6 , Figure 6 This is an optional flowchart illustrating the pose transfer model training method provided in this application embodiment, which will be combined with... Figure 6 The steps shown are explained.

[0122] In step 601, the original video frames from the target person's video are obtained;

[0123] In step 602, the original skeleton features and original model features are extracted from the original video frames;

[0124] In step 603, the original skeleton features and the original model features are input into the pose transfer model to be trained to obtain the predicted video frame; wherein, the pose prediction network is used to generate predicted texture data and predicted coordinate data based on the original skeleton features and the original model features, generate a foreground prediction image based on the predicted texture data and the predicted coordinate data, and generate the predicted video frame based on the foreground prediction image.

[0125] In step 604, the parameters of the pose transfer model to be trained are adjusted based on the original video frames and the predicted video frames to obtain the trained pose transfer model.

[0126] Through the embodiments of this application, for Figure 6 As can be seen from the above exemplary implementation, the pose transfer model trained through the above embodiments can accurately obtain the limb movements of the target person in the current video frame from both the skeleton dimension and the person model dimension. Since the feature information of the target person's limb movements is extracted from the two dimensions of "from the inside out", the pose information input to the pose transfer model can be more accurate, thereby ensuring that the pose transfer model can quickly learn the conversion process between different pose information and different texture information during the training process. Furthermore, since predicted texture data and predicted coordinate data are generated based on the skeleton features and model features respectively, the pose transfer model can generate different predicted texture data and predicted coordinate data according to different input pose data, which improves the restoration of texture details during the object pose transfer process, and the generated predicted video frame details are richer and more realistic.

[0127] See Figure 7A Figure 7A This is an optional flowchart illustrating the pose transfer model training method provided in this application embodiment, based on... Figure 6 , Figure 6 Step 603 in the process can be updated to steps 701, 702, and 703, which will combine Figure 7A The steps shown are explained.

[0128] In step 701, the original skeleton features and the original model features are input into the foreground generation network to obtain the foreground prediction image.

[0129] In step 702, the original skeleton features, the original model features, and the initial background image are input into the background generation network to obtain the background prediction image.

[0130] In some embodiments of this application, the initial background image can be obtained by: acquiring multiple video frames to be processed from the target person's video; performing background segmentation on the multiple video frames to be processed using a segmentation network to obtain multiple first background frames; processing the multiple first background frames using an image completion algorithm to obtain multiple second background frames; and performing average fusion processing on the multiple second background frames to obtain the initial background image.

[0131] The process can involve either frame-by-frame extraction to obtain multiple video frames to be processed from the target person's video, or extraction of a subset of video frames from the target person's video at preset intervals. The segmentation network can be a U-net network, which performs background segmentation on the acquired video frames, specifically segmenting each frame into foreground (person) and background regions, resulting in multiple first background frames with the foreground removed. The image completion algorithm can be the Deepfill v2 algorithm, which performs image completion processing on the multiple first background frames to obtain multiple second background frames. Finally, for each pixel coordinate in the initial background image, the pixel value at that coordinate is averaged across each second background frame to obtain the pixel value at that coordinate, thus completing the averaging and fusion process to generate the initial background image.

[0132] In step 703, the foreground prediction image and the background prediction image are fused to obtain the predicted video frame.

[0133] Please see Figure 7B The pose transfer model 711 includes a foreground generation network 711 and a background generation network 712. The foreground generation network 711 is used to generate a corresponding foreground prediction image based on the input original skeleton features and original model features. The background generation network 712 is used to generate a corresponding background prediction image based on the initial background image, the input original skeleton features and original model features. The pose transfer model 711 is also used to fuse the generated foreground prediction image and background prediction image into a prediction video frame.

[0134] Through the embodiments of this application, for Figure 7A As can be seen from the above exemplary implementation, the pose transfer model in this application is divided into a foreground generation network and a background generation network. This allows the foreground generation network to focus on learning the relationship between human pose and texture image and texture coordinates, while the background generation network can focus on learning the influence of human pose on the initial background image. In this way, the generated foreground and background prediction images become more realistic, resulting in more accurate predicted video frames.

[0135] See Figure 8A , Figure 8AThis is an optional flowchart illustrating the pose transfer model training method provided in this application embodiment, based on... Figure 7A , Figure 7A Step 701 can be implemented through steps 801 to 803, combining Figure 8A The steps shown are explained.

[0136] In step 801, the original skeleton features and the original model features are input into the pre-trained texture coordinate generation sub-network to obtain predicted coordinate data. The predicted coordinate data includes the predicted coordinate information corresponding to each foreground predicted pixel in the foreground predicted image. The predicted coordinate information includes the coordinate information and confidence information of the foreground predicted pixel at each texture location.

[0137] In some embodiments of this application, the pre-training process of the pre-trained texture coordinate generation sub-model includes: obtaining a texture coordinate training set; the texture coordinate training set includes multiple sample pose information and the real coordinate information corresponding to each sample pose information; inputting each sample pose information into the texture coordinate generation sub-network to be trained to obtain the predicted coordinate information corresponding to each sample pose information; inputting the predicted coordinate information and the real coordinate information corresponding to each sample pose information into a preset coordinate loss function to determine the coordinate loss value; using the coordinate loss value to train the texture coordinate generation sub-network until the coordinate loss value converges, and outputting the pre-trained texture coordinate generation sub-network.

[0138] The coordinate loss function can be represented by formula (2-3):

[0139]

[0140] in, It is the cross-entropy loss, and P and C are the prediction results of the texture coordinate generation subnetwork. and It is the ground truth.

[0141] In step 802, the original skeleton features and the original model features are input into the texture image generation sub-network to obtain predicted texture data, which includes the predicted texture map corresponding to each texture part.

[0142] In some embodiments of this application, the texture image generation subnetwork includes an encoding layer and a decoding layer. Steps 8021 to 8023 can be used to input the pose information into the texture image generation subnetwork to obtain a predicted texture map corresponding to each texture region.

[0143] In step 8021, the original model features are projected onto the texture space to obtain the input texture features.

[0144] In step 8022, the input texture features, original skeleton features, and original model features are input into the encoding layer to obtain the encoded features.

[0145] In step 8023, the encoded features are input to the decoding layer to obtain the corresponding texture prediction map for each texture region.

[0146] In some embodiments of this application, the decoding layer includes a local decoder corresponding to each texture region, and the above step 8023 can be implemented in the following way: inputting the encoded features into the local decoder corresponding to each texture region respectively; and obtaining the texture prediction map corresponding to each texture region output by the local decoder corresponding to each texture region.

[0147] In some embodiments of this application, the encoding layer may further include a local encoder corresponding to each texture region, the local encoder corresponding to each texture region outputs the encoded features corresponding to the texture region, and the encoded features corresponding to each texture region are input to the local decoder corresponding to each texture region to obtain the texture prediction map corresponding to each texture region output by the local decoder corresponding to each texture region.

[0148] In step 803, a foreground prediction image is generated based on the predicted texture map corresponding to each texture region, the coordinate information and confidence information of each foreground prediction pixel at each texture region.

[0149] Please see Figure 8B The foreground generation network 711 includes a texture coordinate generation subnetwork 7111 and a texture image generation subnetwork 7112. The texture coordinate generation subnetwork 7111 is used to generate corresponding predicted coordinate data based on the input original skeleton features and original model features. The texture image generation subnetwork 7112 is used to generate corresponding predicted texture data based on the input original skeleton features and original model features. The foreground generation network 711 is also used to generate a foreground prediction image based on the predicted coordinate data and the predicted texture data.

[0150] In some embodiments of this application, based on the above embodiments, Figure 7A Step 703 can be implemented in the following way: based on the confidence information of each foreground predicted pixel being located in each texture region, determine the probability that the output pixel corresponding to each foreground predicted pixel in the predicted video frame belongs to the background predicted image; based on the probability that each output pixel belongs to the background predicted image, the foreground predicted image and the background predicted image, determine the predicted video frame.

[0151] Through the embodiments of this application, for Figure 8AAs can be seen from the above exemplary implementations, the foreground generation network in this application is divided into a texture coordinate generation subnetwork and a texture image generation subnetwork. This allows the texture image generation subnetwork to focus on the relationship between the predicted texture map of each texture part and the model features and skeleton features. Furthermore, it can generate dynamic predicted texture maps corresponding to each texture part based on the input model features and skeleton features, thus enriching the details of the texture image. Simultaneously, the texture coordinate generation subnetwork can focus on learning the influence of human pose on the mapping relationship between texture space and screen space, improving the accuracy of the texture mapping process. In this way, the generated foreground prediction image can be made more realistic.

[0152] See Figure 9 , Figure 9 This is an optional flowchart illustrating the pose transfer model training method provided in this application embodiment, based on... Figure 6 , Figure 6 Step 604 can be achieved through steps 901 to 903, combining... Figure 9 The steps shown are explained.

[0153] In step 901, the value of the first loss function is calculated based on the pose information, the original video frame, and the predicted video frame.

[0154] The first loss function can be represented by formula (1-4):

[0155]

[0156] Among them, I pose I syn and I real Let represent pose information, predicted video frame, and original video frame, respectively. Let D represent the discriminator, and log be the natural logarithm. This represents the expectation based on the training data. (Through...) This allows the discriminator to learn the ability to distinguish between real and fake video frames, through... This enables the pose transfer model to learn and generate the ability to "fool" the discriminator.

[0157] In step 902, the value of the second loss function is calculated based on the original video frame and the predicted video frame.

[0158] The second loss function can be represented by equation (1-5):

[0159]

[0160] Among them, I pose I syn and I realRepresenting pose information, predicted video frame, and original video frame respectively, VGG indicates that a network using VGG-19 extracts features from the image, and λ f and λ t For weight parameters. ||VGG(I syn )-VGG(I real The goal is to make the generated results as similar as possible to the actual results within the feature space constraints. syn -I real ||2 indicates that the generated result is similar to the true result under image space constraints. λ f and λ t This is used to adjust the preceding and following terms to the same order of magnitude. In some embodiments, λ is selected. f =10, λ l =200.

[0161] In step 903, the total loss value is calculated based on the values ​​of the first loss function and the second loss function, and the parameters of the pose transfer model to be trained are adjusted based on the total loss value until a preset convergence condition is reached. The pose transfer model to be trained when the preset convergence condition is reached is taken as the trained pose transfer model.

[0162] In some embodiments of this application, the method further includes: obtaining neighboring predicted video frames of the predicted video frame; the neighboring predicted video frames are the predicted images of the previous frame of the original video frame; inputting the neighboring predicted video frames into a preset optical flow prediction model to obtain the optical flow prediction image of the current frame; and calculating the value of a third loss function based on the optical flow prediction image and the predicted video frame input.

[0163] The third loss function can be represented by formula (1-6):

[0164]

[0165] Among them, f t (·) is a function that transforms adjacent predicted video frames into an optical flow predicted image. D = H × W is the dimension of the output image, and I(k) represents the pixel at position k. k The optical flow confidence level is represented as follows: 0 indicates that the optical flow at that pixel is unreliable, and 1 indicates that it is reliable. Here, the optical flow confidence level is c. k Both optical flow and optical flow are pre-calculated on the training data.

[0166] The above calculation of the total loss value based on the values ​​of the first loss function and the second loss function can be achieved in the following way: the total loss value is calculated based on the values ​​of the first loss function, the second loss function, and the third loss function.

[0167] Through the embodiments of this application, for Figure 9As can be seen from the above exemplary implementations, the embodiments of this application can improve the optimization efficiency of model parameters by adjusting the parameters of the overall pose transfer model by fusing the values ​​of the first loss function, the second loss function, and the third loss function. Simultaneously, it allows the texture coordinate generation subnetwork to focus on learning the influence of human pose on the mapping relationship between texture space and screen space, improving the accuracy of the texture mapping process. Thus, the generated foreground prediction image can be made more realistic.

[0168] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0169] Object pose transfer technology has a wide range of applications, such as the synthesis of virtual anchors or virtual teachers. Specifically, given two sets of videos (source person video and target person video), the goal of human pose transfer is to generate a motion video of the target person performing the same actions as the source person. This application's embodiments primarily combine image translation technology and explicit 3D representation through differentiable rendering and couple it into the video generation framework.

[0170] In this application embodiment, by decoupling the texture map and texture coordinates and predicting them separately through two different modules, the transfer task can be completed more robustly. At the same time, compared with using a static texture map, this application obtains a dynamic texture through a generator, which improves the detail richness of the generated result. In addition, this application also improves the coherence of the generated video by introducing temporal constraints.

[0171] For the field of human pose transfer, the following related technologies exist: (1) Encoding the input image into deep features with strong representational capabilities through neural networks, and then decoding to generate results with rich details; such as Everybody dance now by Chan et al. and Video-to-video synthesis by Wang et al. (2) Human pose transfer is achieved by relying on static texture representation, such as Textured neural avatars by Shysheya et al. (3) Human pose transfer is achieved by relying on accurate 3D human models, such as Neural human video rendering by learning dynamic textures and rendering-to-video translation by Liu et al.

[0172] The applicant's research revealed the following shortcomings in existing human pose transfer techniques: (1) poor robustness, failing to generate reasonable results for poses significantly deviating from the training samples; (2) the use of static texture representation, failing to depict the differences in human image details under different poses; and (3) reliance on sophisticated 3D reconstruction equipment and multi-view data, limiting the model's application scenarios. The human pose transfer method provided in this application, by coupling image translation technology and differentiable rendering-based 3D representation into the video generation framework, better balances model robustness and detail richness, while also avoiding the need for multi-view human data required to reconstruct a sophisticated 3D model.

[0173] Please see Figure 10 In this embodiment of the application, the task of generating the task image frame is decomposed into a foreground (human body) generation task and a background generation task. The foreground generation task can be further decoupled into two sub-tasks: texture generation and texture coordinate prediction.

[0174] exist Figure 10 In this process, pose information A2 needs to be extracted from video frame A1 first. Then, UV generator A31 generates corresponding texture coordinates A32 based on the pose information A2; simultaneously, texture generator A41 generates corresponding dynamic texture A42 based on the pose information A2, where A42 can include dynamic textures corresponding to each of multiple human body parts; then, by combining texture coordinates A32 and dynamic texture A42, the foreground image A5 of the person can be rendered. Since the background lighting changes under different poses, this application uses a separate background generator A6 to generate a predicted background image. Finally, by combining the foreground image A5 of the person and the predicted background image generated by the background generator A6, the complete generated video frame A7 can be obtained.

[0175] In some embodiments of this application, the foreground generation task can be decoupled into two subtasks: texture generation and texture coordinate prediction, both of which are constrained by pose information. Further, the pose information obtained from the video frame includes both two-dimensional features (corresponding to the skeleton features in the above embodiments) and three-dimensional features (corresponding to the model features in the above embodiments): the two-dimensional feature is a three-channel skeleton image obtained by connecting keypoints, and the three-dimensional feature is the projection of the six-channel Laplacian features of each vertex in the reconstructed human model onto a two-dimensional plane. Therefore, the pose information is a nine-channel image.

[0176] The applicant considered the difficulty of directly translating a screen-space pose image into a texture-space texture image. Therefore, in this embodiment, the Laplacian features of the 3D model in the pose information are first unfolded into the texture space and then fed into the texture generator (corresponding to the texture image generation sub-network in the above embodiment) along with the pose image. Figure 11As shown, the Laplacian feature B11 of the 3D model is unfolded to obtain a partial texture map B12 in the texture space. To eliminate the mutual influence between different parts of the human body, the human body can be divided into N parts, each with a separate texture map. Here, N is an integer greater than 2, and can be set to 24 in actual implementation.

[0177] In some embodiments of this application, the process of generating texture images using a texture generator can be as follows: Figure 11 As shown, pose information B3 and a partial texture map B4 are input together to a texture generator B4, which can include an image translation framework comprising an upsampling network, a residual network, and a downsampling network. Different human body parts can share a single encoder B41, but each part corresponds to a different decoder B42; that is, different decoders B42 generate texture images B5 corresponding to different human body parts. This design allows the network to better focus on generating the current human body part and eliminates potential mutual interference between different human body parts.

[0178] After obtaining the texture image, it needs to be rendered from texture space to screen space. Traditional rendering pipelines rely on explicit 3D models to complete this texture mapping step, but detailed 3D models are often difficult to obtain.

[0179] Therefore, this embodiment of the application uses a UV generator (corresponding to the texture coordinate generation sub-network in the above embodiment) to directly obtain the UV coordinates of each pixel in the foreground image of the person from the pose information in the texture map. Further, the output of the UV generator includes two parts: 1) the probability P of each pixel in the foreground image belonging to each human body part. i (i = 1, ..., N); 2) The UV coordinates C of each pixel in the foreground image of the character and the corresponding texture map of the corresponding part. i (i = 1, ..., N).

[0180] After obtaining the output P of the UV generator i and C i Then, the foreground image of the figure can be obtained using the following formula (2-1):

[0181]

[0182] Among them, T i Let φ represent the texture map of the i-th part, and φ is the mapping function that maps the texture map to screen space according to UV coordinates.

[0183] In some embodiments of this application, regarding the background image generation process: although the generated video has a fixed background, different human postures cause slight changes in the background's lighting and shadows. Therefore, this application first uses image completion technology to obtain a static initial background, and then uses a background generator (corresponding to the background generation network in the above embodiments) to characterize these changes in lighting and shadows that change with posture. Further, this application uses a U-net network to segment and subtract the foreground from the human image frame by frame, and then uses the deepfill v2 completion technology to complete the remaining background frames. The initial background can be generated by averaging all the completed background frames. The initial background and the current posture information are then fed into the background generator to obtain the final optimized background image.

[0184] In some embodiments of this application, the fusion process of the background image and the foreground image of the person can generate video frames using the following formula (2-2):

[0185] I syn =I fg ⊙(1-P0)+I bg ⊙P0 formula (2-2);

[0186] In some embodiments of this application, directly stitching together the generated video frames can result in an unrealistic appearance, often inevitably introducing flickering and jitter. To address this issue, the applicant introduces a temporal loss (corresponding to the third loss function in the above embodiments) during the training process. Specifically, this constraint is defined as the L1 loss of the warp results of the current generated frame and the previous generated frame based on the pre-calculated optical flow, as shown in the following formulas (2-3):

[0187]

[0188] Among them, f t (·) is a function that warps the video frame at time t-1 to time t. D = H × W is the dimension of the output image, and I(k) represents the pixel at position k. k The optical flow confidence level is represented as follows: 0 indicates that the optical flow at that pixel is unreliable, and 1 indicates that it is reliable. Here, the optical flow confidence level is c. k Both optical flow and optical flow are pre-calculated on the training data.

[0189] In some embodiments of this application, the UV generator needs to be pre-trained by minimizing the following constraint function, Equation (2-4):

[0190]

[0191] in, This is the cross-entropy loss, where P and C are the prediction results of the UV generator. and It is the ground truth.

[0192] After the UV generator is pre-trained, the entire human pose transfer model participates in the training. Among them, I pose I syn and I real These represent the pose map, the generated result, and the real video frame, respectively. The overall optimization objectives include GAN loss (corresponding to the first loss function in the above embodiments), supervision loss (corresponding to the second loss function in the above embodiments), and temporal loss (corresponding to the third loss function in the above embodiments). The GAN loss is in the form of Equation (2-5):

[0193]

[0194] Among them, I pose I syn and I real These represent the pose map, the generated result, and the real video frame, respectively. D represents the discriminator, and log is the natural logarithm. This represents the expectation based on the training data. (Through...) This allows the discriminator to learn the ability to distinguish between real and fake video frames, through... This enables the pose transfer model to learn and generate the ability to "fool" the discriminator.

[0195] The supervised loss is in the form of Equation (2-6):

[0196]

[0197] Where VGG represents the network using VGG-19 to extract features from an image, λ f and λ t For weight parameters. ||VGG(I syn )-VGG(I real The goal is to make the generated results as similar as possible to the actual results within the feature space constraints. syn -I real ||2 indicates that the generated result is similar to the true result under image space constraints. λ f and λ t This is used to adjust the preceding and following terms to the same order of magnitude. In some embodiments, λ is selected. f =10, λ l =200.

[0198] The following continues to describe the exemplary structure of the object pose transfer device 555 provided in the embodiments of this application as a software module. In some embodiments of this application, such as... Figure 2 As shown, the software modules stored in the object pose transfer device 555 in the memory 550 may include:

[0199] The acquisition module 5551 is used to acquire posture data in the video frames of the source person that represent the limb movements of the source person; the posture data includes skeletal features representing the posture of the person's skeleton and model features representing the posture of the person's model.

[0200] The first generation module 5552 is used to generate dynamic texture data and texture coordinate data of the target character based on skeleton features and model features, respectively.

[0201] The second generation module 5553 is used to generate a foreground migration image for displaying the target person based on dynamic texture data and texture coordinate data; the limb movements of the target person in the foreground migration image are the same as the limb movements of the source person in the video frame.

[0202] The third generation module 5554 is used to generate migration video frames based on the foreground migration image.

[0203] In some embodiments of this application, the acquisition module 5551 is further configured to: acquire a set of skeletal key points of the source character in the video frame; and obtain skeletal features based on the set of skeletal key points.

[0204] In some embodiments of this application, the acquisition module 5551 is further configured to: construct a three-dimensional model of the source character based on video frames; the three-dimensional model includes multiple model vertices; construct a Laplacian matrix based on the multiple model vertices; determine multiple eigenvalues ​​corresponding to the Laplacian matrix and eigenvectors corresponding to each eigenvalue; determine N target eigenvalues ​​that meet preset requirements among the multiple eigenvalues; determine the vertex features of each model vertex based on the N eigenvectors corresponding to the N target eigenvalues; and project the vertex features of each model vertex onto a two-dimensional plane to obtain model features.

[0205] In some embodiments of this application, the first generation module 5552 is further configured to: generate migration coordinate information corresponding to each migration pixel based on skeleton features and model features; the migration coordinate information includes the coordinate information and confidence information of the migration pixel in each texture location; and generate a migration texture map corresponding to each texture location based on skeleton features and model features.

[0206] In some embodiments of this application, the second generation module 5553 is further configured to: generate a foreground migration image based on the migration texture map corresponding to each texture region, the coordinate information and confidence information of each foreground pixel at each texture region.

[0207] In some embodiments of this application, the third generation module 5554 is further configured to: process the initial background image according to the skeleton features and the model features to obtain a background migration image; and fuse the foreground migration image and the background migration image to obtain the migration video frame.

[0208] In some embodiments of this application, the third generation module 5554 is further configured to: determine the probability that a target pixel in the migrated video frame corresponding to each migrated pixel belongs to the background migrated image based on the confidence information of each migrated pixel being located in each texture region; and determine the migrated video frame based on the probability that each target pixel belongs to the background migrated image, the foreground migrated image, and the background migrated image.

[0209] In some embodiments of this application, the object pose transfer device 555 further includes a training module, wherein;

[0210] The training module is used to acquire the original video frames from the target person's video; extract the original skeleton features and original model features from the original video frames; input the original skeleton features and original model features into the pose transfer model to be trained to obtain the predicted video frames; wherein, the pose prediction network is used to generate predicted texture data and predicted coordinate data based on the original skeleton features and original model features, generate a foreground prediction image based on the predicted texture data and predicted coordinate data, and generate the predicted video frames based on the foreground prediction image; based on the original video frames and predicted video frames, the parameters of the pose transfer model to be trained are adjusted to obtain the trained pose transfer model.

[0211] In some embodiments of this application, the training module is further configured to: input the original skeleton features and the original model features into the foreground generation network to obtain a foreground prediction image; input the original skeleton features, the original model features, and the initial background image into the background generation network to obtain a background prediction image; and fuse the foreground prediction image and the background prediction image to obtain a predicted video frame.

[0212] In some embodiments of this application, the training module is further configured to: input the original skeleton features and the original model features into a pre-trained texture coordinate generation sub-network to obtain predicted coordinate data; the predicted coordinate data includes predicted coordinate information corresponding to each foreground predicted pixel in the foreground predicted image; the predicted coordinate information includes the coordinate information and confidence information of the foreground predicted pixel at each texture location; input the original skeleton features and the original model features into a texture image generation sub-network to obtain predicted texture data, the predicted texture data including a predicted texture map corresponding to each texture location; and generate a foreground predicted image based on the predicted texture map corresponding to each texture location, the coordinate information and confidence information of each foreground predicted pixel at each texture location.

[0213] In some embodiments of this application, the training module is further configured to: project the original model features onto the texture space to obtain input texture features; input the input texture features, the original skeleton features, and the original model features into the encoding layer to obtain encoded features; and input the encoded features into the decoding layer to obtain the corresponding texture prediction map for each texture part.

[0214] In some embodiments of this application, the training module is further configured to: input the encoded features into the local decoder corresponding to each texture region; and obtain the texture prediction map corresponding to each texture region output by the local decoder corresponding to each texture region.

[0215] In some embodiments of this application, the training module is further configured to: determine the probability that the output pixel corresponding to each foreground prediction pixel in the predicted video frame belongs to the background prediction image based on the confidence information of each foreground prediction pixel being located in each texture region; and determine the predicted video frame based on the probability that each output pixel belongs to the background prediction image, the foreground prediction image, and the background prediction image.

[0216] In some embodiments of this application, the training module is further configured to: acquire multiple video frames to be processed in the target person's video; perform background segmentation on the multiple video frames to be processed using a segmentation network to obtain multiple first background frames; process the multiple first background frames using an image completion algorithm to obtain multiple second background frames; and perform average fusion processing on the multiple second background frames to obtain an initial background image.

[0217] In some embodiments of this application, the training module is further configured to: acquire a texture coordinate training set; the texture coordinate training set includes multiple sample pose information and the real coordinate information corresponding to each sample pose information; input each sample pose information into the texture coordinate generation sub-network to be trained to obtain the predicted coordinate information corresponding to each sample pose information; input the predicted coordinate information and the real coordinate information corresponding to each sample pose information into a preset coordinate loss function to determine the coordinate loss value; use the coordinate loss value to train the texture coordinate generation sub-network until the coordinate loss value converges, and output the pre-trained texture coordinate generation sub-network.

[0218] In some embodiments of this application, the training module is further configured to: calculate the value of a first loss function based on pose information, original video frames, and predicted video frames; calculate the value of a second loss function based on the original video frames and predicted video frames; calculate a total loss value based on the values ​​of the first and second loss functions; and adjust the parameters of the pose transfer model to be trained based on the total loss value until a preset convergence condition is reached, and take the pose transfer model to be trained when the preset convergence condition is reached as the trained pose transfer model.

[0219] In some embodiments of this application, the training module is further configured to: obtain neighboring predicted video frames of the predicted video frame; the neighboring predicted video frame is the predicted image of the previous frame of the original video frame; input the neighboring predicted video frames into a preset optical flow prediction model to obtain the optical flow prediction image of the current frame; calculate the value of the third loss function based on the optical flow prediction image and the predicted video frame input; and calculate the total loss value based on the values ​​of the first loss function, the second loss function, and the third loss function.

[0220] This disclosure provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the object pose transfer method described in the embodiments of this application.

[0221] This disclosure provides a computer-readable storage medium storing executable instructions. When these executable instructions are executed by a processor, they cause the processor to execute the object pose transfer method provided in this application, for example... Figure 3 , Figure 4 , Figure 5 , Figure 6 , Figure 7A , Figure 8A or Figure 9 The method shown.

[0222] In some embodiments of this application, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a device that includes one or any combination of the above-mentioned memories.

[0223] In some embodiments of this application, executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including being deployed as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0224] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., files that store one or more modules, subroutines, or code sections).

[0225] As an example, executable instructions can be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0226] In summary, the following technical effects can be achieved through the embodiments of this application:

[0227] (1) In this embodiment of the application, by obtaining the skeleton features representing the skeleton posture of the object and the model features representing the model posture of the object in the video frame, the limb movements of the source object in the current video frame can be accurately obtained from the skeleton dimension and the object model dimension. Since the feature information of the limb movements of the source object is extracted from the two dimensions of "from the inside to the outside", the accuracy of the limb movements of the source object to the limb movements of the target object can be improved during the object posture transfer process, and it has strong robustness. Furthermore, since dynamic texture data and texture coordinate data are generated based on the skeleton features and model features respectively, the generated dynamic texture data and texture coordinate data can change with different limb movements, which improves the restoration of texture details during the object posture transfer process, and the generated transfer video frame has richer details and higher realism.

[0228] (2) Based on the skeleton features and the model features, the embodiments of this application generate corresponding migration texture maps for different texture parts of the object. In this way, not only can the generated migration texture map change dynamically with the different limb movements of the source object, but also dynamic texture maps that fit the texture parts more closely can be generated for different texture parts, thereby enriching the details of the texture map and improving the accuracy of the object's posture migration process.

[0229] (3) In this embodiment, the initial background image is processed according to the skeleton features and model features to obtain a background migration image containing light and shadow information, which can improve the realism of the migration video frame. Furthermore, since the light and shadow information is obtained based on the skeleton features and model features, the light and shadow information can be closer to the light and shadow state in the real scene, thus further improving the realism of the migration video frame. At the same time, based on the confidence information of each migration pixel in each texture part, this application determines the probability that the target pixel corresponding to each migration pixel in the migration video frame belongs to the background migration image. This can improve the accuracy of fusion during the fusion of the foreground migration image and the background migration image, thereby improving the accuracy of the object pose migration process.

[0230] (4) The pose transfer model trained through the above embodiments can accurately obtain the limb movements of the target object in the current video frame from the skeleton dimension and the object model dimension. Since the feature information of the target object's limb movements is extracted from the two dimensions of "from the inside out", the pose information input to the pose transfer model can be more accurate, thus ensuring that the pose transfer model can quickly learn the conversion process between different pose information and different texture information during the training process. Furthermore, since the predicted texture data and predicted coordinate data are generated based on the skeleton features and model features respectively, the pose transfer model can generate different predicted texture data and predicted coordinate data according to different input pose data, which improves the restoration of texture details in the object pose transfer process, and the generated predicted video frame details are richer and more realistic.

[0231] (5) In this embodiment, the pose transfer model is divided into a foreground generation network and a background generation network. This allows the foreground generation network to focus on learning the relationship between the object's pose and the texture image and texture coordinates, while the background generation network can focus on learning the influence of the object's pose on the initial background image. This results in more realistic foreground and background prediction images, leading to more accurate predicted video frames.

[0232] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A method for object pose transfer, the method comprising: The method comprises: obtaining pose data in a video frame of a source object for representing limb movement of the source object; the pose data comprises skeleton features representing a skeleton pose of the object and model features representing a model pose of the object; generating, according to the skeleton features and the model features, migration coordinate information corresponding to each migration pixel point of a target object, the migration coordinate information comprising coordinate information and confidence information of each texture part where the migration pixel point is located; generating, according to the skeleton features and the model features, migration texture maps corresponding to each texture part of the target object; generating a foreground migration image for displaying the target object based on the migration texture map corresponding to each texture part, the coordinate information and the confidence information of each texture part where each foreground pixel point is located, the limb movement of the target object in the foreground migration image being the same as that of the source object in the video frame; processing an initial background image according to the skeleton features and the model features to obtain a background migration image; determining a probability that a target pixel point corresponding to each migration pixel point in a migration video frame belongs to the background migration image according to the confidence information of each migration pixel point in each texture part; determining the migration video frame based on the probability that each target pixel point belongs to the background migration image, the foreground migration image and the background migration image.

2. The method of claim 1, wherein the method is implemented by a trained pose migration model; and the training process of the pose migration model comprises: obtaining an original video frame in a target object video; extracting original skeleton features and original model features in the original video frame; inputting the original skeleton features and the original model features into a to-be-trained pose migration model to obtain a predicted video frame; wherein the pose prediction network is configured to generate predicted texture data and predicted coordinate data according to the original skeleton features and the original model features, generate a foreground prediction image according to the predicted texture data and the predicted coordinate data, and generate the predicted video frame based on the foreground prediction image; adjusting parameters of the to-be-trained pose migration model based on the original video frame and the predicted video frame to obtain a trained pose migration model.

3. The method of claim 2, wherein, The pose migration model comprises a foreground generation network and a background generation network; and the inputting of the original skeleton features and the original model features into the to-be-trained pose migration model to obtain the predicted video frame comprises: inputting the original skeleton features and the original model features into the foreground generation network to obtain the foreground prediction image; inputting the original skeleton features, the original model features and an initial background image into the background generation network to obtain a background prediction image; fusing the foreground prediction image and the background prediction image to obtain the predicted video frame.

4. The method of claim 3, wherein, The foreground generation network comprises a texture image generation subnetwork and a texture coordinate generation subnetwork; the inputting of the original skeleton feature and the original model feature into the foreground generation network to obtain the foreground prediction image comprises: inputting the original skeleton feature and the original model feature into a pre-trained texture coordinate generation subnetwork to obtain the predicted coordinate data; the predicted coordinate data comprises predicted coordinate information corresponding to each foreground prediction pixel point in the foreground prediction image; the predicted coordinate information comprises coordinate information and confidence information of each texture part where the foreground prediction pixel point is located; inputting the original skeleton feature and the original model feature into the texture image generation subnetwork to obtain the predicted texture data, the predicted texture data comprising a predicted texture image corresponding to each texture part; based on the predicted texture image corresponding to each texture part, the coordinate information and the confidence information of each texture part where each foreground prediction pixel point is located, the foreground prediction image is generated.

5. The method of claim 4, wherein, The texture image generation subnetwork comprises an encoding layer and a decoding layer; the inputting of the original skeleton feature and the original model feature into the texture image generation subnetwork to obtain the predicted texture image corresponding to each texture part comprises: projecting the original model feature into a texture space to obtain input texture features; inputting the input texture features, the original skeleton feature and the original model feature into the encoding layer to obtain encoding features; inputting the encoding features into the decoding layer to obtain the corresponding texture prediction image of each texture part.

6. The method of claim 3, wherein, The method further comprises: obtaining a plurality of to-be-processed video frames in the target object video; performing background segmentation on the plurality of to-be-processed video frames by using a segmentation network to obtain a plurality of first background frames; processing the plurality of first background frames by using an image completion algorithm to obtain a plurality of second background frames; performing average fusion processing on the plurality of second background frames to obtain the initial background image.

7. The method of claim 4, wherein, The pre-training process of the texture coordinate generation subnetwork comprises: obtaining a texture coordinate training set; the texture coordinate training set comprises a plurality of sample posture information and real coordinate information corresponding to each sample posture information; inputting each sample posture information into a to-be-trained texture coordinate generation subnetwork to obtain predicted coordinate information corresponding to each sample posture information; inputting the predicted coordinate information and the real coordinate information corresponding to each sample posture information into a preset coordinate loss function to determine a coordinate loss value; the to-be-trained texture coordinate generation subnetwork is adjusted by using the coordinate loss value until the coordinate loss value converges, and the pre-trained texture coordinate generation subnetwork is output.

8. The method of claim 2, wherein, The adjustment of the parameters of the to-be-trained posture transfer model based on the original video frame and the predicted video frame to obtain a trained posture transfer model comprises: calculating a value of a first loss function based on the posture information, the original video frame and the predicted video frame; calculating a value of a second loss function based on the original video frame and the predicted video frame; A total loss value is calculated based on the value of the first loss function and the value of the second loss function, and parameters of the posture transfer model to be trained are adjusted based on the total loss value until a preset convergence condition is reached, and the posture transfer model to be trained when the preset convergence condition is reached is taken as a trained posture transfer model.

9. The method of claim 8, wherein, The method further comprises: An adjacent predicted video frame of the predicted video frame is obtained, the adjacent predicted video frame being a predicted image of a previous frame image of the original video frame; the adjacent predicted video frame is input into a preset optical flow prediction model to obtain an optical flow predicted image of the current frame; and a value of a third loss function is calculated based on the optical flow predicted image and the predicted video frame input. The total loss value is calculated based on the value of the first loss function and the value of the second loss function, and parameters of the posture transfer model to be trained are adjusted based on the total loss value until a preset convergence condition is reached, and the posture transfer model to be trained when the preset convergence condition is reached is taken as a trained posture transfer model. The total loss value is calculated based on the value of the first loss function, the value of the second loss function and the value of the third loss function.

10. An object pose transfer apparatus, characterized by comprising: It comprises: An acquisition module is configured to acquire posture data representing limb movement of a source object from a video frame of the source object, wherein the posture data comprises skeleton features representing a skeleton posture of the object and model features representing a model posture of the object; A first generation module is configured to generate, according to the skeleton features and the model features, migration coordinate information corresponding to each migration pixel point of a target object, wherein the migration coordinate information comprises coordinate information and confidence information of each texture part where the migration pixel point is located; and generate, according to the skeleton features and the model features, migration texture maps corresponding to each texture part of the target object; A second generation module is configured to generate, based on the migration texture maps corresponding to each texture part, the coordinate information and the confidence information of each texture part where each foreground pixel point is located, a foreground migration image for displaying the target object, wherein the limb movement of the target object in the foreground migration image is the same as the limb movement of the source object in the video frame; A third generation module is configured to process an initial background image according to the skeleton features and the model features to obtain a background migration image; According to the confidence information of each migration pixel point in each texture part, a probability that a target pixel point corresponding to each migration pixel point in a migration video frame belongs to the background migration image is determined; Based on the probability that each target pixel point belongs to the background migration image, the foreground migration image and the background migration image, the migration video frame is determined.

11. An object pose transfer device, comprising: It comprises: A memory is configured to store executable instructions; A processor is configured to execute the executable instructions stored in the memory to implement the method in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, Executable instructions are stored in the memory, and when executed by the processor, the method in any one of claims 1 to 9 is implemented.

13. A computer program product comprising computer-executable instructions or a computer program, characterized in that, When the computer executable instructions or the computer program are executed by the processor, the method in any one of claims 1 to 9 is implemented.

Images