Virtual object video generation method and apparatus, electronic device, storage medium

Abstract

The application provides a virtual object video generation method and device, an electronic device and a storage medium, and relates to the fields of artificial intelligence and financial technology. The method comprises the following steps: extracting an original virtual object image frame from an original virtual object video; performing clothing feature extraction on the original virtual object image frame to obtain target clothing texture features; extracting a target action image frame from a target action video; performing action feature extraction on the target action image frame to obtain target action features; performing global flow field estimation on the original virtual object video and the target action video to obtain target global flow field features; and performing local attention fusion on the target clothing texture features, the target action features and the target global flow field features to obtain a target virtual object video. The application can generate a virtual object video with arbitrary actions, and the virtual object video has high authenticity.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence technology and financial technology, and in particular to a method and apparatus for generating virtual object videos, an electronic device, and a storage medium. Background Technology

[0002] Virtual objects are virtual avatars used to interact with customers. For example, in the fintech field, virtual objects are customer service representatives who interact with customers via video using virtual avatars.

[0003] Currently, some virtual object generation technologies are limited by fixed action templates. Other virtual object generation technologies use neural network models for action transfer, which can break free from fixed action templates and allow virtual objects to generate videos according to user-specified actions.

[0004] When generating virtual object videos, the aforementioned technologies all struggle to represent the continuity of movements and realistically reproduce the fine details such as the folds and textures of clothing, resulting in low realism. Summary of the Invention

[0005] The main objective of this application is to provide a method, apparatus, electronic device, and storage medium for generating virtual object videos, which can generate virtual object videos with arbitrary actions and high realism.

[0006] To achieve the above objectives, a first aspect of this application proposes a method for generating virtual object videos, the method comprising:

[0007] Extract the original virtual object image frames from the original virtual object video;

[0008] Clothing features are extracted from the original virtual object image frame to obtain the target clothing texture features;

[0009] Extract target action image frames from target action video;

[0010] The target action image frame is subjected to action feature extraction to obtain target action features;

[0011] Global flow field estimation is performed on the original virtual object video and the target motion video to obtain the target global flow field characteristics;

[0012] Local attention fusion is performed on the target clothing texture features, the target motion features, and the target global flow field features to obtain the target virtual object video.

[0013] In some embodiments, the step of performing local attention fusion on the target clothing texture features, the target motion features, and the target global flow field features to obtain the target virtual object video includes:

[0014] A first local attention fusion is performed based on the target global flow field features and the target clothing texture features to obtain the target flow field clothing fusion features;

[0015] A second local attention fusion is performed based on the target global flow field features and the target action features to obtain the target flow field action fusion features;

[0016] The target flow field clothing fusion feature and the target flow field motion fusion feature are multiplied element-wise to obtain the target fused flow field feature;

[0017] Convolution is performed on the target fused flow field features to obtain the target virtual object image frame;

[0018] The target virtual object video is obtained by merging the image frames of the target virtual object.

[0019] In some embodiments, the step of performing a second local attention fusion based on the target global flow field features and the target action features to obtain target flow field action fusion features includes:

[0020] Shallow local attention fusion is performed on the target global flow field features and the target motion features to obtain initial flow field motion fusion features;

[0021] The target action features are decoded to obtain action decoding features;

[0022] The action decoding features are convolved to obtain action convolution features;

[0023] The initial flow field action fusion feature and the action convolution feature are fused by deep local attention to obtain the target flow field action fusion feature.

[0024] In one embodiment, the step of performing local attention fusion on the target clothing texture features, the target motion features, and the target global flow field features to obtain a target virtual object video includes:

[0025] Based on the target global flow field features and the target clothing texture features, a first local attention fusion is performed to obtain the target flow field clothing fusion features;

[0026] A second local attention fusion is performed based on the target global flow field features and the target action features to obtain the target flow field action fusion features;

[0027] The target flow field clothing fusion feature and the target flow field motion fusion feature are multiplied element-wise to obtain the target fused flow field feature;

[0028] The target number of times is obtained by incrementing the number of times the local attention fusion is performed on the original virtual object image frame;

[0029] If the number of target events is less than or equal to the attention fusion number threshold, then the target fused flow field feature is taken as the target global flow field feature, and the target flow field action fusion feature is taken as the target action feature. Then, the step of performing the first local attention fusion based on the target global flow field feature and the target clothing texture feature to obtain the target flow field clothing fusion feature is returned.

[0030] If the number of target iterations is greater than the attention fusion iteration threshold, then the target fusion flow field features are convolved to obtain the target virtual object image frame;

[0031] The target virtual object video is obtained by merging the image frames of the target virtual object.

[0032] In some embodiments, the step of extracting clothing features from the original virtual object image to obtain target clothing texture features includes:

[0033] The original virtual object image frame is subjected to clothing feature extraction by a preset directional filter bank to obtain pixel-level gradient direction features and texture intensity features;

[0034] The target clothing texture feature is obtained by adding the pixel-level gradient direction feature and the texture intensity feature element by element.

[0035] In some embodiments, the directional filter bank includes a contour extraction filter and a structure extraction filter; the step of extracting clothing features from the original virtual object image frame using the preset directional filter bank to obtain pixel-level gradient direction features and texture intensity features includes:

[0036] The original virtual object image frame is subjected to clothing contour feature extraction by the contour extraction filter to obtain the pixel-level gradient direction feature;

[0037] The original virtual object image frame is subjected to clothing structure feature extraction using the structure extraction filter to obtain the texture intensity feature.

[0038] In some embodiments, the step of extracting action features from the target action image frame to obtain target action features includes:

[0039] Key points are extracted from the target action image frame to obtain the target key points;

[0040] By connecting the target key points, the target pose skeleton is obtained;

[0041] Based on the target pose skeleton of multiple target action image frames, determine the target motion features;

[0042] Actions are generated based on the target pose skeleton and the target motion features to obtain the target motion features.

[0043] In some embodiments, determining the target motion features based on the target pose skeleton of a plurality of target motion image frames includes:

[0044] The target pose skeleton of multiple target action image frames is sequentially represented to obtain a pose skeleton time series;

[0045] Motion features are extracted from the time series of the posture skeleton to obtain the target motion features.

[0046] In one embodiment, the original virtual object video contains multiple candidate virtual object image frames, and the target motion video contains multiple candidate motion image frames. Extracting the target motion image frame from the target motion video includes:

[0047] An initial ratio is obtained by calculating the ratio between the number of candidate virtual object image frames and the number of candidate action image frames.

[0048] The initial ratio is rounded up to obtain the target quantity;

[0049] Image frames are extracted from the target action video according to the time sequence to obtain the target number of target action image frames; wherein, the number of the original virtual object image frames is the same as the number of target action image frames.

[0050] To achieve the above objectives, a second aspect of this application provides a virtual object video generation apparatus, the apparatus comprising:

[0051] The raw image extraction module is used to extract raw virtual object image frames from the raw virtual object video.

[0052] The clothing feature extraction module is used to extract clothing features from the original virtual object image frame to obtain the target clothing texture features;

[0053] The motion image extraction module is used to extract target motion image frames from target motion videos;

[0054] The action feature extraction module is used to extract action features from the target action image frame to obtain target action features;

[0055] The global estimation module is used to perform global flow field estimation on the original virtual object video and the target motion video to obtain the target global flow field features;

[0056] The local fusion module is used to perform local attention fusion on the target clothing texture features, the target motion features, and the target global flow field features to obtain the target virtual object video.

[0057] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the virtual object video generation method described in the first aspect.

[0058] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the virtual object video generation method described in the first aspect.

[0059] The virtual object video generation method, apparatus, electronic device, and computer-readable storage medium proposed in this application extract original virtual object image frames from the original virtual object video and target action image frames from the target action video; secondly, clothing feature extraction is performed on the original virtual object image frames to obtain target clothing texture features, and action feature extraction is performed on the target action image frames to obtain target action features; furthermore, global flow field estimation is performed on the original virtual object video and the target action video to obtain target global flow field features; finally, local attention fusion is performed on the target clothing texture features, target action features, and target global flow field features to obtain the target virtual object video. This application, through the interaction of global flow field estimation and local attention fusion, can overcome the limitations in dealing with action changes, assign higher weights to important features, thereby ensuring the continuity of video action, and effectively and realistically reproducing the fine appearance of clothing such as folds and textures. In summary, this application can generate virtual object videos with arbitrary actions and achieve a high degree of realism.

[0060] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0061] Figure 1 This is a flowchart of the virtual object video generation method provided in the embodiments of this application;

[0062] Figure 2 yes Figure 1 The flowchart for step 102 in the document;

[0063] Figure 3 yes Figure 1 The flowchart for step 104 in the document;

[0064] Figure 4 yes Figure 3 The flowchart for step 303 in the document;

[0065] Figure 5 yes Figure 1 The flowchart for step 106 in the document;

[0066] Figure 6 yes Figure 5 The flowchart for step 502 in the document;

[0067] Figure 7 This is a block diagram of the module structure of the virtual object video generation device provided in the embodiments of this application;

[0068] Figure 8 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0069] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0070] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0071] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0072] First, let's analyze some of the terms used in this application:

[0073] Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0074] Natural Language Processing (NLP): NLP uses computers to process, understand, and utilize human language (such as Chinese and English). NLP is a branch of artificial intelligence and an interdisciplinary field of computer science and linguistics, often referred to as computational linguistics. NLP includes syntactic analysis, semantic analysis, and discourse understanding. It is commonly used in machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information and image processing, information extraction and filtering, text classification and clustering, sentiment analysis, and opinion mining. It involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computation.

[0075] Virtual objects are virtual avatars that interact with customers in real time via video, using advanced technologies such as video generation, natural language processing, speech synthesis, and real-time rendering. Virtual objects can conduct live events on various online platforms, chat with viewers in real time, answer questions, and even act as customer service representatives, thus saving on the cost of hiring human customer service personnel.

[0076] Human motion transfer methods involve spatial manipulation of source data and can be divided into the following steps: First, posture and motion data are collected from the source person; the collected raw data is cleaned, denoised, and standardized; then, key features are extracted from the preprocessed data using a network model, typically including the geometric information of the human skeleton and the motion trajectories of joints; next, based on the skeletal proportions and posture characteristics of the target person, the motion features extracted from the source data are spatially adjusted and transformed. Finally, the transformed motion features are combined with the skeletal information of the target person. By iterating and adjusting the above process continuously, the accuracy of motion transfer can be improved.

[0077] Related technologies are often limited by fixed action templates, resulting in a lack of diversity and flexibility in the movement of generated virtual objects (such as virtual anchors). Convolutional neural network (CNN) models, by combining the above, can synthesize virtual avatars based on audio and reference video while breaking free from fixed video templates, allowing virtual objects to generate videos according to user-specified target actions. However, CNNs are inefficient at spatial transformation of inputs, limiting the transformation effect. This can lead to errors in the transferred actions or discontinuous transferred actions, especially in videos with drastic changes in movement (such as dancing), where the effect before and after transfer can differ significantly. Furthermore, when virtual objects wear loose clothing (such as dresses), most current methods may produce noticeable temporal and spatial artifacts, greatly limiting the achievable visual quality. This is because existing technologies struggle to realistically reproduce fine details such as wrinkles and textures on this type of clothing. High-fidelity and temporally consistent human motion transfer is also impossible. These problems result in a lack of realism and scalability in currently synthesized virtual objects, ultimately failing to meet customer expectations in practical business applications.

[0078] This application discloses a method, apparatus, electronic device, and computer-readable storage medium for generating virtual object videos, capable of generating virtual object videos with arbitrary movements and high realism. This application can be used for video digital human generation, enriching the action performance of generated digital humans. It can also be applied to scenarios such as live-streaming sales of insurance products, course training, and customer service, replacing real people and significantly improving efficiency.

[0079] The virtual object video generation method provided in this application can be applied to terminals and servers, or it can be software running on the server. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The software can be an application that implements the virtual object video generation method, but it is not limited to the above forms.

[0080] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include server computers, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0081] This application provides a method for generating virtual object videos, a device for generating virtual object videos, an electronic device, and a computer-readable storage medium. The specific embodiments are described below. First, the method for generating virtual object videos in this application is described.

[0082] It should be noted that in each specific embodiment of this application, when it is necessary to process data related to user identity or characteristics, such as user behavior data, the user's permission or consent will be obtained first. Moreover, the collection, use and processing of this data will comply with relevant laws, regulations and standards.

[0083] Reference Figure 1 , Figure 1 This is an optional flowchart of the virtual object video generation method provided in the embodiments of this application, which may include, but is not limited to, steps 101 to 106.

[0084] Step 101: Extract the original virtual object image frames from the original virtual object video;

[0085] Step 102: Extract clothing features from the original virtual object image frame to obtain the target clothing texture features;

[0086] Step 103: Extract target action image frames from the target action video;

[0087] Step 104: Extract action features from the target action image frame to obtain target action features;

[0088] Step 105: Perform global flow field estimation on the original virtual object video and the target motion video to obtain the target global flow field features;

[0089] Step 106: Perform local attention fusion on the target clothing texture features, target motion features, and target global flow field features to obtain the target virtual object video.

[0090] Steps 101 to 106, as illustrated in the embodiments of this application, utilize the interaction of global flow field estimation and local attention fusion. This approach addresses the limitations of handling motion transformations while assigning higher weights to important features, thereby ensuring the continuity of video motion and effectively and realistically reproducing fine details such as clothing folds and textures. In summary, this application can generate virtual object videos with arbitrary motion and high realism.

[0091] Understandably, the aforementioned virtual object video generation method can generate virtual avatars of specified target actions (also known as target poses), enriching the motion effects of virtual objects while maintaining vivid clothing, textures, and motion details. This can provide customers interacting with virtual objects with a more lifelike experience, improving customer satisfaction.

[0092] In step 101 of some embodiments, the original virtual object video is a virtual object video to be animated. A virtual object video refers to interactive video content created using virtual reality (VR), augmented reality (AR), or mixed reality (MR) technologies. The original virtual object video contains multiple candidate virtual object image frames. In one example, step 101 includes: extracting each candidate virtual object image frame from the original virtual object video in chronological order to obtain the original virtual object image frame.

[0093] In step 102 of some embodiments, clothing features can be extracted from the original virtual object image frame using a clothing texture information extractor to obtain the target clothing texture features. A clothing texture information extractor is a technique or tool used to extract and analyze texture information from an image. For example, a clothing texture information extractor is a directional filter bank, also known as a directional Gabor filter bank.

[0094] It should be noted that different textures generally have different center frequencies and bandwidths. Based on these frequencies and bandwidths, a set of Gabor filters can be designed to filter the image. Each Gabor filter only allows textures corresponding to its frequency to pass through smoothly, while suppressing the energy of other textures. Texture features are analyzed and extracted from the output of each filter for subsequent video synthesis tasks. Extracting texture features using Gabor filters mainly involves two processes: ① designing filters (e.g., function, number, direction, and spacing); ② extracting the effective texture feature set from the filter output. Gabor filters are bandpass filters, and their unit impulse response function (Gabor function) is the product of a Gaussian function and a complex exponential function. It is a function that reaches the lower bound of the time-frequency uncertainty principle, possessing the best ability to balance signal resolution in the time and frequency domains.

[0095] In one embodiment, reference is made to Figure 2 Step 102 includes:

[0096] Step 201: Extract clothing features from the original virtual object image frame using a preset directional filter bank to obtain pixel-level gradient direction features and texture intensity features;

[0097] Step 202: Element-wise summation is performed based on pixel-level gradient direction features and texture intensity features to obtain the target clothing texture features.

[0098] In step 201 of some embodiments, the directional filter bank includes a contour extraction filter and a structure extraction filter. In one embodiment, step 201 may include: extracting clothing contour features from the original virtual object image frame using the contour extraction filter to obtain pixel-level gradient direction features; and extracting clothing structure features from the original virtual object image frame using the structure extraction filter to obtain texture intensity features. Referring to the above, contour extraction filters and structure extraction filters can be designed by adjusting functions, numbers, directions, and intervals, thereby obtaining pixel-level gradient direction features and texture intensity features respectively. This embodiment ensures the consistency of clothing texture over time.

[0099] In step 202 of some embodiments, element-wise addition refers to adding the elements at the same position in two features to obtain a new feature. In one embodiment, step 202 includes: pooling the pixel-level gradient direction feature into a target pixel-level gradient direction feature of a predetermined dimension, and pooling the texture intensity feature into a target texture intensity feature of a predetermined dimension; then, element-wise adding the target pixel-level gradient direction feature and the target texture intensity feature to obtain the target clothing texture feature. This embodiment has high universality.

[0100] In one embodiment, before step 201, step 102 further includes: calculating the similarity between two adjacent original virtual image frames; if the similarity is greater than or equal to a similarity threshold, then the target clothing texture feature of the former of the two adjacent original virtual image frames is used as the target clothing texture feature of the latter; if the similarity is less than the similarity threshold, proceed to steps 201 to 202. This embodiment can reduce the number of images for clothing feature extraction, thereby improving video generation efficiency.

[0101] In step 103 of some embodiments, the target motion video refers to video content containing the target motion. The target motion video can be built into the system for user selection or uploaded from a client. Based on the target motion video, this application can transform the actions of an object in a virtual object video into the actions in the target motion video, increasing the expressiveness of the object. The target motion video contains multiple candidate motion image frames. In one example, step 103 includes: extracting each candidate motion image frame from the target motion video in chronological order to obtain the target motion image frame.

[0102] In one embodiment, step 103 includes: calculating an initial ratio based on the ratio of the number of candidate virtual object image frames to the number of candidate action image frames; rounding the initial ratio up to obtain the target number; extracting image frames from the target action video according to the time sequence to obtain the target number of target action image frames; wherein the number of original virtual object image frames is the same as the number of target action image frames.

[0103] For example, if there are 100 candidate virtual object image frames and 100 candidate action image frames, the initial extraction ratio is 100 / 100 = 1. After rounding up, the target extraction ratio is 1. Therefore, based on the time order and the target extraction ratio of 1, each candidate action image frame is treated as one target action image frame. As another example, if there are 90 candidate virtual object image frames and 100 candidate action image frames, the initial extraction ratio is 90 / 100 = 0.9. After rounding up, the target extraction ratio is 1. Therefore, based on the time order and the target extraction ratio of 1, each candidate action image frame is treated as one target action image frame. Yet another example, if there are 190 candidate virtual object image frames and 100 candidate action image frames, the initial extraction ratio is 190 / 100 = 1.9. After rounding up, the target extraction ratio is 2. Therefore, based on the time order and the target extraction ratio of 2, each candidate action image frame is treated as two target action image frames.

[0104] The advantage of the above embodiments is that they can ensure the correspondence between the original virtual image frames and the target action image frames when the number of candidate virtual object image frames and the number of candidate action image frames are different, which helps to improve the action continuity of the generated video.

[0105] In step 104 of some embodiments, the target action image frame can be processed by a target action extractor to extract action features, thereby obtaining target action features. A target action extractor is a system or algorithm used to identify and extract specific actions from video or real-time input.

[0106] In one embodiment, reference is made to Figure 3 Step 104 includes:

[0107] Step 301: Extract key points from the target action image frame to obtain target key points;

[0108] Step 302: Connect the target key points to obtain the target pose skeleton;

[0109] Step 303: Determine the target motion features based on the target pose skeleton of multiple target action image frames;

[0110] Step 304: Generate motion based on the target pose skeleton and target motion features to obtain target motion features.

[0111] In step 301 of some embodiments, key points can be extracted from the target action image frame using a key point detection model to obtain target key points. Target key points include the nose, eyes, ears, shoulders, elbows, wrists, hips, knees, and ankles. The key point detection model can be YOLOv8.

[0112] In step 302 of some embodiments, key points can be connected based on the known human skeletal structure to form a complete target pose skeleton. Common connection relationships can be represented as edge graphs. The target pose skeleton can be a simple line drawing or a more complex 3D model.

[0113] In step 303 of some embodiments, the target motion features indicate the motion characteristics of the target pose skeleton.

[0114] In one embodiment, reference is made to Figure 4 Step 303 may include:

[0115] Step 401: Sequence representation of the target pose skeleton of multiple target action image frames is performed to obtain the pose skeleton time series;

[0116] Step 402: Extract motion features from the pose skeleton time series to obtain the target motion features.

[0117] In this embodiment, sequence representation specifically refers to using a pose skeleton to calculate the temporal context. This allows for the capture of changes in the pose skeleton (also known as motion) over time. By introducing temporal context, the motion characteristics of the pose can be better understood, and continuous motion can be more accurately simulated and generated during human motion transfer. Target motion characteristics include motion trajectory, velocity, acceleration, angle changes, etc.

[0118] In step 304 of some embodiments, motion is generated based on the target pose skeleton and target motion features, such as by performing feature splicing or element-by-element addition to obtain target motion features.

[0119] The advantage of the embodiments of steps 301 to 304 described above is that, based on the ability to obtain target action features, the continuity of target action features between adjacent target action image frames is improved.

[0120] In step 105 of some embodiments, a global flow field estimator can be used to estimate the global flow field of the original virtual object video and the target motion video to obtain the target global flow field features. The global flow field estimator is used to estimate the global correlation between the original virtual object video and the target motion video to predict the target flow field features. A global flow field estimator is a tool or algorithm used to estimate the motion of an object or region from an image sequence or video. It is usually used to calculate the optical flow of each pixel (i.e., the speed and direction of the object moving in the image) and then infer the global motion features in the scene. The global flow field estimator can use the following networks: (1) Convolutional Neural Network: It estimates optical flow by learning features and is often used in complex scenes. (2) Optical Flow Network: Such as FlowNet, PWC-Net, etc., which can effectively handle large-scale motion and complex backgrounds.

[0121] In step 106 of some embodiments, a target virtual object video can be obtained by performing local attention fusion on the target clothing texture features, target motion features, and target global flow field features using a target motion renderer. The target motion renderer is a neural network model based on a local attention mechanism, capable of performing local attention fusion on the input target clothing texture features, target motion features, and target global flow field features to obtain a target virtual object video.

[0122] It should be noted that global attention mechanisms are suitable for global structures that require comprehensive capture of input data, but they consume more computational resources and generate more computational load. Unlike global attention, local attention only performs calculations on a specific region of the input sequence. Since the focus of this application's embodiments is on actions, using a local attention mechanism can achieve more refined results.

[0123] In one embodiment, reference is made to Figure 5 Step 106 may include:

[0124] Step 501: Perform first local attention fusion based on the target global flow field features and the target clothing texture features to obtain the target flow field clothing fusion features;

[0125] Step 502: Perform second local attention fusion based on the target global flow field features and target action features to obtain target flow field action fusion features;

[0126] Step 503: Multiply the target flow field clothing fusion feature and the target flow field motion fusion feature element by element to obtain the target fused flow field feature;

[0127] Step 504: Convolve the target fused flow field features to obtain the target virtual object image frame;

[0128] Step 505: Merge the image frames of the target virtual object to obtain the video of the target virtual object.

[0129] The first and second local attention fusion methods can be as follows: They mainly involve weighted processing of local regions of the input sequence. First, the length and position of the local region (i.e., the size and position of the sliding window) are specified according to task requirements and the characteristics of the input data. Then, at each step, the correlation between the input data within the window and the current step is calculated. Next, based on the calculated correlation, a weight value is assigned to each input data point within the window. The magnitude of the weight value reflects the importance of the input data at the current step. Finally, the input data within the window and its corresponding weight value are weighted and summed to obtain local context features (including target flow field clothing fusion features and target flow field motion fusion features), ultimately resulting in the target virtual object video.

[0130] The advantage of the embodiments of steps 501 to 505 is that, before generating the target virtual object image frame using convolution, the first local attention calculation and the second local attention calculation are used to effectively fuse three different features (i.e., target global flow field features, target clothing texture features and target action features), which can reduce the amount of computation and ensure the information richness of the target fused flow field features, thereby improving the realism of the video.

[0131] In one embodiment, reference is made to Figure 6 Step 502 may include:

[0132] Step 601: Perform shallow local attention fusion on the global flow field features and the target motion features to obtain the initial flow field motion fusion features;

[0133] Step 602: Decode the target action features to obtain action decoding features;

[0134] Step 603: Convolve the action decoding features to obtain action convolution features;

[0135] Step 604: Perform deep local attention fusion on the initial flow field action fusion features and action convolution features to obtain the target flow field action fusion features.

[0136] Both shallow local attention fusion and deep local attention fusion belong to local attention fusion, but one focuses on shallow features while the other focuses more on deep features. Action decoding features can be obtained by decoding target action features using a decoder. Action convolutional features can be obtained by convolutional networks, where the convolutional network can include multiple cascaded convolutional layers.

[0137] The advantage of the embodiments of steps 601 to 604 is that shallow local attention fusion and deep local attention fusion are introduced, and the action convolution features obtained by decoding and convolution have rich action information, which improves the richness of action information of the obtained target flow field action fusion features.

[0138] In another embodiment, after step 503, step 106 may further include:

[0139] The target number of times is obtained by incrementing the number of times the local attention fusion is performed on the original virtual object image frame;

[0140] If the number of target events is less than or equal to the attention fusion number threshold, then the target fused flow field feature is used as the target global flow field feature, and the target flow field action fusion feature is used as the target action feature, and the process returns to step 501.

[0141] If the number of target attempts exceeds the attention fusion attempt threshold, proceed to step 504.

[0142] The advantage of the above embodiments is that multiple consecutive first local attention fusions and second local attention fusions can be performed, which can improve the realism of the obtained target virtual object image frames, thereby improving the realism of the video.

[0143] In summary, this application achieves at least the following effects: (1) A novel method for generating virtual object videos allows the image of a virtual object to be generated based on any target action, thereby overcoming the limitations of using fixed action templates to generate virtual objects in the past. (2) A method for separating the appearance and clothing information of virtual objects is adopted. By using the outline and structure of the virtual object as an intermediate representation, the encoding of its appearance and clothing information is realized. Such an encoding method has advantages in both time and space. Even if the virtual object is wearing loose clothing, the generated target virtual object video can still maintain high fidelity. (3) In order to ensure that the generated video has high similarity to the input target action video, a global flow field estimator is used to extract global correlation from the input and generate a target global flow field feature to guide the subsequent image processing process. Then, a local attention mechanism is used to perform texture sampling in a local range based on the input target global flow field feature. Different features have different attention levels, which can better preserve high-frequency details and avoid gradient vanishing or gradient exploding problems, thereby improving the quality of the generated video. (4) By making virtual objects in virtual object videos more flexible and with richer body movements, customers’ sense of participation is enhanced, thereby improving customer’s user experience and satisfaction, increasing customer trust, and bringing positive impact to business development.

[0144] Please see Figure 7 This application also provides a virtual object video generation apparatus, which can implement the above-described virtual object video generation method. Figure 7 The present invention provides a block diagram of the module structure of a virtual object video generation device. The device includes: an original image extraction module 701, a clothing feature extraction module 702, an action image extraction module 703, an action feature extraction module 704, a global estimation module 705, and a local fusion module 706. Specifically, the original image extraction module 701 extracts original virtual object image frames from the original virtual object video; the clothing feature extraction module 702 extracts clothing features from the original virtual object image frames to obtain target clothing texture features; the action image extraction module 703 extracts target action image frames from the target action video; the action feature extraction module 704 extracts action features from the target action image frames to obtain target action features; the global estimation module 705 estimates the global flow field of the original virtual object video and the target action video to obtain target global flow field features; and the local fusion module 706 performs local attention fusion on the target clothing texture features, target action features, and target global flow field features to obtain the target virtual object video.

[0145] In one embodiment, the original virtual object video contains multiple candidate virtual object image frames, and the target action video contains multiple candidate action image frames. The action image extraction module is specifically used for: calculating an initial ratio based on the ratio of the number of candidate virtual object image frames to the number of candidate action image frames; rounding the initial ratio up to obtain the target number; and extracting image frames from the target action video according to the time sequence to obtain the target number of target action image frames; wherein the number of original virtual object image frames is the same as the number of target action image frames.

[0146] In one embodiment, the local fusion module 706 is specifically used for:

[0147] The first local attention fusion is performed based on the target global flow field features and the target clothing texture features to obtain the target flow field clothing fusion features;

[0148] A second local attention fusion is performed based on the global flow field characteristics and the target action characteristics to obtain the target flow field action fusion characteristics.

[0149] The target flow field fusion flow field features are obtained by multiplying the clothing fusion features and motion fusion features of the target flow field element by element.

[0150] The target number of times is obtained by incrementing the number of times the local attention fusion is performed on the original virtual object image frame;

[0151] If the number of target events is less than or equal to the threshold for the number of attention fusion events, then the target fused flow field features are taken as the target global flow field features, and the target flow field action fusion features are taken as the target action features. Then, the step of performing the first local attention fusion based on the target global flow field features and the target clothing texture features to obtain the target flow field clothing fusion features is returned.

[0152] If the number of times the target is exceeded is greater than the threshold for the number of times attention is fused, then the flow field features of the target are convolved to obtain the image frame of the target virtual object;

[0153] The target virtual object video is obtained by merging the image frames of the target virtual object.

[0154] It should be noted that the specific implementation of the virtual object video generation device is basically the same as the specific implementation of the virtual object video generation method described above, and will not be repeated here.

[0155] This application also provides an electronic device, which includes: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for communication between the processor and the memory. When the program is executed by the processor, it implements the aforementioned virtual object video generation method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0156] Please see Figure 8 , Figure 8 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0157] The processor 801 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0158] The memory 802 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 802 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 802 and is called and executed by the processor 801 using the virtual object video generation method of the embodiments of this application.

[0159] The 803 input / output interface is used to implement information input and output.

[0160] The communication interface 804 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0161] Bus 805 transmits information between various components of the device (e.g., processor 801, memory 802, input / output interface 803, and communication interface 804);

[0162] The processor 801, memory 802, input / output interface 803, and communication interface 804 are connected to each other within the device via bus 805.

[0163] This application also provides a computer-readable storage medium for computer-readable storage, wherein the storage medium stores one or more programs, which can be executed by one or more processors to implement the above-described virtual object video generation method.

[0164] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0165] The virtual object video generation method, apparatus, electronic device, and computer-readable storage medium proposed in this application extract original virtual object image frames from the original virtual object video and target action image frames from the target action video; secondly, clothing feature extraction is performed on the original virtual object image frames to obtain target clothing texture features, and action feature extraction is performed on the target action image frames to obtain target action features; furthermore, global flow field estimation is performed on the original virtual object video and the target action video to obtain target global flow field features; finally, local attention fusion is performed on the target clothing texture features, target action features, and target global flow field features to obtain the target virtual object video. This application, through the interaction of global flow field estimation and local attention fusion, not only overcomes the limitations of general neural network models in dealing with action transformations, but also assigns higher weights to important features, thereby ensuring the continuity of the video's action and effectively and realistically reproducing the fine appearance of clothing such as folds and textures. In summary, this application can generate virtual object videos with arbitrary actions and achieve a high degree of realism.

[0166] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0167] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0168] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0169] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0170] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0171] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0172] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0173] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0174] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0175] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0176] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for generating virtual object videos, characterized in that, The method includes: Extract the original virtual object image frames from the original virtual object video; Clothing features are extracted from the original virtual object image frame to obtain the target clothing texture features; Extract target action image frames from target action video; The target action image frame is subjected to action feature extraction to obtain target action features; Global flow field estimation is performed on the original virtual object video and the target motion video to obtain the target global flow field features; wherein, a global flow field estimator is used to estimate the global correlation between the original virtual object video and the target motion video to predict the target global flow field features. The global flow field estimator is used to estimate the motion of objects or regions from image sequences or videos, calculate the optical flow of each pixel, and infer the global motion features in the scene. Local attention fusion is performed on the target clothing texture features, the target motion features, and the target global flow field features to obtain the target virtual object video.

2. The method according to claim 1, characterized in that, The step of performing local attention fusion on the target clothing texture features, the target motion features, and the target global flow field features to obtain the target virtual object video includes: A first local attention fusion is performed based on the target global flow field features and the target clothing texture features to obtain the target flow field clothing fusion features; A second local attention fusion is performed based on the target global flow field features and the target action features to obtain the target flow field action fusion features; The target flow field clothing fusion feature and the target flow field motion fusion feature are multiplied element-wise to obtain the target fused flow field feature; Convolution is performed on the target fused flow field features to obtain the target virtual object image frame; The target virtual object video is obtained by merging the image frames of the target virtual object.

3. The method according to claim 2, characterized in that, The step of performing a second local attention fusion based on the target global flow field features and the target action features to obtain target flow field action fusion features includes: Shallow local attention fusion is performed on the target global flow field features and the target motion features to obtain initial flow field motion fusion features; The target action features are decoded to obtain action decoding features; The action decoding features are convolved to obtain action convolution features; The initial flow field action fusion feature and the action convolution feature are fused by deep local attention to obtain the target flow field action fusion feature.

4. The method according to any one of claims 1 to 3, characterized in that, The step of extracting clothing features from the original virtual object image frame to obtain the target clothing texture features includes: The original virtual object image frame is subjected to clothing feature extraction by a preset directional filter bank to obtain pixel-level gradient direction features and texture intensity features; The target clothing texture feature is obtained by adding the pixel-level gradient direction feature and the texture intensity feature element by element.

5. The method according to claim 4, characterized in that, The directional filter bank includes a contour extraction filter and a structure extraction filter; The step of extracting clothing features from the original virtual object image frame using a preset directional filter bank to obtain pixel-level gradient direction features and texture intensity features includes: The original virtual object image frame is subjected to clothing contour feature extraction by the contour extraction filter to obtain the pixel-level gradient direction feature; The original virtual object image frame is subjected to clothing structure feature extraction using the structure extraction filter to obtain the texture intensity feature.

6. The method according to any one of claims 1 to 3, characterized in that, The step of extracting action features from the target action image frame to obtain target action features includes: Key points are extracted from the target action image frame to obtain the target key points; By connecting the target key points, the target pose skeleton is obtained; Based on the target pose skeleton of multiple target action image frames, determine the target motion features; Actions are generated based on the target pose skeleton and the target motion features to obtain the target motion features.

7. The method according to claim 6, characterized in that, Determining the target motion action based on the target pose skeleton of multiple target motion image frames includes: The target pose skeleton of multiple target action image frames is sequentially represented to obtain a pose skeleton time series; Motion features are extracted from the time series of the posture skeleton to obtain the target motion features.

8. A virtual object video generation device, characterized in that, The device includes: The raw image extraction module is used to extract raw virtual object image frames from the raw virtual object video. The clothing feature extraction module is used to extract clothing features from the original virtual object image frame to obtain the target clothing texture features; The motion image extraction module is used to extract target motion image frames from target motion videos; The action feature extraction module is used to extract action features from the target action image frame to obtain target action features; A global estimation module is used to perform global flow field estimation on the original virtual object video and the target motion video to obtain the target global flow field features. Specifically, a global flow field estimator is used to estimate the global correlation between the original virtual object video and the target motion video to predict the target global flow field features. The global flow field estimator is used to estimate the motion of objects or regions from image sequences or videos and calculate the optical flow of each pixel to infer the global motion features in the scene. The local fusion module is used to perform local attention fusion on the target clothing texture features, the target motion features, and the target global flow field features to obtain the target virtual object video.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images