Virtual image driving method, device, equipment and storage medium

Abstract

The embodiment of the present invention discloses a method, device, equipment and storage medium for driving a virtual image. The method comprises: obtaining a target video frame, and inputting the target video frame into a virtual image driving model; the target video frame includes a target object corresponding to the target virtual image; through the virtual image driving model, extracting image features and human key point features corresponding to the target video frame, and performing feature fusion of the image features and the human key point features; obtaining a driving signal predicted by the virtual image driving model according to the fusion features; according to the driving signal, driving the target virtual image to perform the same action as the target object in the target video frame. The technical solution of the embodiment of the present invention drives the virtual image by fusing image features with human key point features, thereby improving the driving accuracy of the virtual image while meeting the real-time requirements.

Description

Technical Field

[0001] The embodiments of the present invention relate to the fields of artificial intelligence and computer vision technology, and in particular to a virtual image driving method, device, equipment and storage medium. Background Art

[0002] In recent years, with the rapid development of computer vision technology based on artificial intelligence, driving virtual images to perform corresponding actions based on images has great application prospects in many fields such as somatosensory games, virtual reality, security, and monitoring. At present, when driving virtual images based on images, if a model with a large amount of calculation is used, there will be a long delay time and it will be difficult to apply to the user client; if a driving model with a small amount of calculation is used, it will be difficult to train and converge, and the driving effect will be poor. Summary of the invention

[0003] The embodiments of the present invention provide a virtual image driving method, device, equipment and storage medium, so as to drive the virtual image by fusing image features with key point features of the human body, thereby improving the driving accuracy of the virtual image while meeting the real-time requirements.

[0004] In a first aspect, an embodiment of the present invention provides a method for driving a virtual image, comprising:

[0005] Acquire a target video frame and input the target video frame into the virtual image driving model; the target video frame includes a target object corresponding to the target virtual image;

[0006] The virtual image driving model is used to extract the image features and human key point features corresponding to the target video frame, and the image features and human key point features are fused;

[0007] Obtaining a driving signal predicted by the virtual image driving model according to the fusion features;

[0008] According to the driving signal, the target virtual image is driven to perform the same action as the target object in the target video frame.

[0009] Optionally, the model is driven by an avatar to extract image features and human key point features corresponding to the target video frame, including:

[0010] Extracting image features corresponding to the target video frame by driving a first feature extraction network in the avatar model;

[0011] The decoder network in the virtual image driving model is used to obtain the coordinates of two-dimensional human body key points corresponding to the image features;

[0012] The second feature extraction network in the virtual image driving model is used to extract features of the two-dimensional human body key point coordinates to obtain the human body key point features.

[0013] Optionally, the first feature extraction network is a MobileNet V3 network; and the second feature extraction network is a six-layer fully connected neural network.

[0014] Optionally, obtaining a driving signal predicted by the virtual image driving model according to the fusion feature includes:

[0015] The fused features are multiplied by weighted matrices of each fully connected layer through the fully connected layer in the virtual image driving model to map the fused features into six-dimensional spatial data;

[0016] The six-dimensional spatial data is converted into quaternion to obtain the driving signal corresponding to the target video frame.

[0017] Optionally, before obtaining the target video frame and inputting the target video frame into the virtual image driving model, the method further includes:

[0018] The COCO dataset is used to pre-train the first feature extraction network in the avatar-driven model for the human key point detection task.

[0019] Optionally, before obtaining the target video frame and inputting the target video frame into the virtual image driving model, the method further includes:

[0020] The loss function of the virtual image driving model is defined as the sum of the loss function of the driving signal and the loss function of the coordinates of the key points of the human body;

[0021] The loss function of the driving signal is the sum of the error of the projected 2D human key points and the error between the predicted driving signal and the real driving signal.

[0022] The loss function of the human body key point coordinates is the mean square error between each predicted two-dimensional human body key point coordinate and the true two-dimensional human body key point coordinate;

[0023] The loss function of the driving signal is independent of the loss function of the coordinates of the key points of the human body.

[0024] Optionally, according to the driving signal, driving the target virtual image to perform the same action as the target object in the target video frame includes:

[0025] According to the driving signal, the target posture data of each key point of the human body is determined;

[0026] The posture data of each human body key point of the target virtual image is updated to the corresponding target posture data to drive the target virtual image to perform the same action as the target object in the target video frame.

[0027] In a second aspect, an embodiment of the present invention further provides a virtual image driving device, comprising:

[0028] A model input module is used to obtain a target video frame and input the target video frame into the virtual image driving model; the target video frame includes a target object corresponding to the target virtual image;

[0029] A feature fusion module is used to extract image features and human key point features corresponding to the target video frame through a virtual image driving model, and perform feature fusion of the image features and the human key point features;

[0030] A prediction module, used to obtain a driving signal predicted by a virtual image driving model according to fusion features;

[0031] The driving module is used to drive the target virtual image to perform the same action as the target object in the target video frame according to the driving signal.

[0032] In a third aspect, an embodiment of the present invention further provides an electronic device, the device comprising:

[0033] one or more processors;

[0034] a storage device for storing one or more programs,

[0035] When one or more programs are executed by one or more processors, the one or more processors implement the virtual image driving method provided by any embodiment of the present invention.

[0036] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the virtual image driving method provided by any embodiment of the present invention.

[0037] In an embodiment of the present invention, a target video frame is obtained and input into a virtual image driving model; the target video frame includes a target object corresponding to the target virtual image; the image features and human key point features corresponding to the target video frame are extracted through the virtual image driving model, and the image features are fused with the human key point features; a driving signal predicted by the virtual image driving model according to the fused features is obtained; according to the driving signal, the target virtual image is driven to perform the same action as the target object in the target video frame, thereby solving the problem in the prior art that both delay time and driving effect cannot be taken into account when driving the virtual image, and the virtual image is driven by fusing the image features with the human key point features, thereby improving the driving accuracy of the virtual image while meeting the real-time requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is a flow chart of a virtual image driving method in Embodiment 1 of the present invention;

[0039] Figure 2ais a flow chart of a virtual image driving method in Embodiment 2 of the present invention;

[0040] Figure 2b It is a network architecture of a virtual image driving model in the second embodiment of the present invention;

[0041] Figure 3 It is a structural schematic diagram of a virtual image driving device in Embodiment 3 of the present invention;

[0042] Figure 4 It is a structural schematic diagram of an electronic device in Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0043] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0044] Embodiment 1

[0045] Figure 1 is a flow chart of a virtual image driving method in the first embodiment of the present invention. This embodiment is applicable to the case where a virtual image is driven to perform corresponding actions according to a video frame. The method can be executed by a virtual image driving device, which can be implemented by hardware and / or software and can generally be integrated in an electronic device that provides virtual image driving services. Figure 1 As shown, the method includes:

[0046] Step 110: Acquire a target video frame, and input the target video frame into the virtual image driving model; the target video frame includes a target object corresponding to the target virtual image.

[0047] In this embodiment, the target video frame can be a picture captured from a live video or a specified video, can be a picture downloaded from the Internet, or can be a picture captured by a camera in real time, etc. The target object can be any person appearing in the target video frame, such as a host in a live video frame, an athlete in a game map, etc. The target virtual object can be a two-dimensional character image corresponding to the target object.

[0048] In this embodiment, after acquiring the target video frame, the target video frame may be input into the virtual image driving model so as to identify the target object in the target video frame through the virtual image driving model and determine the current posture of the target object.

[0049] Step 120: extract image features and human key point features corresponding to the target video frame through the virtual image driving model, and perform feature fusion of the image features and the human key point features.

[0050] In this embodiment, in order to obtain a driving signal with high accuracy through less calculation, after the target video frame is input into the virtual image driving model, the image features can be extracted from the target video frame through the virtual image driving model, and then the human key point features corresponding to the target object are obtained according to the image features. Then, the image features are fused with the human key point features, that is, the feature data of the image features and the human key point features are spliced ​​to obtain the fused features.

[0051] Among them, image features can include color features, texture features, shape features, and spatial relationship features of the image. Color features and texture features are global features that describe the surface properties of the scene corresponding to the image or image area; there are two types of representation methods for shape features, one is contour features and the other is regional features. The contour features of the image are mainly aimed at the outer boundaries of the object, while the regional features of the image are related to the entire shape area; spatial relationship features refer to the spatial position or relative direction relationship between multiple targets segmented from the image. These relationships can also be divided into connection / adjacency relationships, overlap / overlapping relationships, and inclusion / inclusion relationships. Human key point features can be the features of joints on the human body with a certain degree of freedom, such as neck, shoulder, elbow, wrist, waist, knee, ankle, etc.

[0052] In this embodiment, by fusing image features with human body key point features, while obtaining rich feature information such as ambiguity, foreground and background provided by image features, the human body key point features can be used as a description of the region of interest (limb area rather than background area), so that the model focuses on learning the feature information of the region of interest to avoid performance degradation caused by a network with small computational load.

[0053] Step 130: Obtain a driving signal predicted by the virtual image driving model according to the fusion features.

[0054] In this embodiment, after fusing the image features with the key point features of the human body, the virtual image driving model generates a driving signal reflecting the current limb posture of the target object by mapping the fused features through the fully connected layer in the model. The driving signal actually includes the rotation angles of each key point of the human body, for example, the rotation angle of the elbow joint point, the rotation angle of the knee joint point, etc. By determining the rotation angle of each key point of the human body, the limb posture of the human body can be further determined, for example, the swinging posture of the limbs when running, etc.

[0055] Step 140: According to the driving signal, drive the target virtual image to perform the same action as the target object in the target video frame.

[0056] In this embodiment, after obtaining the driving signal output by the virtual image driving model, the target virtual image is driven to change from the initial posture to the same posture as the target object according to the driving signal. For example, the target virtual image changes from a standing posture with both hands raised horizontally to a running posture that is the same as the target object. The initial posture can be a fixed posture pre-set for the target virtual image, or it can be the posture that the target virtual image made when it was driven last time.

[0057] Optionally, according to the driving signal, driving the target virtual image to perform the same action as the target object in the target video frame may include: determining the target posture data of each human body key point according to the driving signal; updating the posture data of each human body key point of the target virtual image to the corresponding target posture data, so as to drive the target virtual image to perform the same action as the target object in the target video frame.

[0058] In this embodiment, the target posture data is the rotation angle of each human key point obtained from the driving signal. The human key points in the target virtual image can be adjusted accordingly according to the rotation angle of each key point, so that when the rotation angles of all human key points of the target virtual image are adjusted to be consistent with the driving signal, the target virtual image changes from the preset standing posture with both hands raised horizontally to the posture of the target object in the target video frame, thereby realizing the driving of the target virtual image.

[0059] In an embodiment of the present invention, a target video frame is obtained and input into a virtual image driving model; the target video frame includes a target object corresponding to the target virtual image; the image features and human key point features corresponding to the target video frame are extracted through the virtual image driving model, and the image features are fused with the human key point features; a driving signal predicted by the virtual image driving model according to the fused features is obtained; according to the driving signal, the target virtual image is driven to perform the same action as the target object in the target video frame, thereby solving the problem in the prior art that both delay time and driving effect cannot be taken into account when driving the virtual image, and the virtual image is driven by fusing the image features with the human key point features, thereby improving the driving accuracy of the virtual image while meeting the real-time requirements.

[0060] Embodiment 2

[0061] Figure 2a This is a flow chart of a virtual image driving method in the second embodiment of the present invention. This embodiment further refines the above embodiment and provides specific steps of extracting image features and human key point features corresponding to the target video frame through the virtual image driving model, and fusing the image features with the human key point features. Figure 2a A virtual image driving method provided in this embodiment is described, which includes the following steps:

[0062] Step 210: Train the preset machine learning model to obtain a virtual image driving model.

[0063] In this embodiment, in order to use less computational effort, the corresponding driving signal can be generated according to the video frame to drive the virtual image to change posture. The labeled video frames can be collected as training sets and verification sets, and the training set is used to iteratively learn and train the preset machine learning model, and the verification set is used to verify the trained model. The model output result can be substituted into the pre-defined model loss function to calculate the corresponding model loss value, and the Adam optimizer can be used to adjust the model parameters according to the model loss value, and the adjusted model is continuously trained until the model reaches the optimal value on the verification set, thereby obtaining the virtual image driving model.

[0064] The Adam optimizer can be used to train for 50 iterations, randomly select data for training for 1,000 steps in each iteration, and take the model with the best result in the validation set as the final avatar driving model. The avatar driving model includes a first feature extraction network, a decoder network, a second feature extraction network, a fully connected layer, etc.

[0065] Optionally, the COCO dataset is used to pre-train the first feature extraction network in the avatar-driven model for a human key point detection task.

[0066] In this embodiment, since the first feature extraction network is a small network model with less computational effort, it is not easy to achieve network convergence. Therefore, the COCO dataset can be used to pre-train the first feature extraction network for the human key point detection task, so that the first feature extraction network can converge quickly and the features extracted by the first feature extraction network can have a certain degree of limb perception ability.

[0067] Among them, the COCO (Common Objects in Context) dataset is a large-scale computer vision dataset proposed by Microsoft Research, which is dedicated to analyzing and evaluating common visual tasks, including object detection, instance segmentation, human key point detection, panoramic segmentation and other tasks.

[0068] Optionally, the loss function of the virtual image driving model is defined as the sum of the loss function of the driving signal and the loss function of the human body key point coordinates; wherein, the loss function of the driving signal is the sum of the error of the projected two-dimensional human body key point and the error between the predicted driving signal and the real driving signal; the loss function of the human body key point coordinates is the mean square error between each predicted two-dimensional human body key point coordinate and the real two-dimensional human body key point coordinate; the loss function of the driving signal and the loss function of the human body key point coordinates are independent of each other.

[0069] In this embodiment, according to the process of generating the driving signal by the virtual image driving model, the loss function of the virtual image driving model can be set to consist of two losses, one is the prediction loss of the driving signal, and the other is the prediction loss of the coordinates of the key points of the human body. drive , including the error generated when projecting two-dimensional human key points to extract the key point features of the human body, and the error between the predicted driving signal and the actual driving signal. kp , including the mean square error between the predicted two-dimensional human key point coordinates and the real two-dimensional human key point coordinates. Among them, the prediction loss of the driving signal is independent of the prediction loss of the human key point coordinates, that is, the prediction loss of the driving signal does not need to be passed to the decoder network when calculating its gradient. Finally, the loss function of the virtual image driving model can be expressed as: Loss = Loss drive +Loss kp .

[0070] In this embodiment, the prediction loss of the human body key point coordinates is kp It can be expressed as: Among them, N 2D is the number of key points of the two-dimensional human body, K 2d G is the real 2D human key point coordinates, K 2d p are the coordinates of the predicted two-dimensional human key points.

[0071] Step 220: Acquire a target video frame, and input the target video frame into the virtual image driving model.

[0072] The target video frame includes a target object corresponding to the target virtual image. The target video frame can be a picture captured from a live video or a specified video, a picture downloaded from the Internet, or a picture collected by a camera in real time. The target object can be any person appearing in the target video frame, such as a host in a live video frame, an athlete in a game map, etc. The target virtual object can be a two-dimensional character image corresponding to the target object.

[0073] Step 230: extract image features and human key point features corresponding to the target video frame through the virtual image driving model, and perform feature fusion of the image features and the human key point features.

[0074] In this embodiment, since the image features include background features and character features, and the human body key point features can accurately reflect the features of the character's limb key points, the two can be combined to achieve the purpose of accurately locating the character's position in the background and determining the character's current posture, avoiding performance degradation due to a small computing network.

[0075] Optionally, extracting image features and human key point features corresponding to the target video frame through the virtual image driving model may include: extracting image features corresponding to the target video frame through the first feature extraction network in the virtual image driving model; obtaining two-dimensional human key point coordinates corresponding to the image features through the decoder network in the virtual image driving model; and performing feature extraction on the two-dimensional human key point coordinates through the second feature extraction network in the virtual image driving model to obtain human key point features.

[0076] In this embodiment, Figure 2b As shown, after the target video frame is input into the virtual image driving model, the image features corresponding to the target video frame can be obtained through the first feature extraction network. For image features, unlike the previous image driving algorithm that directly predicts the driving signal, it is divided into two branches. One branch passes through a decoder network to predict the two-dimensional human body key point coordinates, and through the second feature extraction network, the two-dimensional human body key point coordinates are feature extracted to obtain the human body key point features. Then the obtained human body key point features are spliced ​​with the image features of the other branch to obtain the feature map of the fusion feature.

[0077] Optionally, the first feature extraction network is a MobileNet V3 network; and the second feature extraction network is a six-layer fully connected neural network.

[0078] In this embodiment, in order to balance the amount of computation and network performance, the MobileNet V3 network is selected as the first feature extraction network, which is used to input pictures and output feature maps. Among them, the MobileNet V3 network can be specifically divided into three parts: the starting part includes 1 convolution layer, which extracts features through 3x3 convolution; the middle part includes multiple convolution blocks; the last part includes 512 1*1 convolution kernels. Exemplarily, if a 224*224 resolution, three-channel picture of the first sample number is input into the first feature extraction network, the first feature extraction network can output the first sample number*512 feature maps.

[0079] In this embodiment, the decoder network is actually equivalent to a deconvolution network module. For example, if the feature map with a resolution of 7*7 and 160 channels output by the first feature network is input into the decoder network, a key point heat map with a resolution of 224*224 is output through a three-layer deconvolution network. The key point heat map can quickly obtain the coordinates of the key points by taking the coordinates of the maximum response value.

[0080] Among them, the decoder network can calculate the prediction loss Loss of the key point coordinates based on the predicted key point coordinates and the actual key point coordinates kp , and according to Loss kp Continuous training makes the key point heat map predicted by the decoder network more and more accurate.

[0081] Exemplarily, for the second feature extraction network, if the decoder network outputs the coordinates of 25 two-dimensional human body key points of the COCO model, after the human body key point coordinates of the first sample number * 50 dimensions are input into the second feature extraction network, it can be passed through a four-layer fully connected network in the second feature extraction network, that is, input 50 dimensions and output 128 dimensions, input 128 dimensions and output 256 dimensions, input 256 dimensions and output 256 dimensions, and input 256 dimensions and output 256 dimensions, a total of four layers, and finally the two-dimensional human body key point features of the first sample number * 256 are obtained.

[0082] Subsequently, the two-dimensional human body key point features of the first sample number*256 and the feature map of the first sample number*512 are feature fused to generate a feature map of the first sample number*768.

[0083] Step 240: Obtain the driving signal predicted by the virtual image driving model according to the fusion features.

[0084] Exemplarily, after generating a feature map of the first number of samples*768, a fully connected network can be used to map the fused feature map to generate and output a driving signal of the first number of samples*144.

[0085] Optionally, obtaining the driving signal predicted by the virtual image driving model based on the fusion features may include: multiplying the fusion features with the weighted matrices of each fully connected layer in the virtual image driving model to map the fusion features into six-dimensional spatial data; converting the six-dimensional spatial data into quaternions to obtain a driving signal corresponding to the target video frame.

[0086] Among them, the common driving signal is a quaternion (equivalent to four-dimensional space data), and this embodiment uses the rotation angle of the key point of the human body in three-dimensional space. When the rotation angle of three-dimensional space is used, at least five-dimensional space vectors are required to continuously represent the driving signal, that is, the fusion feature needs to be mapped to at least five-dimensional space data. Based on this situation, considering that the data conversion between six-dimensional space data and quaternion is more convenient, and the use of six-dimensional space data can make the actual network performance better, therefore, the fusion feature can be mapped to six-dimensional space data.

[0087] In this embodiment, the feature map of the first sample number * 768 can be input as an input vector to the fully connected layer, and the product operation is performed with the weighted matrix of each fully connected layer to map the feature map of the fused feature to generate six-dimensional spatial data of the first sample number * 144. Among them, the fully connected layer includes three fully connected layers: input 768 dimensions and output 512 dimensions, input 512 dimensions and output 256 dimensions, and input 256 dimensions and output 144 dimensions. Then the six-dimensional spatial data is converted into a transformation matrix. For example, the six-dimensional spatial data is normalized to obtain a first vector, the six-dimensional spatial data is orthogonalized to obtain a second vector, and the vector product of the first vector and the second vector is used as a rotation matrix. Then, according to the existing conversion method between the rotation matrix and the quaternion, for example, the elements of the rotation matrix are multiplied, divided, and the inverse trigonometric function is performed, and the quaternion is extracted from the rotation matrix to obtain the driving signal output by the network model.

[0088] In this embodiment, the input and output of the fully connected layer are both vectors, and the trained weighted matrix is ​​stored locally. The fully connected layer actually multiplies the stored weighted matrix by the input vector to obtain the output result. The essence is to linearly transform from one feature space to another feature space to extract and integrate useful information. Among them, the fully connected layer continuously improves the value of the weighted matrix through training, so that the product of the weighted matrix and the input vector is closer to the expected true value.

[0089] Step 250: According to the driving signal, the target virtual image is driven to perform the same action as the target object in the target video frame.

[0090] It should be noted that, through experiments, it is found that compared with the solution of driving the virtual image using only image features, the technical solution of this embodiment improves the network performance by 5.1%, while the delay is only increased by 2ms, meeting the real-time requirements. In addition, compared with the virtual image driving solution using only key point prediction, this embodiment has a significant improvement in the ambiguity of the driving signal.

[0091] In an embodiment of the present invention, a target video frame is obtained and input into a virtual image driving model; the target video frame includes a target object corresponding to the target virtual image; the image features and human key point features corresponding to the target video frame are extracted through the virtual image driving model, and the image features are fused with the human key point features; a driving signal predicted by the virtual image driving model according to the fused features is obtained; according to the driving signal, the target virtual image is driven to perform the same action as the target object in the target video frame, thereby solving the problem in the prior art that both delay time and driving effect cannot be taken into account when driving the virtual image, and the virtual image is driven by fusing the image features with the human key point features, thereby improving the driving accuracy of the virtual image while meeting the real-time requirements.

[0092] Embodiment 3

[0093] Figure 3 is a schematic diagram of the structure of a virtual image driving device in the third embodiment of the present invention. This embodiment is applicable to the case where the virtual image is driven to perform corresponding actions according to the video frame. The device can be implemented by hardware and / or software and can generally be integrated into an electronic device that provides virtual image driving services. Figure 3 As shown, the device includes: a model input module 310, a feature fusion module 320, a prediction module 330 and a driving module 340;

[0094] The model input module 310 is used to obtain a target video frame and input the target video frame into the virtual image driving model; the target video frame includes a target object corresponding to the target virtual image;

[0095] A feature fusion module 320 is used to extract image features and human key point features corresponding to the target video frame through the virtual image driving model, and perform feature fusion of the image features and the human key point features;

[0096] Prediction module 330, used to obtain the driving signal predicted by the virtual image driving model according to the fusion feature;

[0097] The driving module 340 is used to drive the target virtual image to perform the same action as the target object in the target video frame according to the driving signal.

[0098] In an embodiment of the present invention, a target video frame is obtained and input into a virtual image driving model; the target video frame includes a target object corresponding to the target virtual image; the image features and human key point features corresponding to the target video frame are extracted through the virtual image driving model, and the image features are fused with the human key point features; a driving signal predicted by the virtual image driving model according to the fused features is obtained; according to the driving signal, the target virtual image is driven to perform the same action as the target object in the target video frame, thereby solving the problem in the prior art that both delay time and driving effect cannot be taken into account when driving the virtual image, and the virtual image is driven by fusing the image features with the human key point features, thereby improving the driving accuracy of the virtual image while meeting the real-time requirements.

[0099] Optionally, the feature fusion module 320 is used to:

[0100] Extracting image features corresponding to the target video frame by driving a first feature extraction network in the avatar model;

[0101] The decoder network in the virtual image driving model is used to obtain the coordinates of two-dimensional human body key points corresponding to the image features;

[0102] The second feature extraction network in the virtual image driving model is used to extract features of the two-dimensional human body key point coordinates to obtain the human body key point features.

[0103] Optionally, the first feature extraction network is a MobileNet V3 network; and the second feature extraction network is a six-layer fully connected neural network.

[0104] Optionally, the prediction module 330 is used to:

[0105] Through the fully connected layer in the virtual image driving model, the fusion feature is multiplied with the weighted matrix of each fully connected layer to map the fusion feature into six-dimensional spatial data; the six-dimensional spatial data is converted into quaternion to obtain a driving signal corresponding to the target video frame.

[0106] Optionally, it also includes: a pre-training module, which is used to pre-train the first feature extraction network in the virtual image driving model for a human key point detection task using the data set COCO before obtaining the target video frame and inputting the target video frame into the virtual image driving model.

[0107] Optionally, it also includes: a loss function definition module, which is used to define the loss function of the virtual image driving model as the sum of the loss function of the driving signal and the loss function of the human body key point coordinates before obtaining the target video frame and inputting the target video frame into the virtual image driving model;

[0108] The loss function of the driving signal is the sum of the error of the projected 2D human key points and the error between the predicted driving signal and the real driving signal.

[0109] The loss function of the human body key point coordinates is the mean square error between each predicted two-dimensional human body key point coordinate and the true two-dimensional human body key point coordinate;

[0110] The loss function of the driving signal is independent of the loss function of the coordinates of the key points of the human body.

[0111] Optionally, the driving module 340 is used to:

[0112] According to the driving signal, the target posture data of each key point of the human body is determined;

[0113] The posture data of each human body key point of the target virtual image is updated to the corresponding target posture data to drive the target virtual image to perform the same action as the target object in the target video frame.

[0114] The virtual image driving device provided in the embodiment of the present invention can execute the virtual image driving method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0115] Embodiment 4

[0116] Figure 4 It is a structural schematic diagram of an electronic device in Embodiment 4 of the present invention. Figure 4 A block diagram of an exemplary device 12 suitable for use in implementing embodiments of the present invention is shown. Figure 4 The device 12 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0117] like Figure 4 As shown, device 12 is in the form of a general purpose computing device. Components of device 12 may include, but are not limited to, one or more processors or processing units 16, system memory 28, and bus 18 that connects various system components (including system memory 28 and processing unit 16).

[0118] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. By way of example, these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0119] Device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by device 12, including volatile and non-volatile media, removable and non-removable media.

[0120] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Figure 4 not shown, usually called a "hard drive"). Although Figure 4 Not shown in the figure, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present invention.

[0121] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in the memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0122] The device 12 may also communicate with one or more external devices 14 (e.g., keyboards, pointing devices, displays 24, etc.), may communicate with one or more devices that enable a user to interact with the device 12, and / or may communicate with any device that enables the device 12 to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed via an input / output (I / O) interface 22. Furthermore, the device 12 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with other modules of the device 12 via a bus 18. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the device 12, including, but not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0123] The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, such as implementing the virtual image driving method provided in the embodiment of the present invention.

[0124] That is, a virtual image driving method is implemented, including:

[0125] Acquire a target video frame and input the target video frame into the virtual image driving model; the target video frame includes a target object corresponding to the target virtual image;

[0126] The virtual image driving model is used to extract the image features and human key point features corresponding to the target video frame, and the image features and human key point features are fused;

[0127] Obtaining a driving signal predicted by the virtual image driving model according to the fusion features;

[0128] According to the driving signal, the target virtual image is driven to perform the same action as the target object in the target video frame.

[0129] Embodiment 5

[0130] Embodiment 5 of the present invention further discloses a computer storage medium, on which a computer program is stored. When the program is executed by a processor, a method for driving a virtual image is implemented, including:

[0131] Acquire a target video frame and input the target video frame into the virtual image driving model; the target video frame includes a target object corresponding to the target virtual image;

[0132] The virtual image driving model is used to extract the image features and human key point features corresponding to the target video frame, and the image features and human key point features are fused;

[0133] Obtaining a driving signal predicted by the virtual image driving model according to the fusion features;

[0134] According to the driving signal, the target virtual image is driven to perform the same action as the target object in the target video frame.

[0135] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0136] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0137] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0138] Computer program code for performing the operations of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0139] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A method for driving a virtual image, characterized in that: include: The first feature extraction network in the avatar-driven model is pre-trained for the human key point detection task using the COCO dataset; Acquire a target video frame, and input the target video frame into a virtual image driving model; the target video frame includes a target object corresponding to the target virtual image; By means of the virtual image driving model, image features and human key point features corresponding to the target video frame are extracted, and the image features and the human key point features are subjected to feature fusion; wherein the image features include color features, texture features, shape features and spatial relationship features of the image; Through the fully connected layers in the virtual image driving model, the fusion features are multiplied by the weighted matrices of each fully connected layer to map the fusion features into six-dimensional spatial data; Convert the six-dimensional spatial data into a quaternion to obtain a driving signal corresponding to the target video frame; wherein the driving signal includes the rotation angle of each key point of the human body; According to the driving signal, the target virtual image is driven to perform the same action as the target object in the target video frame.

2. The method according to claim 1, characterized in that: The image features and human key point features corresponding to the target video frame are extracted by the virtual image driving model, including: Extracting image features corresponding to the target video frame through a first feature extraction network in the virtual image driving model; Acquire the coordinates of two-dimensional human body key points corresponding to the image features through a decoder network in the virtual image driving model; The second feature extraction network in the virtual image driving model is used to extract features of the two-dimensional human body key point coordinates to obtain human body key point features.

3. The method according to claim 2, characterized in that The first feature extraction network is a MobileNet V3 network; the second feature extraction network is a six-layer fully connected neural network.

4. The method according to claim 1, characterized in that: Before obtaining the target video frame and inputting the target video frame into the virtual image driving model, the method further includes: The loss function of the virtual image driving model is defined as the sum of the loss function of the driving signal and the loss function of the two-dimensional human body key point coordinates; The loss function of the driving signal is the sum of the error generated when projecting the two-dimensional human body key points to extract the key point features of the human body and the error between the predicted driving signal and the real driving signal; The loss function of the two-dimensional human body key point coordinates is the mean square error between each predicted two-dimensional human body key point coordinate and the true two-dimensional human body key point coordinate; The loss function of the driving signal and the loss function of the two-dimensional human body key point coordinates are independent of each other.

5. The method according to claim 1, characterized in that According to the driving signal, driving the target virtual image to perform the same action as the target object in the target video frame includes: Determining target posture data of each key point of the human body according to the driving signal; The posture data of each human body key point of the target virtual image is updated to the corresponding target posture data, so as to drive the target virtual image to perform the same action as the target object in the target video frame.

6. A virtual image driving device, characterized in that: include: A pre-training module, used to pre-train the first feature extraction network in the avatar driving model for a human key point detection task using the COCO dataset; A model input module, used to obtain a target video frame and input the target video frame into a virtual image driving model; the target video frame includes a target object corresponding to the target virtual image; A feature fusion module, used to extract image features and human key point features corresponding to the target video frame through the virtual image driving model, and perform feature fusion of the image features and the human key point features; wherein the image features include color features, texture features, shape features and spatial relationship features of the image; A prediction module, configured to perform a product operation on the fusion feature and the weighted matrix of each fully connected layer through the fully connected layer in the virtual image driving model to map the fusion feature into six-dimensional spatial data; convert the six-dimensional spatial data into a quaternion to obtain a driving signal corresponding to the target video frame; wherein the driving signal includes the rotation angle of each key point of the human body; The driving module is used to drive the target virtual image to perform the same action as the target object in the target video frame according to the driving signal.

7. An electronic device, characterized in that: The device comprises: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the virtual image driving method as described in any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the virtual image driving method as described in any one of claims 1-5 is implemented.

Images