Method, device and server for driving a virtual avatar

Abstract

The embodiment of the application provides a virtual image driving method, device and server, and relates to the technical field of video image processing. The first topological image and the second topological image are constructed from the human body detection algorithm end and the target virtual image end respectively, and the target virtual object corresponding to the second topological image can perform the same action with the target person through the action posture data synchronous repair mode. Compared with the scheme that the human body action data obtained from the target person by directly using the human body detection algorithm is used to drive the target virtual image, the technical scheme provided by the application can overcome the driving effect difference when the same human body action data is used to drive the target virtual image with different body characteristics, can optimize the driving effect of the target virtual image, and can improve the interactive experience. In addition, the above process does not involve complex algorithms, has less resource occupation and short time consumption, and can also be compatible with different human body detection algorithms and different target virtual images.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of Internet live broadcast, in particular to a virtual image driving method and device and server. BACKGROUND

[0002] With the continuous development of mobile Internet technology and network communication technology, network live broadcast has been rapidly developed and applied in people's daily work and life. For example, users can watch live content provided by anchors on a live broadcast platform through smart phones, computers, tablet computers and other devices, or users can provide live content on the corresponding live broadcast platform at any time and anywhere through smart phones, computers, tablet computers and other devices for others to watch.

[0003] At present, common live broadcast methods include live broadcast interaction methods based on real anchors, which interact with viewers in the live broadcast room through live broadcast pictures of real anchors. Viewers can interact with anchors through methods such as sending bullet screens, rewarding, sending gifts, and connecting microphones. In addition, in some specific live broadcast scenarios, in order to provide diversified live broadcast experience, virtual live broadcast methods based on virtual images have also been widely applied. Compared with the live broadcast method of real anchors, virtual live broadcast does not require anchors to interact with real people. Anchors can interact with viewers in the background by controlling virtual images to simulate the behavior of the background anchor.

[0004] Among them, the live broadcast method based on virtual images allows anchors to communicate and interact with viewers through virtual images. For example, a virtual image (e.g., a cartoon character image) consistent with the anchor's body movements can be used to interact with viewers in a game to increase the live broadcast effect. In order to achieve better live broadcast effect and user experience, effectively driving virtual images to complete the same or almost consistent interactive actions with anchors is a problem that needs to be focused on by those skilled in the art. SUMMARY

[0005] To solve the above technical problems, the present application provides a virtual image driving method, device and server.

[0006] In a first aspect, the present application provides a virtual image driving method, which comprises:

[0007] Obtaining human body action data of a target person based on a human body detection algorithm, migrating the human body action data to a general skeletal topology structure having a joint mapping relationship with the human body detection algorithm, and obtaining a first topology image;

[0008] acquire body data of the target virtual image, and obtain a second topological image based on a mapping relationship between the body data and the general skeleton topological structure, wherein a driving pose in the human body detection algorithm is semantically aligned with a driving pose in the target virtual image;

[0009] synchronize and repair motion pose data of the second topological image based on the motion pose data of the first topological image;

[0010] drive the target virtual image to move according to the repaired motion pose data of the second topological image.

[0011] In a possible implementation, before the step of acquiring body motion data of a target character based on a human body detection algorithm and migrating the body motion data to a general skeleton topological structure that has a joint mapping relationship with the human body detection algorithm to obtain a first topological image, the method further includes:

[0012] creating the general skeleton topological structure, wherein the general skeleton topological structure includes a head, limbs, fingers, and an expression base;

[0013] defining a number of joints in the general skeleton topological structure, semantics of each joint, and a hierarchical relationship between different joints.

[0014] In a possible implementation, before the step of acquiring body motion data of a target character based on a human body detection algorithm and migrating the body motion data to a general skeleton topological structure that has a joint mapping relationship with the human body detection algorithm to obtain a first topological image, the method further includes:

[0015] creating a first mapping relationship between joints in the human body detection algorithm and joints in the general skeleton topological structure to obtain a first configuration file;

[0016] creating a second mapping relationship between joints in the target virtual image and joints in the general skeleton topological structure to obtain a second configuration file;

[0017] based on a driving pose common to the human body detection algorithm and the target virtual image, creating driving pose information and collision body parameter information generated according to characteristics of the target virtual image to obtain a third configuration file.

[0018] In a possible implementation, the step of acquiring body motion data of a target character based on a human body detection algorithm and migrating the body motion data to a general skeleton topological structure that has a joint mapping relationship with the human body detection algorithm to obtain a first topological image includes:

[0019] mapping relationship, the human action data is migrated into the general skeleton topology to obtain the first topological image, the human action data including joint local rotation data and joint local translation data of the target character;

[0020] The step of obtaining the body data of the target virtual image and obtaining the second topological image based on a mapping relationship between the body data and the general skeleton topology includes:

[0021] The body data is migrated into the general skeleton topology based on the second mapping relationship to obtain the second topological image, the body data including joint local translation data of the virtual image.

[0022] In a possible implementation, the synchronization repair includes a slide repair and a collision repair, and the step of synchronously repairing the motion posture data of the second topological image based on the motion posture data of the first topological image includes:

[0023] Based on the human action data, ground contact information of whether the feet of the first topological image contact the ground is obtained, and the second topological image is repaired based on the human action data, root joint translation data, joint local translation data of the target virtual image, and the ground contact information.

[0024] Collision relationships of different joint collision bodies in the second topological image after the motion posture data of the first topological image is migrated into the second topological image are detected, and based on a pre-set joint collision body priority, an obstacle and a collision repair object are determined from the different joint collision bodies, and the collision repair object is repaired.

[0025] In a possible implementation, the step of obtaining the ground contact information of whether the feet of the first topological image contact the ground based on the human action data, and repairing the second topological image based on the human action data, root joint translation data, joint local translation data of the target virtual image, and the ground contact information includes:

[0026] Based on the human action data, a forward kinematics is used to obtain a foot position of the first topological image in each frame of image and a length of a lower body.

[0027] A ratio between a distance between the same foot position in a current frame and a preset frame and the length of the lower body is calculated, and a height difference value between heights of two feet in the current frame is calculated.

[0028] The ratio and the height difference value are compared with a first threshold value and a second threshold value respectively, and ground contact information of whether the feet of the first topological image contact the ground is obtained according to a comparison result.

[0029] scaling the root joint translation data based on a ratio between a length of a lower body of the first topological figure and a length of a lower body of the second topological figure;

[0030] correcting a joint kinematic chain of the lower body of the second topological figure based on the ground contact information to make the second topological figure and the first topological figure touch or leave the ground at the same time.

[0031] In a possible implementation, the step of comparing the ratio and the height difference value with a first threshold value and a second threshold value respectively, and obtaining ground contact information of whether the feet of the first topological figure are in contact with the ground according to a comparison result, comprises:

[0032] if the ratio is less than the first threshold value and the height difference value is less than the second threshold value, it is determined that both feet of the first topological figure are in contact with the ground;

[0033] if the ratio is less than the first threshold value and the height difference value is not less than the second threshold value, it is determined that the foot of the first topological figure with a lower height is in contact with the ground;

[0034] if the ratio is not less than the first threshold value, it is determined that both feet of the first topological figure are in contact with the ground.

[0035] In a possible implementation, the step of detecting collision relationships of different joint collision bodies in the second topological figure after the motion pose data of the first topological figure is migrated to the second topological figure, determining an obstacle and a collision repair object in the different joint collision bodies based on a preset joint collision body priority, and performing collision repair on the collision repair object, comprises:

[0036] generating joint collision bodies on joints of the second topological figure;

[0037] calculating collision depths and collision directions between different joint collision bodies of the second topological figure after the motion pose data of the first topological figure is migrated to the second topological figure;

[0038] determining an obstacle and a collision repair object in the different joint collision bodies based on a preset joint collision body priority, and calculating a position of the collision repair object after collision repair based on the collision depths and the collision directions between the different joint collision bodies;

[0039] adjusting local rotation data of joints in a joint kinematic chain of the collision repair object based on the position of the collision repair object after collision repair to obtain motion pose data after collision repair.

[0040] In a second aspect, the embodiments of the present application further provide a virtual image driving device, the device comprising:

[0041] a first determining module configured to obtain human body action data of a target person based on a human body detection algorithm, and migrate the human body action data to a general skeleton topology structure having a joint mapping relationship with the human body detection algorithm to obtain a first topology image;

[0042] a second determining module configured to obtain body data of a target virtual image, and obtain a second topology image based on a mapping relationship between the body data and the general skeleton topology structure, wherein a driving pose in the human body detection algorithm is aligned with a driving pose in the target virtual image;

[0043] a repairing module configured to synchronize and repair action pose data of the second topology image based on action pose data of the first topology image;

[0044] a driving module configured to drive the target virtual image to move according to the repaired action pose data of the second topology image.

[0045] In a third aspect, the embodiments of the present application further provide a server, the server comprising a processor, a communication unit and a computer readable storage medium, the processor, the communication unit and the computer readable storage medium are connected through a bus system, the communication unit is configured to connect an electronic device to realize data interaction between the server and the electronic device, the computer readable storage medium is configured to store programs, instructions or codes, and the processor is configured to execute the programs, instructions or codes in the computer readable storage medium to realize the virtual image driving method in any one of the possible implementation manners of the first aspect.

[0046] In a fourth aspect, the embodiments of the present application provide a computer readable storage medium, the computer readable storage medium stores instructions, when the instructions are executed, the computer executes the virtual image driving method in the first aspect or any one of the possible implementation manners of the first aspect.

[0047] Based on any one of the above aspects, the virtual image driving method, device and server provided by the embodiments of the present application can overcome the driving effect difference when the same human body action data drives different body feature target virtual images, can optimize the driving effect of the target virtual image, and improve the interactive experience, compared with the scheme in the prior art that directly uses the human body action data obtained from the target person by the human body detection algorithm to drive the target virtual image. BRIEF DESCRIPTION OF DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required to be invoked in the embodiments will be briefly introduced as follows. It should be understood that the following drawings only show some of the embodiments of the present application, and therefore should not be regarded as a limitation on the scope, and for those of ordinary skill in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0049] Figure 1 An interaction scene schematic diagram of a virtual image driving system provided by the embodiments of the present application;

[0050] Figure 2 A flowchart of a virtual image driving method provided by the embodiments of the present application;

[0051] Figure 3 A driving posture schematic diagram and a schematic diagram of a collision body in the driving posture provided by the embodiments of the present application;

[0052] Figure 4 A flowchart of a step S23 in the method provided by the embodiments of the present application; Figure 2 A flowchart of a step S231 in the method provided by the embodiments of the present application;

[0053] Figure 5 A flowchart of a step S232 in the method provided by the embodiments of the present application; Figure 4 A flowchart of a step S231 in the method provided by the embodiments of the present application;

[0054] Figure 6 A flowchart of a step S232 in the method provided by the embodiments of the present application; Figure 4 A flowchart of a step S231 in the method provided by the embodiments of the present application;

[0055] Figure 7 A schematic diagram of a collision of a collision body provided by the embodiments of the present application;

[0056] Figure 8 A schematic diagram of a collision of a collision body provided by the embodiments of the present application;

[0057] Figure 9 A functional module schematic diagram of a virtual image driving device provided by the embodiments of the present application;

[0058] Figure 10 A possible structure schematic diagram of a server provided by the embodiments of the present application. DETAILED DESCRIPTION

[0059] To make the purposes, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. It should be appreciated that the drawings in the present application are only intended to illustrate and describe the purposes, and are not used to limit the protection scope of the present application. In addition, it should be appreciated that the schematic drawings are not drawn according to the actual proportions. The flowcharts used in the present application show the operations implemented according to some embodiments of the present application. It should be appreciated that the operations of the flowcharts can not be implemented in sequence, and the steps without logical context relationship can be reversed in sequence or implemented simultaneously. In addition, one or more other operations can be added to the flowcharts or one or more operations can be removed from the flowcharts under the guidance of the content of the present application.

[0060] In addition, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0061] To solve the technical problems mentioned in the foregoing background, in one possible implementation in the prior art, a human body detection algorithm can be used first to obtain human body motion data (such as joint motion data) of a target person, and then the human body motion data is applied to corresponding joints of a virtual image to drive the virtual image.

[0062] In the above solution, the inventors find that, because the differences in physical characteristics (such as the differences in height, weight, and body shape or the differences in length of the same joint) of different virtual images are not considered, driving different virtual images with the same joint motion data of a target person will cause inconsistent driving effects of the virtual images (such as the same joint motion data will cause the virtual image to be worn, for example, the hands of a virtual image with a fat body shape are more likely to be inserted into the head or body than the hands of a virtual image with a thin body shape). In addition, because the joint lengths defined in the human body detection algorithm are inconsistent with the joint lengths of the corresponding joints in the virtual image, after performing forward kinematics processing, the positions of the end joints (such as the feet) will be different, thereby causing the technical problem of foot sliding.

[0063] In order to solve the above technical problems in the prior art, the embodiment of the present application provides a virtual image driving scheme. In order to facilitate the understanding of the present application scheme, the driving system of the virtual image to which the present application can be applied is introduced first. It can be understood that the driving system of the virtual image introduced below is only to illustrate the possible application scenarios of the present application scheme, and the present application scheme can also be applied in other application scenarios other than the following scenarios.

[0064] Please refer to Figure 1 , Figure 1 A possible interaction scenario of the virtual image driving system provided by the present application scheme is shown. The virtual image driving system 10 can include a server 100, a host end 200 and a viewer end 300 connected in communication. The server 100 can provide video image processing support for the host end 200, such as storing a human body detection algorithm in the server 100 to obtain human body motion data by processing live video.

[0065] In the embodiment of the present application, the host end 200 and the viewer end 300 can be, but are not limited to, a smart phone, a personal digital assistant, a tablet computer, a personal computer, a notebook computer, a virtual reality terminal device, an augmented reality terminal device, etc. In the specific implementation process, there can be multiple host ends 200 and viewer ends 300 accessing the server 100, Figure 1 only one host end 200 and two viewer ends 300 are shown in the server 100. Among them, the host end 200 and the viewer end 300 can install a live service program, for example, the service program can be an application APP or a mini program related to Internet live in a computer or a smart phone.

[0066] In the embodiment of the present application, the server 100 can be a single physical server, or a server group composed of multiple physical servers for performing different data processing functions. The server group can be centralized or distributed (for example, the server 100 can be a distributed system). In some possible embodiments, if the server 100 adopts a single physical server, different logical server components can be allocated to the physical server based on different business functions.

[0067] It can be understood that Figure 1 the live scene shown is only one possible example, and in other possible embodiments, the live scene can also include only a part of the components shown or can also include other components. Figure 1

[0068] The virtual image driving method provided by the present application embodiment will be exemplarily described below in combination with the application scenario shown in Figure 1 Figure 2 ​​The driving method of the virtual image provided in the embodiments of the present application can be executed by the foregoing server 100, the order of some steps in the driving method of the virtual image in the embodiments of the present application can be exchanged with each other according to actual needs, or some steps therein can be omitted or deleted, and the detailed steps of the driving method of the virtual image executed by the server 100 are introduced as follows.

[0069] In step S21, the human body action data of the target person is obtained based on a human body detection algorithm, and the human body action data is migrated to a general skeleton topology structure having a joint mapping relationship with the human body detection algorithm to obtain a first topology image.

[0070] The human body detection algorithm can include a light capture human body detection algorithm, an inertial capture human body detection algorithm, a visual human body detection algorithm, and the like. In the scenario provided in the embodiments of the present application, the human body detection algorithm can be a human body detection algorithm based on an RGB video image. For example, after the server 100 obtains the live video frame image of the anchor end 200, the live video frame image is subjected to human body detection to obtain the human body action data of the target person. Each joint of the human body is defined in the human body detection algorithm, and a mapping between the joint defined in the human body detection algorithm and the joint in the general skeleton topology structure can be established in advance to establish a corresponding relationship between the joints having the same joint semantics. More joints can be defined in the general skeleton topology structure than in the human body detection algorithm to ensure that the human body action data can be completely migrated to the joint in the general skeleton topology structure corresponding to the joint in the human body detection algorithm in the process of migrating the human body action data to the skeleton topology structure, thereby avoiding loss of the human body action data in the migration process.

[0071] In step S22, the body data of the target virtual image is obtained, and a second topology image is obtained based on a mapping relationship between the body data and the general skeleton topology structure.

[0072] The target virtual image can be a humanoid virtual image having a body structure similar to that of a human body. The driving poses in the human body detection algorithm are semantically aligned with the driving poses in the target virtual image. In each detection, the driving poses in the human body detection algorithm and the driving poses in the target virtual image need to be semantically aligned to ensure that the human body action data obtained by the human body detection algorithm can make the target virtual image complete the same action as the target person from a static driving pose. Based on the mapping relationship between the joints in the body data of the target virtual image and the joints in the general skeleton topology structure, the second topology image can be obtained. More joints can be defined in the general skeleton topology structure than in the target virtual image to ensure that the second topology image has a body structure corresponding to the target virtual object.

[0073] In step S23, the action pose data of the second topology image is synchronously repaired based on the action pose data of the first topology image.

[0074] In the embodiment of the present application, after the human body action data is migrated to the general skeleton topology, a first topological image of the motion is formed. The action posture data based on the first topological image can be used to synchronously repair the action posture data of the second topological image, so that the second topological image performs the same action as the first topological image. Since the first topological image and the second topological image are obtained based on different mapping relationships, the second topological image can perform the same action as the first topological image through the above-mentioned synchronous repair, avoiding the technical problem that the second topological image may slide and / or be inserted when the motion posture data of the first topological image is directly migrated to the second topological image.

[0075] In step S24, the target virtual image is driven to move according to the action posture data of the repaired second topological image.

[0076] In the embodiment of the present application, the target virtual image is driven to move based on the action posture data of the repaired second topological image, which can ensure that the second topological image can be consistent with the action of the target person and will not have the technical problem that the driving effect is poor due to the difference in body features of the target virtual image. After the driving of the target virtual image is completed, the server 100 can send the video frame image formed after the target virtual image is driven to the audience end 300, and the audience end 300 displays the action of the target virtual image.

[0077] The above-mentioned scheme provided by the embodiment of the present application first obtains the human body action data of the target person through a human body detection algorithm, and migrates the human body action data to the general skeleton topology to obtain a first topological image. Then, based on the mapping relationship between the body data of the target virtual image and the general skeleton topology, a second topological image is obtained. Then, the action posture data of the second topological image is synchronously repaired using the action posture data of the first topological image. Finally, the target virtual image is driven to move according to the action posture data of the repaired second topological image. The above-mentioned process constructs the first topological image and the second topological image from the human body detection algorithm end and the target virtual image end respectively, and ensures that the target virtual object corresponding to the second topological image can perform the same action as the target person through the synchronous repair of the action posture data, and will not have poor driving effect. Compared with the scheme in the prior art that directly uses the human body action data obtained from the target person by the human body detection algorithm to drive the target virtual image, the technical scheme provided by the present application can overcome the driving effect difference when the same human body action data is used to drive target virtual images with different body features, can optimize the driving effect of the target virtual image, and improve the interactive experience. In addition, the above-mentioned process does not involve complex algorithms (such as neural network learning models), has less resource occupation and shorter time consumption, and can also be compatible with different human body detection algorithms and different target virtual images.

[0078] Further, before step S21, the virtual image driving method provided in the embodiments of the present application can further include a step of defining a general skeleton topology, which can be implemented in the following way.

[0079] First, a general skeleton topology is created.

[0080] In the embodiments of the present application, the skeleton topology is a human skeleton structure, and the general skeleton topology can include a head, limbs, fingers and an expression base, wherein the head and the limbs are used for limb driving, the fingers are used for gesture driving, and the expression base is used for expression driving.

[0081] Then, the number of joints in the general skeleton topology, the semantics of each joint and the hierarchical relationship between different joints are defined.

[0082] In the embodiments of the present application, the number of joints defined in the general skeleton topology can be greater than the number of joints in the human body detection algorithm and the number of joints in the virtual character. Taking the spine joint as an example, the number of spine joints in the human body detection algorithm and the virtual character can be 3, while the number of spine joints defined in the general skeleton topology can be 4. At the same time, the semantics of the same joint in the human body detection algorithm, the virtual character and the general skeleton topology are the same. In this way, it can be ensured that the joint data in the human body detection algorithm and the joint data of the virtual character will not be lost when they are mapped to the general skeleton topology. Exemplarily, the hierarchical relationship between different joints can be represented by a joint motion chain. Taking the arm as an example, the joint motion chain of the arm can be represented as shoulder-elbow-hand-finger.

[0083] Further, the virtual image driving method provided in the embodiments of the present application can further include creating a mapping relationship between the joints in the human body detection algorithm, the joints in the general skeleton topology and the joints in the target virtual image, and the specific process can be as follows.

[0084] First, a first mapping relationship between the joints in the human body detection algorithm and the joints in the general skeleton topology is created, obtaining a first configuration file;

[0085] Then, a second mapping relationship between the joints in the target virtual image and the joints in the general skeleton topology is created, obtaining a second configuration file;

[0086] Finally, based on the driving poses common to the human body detection algorithm and the target virtual image, driving pose information and collision body parameter information generated according to the characteristics of the target virtual image are created, obtaining a third configuration file.

[0087] In the above process, the first configuration file and the second configuration file can be obtained through user editing. For example, the first configuration file and the second configuration file can be edited manually, and then semantic alignment is performed according to the joint names to detect whether an error is made in establishing the mapping between different joints during editing. When an error is made in establishing the mapping, an error prompt is displayed to remind the user to adjust the joint with the error mapping, so as to avoid errors caused by manual editing. The third configuration file can automatically generate configuration information according to the driving pose common to the human body detection algorithm and the target virtual image. The configuration information includes driving pose information and collision body parameter information generated according to the characteristics of the target virtual image. The characteristics of the target virtual image include the driving pose of the target virtual image and the body shape of the target virtual image. Please refer to Figure 3 , the driving pose can be an A POSE driving pose or a T POSE driving pose. For example, the driving pose information can include pose data corresponding to the switching of the target virtual image between the A POSE driving pose and the T POSE driving pose. The collision body parameter information can include parameters for generating a collision body corresponding to the body shape of the target virtual image. In the embodiments of the present application, automatic analysis is supported for the uploaded target virtual image to obtain animation data of the target virtual image and the driving pose of the human body detection algorithm. The animation data includes joint local rotation data and joint local translation data. By superimposing the action pose data of the second topology image after repair on the animation data, the target virtual image can perform the same action as the target person.

[0088] As a possible implementation, step S21 in the embodiments of the present application can migrate the human action data into the general skeleton topology structure based on the first mapping relationship to obtain the first topology image. The human action data includes joint local rotation data and joint local translation data of the target person. After the human action data is migrated into the general skeleton topology structure, the first topology image consistent with the action of the target person can be obtained.

[0089] As a possible implementation, step S22 in the embodiments of the present application can migrate the body data into the general skeleton topology structure based on the second mapping relationship to obtain the second topology image. The body data includes joint local translation data of the virtual image.

[0090] Please refer to Figure 4 In the embodiments of the present application, the synchronization repair includes slide repair and collision repair. Step S23 can be implemented by the following steps.

[0091] In step S231, ground contact information of whether the foot of the first topological image contacts the ground is obtained based on the human action data, and the second topological image is repaired for sliding based on the human action data, the root joint translation data, the joint local translation data of the target virtual image, and the ground contact information.

[0092] The root joint translation data can refer to translation data of a root joint in a general skeletal topological structure, and the root joint corresponds to a hip joint in the general skeletal topological structure. In the embodiments of the present application, the root joint is different from other joints (such as finger joints), and the translation data and rotation data of the root joint are global data. The ground contact information includes double-foot ground contact information, single-foot ground contact information, or double-foot take-off information.

[0093] In step S232, the collision relationship of different joint collision bodies in the second topological image after the action posture data of the first topological image is migrated to the second topological image is detected, the obstacle and the collision repair object are determined in the different joint collision bodies based on the pre-set joint collision body priority, and the collision repair object is repaired for collision.

[0094] Further, in the embodiments of the present application, please refer to Figure 5 Step S231 can be implemented through the following sub-steps.

[0095] In sub-step S2311, the foot position and the length of the lower body of the first topological image in each frame of image are obtained based on the human action data by using forward kinematics.

[0096] In this sub-step, the foot position and the length of the lower body of the second topological image in each frame of image can be calculated based on the joint local rotation data and the joint local translation data by using forward kinematics.

[0097] In sub-step S2312, a ratio between the distance between the same foot position in the current frame and a preset frame and the length of the lower body is calculated, and a height difference value between the heights of the two feet in the current frame is calculated.

[0098] For example, the ratio in this sub-step can be a ratio between the distance between the same foot position (such as the left ankle and the left toe) in the current frame and a preset frame (such as 5 frames ago) and the length of the lower body.

[0099] In sub-step S2313, the ratio and the height difference value are compared with a first threshold value and a second threshold value respectively, and the ground contact information of whether the foot of the first topological image contacts the ground is obtained according to the comparison result.

[0100] Exemplarily, if the ratio is less than a first threshold value and the height difference value is less than a second threshold value, it is determined that the two feet of the first topological image are in contact with the ground; if the ratio is less than the first threshold value and the height difference value is not less than the second threshold value, it is determined that the foot of the first topological image with lower height is in contact with the ground; if the ratio is not less than the first threshold value, it is determined that the two feet of the first topological image are off the ground. The first threshold value can be 0.05, and the second threshold value can be in the range of 5 cm to 10 cm.

[0101] In substep S2314, the root joint translation data is scaled based on a ratio between a length of the lower body of the first topological image and a length of the lower body of the second topological image.

[0102] Through the substep, the global motion of the first topological image and the second topological image can be matched.

[0103] In substep S2315, the joint motion chain of the lower body of the second topological image is corrected based on the ground contact information, so that the second topological image and the first topological image touch or leave the ground at the same time.

[0104] In the substep, when a foot is detected to touch the ground, the position and direction of the ankle and the foot of the foot at this time are saved as an optimization target Goal, and the position of the foot of each subsequent image needs to be moved to the optimization target Goal by using an inverse kinematics algorithm (such as a PBIK algorithm) until the foot is detected to be off the ground. When the foot is detected to touch the ground again, the optimization target Goal is updated again. The joint motion chain of the lower body mainly involved in the slide repair, such as the hip (left up leg)-knee (left leg)-ankle (left foot)-toe (left toe base), is corrected in the repair. The local rotation data of each joint on the joint motion chain is corrected to ensure that the position of the foot on the target virtual image is fixed and does not slide, and to minimize the damage to the original animation and ensure the smoothness and naturalness of the motion.

[0105] Further, in the embodiment of the present application, please refer to Figure 6 Step S232 can be implemented through the following substeps.

[0106] In substep S2321, a joint collision body is generated on the joint of the second topological image.

[0107] In the substep, the collision body of each joint can be generated on the joint of the second topological image based on the image character skinning, such as Figure 3The T POSE collider in the T POSE collider. The collider can be a capsule or a sphere. The collider is bound to the corresponding joint. The size of the collider can be controlled by adjusting the threshold of the skin weight. When the body type of the target virtual image is relatively fat, the size of the collider can be increased. When the body type of the target virtual image is relatively thin, the size of the collider can be decreased.

[0108] In substep S2322, the collision depth and the collision direction between different joint colliders of the second topological image are calculated after the motion pose data of the first topological image is migrated to the second topological image.

[0109] Since the colliders are bound to the corresponding joints, the movement of the joints during driving also causes the movement of the colliders (for example, the hand colliders also move when the hand is driven). The collision conditions of the joint colliders can be calculated during driving. In the embodiments of the present application, the colliders can include two types of capsules and spheres. Assuming that the parameters of the capsule are endpoints P1 and P2 and a radius R, the parameters of the sphere are a sphere center v and a radius r, the distance between points is represented as | |, the shortest distance between a point and a line segment is represented as || | |, and the shortest distance between line segments is represented as || | | |. When two spheres collide, | v1-v2 |< = r1+r2, and the collision depth is r1+r2-| v1-v2 |. When a sphere collides with a capsule, || v-(P1,P2) ||< = r+R, and the collision depth is r+R-|| v-(P1,P2) ||. When two capsules collide, || | | (P1,P2)-(P3,P4) | | |< = R1+R2, and the collision depth is R1+R2-|| | | (P1,P2)-(P3,P4) | | |.

[0110] In substep S2323, based on the pre-set joint collider priority, obstacles and collision repair objects are determined in different joint colliders. The position of the collision repair object after collision repair is calculated based on the collision depth and the collision direction between different joint colliders.

[0111] In substep S2324, based on the position of the collision repair object after collision repair, the local rotation data of the joints in the joint kinematic chain of the collision repair object is adjusted to obtain the motion pose data after collision repair.

[0112] Taking the collision between the limbs and other parts of the body as an example, the joint collision priority of the other parts of the body (for example, the head and the human body) can be set to be relatively high, and the joint collision priority of the limbs can be set to be relatively low. In the embodiments of the present application, the other parts of the body can be used as obstacles, and the limbs can be used as collision repair objects. Please refer to Figure 7When the hand collides with the head, the collision depth and direction are d1 and n1 respectively, and the hand collides with the chest at the same time, the collision depth and direction are d2 and n2 respectively, then the final collision result is d=d1*n1+d2*n2, and the position of the hand at this time is p, then the repaired position is p+d, and the repaired position is taken as the optimization target Goal of the penetration repair, and then the local rotation data of the motion joint chain shoulder-elbow-hand is modified based on the inverse motion algorithm optimization, so as to avoid the penetration of the hand and other parts. When the limbs collide with each other, such as when the hands cross, please refer to Figure 8 , the collision direction is selected from any one of the two directions, and the collision results of the hands are dm and-dm respectively, the positions of the hands are p1 and p2, and the collision results of the hands with other parts of the body are d1 and d2 respectively, then the repaired positions of the hands are p1+d1+dm / 2 and p2+d2-dm / 2 respectively, that is, the positions of the hands are modified, and the penetration optimization of the hands is performed based on the same process as Figure 7 .

[0113] The driving method of the virtual image provided in the embodiment of the application is used in the case that the driving posture of the human body detection algorithm is unchanged, the first configuration file is related to the human body detection algorithm, and the second configuration file and the third configuration file are related to the target virtual image, so that the human body detection algorithm and the target virtual image are decoupled on both sides, and therefore the second configuration file and the third configuration file are kept unchanged, the human body detection algorithm is replaced by replacing the first configuration file, or the first configuration file is kept unchanged to support the driving optimization of other target virtual images by replacing the second configuration file and the third configuration file, so that the scheme provided in the application is a general virtual image driving method.

[0114] Further, please refer to Figure 9 , Figure 9 , the virtual image driving device 400 provided in the embodiment of the application, the virtual image driving device 400 can be divided into functional modules according to the method embodiment executed by the server, that is, the following various functional modules corresponding to the virtual image driving device 400 can be used to execute the various method embodiments. Among them, the virtual image driving device 400 can include a first determination module 410, a second determination module 420, a repair module 430 and a driving module 440, and the functions of each functional module of the virtual image driving device 400 will be described in detail below.

[0115] The first determining module 410 is configured to obtain human body action data of the target character based on a human body detection algorithm, and migrate the human body action data to a general skeleton topology structure having a joint mapping relationship with the human body detection algorithm to obtain a first topological image.

[0116] The human body detection algorithm can include a light capture human body detection algorithm, an inertial capture human body detection algorithm, a visual human body detection algorithm, etc. In the scenario provided in the embodiments of the present application, the human body detection algorithm can be a human body detection algorithm based on an RGB video image. For example, after the server 100 obtains the live video frame image of the anchor end 200, the live video frame image is subjected to human body detection to obtain the human body action data of the target character. Each joint of the human body is defined in the human body detection algorithm. The mapping between the joints defined in the human body detection algorithm and the joints in the general skeleton topology structure can be established in advance to establish the corresponding relationship between the joints having the same joint semantics. More joints can be defined in the general skeleton topology structure than the joints in the human body detection algorithm to ensure that the human body action data can be completely migrated to the joints in the general skeleton topology structure corresponding to the joints in the human body detection algorithm in the process of migrating the human body action data to the skeleton topology structure, avoiding the loss of the human body action data in the migration process.

[0117] In the embodiments, the first determining module 410 can be configured to perform the step S21 described above. The detailed implementation of the first determining module 410 can be referred to the detailed description of the step S21.

[0118] The second determining module 420 is configured to obtain body data of the target virtual image, and obtain a second topological image based on the mapping relationship between the body data and the general skeleton topology structure.

[0119] The target virtual image can be a humanoid virtual image having a body structure similar to that of a human body. The driving pose in the human body detection algorithm is semantically aligned with the driving pose in the target virtual image. The semantic alignment between the driving pose in the human body detection algorithm and the driving pose in the target virtual image is required at each detection to ensure that the human body action data obtained by the human body detection algorithm can enable the target virtual image to complete the same action as the target character from the static driving pose. Based on the mapping relationship between the joints in the body data of the target virtual image and the joints in the general skeleton topology structure, the second topological image can be obtained. More joints can be defined in the general skeleton topology structure than the joints in the target virtual image to ensure that the second topological image has a body structure corresponding to the target virtual object.

[0120] In the embodiments, the second determining module 420 can be configured to perform the step S22 described above. The detailed implementation of the second determining module 420 can be referred to the detailed description of the step S22.

[0121] The repair module 430 is configured to synchronize and repair the motion posture data of the second topological image based on the motion posture data of the first topological image.

[0122] In the embodiment of the present application, after the human motion data is migrated to the general skeleton topological structure, the first topological image of the motion is formed, and the motion posture data of the first topological image can be used to synchronize and repair the second topological image, so that the second topological image performs the same motion as the first topological image. Since the first topological image and the second topological image are obtained based on different mapping relationships, the above-mentioned synchronization and repair can make the second topological image perform the same motion as the first topological image, avoiding the technical problems of sliding and / or interleaving of the second topological image when directly migrating the motion posture data of the first topological image to the second topological image.

[0123] The repair module 430 in the embodiment can be used to perform the above-mentioned step S23, and the detailed implementation of the repair module 430 can refer to the detailed description of step S23.

[0124] The driving module 440 is configured to drive the target virtual image to move according to the motion posture data of the repaired second topological image.

[0125] The driving module 440 in the embodiment can be used to perform the above-mentioned step S24, and the detailed implementation of the driving module 440 can refer to the detailed description of step S24.

[0126] It should be noted that the division of each module in the above device or system is only a logical division of functions, and all or part of the modules can be integrated into one physical entity, or can be physically separated. These modules can all be implemented in the form of software (such as open source software) that can be called by a processor; all can be implemented in the form of hardware; some modules can be implemented in the form of software called by a processor, and some modules can be implemented in the form of hardware. As an example, the repair module 430 can be implemented by a separate processor, and the functions of the repair module 430 can be stored in the memory of the above device or system in the form of program code, and can be called and executed by a processor of the above device or system. The implementation of other modules is similar, and will not be described here. In addition, these modules can be integrated together or implemented independently. The processor described herein can be an integrated circuit with signal processing capability, and each step or module in the above technical solution can be completed by integrated logic circuit or software execution in the processor.

[0127] Please refer to Figure 10 , Figure 10A hardware structure diagram of the server 100 for implementing the driving method of the virtual image is shown. As shown in the figure, the server 100 can include a processor 110, a computer readable storage medium 120, a bus 130, and a communication unit 140. Figure 10

[0128] In the implementation process, the processor 110 executes the computer execution instructions (for example, the various modules in the driving device 400 of the virtual image shown in the figure) stored in the computer readable storage medium 120, so that the processor 110 can execute the video coding parameter combination determination method of the above method embodiment, wherein the processor 110, the computer readable storage medium 120, and the communication unit 140 can be connected through the bus 130. Figure 9

[0129] The specific implementation process of the processor 110 can refer to the various method embodiments executed by the server 100 described above, which has similar implementation principles and technical effects, and will not be described here in detail.

[0130] The computer readable storage medium 120 can be, but is not limited to, a random access memory (RAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc. The memory 111 is used to store programs or data.

[0131] The bus 130 can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, the bus in the drawings of the present application does not limit only one bus or one type of bus.

[0132] In the interactive scene provided by the embodiments of the present application, the communication unit 140 can be used for communication with the anchor end 200 and the audience end 300, so as to realize the data interaction between the server 100 and the anchor end 200 and the audience end 300.

[0133] In addition, the embodiments of the present application also provide a readable storage medium, and the readable storage medium stores computer execution instructions. When the processor executes the computer execution instructions, the driving method of the virtual image is realized.

[0134] ​​In summary, the virtual image driving method, device and server provided by the embodiments of the present application first obtain the human body action data of the target person through a human body detection algorithm, and migrate the human body action data to a general skeleton topology to obtain a first topology image; then, based on the mapping relationship between the body data of the target virtual image and the general skeleton topology, a second topology image is obtained; then, the action posture data of the first topology image is used to synchronize and repair the action posture data of the second topology image; finally, the target virtual image is driven to move according to the repaired action posture data of the second topology image. The above process constructs the first topology image and the second topology image from the human body detection algorithm side and the target virtual image side respectively, and ensures that the target virtual object corresponding to the second topology image can perform the same action as the target person through the action posture data synchronization repair method, and the driving effect is not poor. Compared with the prior art scheme of directly using the human body action data obtained from the target person by the human body detection algorithm to drive the target virtual image, the technical scheme provided by the present application can overcome the driving effect difference when the same human body action data is used to drive target virtual images with different body characteristics, can optimize the driving effect of the target virtual image, and improve the interactive experience. In addition, the above process does not involve complex algorithms (such as neural network learning models), has less resource occupation and short time consumption, and can also be compatible with different human body detection algorithms and different target virtual images.

[0135] The above-described embodiments are only some of the embodiments of the present application, not all the embodiments. The components of the embodiments of the present application described and shown in the drawings can be arranged and designed in various different configurations. Therefore, the detailed description of the embodiments of the present application provided in the drawings is not intended to limit the protection scope of the present application, but only represents selected embodiments of the present application. Based on this, the protection scope of the present application should be subject to the protection scope of the claims. In addition, based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor shall fall within the scope of protection of the present application.

Claims

1. A method for driving a virtual image, characterized in that: The method comprises: Acquiring human motion data of a target person based on a human body detection algorithm, migrating the human motion data to a universal skeletal topological structure having a joint mapping relationship with the human body detection algorithm, and obtaining a first topological image; Obtaining body data of a target avatar, and obtaining a second topological image based on a mapping relationship between the body data and the universal skeletal topological structure, wherein a driving posture in the human body detection algorithm is semantically aligned with a driving posture in the target avatar; Synchronously repairing the motion posture data of the second topological image based on the motion posture data of the first topological image, wherein the synchronous repairing includes sliding repair and collision repair; The target virtual image is driven to move according to the restored motion posture data of the second topological image.

2. The method for driving a virtual image according to claim 1, wherein: Before the step of acquiring human motion data of a target person based on a human body detection algorithm and migrating the human motion data to a universal skeletal topological structure having a joint mapping relationship with the human body detection algorithm to obtain a first topological image, the method further includes: Creating the general skeleton topology structure, wherein the general skeleton topology structure includes a head, limbs, fingers, and an expression base; The number of joints in the general skeleton topology, the semantics of each joint, and the hierarchical relationship between different joints are defined.

3. The method for driving a virtual image according to claim 2, wherein: Before the step of acquiring human motion data of a target person based on a human body detection algorithm and migrating the human motion data to a universal skeletal topological structure having a joint mapping relationship with the human body detection algorithm to obtain a first topological image, the method further includes: Creating a first mapping relationship between joints in the human body detection algorithm and joints in the universal skeletal topology structure to obtain a first configuration file; Creating a second mapping relationship between joints in the target avatar and joints in the universal skeletal topology structure to obtain a second configuration file; Based on the human body detection algorithm and the common driving posture of the target virtual image, driving posture information and collision body parameter information generated according to the characteristics of the target virtual image are created to obtain a third configuration file.

4. The method for driving a virtual image according to claim 3, wherein: The step of obtaining human motion data of a target person based on a human body detection algorithm, migrating the human motion data to a universal skeletal topological structure having a joint mapping relationship with the human body detection algorithm, and obtaining a first topological image includes: Based on the first mapping relationship, the human motion data is migrated to the universal skeleton topology structure to obtain the first topological image, wherein the human motion data includes local rotation data and local translation data of the joints of the target person; The step of obtaining the body data of the target avatar and obtaining the second topological image based on the mapping relationship between the body data and the universal skeletal topological structure includes: Based on the second mapping relationship, the body data is migrated to the universal skeletal topology structure to obtain the second topological image, where the body data includes local translation data of the joints of the virtual image.

5. The method for driving a virtual image according to any one of claims 1 to 4, wherein: The step of synchronously repairing the motion posture data of the second topological image based on the motion posture data of the first topological image comprises: obtaining ground contact information of whether a foot of the first topological image is in contact with the ground based on the human motion data, and performing sliding repair on the second topological image based on the human motion data, the root joint translation data, the joint local translation data of the target avatar, and the ground contact information; Detect collision relationships between different joint collision bodies in the second topological image after the motion posture data of the first topological image is migrated to the second topological image, determine obstacles and collision repair objects in the different joint collision bodies based on pre-set joint collision body priorities, and perform collision repair on the collision repair objects.

6. The method for driving a virtual image according to claim 5, wherein: The step of obtaining ground contact information of whether the foot of the first topological image is in contact with the ground based on the human motion data, and performing sliding repair on the second topological image according to the human motion data, root joint translation data, joint local translation data of the target virtual image, and the ground contact information includes: Based on the human motion data, forward kinematics is used to obtain the foot position and the length of the lower body of the first topological image in each frame of the image; Calculating a ratio between a distance between a position of the same foot in a current frame and a position of the same foot in a previous preset frame and a length of the lower body, and calculating a height difference between the heights of the two feet in the current frame; Comparing the ratio and the height difference with a first threshold and a second threshold respectively, and obtaining ground contact information of whether the foot of the first topological image is in contact with the ground according to the comparison results; scaling the root joint translation data based on a ratio between a length of a lower body of the first topological figure and a length of a lower body of the second topological figure; The joint kinematic chain of the lower body of the second topological figure is modified based on the ground contact information, so that the second topological figure touches the ground or leaves the ground at the same time as the first topological figure.

7. The method for driving a virtual image according to claim 6, wherein: The step of comparing the ratio and the height difference value with a first threshold and a second threshold respectively, and obtaining ground contact information of whether the foot of the first topological image is in contact with the ground according to the comparison results, includes: If the ratio is smaller than the first threshold value, and the height difference is smaller than the second threshold value, it is determined that both feet of the first topological figure are in contact with the ground; If the ratio is less than the first threshold value and the height difference value is not less than the second threshold value, it is determined that the lower foot of the first topological figure is in contact with the ground; If the ratio is not less than the first threshold, it is determined that both feet of the first topological image are off the ground.

8. The method for driving a virtual image according to claim 5, wherein: The step of detecting collision relationships between different joint collision bodies in the second topological image after the motion posture data of the first topological image is transferred to the second topological image, determining obstacles and collision repair objects in the different joint collision bodies based on a preset joint collision body priority, and performing collision repair on the collision repair objects includes: generating a joint collision body on the joint of the second topological image; calculating the collision depth and collision direction between different joint collision bodies of the second topological image after the motion posture data of the first topological image is transferred to the second topological image; Based on the preset joint collision body priorities, obstacles and collision repair objects are determined in the different joint collision bodies, and the positions of the collision repair objects after collision repair are calculated based on the collision depths and collision directions between the different joint collision bodies; Based on the position of the collision repair object after collision repair, the local rotation data of the joint in the joint motion chain where the collision repair object is located is adjusted to obtain the motion posture data after collision repair.

9. A driving device for a virtual image, characterized in that: The device comprises: A first determination module is configured to obtain human motion data of a target person based on a human detection algorithm, and to transfer the human motion data to a general skeleton topology structure having a joint mapping relationship with the human detection algorithm to obtain a first topological image; a second determination module configured to obtain body data of a target avatar and obtain a second topological image based on a mapping relationship between the body data and the universal skeletal topological structure, wherein a driving posture in the human body detection algorithm is semantically aligned with a driving posture in the target avatar; a repair module, configured to synchronously repair the motion posture data of the second topological image based on the motion posture data of the first topological image, wherein the synchronous repair includes sliding repair and collision repair; A driving module is used to drive the target virtual image to move according to the restored motion posture data of the second topological image.

10. A server, characterized in that: The server includes a processor, a communication unit and a computer-readable storage medium. The processor, the communication unit and the computer-readable storage medium are connected via a bus system. The communication unit is used to connect electronic devices to realize data interaction between the server and the electronic devices. The computer-readable storage medium is used to store programs, instructions or codes. The processor is used to execute the programs, instructions or codes in the computer-readable storage medium to realize the driving method of the virtual image described in any one of claims 1-8.

Images