Methods, devices, storage media and electronic devices for driving virtual humans

Abstract

This application proposes a method, apparatus, storage medium, and electronic device for driving a virtual human. The method for driving the virtual human includes: determining a first rotation matrix of human joints based on a first acquired human body image; adding a constraint matrix to the first rotation matrix to determine a second rotation matrix of the human joints, the constraint matrix corresponding to the physical constraints of the human joints; smoothing the second rotation matrix to determine a target rotation matrix of the human joints; and driving the virtual human to move according to the target rotation matrix.

Description

Technical Field

[0001] This application relates to the field of virtual human technology, and more specifically, to a driving method, apparatus, storage medium, and electronic device for a virtual human. Background Technology

[0002] Using real human movements to drive virtual humans is the main method currently used for virtual human driving. How to make the driven virtual human movements more natural and fluid is one of the main problems to be solved in virtual human driving. Summary of the Invention

[0003] This application aims to address at least one of the technical problems existing in the related art.

[0004] Therefore, the first aspect of this application is to propose a method for driving a virtual human.

[0005] The second aspect of this application is to propose a driving device for a virtual human.

[0006] The third aspect of this application is to propose another driving device for virtual humans.

[0007] The fourth aspect of this application is to propose a readable storage medium.

[0008] The fifth aspect of this application is to propose an electronic device.

[0009] The sixth aspect of this application is to provide a computer program product.

[0010] In view of this, according to one aspect of this application, a driving method for a virtual human is proposed, the driving method comprising: determining a first rotation matrix of human joints based on a first acquired human body image; adding a constraint matrix to the first rotation matrix to determine a second rotation matrix of human joints, the constraint matrix corresponding to the physical constraints of human joints; smoothing the second rotation matrix to determine a target rotation matrix of human joints; and driving the virtual human to move according to the target rotation matrix.

[0011] It should be noted that the execution subject of the virtual human driving method proposed in this application can be the virtual human driving device. In order to more clearly explain the virtual human driving method proposed in this application, the following technical solution uses the virtual human driving device as the execution subject of the virtual human driving method for illustrative purposes.

[0012] In this technical solution, the first human image refers to a frame of image captured in a video; the constraint matrix refers to the physical constraint conditions for human joints, which restrict the movement of human joints and prevent their movements from exceeding certain limits; the rotation matrix refers to a mathematical expression of human joint posture; the first rotation matrix refers to the rotation matrix of human joints detected by the human posture detection network; and the target rotation matrix refers to the second rotation matrix after smoothing.

[0013] Specifically, the driving device first determines the rotation matrix of each joint of the virtual human body, i.e., the aforementioned first rotation matrix, based on the first human body image captured from the video. Specifically, the driving device can determine the first rotation matrix by inputting the aforementioned first human body image into a human pose detection network, but is not limited to this method.

[0014] Furthermore, the detection device adds a constraint matrix to the first rotation matrix of the human joint to determine the physically corrected rotation matrix, namely the aforementioned second rotation matrix. Specifically, the first rotation matrix clearly defines the movement of each human joint along the three spatial axes X, Y, and Z. By adding the constraint matrix, the movement of the human joint along which of the X, Y, and Z axes is restricted, as well as the limits of movement on each axis. Therefore, the drive device can correct and restrict the first rotation matrix by adding a constraint matrix to obtain the corrected second rotation matrix.

[0015] Furthermore, the driving device smooths the second rotation matrix obtained after adding the constraint matrix to determine the target rotation matrix used to drive the aforementioned virtual human limb movements.

[0016] Specifically, if the virtual human limbs are driven directly based on the second rotation matrix, the virtual human limbs may vibrate due to unreasonable relationships between some parameters in the second rotation matrix. For example, the relationship between the acceleration of the human joint movement and the external force on the human joint may be unreasonable. Therefore, the driving device needs to smooth the second rotation matrix to obtain the smoothed target rotation matrix.

[0017] Furthermore, the driving device drives the virtual human according to the target rotation matrix, so that the limbs of the virtual human exhibit movements that are adapted to the first human body image.

[0018] In this technical solution, the driving device can determine the first rotation matrix of each joint of the virtual human body based on the first human body image captured from the video, add a constraint matrix to the first rotation matrix to determine the second rotation matrix, smooth the parameters in the second rotation matrix to determine the target rotation matrix, and drive the virtual human body to move according to the target rotation matrix. In the technical solution of this application, the driving device adds a constraint matrix and performs smoothing on the target matrix used to drive the virtual human body to move. This ensures that the movements made by the virtual human do not violate physical laws and there is no jitter during the movement. This solves the problem in related technologies where directly using the detection results of the human posture detection network to drive the virtual human body to move may violate physical laws and cause jitter, thus ensuring the driving effect of the virtual human body.

[0019] Furthermore, the virtual human driving method proposed according to the above-described technical solution of the present invention may also have the following additional technical features:

[0020] In the above technical solution, the step of adding a constraint matrix to the first rotation matrix to determine the second rotation matrix of the human joint specifically includes: using the first rotation matrix of the sub-joints of the human joint as the constraint matrix; compensating the constraint matrix to the first rotation matrix of the human joint to determine the second rotation matrix.

[0021] In this technical solution, the driving device can physically constrain the virtual human's limb movements by using the first rotation matrix corresponding to the sub-joint as a constraint matrix and compensating the first rotation matrix of the human joint with this constraint matrix, thereby determining the second rotation matrix of each human joint in the virtual human. This restricts the rotation of the human joints along the XYZ axes and the positional relationship of the human joints relative to the human body, ensuring that no physical violations occur when driving the virtual human's limb movements.

[0022] In the above technical solution, the first rotation matrix of different sub-joints is different.

[0023] In this technical solution, the sub-joints of different human joints are different. For example, the sub-joint of the thigh is the calf, and the sub-joint of the elbow is the arm. Understandably, the first rotation matrix of different sub-joints is also different, so the constraint matrices added to different human joints are also different. In the above technical solution, the step of smoothing the second rotation matrix to determine the target rotation matrix of the human joint specifically includes: determining the first acceleration of the human joint movement and the first external force acting on the human joint based on the second rotation matrix; determining the first position of the human joint based on the first acceleration and the first external force; and determining the target rotation matrix based on the first position.

[0024] In this technical solution, the driving device can calculate the first acceleration and the first external force of the human joint based on the second rotation matrix. By solving for the optimal relationship between the first acceleration and the first external force, the first position of the smoothed human joint can be determined, and the target rotation matrix can be derived from the first position. In this way, the target rotation matrix used to drive the virtual human limb movements is smoothed, ensuring that there is no jitter during the process of driving the virtual human limb movements.

[0025] In the above technical solution, the steps of determining the first acceleration of the human joint movement and the first external force on the human joint based on the second rotation matrix specifically include: determining the second position of the human joint based on the second rotation matrix; acquiring a second human image and a third human image, as well as the acquisition time information of the second human image and the third human image; determining the first historical position of the human joint in the second human image and the second historical position of the human joint in the third human image; determining the first velocity and the first acceleration of the human joint movement based on the first historical position, the second historical position, the second position and the acquisition time information; and determining the first external force based on the second position, the first velocity and the first acceleration; wherein, the second human image and the third human image are images acquired before the first human image.

[0026] In this technical solution, the driving device solves for the first external force and the first acceleration of each joint of the virtual human body based on the second rotation matrix obtained after adding the constraint matrix. This ensures the accuracy of the determined first external force and second acceleration, and thus ensures the accuracy of the target rotation matrix determined in subsequent steps.

[0027] In the above technical solution, the steps of determining the first velocity and the first acceleration of human joint movement based on the first historical position, the second historical position, and the second position specifically include: determining the first velocity based on the second historical position, the second position, and the acquisition time information; determining the second velocity based on the first historical position, the second historical position, and the acquisition time information; and determining the first acceleration based on the difference between the first velocity and the second velocity.

[0028] In this technical solution, the driving device calculates the velocity and acceleration of each joint in the first human body image based on the positions of the joints in the human body image acquired before the first human body image. In other words, the driving device calculates the velocity and acceleration within the most recent timeframe. This ensures the accuracy of determining the first acceleration and the first velocity.

[0029] In the above technical solution, the step of determining the first external force based on the second position, the first velocity, and the first acceleration specifically includes: determining the first external force based on the second position, the first velocity, and the first acceleration using the rigid body motion equation and the friction cone constraint condition.

[0030] In this technical solution, the driving device can solve for the first external force that satisfies the friction cone constraint condition by using the rigid body motion equation, based on the first acceleration, the first velocity, and the second position, thus ensuring the accuracy of determining the first external force.

[0031] In the above technical solution, the step of determining the first position of the human joint based on the first acceleration and the first external force specifically includes: solving for the first position based on the first acceleration and the first external force using an iterative algorithm.

[0032] In this technical solution, the first position is the optimal position obtained by an iterative algorithm. This ensures the accuracy of the determined first position and, consequently, ensures that when the target rotation matrix derived from the first position drives the virtual human limb movements, there will be no shaking or violation of physical laws.

[0033] According to a second aspect of the present invention, a driving device for a virtual human is provided, the driving device comprising: a first processing module for determining a first rotation matrix of a human joint based on a first acquired human image; a second processing module for adding a constraint matrix to the first rotation matrix to determine a second rotation matrix of the human joint, the constraint matrix corresponding to the physical constraints of the human joint; a third processing module for smoothing the second rotation matrix to determine a target rotation matrix of the human joint; the third processing module is further configured to drive the virtual human to move according to the target rotation matrix.

[0034] In this technical solution, the first human body image refers to a frame of image captured in a video; the constraint matrix refers to the physical constraint conditions for human body joints, which restrict the movement of human body joints and prevent their movements from exceeding certain limits; the rotation matrix refers to a mathematical expression of human body joint posture; the first rotation matrix refers to the rotation matrix of human body joints detected by the human body posture detection network; and the target rotation matrix refers to the second rotation matrix after smoothing.

[0035] Specifically, the first processing module first determines the rotation matrix of each joint of the virtual human body, i.e., the aforementioned first rotation matrix, based on the first human body image captured from the video. Specifically, the first processing module can determine the first rotation matrix by inputting the aforementioned first human body image into a human pose detection network, but is not limited to this method.

[0036] Furthermore, the second processing module adds a constraint matrix to the first rotation matrix of the human joint to determine the physically corrected rotation matrix, namely the second rotation matrix mentioned above.

[0037] Specifically, the first rotation matrix clearly defines the movement of each human joint across the XYZ axes. By adding constraint matrices, the movement of these joints on which axes of the XYZ axes can be restricted, as well as the limits of movement on each axis. Therefore, the second processing module can modify and restrict the first rotation matrix by adding constraint matrices to the first rotation matrix of the human joints, thus obtaining a modified second rotation matrix.

[0038] Furthermore, the third processing module smooths the second rotation matrix obtained after adding the constraint matrix to determine the target rotation matrix used to drive the aforementioned virtual human limb movements.

[0039] Specifically, if the virtual human limb movements are driven directly based on the second rotation matrix, the relationship between some parameters in the second rotation matrix may be unreasonable. For example, the relationship between the acceleration of the human joint movement and the external force on the human joint may be unreasonable, which may cause shaking when driving the virtual human limb movements. Therefore, the third processing module needs to smooth the second rotation matrix to obtain the smoothed target rotation matrix.

[0040] Furthermore, the third processing module drives the virtual human according to the target rotation matrix, so that the limbs of the virtual human exhibit movements that are adapted to the first human body image.

[0041] In this technical solution, the first processing module determines the first rotation matrix of each joint of the virtual human body based on the first human body image captured from the video. The second processing module adds a constraint matrix to the first rotation matrix to determine the second rotation matrix. The third processing module smooths the parameters in the second rotation matrix to determine the target rotation matrix and drives the virtual human body's limb movements based on the target rotation matrix. In this application's technical solution, the third processing module adds a constraint matrix and performs smoothing on the target rotation matrix used to drive the virtual human body's limb movements. This ensures that the virtual human's movements do not violate physical laws and do not jitter during the movement. This solves the problem in related technologies where directly using the detection results of a human pose detection network to drive the virtual human's movements may violate physical laws and cause jitter, thus ensuring the driving effect of the virtual human.

[0042] According to the third aspect of this application, another virtual human driving device is proposed, comprising: a memory storing a program or instructions; and a processor executing the program or instructions stored in the memory to implement the steps of the virtual human driving method proposed in the above-described technical solutions of this application, thus possessing all the beneficial technical effects of the virtual human driving method proposed in the above-described technical solutions of this application, which will not be elaborated further here.

[0043] According to the fourth aspect of this application, a readable storage medium is proposed, on which a program or instructions are stored. When the program or instructions are executed by a processor, they implement the steps of the virtual human driving method proposed in the above-mentioned technical solution of this application, and thus have all the beneficial technical effects of the virtual human driving method proposed in the above-mentioned technical solution of this application, which will not be elaborated further here.

[0044] According to the fifth aspect of this application, an electronic device is proposed, including a driving device for a virtual human as proposed in the above-described technical solution of this invention, and / or a readable storage medium as proposed in the above-described technical solution of this invention. Therefore, the electronic device has all the beneficial effects of the driving device for a virtual human as proposed in the above-described technical solution of this invention and / or the readable storage medium as proposed in the above-described technical solution of this invention, which will not be repeated here.

[0045] According to the sixth aspect of this application, a computer program product is proposed, comprising a computer program that, when executed by a processor, implements the steps of the virtual human driving method proposed in the above-described technical solution of this application, and thus possesses all the beneficial technical effects of the virtual human driving method proposed in the above-described technical solution of this application, which will not be elaborated further here.

[0046] Additional aspects and advantages of this application will become apparent in the following description or may be learned by practice of this application. Attached Figure Description

[0047] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0048] Figure 1 One of the flowcharts illustrating the driving method for a virtual human according to an embodiment of this application is shown;

[0049] Figure 2 A second schematic flowchart of the virtual human driving method according to an embodiment of this application is shown;

[0050] Figure 3 The third schematic flowchart illustrates the virtual human driving method according to an embodiment of this application;

[0051] Figure 4 One of the schematic block diagrams of the driving device for the virtual human according to an embodiment of this application is shown;

[0052] Figure 5 A second schematic block diagram of the driving device for the virtual human according to an embodiment of this application is shown;

[0053] Figure 6 A flowchart illustrating the overall process of driving a virtual human according to an embodiment of this application is shown. Detailed Implementation

[0054] To better understand the above-mentioned objectives, features, and advantages of this application, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0055] Many specific details are set forth in the following description in order to provide a full understanding of this application. However, this application may also be implemented in other ways different from those described herein. Therefore, the scope of protection of this application is not limited to the specific embodiments disclosed below.

[0056] The following is combined Figures 1 to 6 The present application provides a detailed description of a virtual human driving method, apparatus, storage medium, and electronic device through specific embodiments and application scenarios.

[0057] Example 1:

[0058] Figure 1 A flowchart illustrating a virtual human driving method according to an embodiment of this application is shown. The driving method includes:

[0059] S102, Determine the first rotation matrix of the human joints based on the acquired first human image;

[0060] S104, Add a constraint matrix to the first rotation matrix to determine the second rotation matrix of the human joint. The constraint matrix corresponds to the physical constraint conditions of the human joint.

[0061] S106, Smooth the second rotation matrix to determine the target rotation matrix of the human joint;

[0062] S108, drives the virtual human's movements according to the target rotation matrix.

[0063] In this embodiment, the first human body image refers to a frame of image captured in a video, the constraint matrix refers to the physical constraint conditions for human body joints, and the physical constraint conditions refer to the conditions that restrict the movement of human body joints and prevent their movements from exceeding certain limits; the rotation matrix refers to a mathematical expression of human body joint posture, the first rotation matrix refers to the rotation matrix of human body joints detected by the human body posture detection network, and the target rotation matrix refers to the second rotation matrix after smoothing.

[0064] Specifically, the driving device first determines the rotation matrix of each joint of the virtual human body, i.e., the aforementioned first rotation matrix, based on the first human body image captured from the video. Specifically, the driving device can determine the first rotation matrix by inputting the aforementioned first human body image into a human posture detection network, but it is not limited to this; other human posture detection methods are also applicable to this application. Further, the detection device adds constraint matrices to the first rotation matrix of the human body joints to determine the physically corrected rotation matrix, i.e., the aforementioned second rotation matrix. Specifically, the first rotation matrix clearly defines the movement of each human body joint along the three spatial axes X, Y, and Z. By adding constraint matrices, it is possible to limit the movement of the aforementioned human body joints on which of the X, Y, and Z axes, and the limits of movement on each axis. Therefore, the driving device can correct and limit the first rotation matrix by adding constraint matrices to the first rotation matrix of the human body joints to obtain the corrected second rotation matrix.

[0065] For example, taking the lower leg as an example of a human joint, decomposing its first rotation matrix along the XYZ axes reveals that the lower leg only rotates about the Y-axis. However, directly deleting the rotation of the lower leg along the XZ axes might cause a shift in the lower leg's position. Therefore, physical constraints can be imposed on the rotation of the lower leg along the XZ axes, and the relationship between the lower leg and the human body's position can be restricted; that is, a constraint matrix can be added to the first rotation matrix of the lower leg. In this way, the second rotation matrix of the lower leg can be obtained.

[0066] Furthermore, the driving device smooths the second rotation matrix obtained after adding the constraint matrix to determine the target rotation matrix used to drive the aforementioned virtual human limb movements.

[0067] Specifically, if the virtual human limbs are driven directly based on the second rotation matrix, the virtual human limbs may vibrate due to unreasonable relationships between some parameters in the second rotation matrix. For example, the relationship between the acceleration of the human joint movement and the external force on the human joint may be unreasonable. Therefore, the driving device needs to smooth the second rotation matrix to obtain the smoothed target rotation matrix.

[0068] Furthermore, the driving device drives the virtual human according to the target rotation matrix, so that the limbs of the virtual human exhibit movements that are adapted to the first human body image.

[0069] According to the applicant's research, directly using the detection results of a human pose detection network to drive virtual human movements may result in movements that violate physical laws and exhibit severe jitter. Therefore, in the embodiments of this application, the driving device adds a constraint matrix to the target matrix used to drive the virtual human's limb movements and performs smoothing processing. This ensures that the virtual human's movements do not violate physical laws and do not jitter during the movement, thus guaranteeing the driving effect of the virtual human.

[0070] Figure 2 A flowchart illustrating a virtual human driving method according to an embodiment of this application is shown. The driving method includes:

[0071] S202, Determine the first rotation matrix of the human joints based on the acquired first human image;

[0072] S204, using the first rotation matrix of the sub-joints of the human body joint as the constraint matrix, and compensating the constraint matrix to the first rotation matrix of the human body joint to determine the second rotation matrix;

[0073] S206, Smooth the second rotation matrix to determine the target rotation matrix of the human joint;

[0074] S208, drives the virtual human's movements according to the target rotation matrix.

[0075] In this embodiment, the process of determining the second rotation matrix is ​​as follows: the driving device first determines the sub-joints of the human joint, and then determines the constraint matrix of the first rotation matrix of the human joint based on the sub-joints of the human joint.

[0076] Furthermore, after identifying the sub-joints of the human body, the drive device uses the first rotation matrix corresponding to the sub-joint as a constraint matrix and compensates for this constraint matrix with the first rotation matrix of the human body joint to determine the second rotation matrix of each human body joint. Specifically, after compensating the first rotation matrix corresponding to the sub-joint with the first rotation matrix of the human body joint, the resulting second rotation matrix restricts the rotation of the human body joint on the XYZ axes and the positional relationship of the human body joint relative to the human body.

[0077] For example, consider the thigh as a joint in the human body, and the lower leg as its sub-joint. Let the first rotation matrix of the thigh be C0, and the first rotation matrix of the lower leg be C1. Since the lower leg can only rotate along the Y-axis, we can decompose the first rotation matrix of the lower leg sequentially along the XYZ axes, resulting in C1 = C... Y1 C X1 C Z1 By compensating C1 for C0, the second rotation matrix of the thigh and calf can be obtained as C′1=C Y1 ,C′0=C X1 CZ1 C0 ensures that the lower leg can rotate around the Y-axis while maintaining its position relative to the body.

[0078] In this embodiment, the driving device can physically constrain the virtual human's limb movements by using the first rotation matrix corresponding to the sub-joint as a constraint matrix and compensating the first rotation matrix of the human joint with this constraint matrix, thereby determining the second rotation matrix of each human joint in the virtual human. This restricts the rotation of the human joints along the XYZ axes and the positional relationship of the human joints relative to the human body, ensuring that no physical violations occur when driving the virtual human's limb movements.

[0079] In the above embodiments, the first rotation matrix of different sub-joints is different.

[0080] In this embodiment, the sub-joints of different human joints are different. For example, the sub-joint of the thigh is the calf, and the sub-joint of the elbow is the arm. It is understood that the first rotation matrix of different sub-joints is also different, so the constraint matrices added to different human joints are also different.

[0081] Figure 3 A flowchart illustrating a virtual human driving method according to an embodiment of this application is shown. The driving method includes:

[0082] S302, Determine the first rotation matrix of the human joints based on the acquired first human image;

[0083] S304, Add a constraint matrix to the first rotation matrix to determine the second rotation matrix of the human joint. The constraint matrix corresponds to the physical constraint conditions of the human joint.

[0084] S306, determine the first acceleration of the human joint movement and the first external force on the human joint according to the second rotation matrix;

[0085] S308, determine the first position of the human joint based on the first acceleration and the first external force;

[0086] S310, Determine the target rotation matrix based on the first position;

[0087] S312, drives the virtual human's movements according to the target rotation matrix.

[0088] In this embodiment, the first external force refers to the external force on the human joint, such as the reaction force of the ground on the lower leg, and the first position refers to the position of the human joint determined after physical constraints and smoothing.

[0089] Specifically, the drive device first calculates the current acceleration of the human joint, i.e., the first acceleration, and the current external force on the human joint, i.e., the first external force, based on the second rotation matrix.

[0090] Specifically, the current position of the human joint can be calculated based on the parameters in the second rotation matrix. Combining this position information with the time it took to acquire the first human image, and the position and time of the human joint before acquiring the first human image, the velocity and acceleration of the human joint from its base position to its current position can be determined. Based on the velocity and acceleration, the external force acting on the human joint can be calculated. Therefore, the drive device can determine the first acceleration and the first external force based on the second rotation matrix.

[0091] Furthermore, the drive device calculates the smoothed position of the human joint, i.e., the first position, using the aforementioned first acceleration and the aforementioned first external force. Specifically, the drive device can calculate the optimal relationship between the acceleration of the human joint and the external force it experiences based on the aforementioned first acceleration and the aforementioned first external force, and can calculate the smoothed first position of the human joint based on this optimal relationship.

[0092] Furthermore, the drive device derives a rotation matrix for controlling the virtual human's limb movements, namely the aforementioned target rotation matrix, based on the determined first position.

[0093] In this embodiment, the driving device can calculate the first acceleration and the first external force of the human joint based on the second rotation matrix. By solving for the optimal relationship between the first acceleration and the first external force, the first position of the smoothed human joint can be determined, and the target rotation matrix can be derived from the first position. In this way, the target rotation matrix used to drive the virtual human limb movements is smoothed, ensuring that there is no jitter during the driving of the virtual human limb movements.

[0094] In the above embodiments, the steps of determining the first acceleration of the human joint movement and the first external force on the human joint according to the second rotation matrix specifically include: determining the second position of the human joint according to the second rotation matrix; acquiring a second human image and a third human image, as well as the acquisition time information of the second human image and the third human image; determining the first historical position of the human joint in the second human image and the second historical position of the human joint in the third human image; determining the first velocity and the first acceleration of the human joint movement according to the first historical position, the second historical position, the second position and the acquisition time information; and determining the first external force according to the second position, the first velocity and the first acceleration; wherein, the second human image and the third human image are images acquired before the first human image.

[0095] In this embodiment, the second human body image and the third human body image represent images captured before the first human body image, and the frame sequence of the second human body image is earlier than that of the third human body image; the first historical position represents the position of the human body joint in the second human body image, the second historical position represents the position of the human body joint in the third human body image, and the second position represents the position of the human body joint in the first human body image after physical constraints.

[0096] It should be noted that the aforementioned human joints include each human joint of the virtual human. To more clearly describe the solution of this application, human joint A in the human joints will be used as an example below.

[0097] Specifically, the process of determining the first external force and the first acceleration is as follows: the driving device first solves the second position of human joint A after physical constraints based on the second rotation matrix.

[0098] Furthermore, the driving device acquires the second and third human body images, as well as the acquisition time information of the second and third human body images, and determines the first and second historical positions of human joint A. Specifically, the driving device can first input the second and third human body images into the human posture detection network to determine the rotation matrix of human joint A in the second human body image and the rotation matrix of human joint A in the third human body image, and then solve for the first and second historical positions based on these two rotation matrices.

[0099] Furthermore, the driving device uses the aforementioned second position, the aforementioned first historical position, the aforementioned second historical position, and the aforementioned acquisition time information to calculate the first acceleration and the first velocity of human joint A. Specifically, the driving device can determine the distance based on any two of the aforementioned three positions, and then calculate the aforementioned first acceleration and the first velocity using this distance and the aforementioned acquisition time information.

[0100] Furthermore, the drive device calculates the first external force acting on human joint A based on the aforementioned first acceleration, first velocity, and first external force. Specifically, the first external force acting on human joint A has a certain relationship with the aforementioned first acceleration, first velocity, and first external force, such as the relationship constrained by the rigid body motion equation. Based on this relationship, the drive device can calculate the first external force acting on human joint A using the aforementioned first acceleration, first velocity, and first external force.

[0101] In this embodiment, the driving device solves for the first external force and the first acceleration of each joint of the virtual human body based on the second rotation matrix obtained after adding the constraint matrix. This ensures the accuracy of the determined first external force and second acceleration, and thus ensures the accuracy of the target rotation matrix determined in subsequent steps.

[0102] In the above embodiments, the steps of determining the first velocity and the first acceleration of human joint movement based on the first historical position, the second historical position, the second position and the acquisition time information specifically include: determining the first velocity based on the second historical position, the second position and the acquisition time information; determining the second velocity based on the first historical position, the second historical position and the acquisition time information; and determining the first acceleration based on the difference between the first velocity and the second velocity.

[0103] In this embodiment, the process of determining the first velocity and the first acceleration is as follows: the driving device first calculates the first velocity based on the second position, the second historical position, and the acquisition time information. Specifically, the distance between the two positions can be determined based on the second position and the second historical position, and the driving device can calculate the first velocity based on this distance and the acquisition time information.

[0104] Furthermore, the drive device calculates the second speed based on the first historical position, the second historical position, and the acquisition time information. Specifically, the distance between the two positions can be determined based on the first historical position and the second historical position, and the drive device can calculate the second speed based on this distance and the acquisition time information.

[0105] Furthermore, the drive device calculates the difference between the first speed and the second speed to determine the first acceleration.

[0106] For example, the drive device can also calculate the aforementioned first acceleration using a PD (proportional-derivative) controller, with the calculation formula being: q″ d =q″+k d (q′ d -q′)+k p (q d -q), where q represents the second position mentioned above, q′ represents the first velocity mentioned above, q″ represents the first acceleration mentioned above, and q d ,q′ d and q″ d Based on the pre-designed planned route, k p k represents the proportionality coefficient. d This represents the differential coefficient.

[0107] In this embodiment, the driving device calculates the velocity and acceleration of each joint in the first human body image based on the positions of the joints in the human body image acquired prior to the first human body image. In other words, the driving device calculates the velocity and acceleration within the most recent timeframe. This ensures the accuracy of determining the first acceleration and the first velocity.

[0108] In the above embodiment, the step of determining the first external force based on the second position, the first velocity, and the first acceleration specifically includes: determining the first external force based on the second position, the first velocity, and the first acceleration using the rigid body motion equation and the friction cone constraint condition.

[0109] In this embodiment, the process of determining the first external force is as follows: the driving device adopts the rigid body motion equation, and under the condition of satisfying the friction cone constraint, the first external force is solved based on the first acceleration, the first velocity, and the second position.

[0110] Specifically, the above equation of rigid body motion is: τ=M(q)q″+c(q,q′), which can be transformed to obtain the equation f=‖Mq″. d +c(q,q′)-JGλ‖, where q represents the second position mentioned above, q′ represents the first velocity mentioned above, q″ represents the first acceleration mentioned above, and q″ represents the second acceleration mentioned above. d Let λ represent the first acceleration after smoothing, M represent the moment of inertia matrix, J represent the Jacobian matrix describing the relationship between the forces acting on the virtual human and each joint, G is the transformation matrix that converts the first external force λ into linear forces and torques, c represents the non-external forces acting on the system, and τ and f represent known functions of time and rigid body motion state.

[0111] Specifically, the formula for the above friction cone constraint condition is as follows: Where F represents the friction cone constraint, λ n λ represents the normal component. t and λ b Indicates the tangential component. μ represents the friction coefficient, and R represents a three-dimensional vector.

[0112] Furthermore, by limiting min f stF, the first external force mentioned above can be solved, where st stands for subjectto, meaning to make xxx satisfy xxx, that is, the above formula means to make the minimum f satisfy F.

[0113] In this embodiment, the driving device can solve for the first external force that satisfies the friction cone constraint condition by using the rigid body motion equation, based on the first acceleration, the first velocity, and the second position, thus ensuring the accuracy of determining the first external force.

[0114] In the above embodiments, the step of determining the first position of a human joint based on the first acceleration and the first external force specifically includes: solving for the first position based on the first acceleration and the first external force using an iterative algorithm.

[0115] In this embodiment, the process of determining the first position is as follows: using an iterative algorithm, the optimal combination relationship between the first external force and the first acceleration is solved to determine the optimal first position.

[0116] Specifically, the condition for the above iterative algorithm is min‖q″-q″ d ‖+‖τ‖stMq″-τ=JGλ-c(q,q′), even if min‖q″-q″ d ‖+‖τ‖ satisfies q″-τ=JGλ-c(q,q′), where the meanings of the letters in this expression are the same as those in the above expression, and will not be repeated here.

[0117] Specifically, the expression for the velocity of the virtual human's limb movements is: v = Jq′. To prevent the limb movements from penetrating the ground, this application stipulates that when the limb contacts the ground, the velocity perpendicular to the ground must be greater than 0, and the tangential velocity should also be constrained. Therefore, in the above iterative algorithm, 0 ≤ v should also be considered. n ,|v t |≤σ,|v b The constraint condition is |≤σ, where v represents the speed of the limb movement, v n The velocity v represents the velocity perpendicular to the ground. b and v t σ represents the tangential velocity, which is a preset constant.

[0118] In this embodiment, the first position is the optimal position obtained by an iterative algorithm. This ensures the accuracy of the determined first position and, consequently, ensures that when the target rotation matrix derived from the first position drives the virtual human limb movements, there will be no shaking or violation of physical laws.

[0119] Example 2:

[0120] Figure 4 A schematic block diagram of a virtual human driving device according to an embodiment of this application is shown. The virtual human driving device 400 includes: a first processing module 402, used to determine a first rotation matrix of human joints based on a first acquired human body image; a second processing module 404, used to add a constraint matrix to the first rotation matrix to determine a second rotation matrix of human joints, the constraint matrix corresponding to the physical constraints of human joints; a third processing module 406, used to smooth the second rotation matrix to determine a target rotation matrix of human joints; the third processing module 406 is also used to drive the virtual human to move according to the target rotation matrix.

[0121] In this embodiment, the first human body image refers to a frame of image captured in a video, the constraint matrix refers to the physical constraint conditions for human body joints, and the physical constraint conditions refer to the conditions that restrict the movement of human body joints and prevent their movements from exceeding certain limits; the rotation matrix refers to a mathematical expression of human body joint posture, the first rotation matrix refers to the rotation matrix of human body joints detected by the human body posture detection network, and the target rotation matrix refers to the second rotation matrix after smoothing.

[0122] Specifically, the first processing module 402 first determines the rotation matrix of each joint of the virtual human body, i.e., the aforementioned first rotation matrix, based on the first human body image captured from the video. Specifically, the first processing module 402 can determine the first rotation matrix by inputting the aforementioned first human body image into a human pose detection network, but is not limited to this method.

[0123] Furthermore, the second processing module 404 adds a constraint matrix to the first rotation matrix of the human joint to determine the physically corrected rotation matrix, namely the aforementioned second rotation matrix.

[0124] Specifically, the first rotation matrix clearly defines the movement of each human joint across the XYZ axes. By adding constraint matrices, the movement of these joints on which axes of the XYZ axes can be restricted, as well as the limits of movement on each axis. Therefore, the second processing module 404 can modify and restrict the first rotation matrix by adding constraint matrices to derive a modified second rotation matrix.

[0125] Furthermore, the third processing module 406 smooths the second rotation matrix obtained after adding the constraint matrix to determine the target rotation matrix used to drive the above-mentioned virtual human limb movements.

[0126] Specifically, if the virtual human limb movements are driven directly based on the second rotation matrix, the relationship between some parameters in the second rotation matrix may be unreasonable. For example, the relationship between the acceleration of the human joint movement and the external force on the human joint may be unreasonable, which may cause shaking when driving the virtual human limb movements. Therefore, the third processing module 406 needs to smooth the second rotation matrix to obtain the smoothed target rotation matrix.

[0127] Furthermore, the third processing module 406 drives the virtual human according to the target rotation matrix so that the limbs of the virtual human exhibit movements that are adapted to the first human body image.

[0128] In this embodiment, the first processing module 402 can determine the first rotation matrix of each joint of the virtual human body based on the first human body image captured from the video. The second processing module 404 can add a constraint matrix to the first rotation matrix to determine the second rotation matrix. The third processing module 406 can smooth the parameters in the second rotation matrix to determine the target rotation matrix and drive the virtual human body to move according to the target rotation matrix. In this embodiment, the third processing module 406 adds a constraint matrix and performs smoothing on the target rotation matrix used to drive the virtual human body to move. This ensures that the virtual human body's movements do not violate physical laws and there is no jitter during the movement. This solves the problem in related technologies where directly using the detection results of the human posture detection network to drive the virtual human body to move may violate physical laws and cause jitter, thus ensuring the driving effect of the virtual human body.

[0129] In the above embodiment, the second processing module 404 is specifically used to use the first rotation matrix of the sub-joint of the human body joint as the constraint matrix; compensate the constraint matrix to the first rotation matrix of the human body joint, and determine the second rotation matrix.

[0130] In this embodiment, the second processing module 404 can physically constrain the virtual human's limb movements by using the first rotation matrix corresponding to the sub-joint as a constraint matrix and compensating the first rotation matrix of the human joint with this constraint matrix, thereby determining the second rotation matrix of each human joint in the virtual human. This restricts the rotation of the human joints along the XYZ axes and the positional relationship of the human joints relative to the human body, ensuring that no physical violations occur when driving the virtual human's limb movements.

[0131] In the above embodiments, the first rotation matrix of different sub-joints is different.

[0132] In this embodiment, the sub-joints of different human joints are different. For example, the sub-joint of the thigh is the calf, and the sub-joint of the elbow is the arm. It is understood that the first rotation matrix of different sub-joints is also different, so the constraint matrices added to different human joints are also different.

[0133] In the above embodiment, the third processing module 406 is specifically used to determine the first acceleration of the human joint movement and the first external force on the human joint according to the second rotation matrix; determine the first position of the human joint according to the first acceleration and the first external force; and determine the target rotation matrix according to the first position.

[0134] In this embodiment, the third processing module 406 can solve for the first acceleration and the first external force of the human joint based on the second rotation matrix. By solving for the optimal relationship between the first acceleration and the first external force, the first position of the smoothed human joint can be determined, and the target rotation matrix can be derived from the first position. In this way, the target rotation matrix used to drive the virtual human limb movements is smoothed, ensuring that there is no jitter during the process of driving the virtual human limb movements.

[0135] In the above embodiments, the virtual human driving device 400 further includes an acquisition module 408, and a third processing module 406 is specifically used to determine the second position of the human joint according to the second rotation matrix; the acquisition module 408 is used to acquire the second human image and the third human image, as well as the acquisition time information of the second human image and the third human image; the third processing module 406 is specifically used to determine the first historical position of the human joint in the second human image and the second historical position of the human joint in the third human image; determine the first velocity and the first acceleration of the human joint movement according to the first historical position, the second historical position, the second position and the acquisition time information; determine the first external force according to the second position, the first velocity and the first acceleration; wherein, the second human image and the third human image are images acquired before the first human image.

[0136] In this embodiment, the third processing module 406 solves for the first external force and the first acceleration of each joint of the virtual human body based on the second rotation matrix obtained after adding the constraint matrix. This ensures the accuracy of the determined first external force and second acceleration, and thus ensures the accuracy of the target rotation matrix determined in subsequent steps.

[0137] In the above embodiments, the third processing module 406 is specifically used to determine a first velocity based on the second historical position, the second position and the acquisition time information; determine a second velocity based on the first historical position, the second historical position and the acquisition time information; and determine a first acceleration based on the difference between the first velocity and the second velocity.

[0138] In this embodiment, the third processing module 406 calculates the velocity and acceleration of each joint in the first human body image based on the positions of the joints in the human body image acquired before the first human body image. In other words, the driving device calculates the velocity and acceleration within the most recent timeframe. This ensures the accuracy of determining the first acceleration and the first velocity.

[0139] In the above embodiment, the third processing module 406 is specifically used to determine the first external force based on the second position, the first velocity, and the first acceleration by using the rigid body motion equation and the friction cone constraint condition.

[0140] In this embodiment, the third processing module 406 can solve for the first external force that satisfies the friction cone constraint condition by using the rigid body motion equation, based on the first acceleration, the first velocity, and the second position, thus ensuring the accuracy of determining the first external force.

[0141] In the above embodiment, the third processing module 406 is specifically used to solve for the first position based on the first acceleration and the first external force through an iterative algorithm.

[0142] In this embodiment, the first position is the optimal position obtained by the third processing module 406 through an iterative algorithm. This ensures the accuracy of the determined first position and, consequently, ensures that when the target rotation matrix derived from the first position drives the virtual human limb movements, there will be no shaking or violation of physical laws.

[0143] Example 3:

[0144] Figure 5 A schematic block diagram of another virtual human driving device 500 according to an embodiment of this application is shown. The virtual human driving device 500 includes: a memory 502, which stores programs or instructions; and a processor 504, which executes the programs or instructions stored in the memory 502 to implement the steps of the virtual human driving method proposed in the above embodiments of this application. Therefore, it has all the beneficial technical effects of the virtual human driving method proposed in the above embodiments of this application, and will not be described in detail here.

[0145] Example 4:

[0146] According to the fourth embodiment of this application, a readable storage medium is proposed, on which a program or instructions are stored. When the program or instructions are executed by a processor, they implement the steps of the virtual human driving method proposed in the above embodiments of this application, and thus have all the beneficial technical effects of the virtual human driving method proposed in the above embodiments of this application, which will not be elaborated further here.

[0147] Example 5:

[0148] According to the fifth embodiment of this application, an electronic device is proposed, including a driving device for a virtual human as proposed in the above embodiments of the present invention, and / or a readable storage medium as proposed in the above embodiments of the present invention. Therefore, the electronic device has all the beneficial effects of the driving device for a virtual human as proposed in the above embodiments of the present invention and / or the readable storage medium as proposed in the above embodiments of the present invention, which will not be repeated here.

[0149] Example 6:

[0150] According to the sixth embodiment of this application, a computer program product is proposed, including a computer program. When the computer program is executed by a processor, it implements the steps of the virtual human driving method proposed in the above embodiments of this application, and thus has all the beneficial technical effects of the virtual human driving method proposed in the above embodiments of this application, which will not be elaborated further here.

[0151] Example 7:

[0152] This embodiment combines Figure 6 The driving method for the virtual human proposed in this application is illustrated by way of example. Specifically, as... Figure 6 As shown, the drive device first uses the acquired human body image and a human posture detection network to determine the predicted posture of the human body, that is, to determine the first rotation matrix of the human body joints. Then, a constraint matrix is ​​added to the rotation matrix to determine the second rotation matrix after physical constraints.

[0153] Furthermore, the driving device inputs the determined second rotation matrix into the PD controller to solve for the acceleration of the human body shutdown, and uses this acceleration to solve for the first external force of the human body joints. Then, it uses an iterative algorithm to solve for the optimal combination of acceleration and the first external force, and smooths the second rotation matrix according to the optimal combination to determine the target rotation matrix used to drive the virtual human's movements.

[0154] In the description of this specification, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance, unless otherwise expressly specified and limited. The terms "connection," "installation," and "fixing," etc., should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0155] In the description of this specification, the terms "one embodiment," "some embodiments," "specific embodiment," etc., refer to a specific feature, structure, material, or characteristic described in connection with that embodiment or example, which is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0156] Furthermore, the embodiments of this application can be combined with each other, but only if they are based on what those skilled in the art can do. If the combination of embodiments is contradictory or cannot be implemented, it should be considered that such combination of embodiments does not exist and is not within the scope of protection claimed by this application.

[0157] The above are merely preferred embodiments of this application and are not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for driving a virtual human, characterized in that, include: Based on the acquired first human image, determine the first rotation matrix of the human joints; Add a constraint matrix to the first rotation matrix to determine the second rotation matrix of the human joint, wherein the constraint matrix corresponds to the physical constraint conditions of the human joint; The second rotation matrix is ​​smoothed to determine the target rotation matrix of the human joint; The virtual human's movements are driven according to the target rotation matrix; The step of smoothing the second rotation matrix to determine the target rotation matrix of the human joint specifically includes: The first acceleration of the human joint movement and the first external force on the human joint are determined according to the second rotation matrix. The first position of the human joint is determined based on the first acceleration and the first external force; The target rotation matrix is ​​determined based on the first position; Determining the first position of the human joint based on the first acceleration and the first external force specifically includes: Using the rigid body motion equations, the first external force is solved while satisfying the friction cone constraint conditions; The first position is determined by an iterative algorithm based on the first acceleration and the first external force.

2. The virtual human driving method according to claim 1, characterized in that, The step of adding a constraint matrix to the first rotation matrix to determine the second rotation matrix of the human joint specifically includes: The first rotation matrix of the sub-joints of the human body joint is used as the constraint matrix; The constraint matrix is ​​compensated to the first rotation matrix of the human joint, and the second rotation matrix is ​​determined.

3. The virtual human driving method according to claim 2, characterized in that, The first rotation matrix is ​​different for different sub-joints.

4. The virtual human driving method according to claim 1, characterized in that, The step of determining the first acceleration of the human joint movement and the first external force on the human joint based on the second rotation matrix specifically includes: The second position of the human joint is determined according to the second rotation matrix; Acquire a second human body image and a third human body image, as well as the acquisition time information of the second human body image and the third human body image; Determine the first historical position of the human joint in the second human image and the second historical position of the human joint in the third human image; Based on the first historical position, the second historical position, the second position, and the acquisition time information, the first velocity and the first acceleration of the human joint movement are determined; The first external force is determined based on the second position, the first velocity, and the first acceleration. The second human body image and the third human body image are images captured before the first human body image.

5. The virtual human driving method according to claim 4, characterized in that, The step of determining the first velocity and first acceleration of the human joint movement based on the first historical position, the second historical position, the second position, and the acquisition time information specifically includes: The first speed is determined based on the second historical location, the second location, and the acquisition time information; The second speed is determined based on the first historical location, the second historical location, and the acquisition time information; The first acceleration is determined based on the difference between the first velocity and the second velocity.

6. The virtual human driving method according to claim 4, characterized in that, Determining the first external force based on the second position, the first velocity, and the first acceleration specifically includes: The first external force is determined based on the rigid body motion equations and the friction cone constraint conditions, according to the second position, the first velocity, and the first acceleration.

7. A driving device for a virtual human, characterized in that, include: The first processing module is used to determine the first rotation matrix of the human joints based on the acquired first human image. The second processing module is used to add a constraint matrix to the first rotation matrix and determine the second rotation matrix of the human joint, wherein the constraint matrix corresponds to the physical constraint conditions of the human joint. The third processing module is used to smooth the second rotation matrix and determine the target rotation matrix of the human joint. The third processing module is also used to drive the virtual human's movements according to the target rotation matrix; The third processing module is specifically used to determine the first acceleration of the human joint movement and the first external force on the human joint based on the second rotation matrix; determine the first position of the human joint based on the first acceleration and the first external force; determine the target rotation matrix based on the first position; solve for the first external force using rigid body motion equations while satisfying friction cone constraint conditions; and solve for the first position based on the first acceleration and the first external force using an iterative algorithm.

8. A driving device for a virtual human, characterized in that, include: A memory and a processor, the memory storing a program, the processor executing the program to implement the steps of the virtual human driving method as described in any one of claims 1 to 6.

9. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the virtual human driving method as described in any one of claims 1 to 6.

10. An electronic device, characterized in that, include: The driving device for the virtual human as described in claim 7 or 8; and / or The readable storage medium as described in claim 9.

11. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the steps of the driving method for a virtual human as described in any one of claims 1 to 6.

Images