Virtual human action generation method

Abstract

The application belongs to the technical field of virtual reality, and relates to a virtual human action generation method. The method comprises the following steps: constructing a human skeleton model, calibrating a pose zero point of the human skeleton model, and establishing a pose constraint relationship of the human skeleton model; acquiring pose data of a target person by using a sensor, and defining the pose data as first data; according to the first data and the pose constraint relationship, solving unknown actions of virtual upper limbs of the human skeleton model to obtain virtual upper limb predicted pose data; constructing a virtual lower limb action library, and acquiring virtual lower limb predicted pose data in the virtual lower limb action library according to the first data; and generating and outputting a virtual human body action according to the first data, the virtual upper limb predicted pose data and the virtual lower limb predicted pose data. In the application, only a small amount of end sensors are relied on, and virtual human body whole body actions can be accurately generated, thereby solving technical pain points in the prior art, such as high dependence on sensors, high use cost and difficulty in reproducing complex actions.

Description

Technical Field

[0001] This application relates to the field of virtual reality technology, specifically to a method for generating virtual human motion. Background Technology

[0002] With the deep integration of VR technology and virtual human applications, virtual human-based operation and maintenance guidance in virtual environments has been successfully implemented in various fields such as shipbuilding. However, current virtual human full-body motion generation technology still faces many bottlenecks. Existing technologies sometimes rely on image recognition combined with multiple sensors to collect pose data, but this often requires 12 or more sensors, making it cumbersome to wear and significantly increasing hardware costs. Other technologies directly collect data from sensors on each joint of the virtual human and assign it to those joints. While this ensures pose accuracy, the excessive number of sensors makes it inconvenient to use. Even simplified solutions require four sensors for the head, waist, and feet, and can only recognize simple movements such as walking, failing to meet the motion reproduction requirements of complex maintenance operations.

[0003] Mainstream VR hardware typically only comes with a headset and controllers, lacking additional sensor support. This makes it difficult to obtain the user's full-body posture information and to fully demonstrate the action specifications of key maintenance operations and human-machine interference scenarios. Adding additional sensors would increase the difficulty of use and the cost burden. Summary of the Invention

[0004] In view of the shortcomings of the prior art described above, the purpose of this application is to provide a virtual human motion generation method that can accurately generate reasonable full-body movements of a virtual human by relying on only a small amount of end-sensor data, so as to overcome the technical pain points of the prior art, such as reliance on multiple sensors, high cost of use, and difficulty in reproducing complex movements.

[0005] This application provides a method for generating virtual human actions, including the following steps: A human skeleton model is constructed, which includes a virtual head, virtual hands, virtual upper limbs, and virtual lower limbs; The pose zero point of the human skeleton model is calibrated, and the pose constraint relationship of the human skeleton model is established. Obtain the pose data of the target person and define it as the first data; Based on the first data, the human skeletal model is assigned a pose value; Based on the pose constraint relationship, the virtual upper limb unknown movement is solved on the assigned human skeleton model, and the pose is assigned to the human skeleton model based on the solved virtual upper limb predicted pose data. Construct a virtual lower limb motion library, and obtain virtual lower limb predicted pose data from the virtual lower limb motion library based on the first data; Based on the first data, the virtual upper limb predicted pose data, and the virtual lower limb predicted pose data, virtual human motion is generated and output.

[0006] In an optional embodiment, the virtual upper limb includes virtual upper limb joints, the first data includes head pose data and hand pose data, the hand pose data includes left hand pose data and right hand pose data, and the pose data includes spatial position data and rotation angle data. The steps for solving the unknown virtual upper limb movements of the human skeleton model include: Initial pose data are assigned to the virtual upper limb joints; Based on the initial pose data of the virtual upper limb joints, the pose data of the virtual head and the virtual hand are obtained through forward kinematics calculation and defined as the second data; The positional error is calculated as the sum of the squared differences between two head spatial position data, two left-hand spatial position data, and two right-hand spatial position data in the first and second data. The rotation error is calculated as the sum of the squared differences between the two head rotation angle data, the squared differences between the two left hand rotation angle data, and the squared differences between the two right hand rotation angle data in the first data and the second data. Substitute the position error and the rotation error into the following formula to solve for the total error F: F=u p ·E p +u r ·E r +u c ·E c Where F is the total error, E p E represents the position error. r E represents the rotational error. c To constrain punishment, u p u r u c Position error E p Rotational error E r and constraint punishment E c The weight value; Adjust the value of the initial pose data, repeat the above steps until the total error F is less than the preset error threshold, and use the corresponding initial pose data as the virtual upper limb predicted pose data.

[0007] In an optional embodiment, the error threshold is 0.1.

[0008] In an optional embodiment, when the initial pose data satisfies the pose constraint relationship of the human skeleton model, the value of the constraint penalty is 0; when the initial pose data violates the pose constraint relationship, the value of the constraint penalty is a positive number, and the greater the degree to which the initial pose data violates the pose constraint relationship, the greater the value of the constraint penalty.

[0009] In an optional embodiment, the weighting relationship between the position error, rotation error, and constraint penalty is u. p ≥u r >u c .

[0010] In one optional embodiment, the virtual lower limb motion library includes at least the actions of “standing upright,” “walking,” “squatting,” and “standing on an obstacle.”

[0011] In an optional embodiment, during the step of obtaining the virtual lower limb predicted pose data from the virtual lower limb action library, when the pose data of the hand in the first data constitutes a "waving" action, the "walking" action in the virtual lower limb action library is invoked.

[0012] In an optional embodiment, during the step of obtaining the virtual lower limb predicted pose data from the virtual lower limb action library, when the spatial position data of the head pose data becomes lower than 1m, the "squatting" action in the virtual lower limb action library is invoked.

[0013] In an optional embodiment, the human skeleton model is constructed in a virtual environment with environmental constraints, and virtual lower limb predicted pose data is obtained from the virtual lower limb motion library according to the environmental constraints.

[0014] In an optional embodiment, in the step of obtaining the virtual lower limb predicted pose data from the virtual lower limb action library, when the environmental constraint is identified as an "obstacle", if the height of the obstacle is less than 1m, the "standing on the obstacle" action in the virtual lower limb action library is invoked; if the height of the obstacle is greater than 1m, the "standing upright" action in the virtual lower limb action library is invoked.

[0015] As described above, compared with the prior art, the virtual human motion generation method provided in this application has at least the following beneficial effects: Based on the acquired pose data of the target person, the virtual upper limb predicted pose data is obtained by solving the forward kinematics calculation. The virtual lower limb predicted pose data is obtained by constructing a virtual lower limb motion library. The accurate generation of virtual human body full-body motion can be achieved by relying on only a small amount of end sensor data, which solves the technical pain points of existing technologies such as reliance on multiple sensors, high cost of use, and difficulty in reproducing complex movements. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 The diagram shown is a flowchart illustrating a virtual human motion generation method provided in this application. Detailed Implementation

[0018] To make the technical objectives, technical solutions, and technical effects of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with embodiments. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0019] Therefore, the following detailed description of embodiments of this application is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0020] In the description of this application, it should be noted that the terms "center", "longitudinal", "lateral", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicating the orientation or positional relationship are based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing this application and simplifying the description, and are not intended to indicate or imply that the device referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application.

[0021] In the description of this application, unless otherwise expressly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly, for example, referring to both fixed connections and detachable connections. Furthermore, the descriptions using terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples" indicate that a specific feature, structure, material, or characteristic described in connection with an implementation or example is included in at least one implementation or example of this application. In this specification, illustrative expressions of the above terms do not necessarily refer to the same implementation or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more implementations or examples.

[0022] To address the problems of excessive reliance on sensors, high operating costs, and difficulty in reproducing complex movements in the existing technologies described above, this embodiment provides a method for generating virtual human movements that can generate full-body movements of a virtual human using only a small amount of end-sensor data. This method includes the following steps: A human skeleton model is constructed, which includes a virtual head, virtual hands, virtual upper limbs, and virtual lower limbs; The pose zero point of the human skeleton model is calibrated, and the pose constraint relationship of the human skeleton model is established. Obtain the pose data of the target person and define it as the first data; Based on the first data, the human skeletal model is assigned a pose value; Based on the pose constraint relationship, the virtual upper limb unknown movement is solved on the assigned human skeleton model, and the pose is assigned to the human skeleton model based on the solved virtual upper limb predicted pose data. Construct a virtual lower limb motion library, and obtain virtual lower limb predicted pose data from the virtual lower limb motion library based on the first data; Based on the first data, the virtual upper limb predicted pose data, and the virtual lower limb predicted pose data, virtual human motion is generated and output.

[0023] In practical applications, virtual human full-body motion can be generated simply by acquiring the real-time pose data of the target person's head and hands, without the need for costly equipment such as full-body motion capture. This solves the technical problems of high sensor dependence and high usage costs in existing technologies.

[0024] In this embodiment, a human skeletal model is first constructed, which includes at least a virtual head, virtual hands, virtual upper limbs, and virtual lower limbs. The human skeletal model can be constructed in a virtual environment, which can be a scene in a virtual simulation engine, such as Unreal Engine 5, Unreal Engine 5, or other suitable virtual simulation engines, to realize the construction of the human skeletal model and the simulation of the virtual environment and virtual human movements. The virtual upper limbs of the human skeletal model include virtual upper limb joints, such as virtual wrist joints, virtual shoulder joints, and virtual elbow joints. Optionally, the virtual lower limbs of the human skeletal model can include virtual lower limb joints, such as virtual hip joints and virtual knee joints. Specifically, whether or not to construct virtual lower limb joints in the human skeletal model can be selected according to the needs of calculation or action output.

[0025] Next, the pose zero point of the human skeleton model needs to be calibrated, and the pose constraints of the human skeleton model need to be established. The pose constraints of the human skeleton model include the range constraints of the spatial position and rotation angle of the virtual head, virtual hands, virtual upper limbs, and virtual lower limbs. The spatial position constraints can be calculated, for example, using the pose data of the target person's head and hands obtained through sensors. Specifically, when the target person is in an upright and motionless state, the height data of the target person can be obtained through the spatial position data of the target person's head, thereby constraining the range of the spatial position data of the virtual head of the human skeleton model; the shoulder width data of the target person can be calculated through the spatial position data of the target person's hands, thereby constraining the range of the spatial position data of the two virtual shoulder joints of the human skeleton model; based on the aforementioned height and shoulder width data of the target person, the length data of the target person's forearm and upper arm can be calculated using the formulas "height = shoulder width + 2 (forearm length + upper arm length)" and "forearm length: upper arm length = 1:1", thereby constraining the position of the virtual shoulder joint, virtual elbow joint, and virtual wrist joint of the human skeleton model. The range of rotation angles can be referenced to normal human activity. For example, the rotation angle of the virtual head can be constrained to rotate 90° to the left and 90° to the right, and to rotate 90° upwards and downwards. The rotation angle of the right virtual elbow joint can be constrained to rotate 0 to 180° to the left around the forearm axis, and the rotation angle of the left virtual elbow joint can be constrained to rotate 0 to 180° to the right around the forearm axis.

[0026] In this embodiment, the pose data of the target person in an upright state can be used as the pose zero point of the human skeleton model. Specifically, for example, by giving the target person a command to "stay upright and motionless", the target person is kept in an upright and motionless state. The pose data of the target person's head and hands are acquired by sensors. Based on the pose data of the target person's head and hands, and using the above-mentioned pose constraint relationship, the pose data of the virtual wrist joint, virtual elbow joint and virtual shoulder joint when the human skeleton model is in an upright and motionless state are calculated, and these data are calibrated as the pose zero point of the human skeleton model.

[0027] Next, the pose data of the target person when performing a certain action is acquired in real time and defined as the first data. This real-time pose data can be acquired by placing sensors on the target person or other suitable methods. In this embodiment, a VR headset and controllers are worn on the target person's head and hands respectively. The sensors in the VR headset and controllers can track the pose data of the target person's head and hands in real time. The first data includes head pose data and hand pose data, and the hand pose data includes left hand pose data and right hand pose data. The pose data includes spatial position data and rotation angle data. A computer is connected to the VR headset and controllers to acquire the real-time pose data of the target person's head and hands, and this acquired pose data is defined as the first data. Simultaneously, pose values ​​are assigned to the virtual head and virtual hands in the human skeletal model. In this embodiment, the sensors can be one or more combinations of inertial sensors, motion capture sensors, or other suitable sensors to sense the spatial position, rotation angle, and movement state of the target person's head and hands. The acquisition of pose data from sensors is not limited to the method of connecting VR headsets and controllers to a computer via wired or wireless connection. As long as pose data from sensors can be acquired, it is acceptable. The specific method can be adjusted according to the actual situation and specific needs. No specific limitation is made here.

[0028] Next, based on the pose constraints, the unknown virtual upper limb movements are solved on the assigned human skeletal model to obtain the predicted pose data of the virtual upper limbs, and then the pose is assigned to the human skeletal model. In this step, a set of initial pose data is first randomly assigned to the virtual upper limb joints, such as the pose data of the virtual wrist, elbow, and shoulder joints when the human body is upright and motionless. Based on this set of initial pose data, the pose data of the virtual head and virtual hand in the human skeletal model can be obtained through forward kinematics calculation, and this is defined as the second data. Then, the position error E is obtained by calculating the sum of the squared differences between the two head spatial position data, the two left hand spatial position data, and the two right hand spatial position data in the first and second data. pThe value of ; in the first data and the second data, the rotation error E is obtained by calculating the sum of the squared differences between the two head rotation angle data, the squared differences between the two left-hand rotation angle data, and the squared differences between the two right-hand rotation angle data. r The value of . Then the position error E p and rotational error E r Substitute the value of into the following formula to solve for the total error F: F=u p ·E p +u r ·E r +u c ·E c Where F is the total error, E p E represents the position error. r E represents the rotational error. c To constrain punishment, u p u r u c Position error E p Rotational error E r and constraint punishment E c The weight values ​​are then calculated. Finally, an inverse kinematics solver based on particle swarm optimization is used to solve for the initial pose data that satisfies the total error F being less than a preset error threshold. This initial pose data is the virtual upper limb predicted pose data. It is worth noting that the virtual upper limb predicted pose data is a set of pose data including the pose data of all virtual upper limb joints, such as the virtual wrist joint, virtual elbow joint, and virtual shoulder joint, rather than the pose data of a single virtual upper limb joint. The specific value of the preset error threshold can be set according to actual needs. For example, in this embodiment, the preset error threshold is 0.1. That is, when solving for the virtual upper limb predicted pose data using the inverse kinematics solver based on particle swarm optimization, the initial pose data that satisfies the total error F < 0.1 is the virtual upper limb predicted pose data.

[0029] When the initial pose data satisfies the pose constraints of the human skeletal model, the constraint penalty is 0. When the initial pose data does not satisfy the pose constraints, the constraint penalty is a positive number, and the greater the degree to which the initial pose data violates the pose constraints, the larger the constraint penalty. During calculation, u can be first... p and u r Set the value of u to a larger value. c The value is set to a smaller value, for example, it can be u. p =1, u r =0.9, u c=0.1, this weighting ratio allows the initial pose data to quickly approximate the virtual upper limb predicted pose data that satisfies the condition that the total error F is less than a preset error threshold. Then, the position error E... p Rotational error E r and constraint punishment E c weight value u p u r u c Set to three fixed values, position error E p Rotational error E r and constraint punishment E c weight value u p u r u c The settings can be configured according to actual needs to reflect the importance of spatial position, rotation angle and pose constraints during the calculation process, so as to select the optimal solution from multiple possible solutions based on the weight ratio.

[0030] The process of solving the virtual upper limb predicted pose data using a particle swarm optimization-based inverse kinematics solver can be understood as a process of multiple iterative calculations. During the calculation process, multiple particles will comprehensively adjust the input value of the next calculation based on the output value of the iteration process, so that the input value is more likely to satisfy the solution that the total error F is less than the preset error threshold. The iterative calculation ends when the output F is less than the preset error threshold. The initial pose data corresponding to the condition that F is less than the preset error threshold is the virtual upper limb predicted pose data.

[0031] Next, a virtual lower limb motion library was constructed. This library can include lower limb movements such as "standing upright," "walking," "squatting," "standing on an obstacle," and "jumping." Specifically, the types and number of movements in the virtual lower limb motion library can be set according to actual needs and are not specifically limited here. Environmental constraints are set within the virtual environment containing the human skeleton model. During the generation of virtual human movements, different environmental constraints will correspondingly call upon and retrieve different virtual lower limb movements from the virtual lower limb motion library, resulting in different virtual lower limb predicted pose data. Similarly, different head and hand pose data of the target person acquired from sensors (i.e., different initial data) will also correspondingly call upon and retrieve different virtual lower limb movements from the virtual lower limb motion library, resulting in different virtual lower limb predicted pose data. Specifically, the correspondence between environmental constraints and the first data, and the virtual lower limb movements in the virtual lower limb movement library, can be set according to actual needs. Environmental constraints can be prioritized, the first data can be prioritized, or both can be considered simultaneously. For example, environmental constraints can be prioritized. When the environmental constraint is identified as an "obstacle," if the obstacle's height is less than 1m, the "standing on the obstacle" virtual lower limb movement from the virtual lower limb movement library is invoked; if the obstacle's height is greater than 1m, the "standing upright" virtual lower limb movement from the virtual lower limb movement library is invoked. The "standing on the obstacle" or "standing on the obstacle" virtual lower limb movement can be used. The virtual lower limb pose data for the "standing" action is the virtual lower limb predicted pose data. When the environmental constraint is identified as "no obstacles" and the pose data of the target person's hand in the first data constitutes a "waving" action, the virtual lower limb action of "walking" in the virtual lower limb action library is called, and the virtual lower limb pose data of the "walking" action is the virtual lower limb predicted pose data. When the environmental constraint is identified as "no obstacles" and the position data of the target person's head in the first data becomes lower than 1m, the virtual lower limb action of "squatting" in the virtual lower limb action library is called, and the virtual lower limb pose data of the "squatting" action is the virtual lower limb predicted pose data.

[0032] Finally, based on the pose data of the target person obtained by the sensor in the above steps, the virtual upper limb predicted pose data calculated by the inverse kinematics solver, and the virtual lower limb predicted pose data obtained from the virtual lower limb motion library, the full-body motion of the virtual human body is generated and output.

[0033] It should be noted that the above-described virtual human motion generation steps only represent the generation of one frame of motion, that is, the process of generating a corresponding frame of virtual human motion after acquiring a certain frame of motion of the target person. By installing sensors on the target person or using other suitable methods to acquire the target person's real-time pose data, each frame of the target person's real-time motion can be acquired. Each frame of the target person's real-time motion can then be used to generate a corresponding frame of virtual human motion using the above-described virtual human motion generation method, thereby achieving continuous generation of virtual human motion and a high degree of consistency with the target person's motion.

[0034] The virtual human motion generation process can be implemented through code programming. The programming tool can be Visual Studio or other suitable programming tools to achieve real-time and efficient calculation of virtual human pose data and synchronous presentation of the target person's real-time motion in the virtual environment.

[0035] The above description is only a partial preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for generating virtual human actions, characterized in that, Includes the following steps: A human skeleton model is constructed, which includes a virtual head, virtual hands, virtual upper limbs, and virtual lower limbs; The pose zero point of the human skeleton model is calibrated, and the pose constraint relationship of the human skeleton model is established. Obtain the pose data of the target person and define it as the first data; Based on the first data, the human skeletal model is assigned a pose value; Based on the pose constraint relationship, the virtual upper limb unknown movement is solved on the assigned human skeleton model, and the pose is assigned to the human skeleton model based on the solved virtual upper limb predicted pose data. Construct a virtual lower limb motion library, and obtain virtual lower limb predicted pose data from the virtual lower limb motion library based on the first data; Based on the first data, the virtual upper limb predicted pose data, and the virtual lower limb predicted pose data, virtual human body movements are generated and output.

2. The virtual human motion generation method according to claim 1, characterized in that, The virtual upper limb includes virtual upper limb joints. The first data includes head pose data and hand pose data. The hand pose data includes left hand pose data and right hand pose data. The pose data includes spatial position data and rotation angle data. The steps for solving the unknown virtual upper limb movements of the human skeleton model include: Initial pose data are assigned to the virtual upper limb joints; Based on the initial pose data of the virtual upper limb joints, the pose data of the virtual head and the virtual hand are obtained through forward kinematics calculation and defined as the second data; The positional error is calculated as the sum of the squared differences between two head spatial position data, two left-hand spatial position data, and two right-hand spatial position data in the first and second data. The rotation error is calculated as the sum of the squared differences between the two head rotation angle data, the squared differences between the two left hand rotation angle data, and the squared differences between the two right hand rotation angle data in the first data and the second data. Substitute the position error and the rotation error into the following formula to solve for the total error F: F=u p ·HAVE BEEN p +u r ·HAVE BEEN r +u c ·HAVE BEEN c Where F is the total error, E p E represents the position error. r E represents the rotational error. c To constrain punishment, u p u r u c Position error E p Rotational error E r and constraint punishment E c The weight value; Adjust the value of the initial pose data, repeat the above steps until the total error F is less than the preset error threshold, and use the corresponding initial pose data as the virtual upper limb predicted pose data.

3. The virtual human motion generation method according to claim 2, characterized in that, The error threshold is 0.

1.

4. The virtual human motion generation method according to claim 2, characterized in that, When the initial pose data satisfies the pose constraint relationship of the human skeleton model, the constraint penalty value is 0. When the initial pose data violates the pose constraint relationship, the constraint penalty value is a positive number. The greater the degree to which the initial pose data violates the pose constraint relationship, the greater the constraint penalty value.

5. The virtual human motion generation method according to claim 2, characterized in that, The weighting relationship between the position error, rotation error, and constraint penalty is u. p ≥u r >u c .

6. The virtual human motion generation method according to claim 1, characterized in that, The virtual lower limb motion library includes at least the actions of "standing upright", "walking", "squatting" and "standing on an obstacle".

7. The virtual human motion generation method according to claim 6, characterized in that, In the step of obtaining the virtual lower limb predicted pose data from the virtual lower limb action library, when the pose data of the hand in the first data constitutes a "waving hand" action, the "walking" action in the virtual lower limb action library is invoked.

8. The virtual human motion generation method according to claim 6, characterized in that, In the step of obtaining the predicted pose data of the virtual lower limb from the virtual lower limb action library, when the spatial position data of the head pose data becomes lower than 1m, the "squat" action in the virtual lower limb action library is invoked.

9. The virtual human motion generation method according to claim 6, characterized in that, The human skeleton model is constructed in a virtual environment with environmental constraints. Based on the environmental constraints, the predicted pose data of the virtual lower limbs is obtained from the virtual lower limb motion library.

10. The virtual human motion generation method according to claim 9, characterized in that, In the step of obtaining the virtual lower limb predicted pose data from the virtual lower limb action library, when the environmental constraint is identified as an "obstacle", if the height of the obstacle is less than 1m, the "standing on the obstacle" action in the virtual lower limb action library is invoked; if the height of the obstacle is greater than 1m, the "standing upright" action in the virtual lower limb action library is invoked.

Images