Virtual image driving method and related equipment

Abstract

The invention provides a virtual image driving method. The method comprises the following steps: acquiring three-dimensional coordinates of foot key points of a human body in an image at a first moment and a second moment; respectively projecting the three-dimensional coordinates of the foot key points at the first moment and the second moment to a two-dimensional plane to obtain two-dimensional coordinates of the foot key points at the first moment and the second moment; determining the grounding state of the foot of the human body at the second moment based on the two-dimensional coordinates of the foot key points at the first moment and the second moment; determining the position offset of the root node of the virtual image at the second moment based on the grounding state of the foot of the human body at the second moment and the three-dimensional coordinates of the foot key points at the first moment and the second moment; and adjusting the position parameter of the root node of the virtual image at the second moment based on the position offset of the root node of the virtual image at the second moment. Based on the method, the invention further provides a virtual image driving device, a storage medium and a program product.

Description

Technical Field

[0001] The present disclosure relates to the field of Internet technology, and in particular to a virtual image driving method, device, storage medium and program product. Background Art

[0002] In the three-dimensional (3D) human body full-body driving scenario, the 3D coordinates of the human body key points and the position parameters of the human body root nodes output by the 3D human body key point detection model are usually inaccurate and unstable. As a result, the virtual image driven by human body movements may slip, float above the ground, or sink into the ground during its movements, which greatly affects the user experience. Summary of the invention

[0003] In view of this, an embodiment of the present disclosure provides a method for driving a virtual image, which detects the grounding state of the human body's feet in the acquired image, and adjusts the position of the root node of the virtual image driven by the human body based on the detected grounding state of the human body's feet, thereby achieving the purpose of stably placing the feet in the ground state, thereby avoiding the virtual image from sliding or the feet sinking into the ground during the action, thereby improving the user experience.

[0004] The virtual image driving method described in the embodiment of the present disclosure may include: obtaining the three-dimensional coordinates of the key points of the feet of the human body in the image at the first moment and the second moment; projecting the three-dimensional coordinates of the key points of the feet at the first moment and the second moment onto a two-dimensional plane respectively to obtain the two-dimensional coordinates of the key points of the feet at the first moment and the second moment; determining the grounding state of the human body's feet at the second moment based on the two-dimensional coordinates of the key points of the feet at the first moment and the second moment; determining the position offset of the root node of the virtual image at the second moment based on the grounding state of the human body's feet at the second moment and the three-dimensional coordinates of the key points of the feet at the first moment and the second moment; and adjusting the position parameters of the root node of the virtual image at the second moment based on the position offset.

[0005] In an embodiment of the present disclosure, projecting the three-dimensional coordinates of the foot key points at the first moment and the second moment onto a two-dimensional plane respectively includes: projecting the three-dimensional coordinates of the foot key points at the first moment and the second moment respectively onto a two-dimensional plane composed of a horizontal direction and a vertical direction.

[0006] In an embodiment of the present disclosure, determining the ground contact state of the human body's foot at the second moment based on the two-dimensional coordinates of the foot key points at the first moment and the second moment includes: determining the moving speed of the foot key points at the second moment based on the two-dimensional coordinates of the foot key points at the first moment and the second moment; and determining the ground contact state of the human body's foot at the second moment based on the moving speed.

[0007] In an embodiment of the present disclosure, determining the moving speed of the foot key point at the second moment includes: determining the distance moved by the foot key point from the first moment to the second moment on the two-dimensional plane based on the two-dimensional coordinates of the foot key point at the first moment and the second moment; determining the distance moved by the foot key point in unit time based on the difference between the first moment and the second moment and the distance; and using the distance moved in the unit time as the moving speed of the foot key point.

[0008] In an embodiment of the present disclosure, determining the ground contact state of the foot of the human body at the second moment based on the moving speed includes: comparing the moving speed of the key point of the foot with the foot ground contact threshold, and in response to determining that the moving speed of the key point of the foot is greater than the foot ground contact threshold, determining that the foot where the key point of the foot is located is in a non-ground contact state at the second moment; and in response to determining that the moving speed of the key point of the foot is less than or equal to the foot ground contact threshold, determining that the foot where the key point of the foot is located is in a ground contact state at the second moment.

[0009] In an embodiment of the present disclosure, determining the ground contact state of the human body's foot at the second moment based on the moving speed further includes: acquiring a depth parameter of the human body in the image; based on the depth parameter, projecting a preset average human body height parameter onto the two-dimensional plane to obtain a projected height; and taking the product of the projected height and a preset first ratio as the foot ground contact threshold.

[0010] In an embodiment of the present disclosure, determining the grounding state of the feet of the human body at the second moment based on the moving speed further includes: determining whether both feet of the human body in the image are in a grounding state or whether both are jumping, and in response to determining that both feet of the human body in the image are jumping, setting both feet of the human body in the image to be in a non-grounding state at the second moment; in response to determining that both feet of the human body in the image are not both grounded and not both jumping, comparing the moving speeds of the key points of the feet of the two feet, setting the foot with a smaller moving speed of the key points of the feet to be in a grounding state at the second moment, and setting the other foot to be in a non-grounding state at the second moment.

[0011] In an embodiment of the present disclosure, determining whether the foot of the human body in the image jumps includes: determining the moving speed of the foot key point in the vertical direction based on the two-dimensional coordinates of the foot key point at the first moment and the second moment; comparing the moving speed of the foot key point in the vertical direction with a foot jumping threshold, and in response to determining that the moving speed of the foot key point in the vertical direction is greater than the foot jumping threshold, determining that the foot where the foot key point is located has jumped; and in response to determining that the moving speed of the foot key point in the vertical direction is less than or equal to the foot jumping threshold, determining that the foot where the foot key point is located has not jumped.

[0012] In an embodiment of the present disclosure, determining whether the feet of a human body in the image jump up further includes: acquiring a depth parameter of the human body in the image; based on the depth parameter, projecting a preset average height parameter of the human body onto the two-dimensional plane to obtain a projected height; and taking the product of the projected height and a preset second ratio as the foot jumping threshold.

[0013] In an embodiment of the present disclosure, determining the position offset of the root node of the virtual image at the second moment based on the grounding state of the human body's feet at the second moment and the three-dimensional coordinates of the key points of the feet at the first moment and the second moment includes: in response to determining that both feet of the human body in the image are in the grounding state at the second moment, determining the three-dimensional coordinates of the midpoints of the two feet at the first moment and the second moment respectively based on the three-dimensional coordinates of the key points of the feet at the first moment and the second moment, and using the difference between the three-dimensional coordinates of the midpoints of the two feet at the first moment and the second moment as the position offset of the root node of the virtual image at the second moment; in response to determining that both feet of the human body in the image are in the non-grounding state at the second moment, determining the three-dimensional coordinates of the midpoints of the two feet at the first moment and the second moment based on the three-dimensional coordinates of the key points of the feet at the first moment and the second moment The position offset of the root node of the virtual image at the first moment determines the position offset of the root node of the virtual image at the second moment; in response to determining that the first foot of the human body in the image is in a grounded state at the second moment and the second foot is in a non-grounded state at the second moment, the difference between the three-dimensional coordinates of the key points of the foot of the first foot at the first moment and the second moment is used as the position offset of the root node of the virtual image at the second moment; and in response to determining that the second foot of the human body in the image is in a grounded state at the second moment and the first foot is in a non-grounded state at the second moment, the difference between the three-dimensional coordinates of the key points of the foot of the second foot at the first moment and the second moment is used as the position offset of the root node of the virtual image at the second moment.

[0014] In an embodiment of the present disclosure, determining the position offset of the root node of the virtual image at the second moment based on the position offset of the root node of the virtual image at the first moment includes: taking the product of the position offset of the root node of the virtual image at the first moment and a preset penalty coefficient as the position offset of the root node of the virtual image at the second moment.

[0015] In an embodiment of the present disclosure, determining the position offset of the root node of the virtual image at the second moment based on the grounding state of the human body's feet at the second moment and the three-dimensional coordinates of the key points of the feet at the first moment and the second moment further includes: determining the height of the virtual ground; and in response to determining that at least one of the feet of the human body is in a grounding state at the second moment, adjusting the position offset of the root node of the virtual image at the second moment based on the height of the virtual ground and the three-dimensional coordinates of the key points of the feet at the second moment.

[0016] In an embodiment of the present disclosure, determining the height of the virtual ground includes: acquiring a predetermined number of grounded image frames; wherein at least one of the two feet of a human body in the grounded image frames is in a grounded state; respectively acquiring the three-dimensional coordinates of the key points of the feet in the predetermined number of grounded image frames that are in a grounded state; using the coordinate values ​​in the vertical direction of the three-dimensional coordinates as the ground heights corresponding to each grounded image frame; and determining the height of the virtual ground based on the ground heights corresponding to each grounded image frame.

[0017] In an embodiment of the present disclosure, determining the height of the virtual ground based on the ground heights corresponding to each ground image frame includes: performing sliding averaging on the ground heights corresponding to each ground image frame; and using the obtained sliding average as the height of the virtual ground.

[0018] In an embodiment of the present disclosure, adjusting the position offset of the root node of the virtual image at the second moment based on the height of the virtual ground and the three-dimensional coordinates of the foot key points at the second moment includes: determining the lowest height of both feet at the second moment according to the three-dimensional coordinates of the foot key points at the second moment; determining a correction amount of the position offset of the root node of the virtual image at the second moment based on the height of the virtual ground, the lowest height of both feet at the second moment and the position offset of the root node of the virtual image at the second moment; and adjusting the position offset of the root node of the virtual image at the second moment based on the correction amount.

[0019] In an embodiment of the present disclosure, determining a correction amount for the position offset of the root node of the virtual image at the second moment based on the virtual ground height, the lowest height of the feet at the second moment, and the position offset of the root node of the virtual image at the second moment includes: summing the component of the position offset of the root node of the virtual image at the second moment in the vertical direction with the lowest height of the feet at the second moment to obtain a height difference; and using the difference between the virtual ground height and the height difference as the correction amount for the position offset of the root node of the virtual image at the second moment.

[0020] In an embodiment of the present disclosure, determining the correction amount of the root node offset of the virtual image at the second moment in the vertical direction based on the virtual ground height, the lowest height of the feet at the second moment, and the root node offset of the virtual image at the second moment includes: summing the component of the position offset of the root node of the virtual image at the second moment in the vertical direction with the lowest height of the feet at the second moment to obtain a height difference; taking the product of the height difference and a preset correction speed coefficient as a single-frame correction; and taking the difference between the virtual ground height and the single-frame correction as the correction amount of the position offset of the root node of the virtual image at the second moment.

[0021] In an embodiment of the present disclosure, adjusting the position parameters of the root node of the virtual image at the second moment based on the position offset includes: obtaining the position parameters of the root node of the human body at the second moment; and summing the position parameters of the root node of the human body at the second moment with the position offset to obtain the adjusted position parameters of the root node of the virtual image at the second moment.

[0022] Based on the above method, an embodiment of the present disclosure further provides a virtual image driving device, including:

[0023] An acquisition module, used for acquiring the three-dimensional coordinates of the key points of the feet of the human body in the image at the first moment and the second moment;

[0024] A projection module, used to project the three-dimensional coordinates of the key point of the foot at the first moment and the second moment onto a two-dimensional plane respectively, to obtain the two-dimensional coordinates of the key point of the foot at the first moment and the second moment;

[0025] A grounding detection module, used to determine the grounding state of the foot of the human body at the second moment based on the two-dimensional coordinates of the key points of the foot at the first moment and the second moment;

[0026] a position offset determination module, configured to determine a position offset of a root node of the avatar at the second moment based on a grounding state of the foot of the human body at the second moment and three-dimensional coordinates of key points of the foot at the first moment and the second moment; and

[0027] A position parameter adjustment module is used to adjust the position parameter of the root node of the virtual image at the second moment based on the position offset.

[0028] An embodiment of the present disclosure further provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned virtual image driving method when executing the computer program.

[0029] An embodiment of the present disclosure further provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the above-mentioned virtual image driving method.

[0030] An embodiment of the present disclosure further provides a computer program product, including computer program instructions, which, when executed on a computer, enable the computer to execute the above-mentioned virtual image driving method.

[0031] The above-mentioned virtual image driving method and related equipment first project the 3D coordinates of the key points of the foot into more stable and accurate two-dimensional coordinates, which can effectively avoid disturbances in the depth direction, so as to accurately determine the grounding state of the human foot. Next, the above-mentioned method further adjusts the position of the root node of the virtual image driven by the human body movement based on the determined grounding state of the human foot, so as to achieve the purpose of stably placing the foot in the grounding state on the ground, thereby avoiding the virtual image from sliding or sinking into the ground during the action, achieving the virtual image's real sense of stepping on the ground and improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the present disclosure or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0033] Figure 1 A schematic diagram showing a 3D human body full body driving scene according to an embodiment of the present disclosure is shown;

[0034] Figure 2 The implementation process of the virtual image driving method described in some embodiments of the present disclosure is shown;

[0035] Figure 3 The implementation process of the method for determining the grounding state of a foot of a human body at a second moment according to some embodiments of the present disclosure is shown;

[0036] Figure 4 The implementation process of the method for determining the grounding state of a human foot at a second moment based on the moving speed of a key point of a foot on a two-dimensional plane according to some embodiments of the present disclosure is shown;

[0037] Figure 5 The implementation process of the method for correcting the foot grounding state according to some embodiments of the present disclosure is shown;

[0038] Figure 6 The implementation process of the method for determining whether the feet of a human body in an image jumps up according to some embodiments of the present disclosure is shown;

[0039] Figure 7 The implementation process of the method for determining the position offset of the root node of the virtual image at the second moment described in some embodiments of the present disclosure is shown;

[0040] Figure 8 The implementation process of the method for correcting the position offset of the root node of the virtual image at the second moment according to some embodiments of the present disclosure is shown;

[0041] Fig. 9 The implementation process of the method for adjusting the position offset of the root node of the virtual image at the second moment based on the height of the virtual ground and the 3D coordinates of the key points of the feet at the second moment described in some embodiments of the present disclosure is shown;

[0042] Fig.10 A schematic diagram showing the internal structure of a virtual image driven device according to some embodiments of the present disclosure is shown;

[0043] Fig.11 A more specific schematic diagram of the hardware structure of an electronic device described in some embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0044] In order to make the objectives, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0045] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the embodiments of the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0046] It is understandable that before using the technical solutions of each embodiment of the present disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved will be informed to the user in an appropriate manner, and the user's authorization will be obtained.

[0047] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly remind the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can independently choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0048] As an optional but non-limiting implementation, in response to receiving the user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0049] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0050] Figure 1 A schematic diagram of a 3D human body full body driving scene according to an embodiment of the present disclosure is shown. Figure 1In the scene shown, the user can first collect images containing human bodies through the image acquisition device 110. The above-mentioned image generally refers to a video containing multiple image frames. Next, the above-mentioned image acquisition device 110 can transmit the collected image to the 3D human key point detection device 120, and the 3D human key point detection device 120 processes the received image to obtain 3D information of each key point of the human body at each moment. Further, the above-mentioned 3D human key point detection device 120 outputs the 3D information of each key point of the human body at each moment to the virtual image driving device 130, and the virtual image driving device 120 generates a virtual image 140 that can follow the movement of the human body in a right-handed coordinate system based on the 3D information of each key point of the human body at each moment.

[0051] Typically, the 3D human key point detection device 120 may include a 3D human key point detection model. The 3D human key point detection model is mainly used to detect the key points of the human body in the image and output the 3D information of each human key point at each time. Thereafter, the virtual image driving device 120 will further drive various actions of the virtual image according to the 3D information of each human key point at each time output by the 3D human key point detection model.

[0052] As mentioned above, in the above-mentioned 3D human body full-body driving scenario, since the 3D coordinates of each key point in the 3D information of the human body key points output by the 3D human body key point detection model and the position parameters of the root node are usually inaccurate and unstable, especially the coordinates in the depth direction are inaccurate or unstable, the driven virtual image may slip, the feet may float above the ground or sink into the ground during the action, which seriously affects the user experience.

[0053] In order to solve the above problems, an embodiment of the present invention provides a virtual image driving method, which first projects the acquired 3D coordinates of the key points of the human foot onto a two-dimensional plane to obtain two-dimensional coordinates to eliminate the disturbance of the depth direction coordinates; then, based on the projected two-dimensional coordinates, a grounding detection of the human foot is performed to determine the grounding state of the human foot; finally, based on the grounding state of the human foot, the position of the root node of the virtual image driven by the human body is adjusted, thereby achieving the purpose of stably placing the grounded foot on the ground, avoiding the virtual image from sliding or sinking into the ground during the movement, and achieving a real sense of stepping on the ground for the virtual image.

[0054] Figure 2 The implementation process of the virtual image driving method described in the embodiment of the present disclosure is shown. In the embodiment of the present disclosure, the virtual image driving method can be executed by the virtual image driving device 130. Figure 2As shown, the virtual image driving method described in the embodiment of the present disclosure may include the following steps:

[0055] In step 202, the 3D coordinates of the key points of the feet of the human body in the image at the first moment and the second moment are obtained.

[0056] In step 204, the 3D coordinates of the key points of the foot at the first moment and the second moment are projected onto a two-dimensional plane respectively to obtain the two-dimensional coordinates of the key points of the foot at the first moment and the second moment.

[0057] In step 206, the grounding state of the foot of the human body at the second moment is determined based on the two-dimensional coordinates of the key points of the foot at the first moment and the second moment.

[0058] In step 208, the position offset of the root node of the virtual image at the second moment is determined based on the grounding state of the human foot at the second moment and the 3D coordinates of the key points of the foot at the first moment and the second moment.

[0059] In step 210, the position parameter of the root node of the avatar at the second moment is adjusted based on the position offset of the root node of the avatar at the second moment.

[0060] The following further describes in detail each step of the above virtual image driving method in conjunction with the accompanying drawings and specific examples.

[0061] As mentioned above, in a 3D human body full-body driving scene, the 3D human body key point detection model in the above-mentioned 3D human body key point detection device 120 can perform human body key point detection on the image containing the human body captured by the image acquisition device 110, and output the 3D information of each human body key point at each moment.

[0062] Those skilled in the art will appreciate that the above-mentioned key points of the human body may include key points of the head, neck, shoulders, arms, hands, hips, legs, feet, etc. The key points of each human body detected will correspond one-to-one to the key points on the limbs of the virtual image driven by the human body.

[0063] In addition, it can be understood that a tree structure can be formed between the key points of the human body. Usually, one of the key points of the human body can be pre-defined as the root node of the tree. For example, the hip key point among the key points of the human body can be defined as the root node of the tree, and other key points can be used as child nodes of the root node or child nodes of other child nodes. Usually, the position of each key point on the limbs of the virtual image in the above right-hand coordinate system and the rotation posture and other information can be determined based on the 3D information of the root node.

[0064] In some embodiments, the 3D information of each key point of the human body at each moment output by the 3D human key point detection model may include: the quaternion and 3D coordinates of each key point at each moment. The 3D information of each key point of the human body at each moment may also include: the position parameters of the root node at each moment. Among them, the quaternion represents the quaternion of each key point relative to the root node; the 3D coordinates represent the relative coordinates of each key point relative to the root node; and the position parameters of the root node represent the absolute coordinates of the root node in the right-handed coordinate system. In this way, when the position parameters of the root node of the virtual image are determined, the absolute position of each key point in the right-handed coordinate system can be obtained based on the 3D coordinates of each key point. Based on this, it can be understood that when the 3D information of the key point of the human body at a certain moment output by the 3D human key point detection model is inaccurate or unstable, the virtual image may slide, the feet may float on the ground, or the feet may sink.

[0065] Based on the above content, in the above step 202, the 3D coordinates of the key points of the human foot in the image are first obtained. As mentioned above, the 3D coordinates of the key points of the human foot in the above image are contained in the 3D information of each key point of the human body at each time output by the 3D human key point detection model. Therefore, the 3D coordinates of the key points of the human foot can be directly extracted from the 3D information of each key point of the human body at each time output by the above 3D human key point detection model.

[0066] In addition, it can be understood that, usually, the human body includes two feet: a left foot and a right foot. For the convenience of description, when only one of the two feet of the human body is distinguished from the other, and the specific direction of left or right is not distinguished, in the embodiments of the present disclosure, the left foot can be called the first foot; the right foot can be called the second foot. It should be noted that, without loss of generality, the right foot can also be called the first foot, and the left foot can be called the second foot. The embodiments of the present disclosure do not limit which specific foot is the first foot. In addition, it can be understood that in the embodiments of the present disclosure, one or more key points can generally be set on each foot of the human body. The one or more key points set on the foot can be regarded as the key points of the foot described in the embodiments of the present disclosure, and the embodiments of the present disclosure do not limit the key points of the specific position of the foot.

[0067] In addition, it should be noted that in the embodiments of the present disclosure, the above-mentioned first moment and second moment may refer to the moments corresponding to the two continuous images being processed. Normally, the first moment usually refers to the previous moment of the second moment. In some examples, the first moment may also be described as the t-1 moment or the previous moment; and the second moment may be described as the t+1 moment or the current moment. Moreover, since the method described in the embodiments of the present disclosure is repeatedly executed for each moment corresponding to the image frame, when the time moves to the next moment of the aforementioned second moment and the above-mentioned method needs to be executed again, the previous moment of the current moment (that is, the original second moment) can be used as the new first moment, and the current moment (that is, the next moment of the original second moment) can be used as the new second moment.

[0068] For the above step 204, specifically, the 3D coordinates of the above foot key points at the first moment and the second moment can be projected onto a two-dimensional plane composed of a horizontal direction and a vertical direction, respectively. In a right-handed coordinate system, the above two-dimensional plane is a plane composed of a coordinate axis in the X direction and a coordinate axis in the Y direction. In an embodiment of the present disclosure, the above two-dimensional plane composed of a horizontal direction and a vertical direction can also be referred to as a UV plane. As for how to project the 3D coordinates onto the UV plane, it can be determined according to the specific form of the 3D information of each human key point at each moment output by the above 3D human key point detection model, and the embodiment of the present disclosure does not limit this. In an embodiment of the present disclosure, the above two-dimensional coordinates obtained after projection can also be referred to as the UV coordinates of the key points.

[0069] With respect to the above step 206 , the grounding state of the human body's foot at the second moment may be determined based on the two-dimensional coordinates of the key points of the foot at the first moment and the second moment by various methods.

[0070] Figure 3 The implementation process of the method for determining the grounding state of the human foot at the second moment described in some embodiments of the present disclosure is shown. Figure 3 As shown, the above method may include the following steps:

[0071] In step 302 , a moving speed of the foot key point on a two-dimensional plane is determined based on the two-dimensional coordinates of the foot key point at the first moment and the second moment.

[0072] Specifically, in some embodiments, the distance moved by the foot key point from the first moment to the second moment on the above-mentioned two-dimensional plane can be first determined based on the two-dimensional coordinates of the foot key point at the first moment and the second moment; then, the distance moved by the foot key point in unit time can be determined based on the difference between the above-mentioned first moment and the second moment and the above-mentioned distance; finally, the distance moved in the unit time is used as the moving speed of the foot key point.

[0073] In step 304, the ground contact state of the human body's foot at the second moment is determined based on the movement speed.

[0074] Specifically, in some embodiments, the specific implementation method of determining the grounding state of the human foot at the second moment based on the moving speed of the key point of the foot on the two-dimensional plane described in step 304 can be as follows: Figure 4 As shown, specifically including:

[0075] In step 402, the determined moving speed of the foot key points on the two-dimensional plane is compared with a foot contact threshold.

[0076] In step 404 , in response to determining that the moving speed of the key point of the foot is greater than the foot ground contact threshold, it is determined that the foot where the key point of the foot is located is in a non-ground contact state at a second moment.

[0077] In step 406 , in response to determining that the moving speed of the foot key point is less than or equal to the foot ground contact threshold, it is determined that the foot where the foot key point is located is in a ground contact state at a second moment.

[0078] In the above method, the foot grounding state can be detected based on the relationship between the foot movement speed and the foot grounding threshold. When the foot movement speed is high, the foot is determined to be in a non-grounding state, and when the foot movement speed is low, the foot is determined to be in a grounding state.

[0079] However, in the embodiments of the present disclosure, the above-mentioned foot movement speed is the movement speed on the two-dimensional plane after being projected onto the two-dimensional plane. This speed is not only related to the actual movement speed of the human foot, but also to the resolution of the image and the distance (i.e., depth) of the human body in the image. Therefore, it can be understood that in the embodiments of the present disclosure, if a fixed foot grounding threshold is used to judge the foot grounding state, it will bring a large error. Based on the above considerations, in some embodiments of the present disclosure, when executing the above-mentioned Figure 4Before the method shown, the depth parameter of the human body in the image can be first obtained; and then the foot grounding threshold is determined based on the above depth parameter. Specifically, the above depth parameter generally refers to the coordinate value of the human body in the depth direction of the right-hand coordinate system, that is, the Z direction. In some embodiments, the above depth parameter can be determined according to the coordinate value of the Z direction in the position parameter of the root node. In addition, when determining the foot grounding threshold based on the above depth parameter, the preset average height parameter of the human body can be first projected onto the above two-dimensional plane based on the above depth parameter to obtain a projected height; then, the product of the above projected height and the preset first ratio is used as the above foot grounding threshold. For example, the average height parameter of the human body can be set to 1.7m, and the first ratio can be set to 10%. In this way, the above foot grounding threshold is 10% of the projected height obtained by projecting the height of 1.7m onto the two-dimensional plane based on the depth parameter of the human body in the acquired image. It should be noted that the specific values ​​of the above average height parameter and the second ratio are only examples, and the embodiments of the present disclosure are not limited to the above values. Through the flexible setting method of the foot grounding threshold based on the depth information of the human body in the image, the set foot grounding threshold can adapt to images of different sizes or resolutions, and can also adapt to the postures of the human body at different distances. Therefore, the judgment of the foot grounding state based on the foot grounding threshold set by this method can obtain a higher accuracy.

[0080] In addition, it should be noted that, as mentioned above, each foot of the human body may have more than one foot key point. In this case, in the embodiments of the present disclosure, the above-mentioned grounding state detection can be performed according to the multiple foot key points of each foot, and the grounding state of the foot can be determined according to the multiple grounding states obtained. For example, in some examples, if at least one of the multiple grounding states obtained is a grounding state, it can be determined that the foot is in a grounding state. Of course, other judgment strategies can also be adopted according to actual application conditions, and the present disclosure does not limit this.

[0081] Experiments have shown that the above grounding detection strategy has high robustness in most cases. However, when the feet switch quickly between grounding and ungrounding, such as when the human body moves quickly with a small amplitude, the stability is poor. This is mainly because the movement speed of the feet is always fast during fast movement. The above grounding detection method usually leads to the conclusion that the feet are always in an ungrounded state, and it is basically impossible to determine the grounding state of the feet, which ultimately causes the driven virtual image to appear to be suspended visually.

[0082] To address this problem, other embodiments of the present disclosure are described in the above Figure 4 Specifically, after executing the above steps 402-406, the following steps may be further performed: Figure 5The following method of correcting the foot grounding state is shown.

[0083] In step 502, it is determined whether both feet of the human body in the image are in a grounded state or both feet are in a jumping state at the second moment.

[0084] In step 504, in response to determining that both feet of the human body in the image are in a grounded state at the second moment, both feet of the human body in the image are kept in a grounded state at the second moment.

[0085] In step 506, in response to determining that both feet of the human body in the image jump up at the second moment, both feet of the human body in the image are set to be in a non-grounded state at the second moment.

[0086] In step 508 , in response to determining that both feet of the human body in the image are not both grounded and not both jumping at the second moment, the moving speeds of the key points of the feet of the both feet are compared, and then step 510 is executed.

[0087] In step 510, the foot with the slower moving speed of the foot key point is set to be in the ground contact state at the second moment, and the other foot is set to be in the non-ground contact state at the second moment.

[0088] In an embodiment of the present disclosure, the method for determining whether the foot of a person in an image jumps up in step 502 may be as follows: Figure 6 As shown, the following steps are included:

[0089] In step 602, the moving speed of the foot key point in the vertical direction is determined based on the two-dimensional coordinates of the foot key point at the first moment and the second moment.

[0090] Specifically, the moving speed of the key point of the foot in the vertical direction refers to the distance that the key point of the foot moves in the vertical direction per unit time. The moving distance can be determined based on the difference between the two-dimensional coordinates at the first moment and the second moment in the vertical direction, i.e., the Y direction. The moving time can be determined by the difference between the first moment and the second moment.

[0091] In step 604, the moving speed of the foot key points in the vertical direction is compared with the foot jump threshold.

[0092] Similarly, similar to the above-mentioned foot grounding threshold, due to the difference in the resolution of the input image, the position of the human body in the image is also different. Therefore, if a fixed foot jump threshold is used to determine whether the foot jumps, it will also bring a large error. Considering this situation, in some embodiments of the present disclosure, the depth parameter of the human body in the image can also be first obtained; and then the foot jump threshold can be dynamically determined based on the depth parameter. Similarly, when determining the foot jump threshold based on the depth parameter, the preset human average height parameter can be first projected onto the two-dimensional plane based on the depth parameter to obtain a projected height; then, the product of the projected height and the preset second ratio is used as the foot jump threshold. For example, the average height parameter of the human body can also be set to 1.7m, and the second ratio can be set to 6%. In this way, the foot jump threshold is 6% of the projected height obtained by projecting the height of 1.7m onto the two-dimensional plane based on the depth parameter of the human body in the acquired image. It should be noted that the specific values ​​of the average height parameter and the second ratio are only examples, and the embodiments of the present disclosure are not limited to the above-mentioned values. It can be seen that through the flexible setting method of the above-mentioned foot jump threshold, the set foot jump threshold can adapt to images of different sizes or resolutions, and can also adapt to the postures of people at different distances. Therefore, the judgment of whether the foot jumps based on the foot jump threshold set in this way can obtain a higher accuracy.

[0093] In step 606, in response to determining that the moving speed of the foot key point in the vertical direction is greater than the foot jump threshold, it is determined that the foot where the foot key point is located jumps.

[0094] In step 608, in response to determining that the moving speed of the foot key point in the vertical direction is less than or equal to the foot jump threshold, it is determined that the foot where the foot key point is located has not jumped.

[0095] In addition, it should be noted that, as mentioned above, each foot of the human body may have more than one foot key point. In this case, in the embodiments of the present disclosure, the above-mentioned judgment of whether to jump up can be performed separately according to multiple foot key points of each foot, and whether the foot jumps up can be determined according to the obtained judgment results. For example, in some examples, if all of the multiple judgment results of whether to jump up obtained according to multiple foot key points of a foot are jumping up, it can be determined that the foot jumps up. Of course, other judgment strategies can also be adopted according to actual application conditions, and the present disclosure does not limit this.

[0096] Through the above Figure 6 The method shown can determine whether a certain foot of a human body jumps up. After performing the above judgment on the two feet of the human body respectively, it can be determined whether both feet of the human body jump up.

[0097] It can be seen that through Figure 5 The foot grounding state correction method shown can set at least one foot of the human body to be in a grounding state when not both feet of the human body in the image are jumping, thereby generating an accurate, stable and continuous grounding state, thereby effectively avoiding the problem that when the human body moves quickly and with a small amplitude, the grounding state of the feet cannot be determined, causing the virtual image to be visually suspended.

[0098] With respect to the above step 208, in an embodiment of the present disclosure, a plurality of methods may be used to determine the position offset of the root node of the avatar at the second moment.

[0099] In the embodiment of the present disclosure, the position offset of the root node of the virtual image at the second moment refers to the offset between the position parameter of the root node of the virtual image at the second moment output by the 3D human key point detection model obtained by prediction and the real position parameter of the root node of the virtual image at the second moment. If the prediction of the position offset is accurate, the position parameter of the root node of the virtual image at the second moment is compensated based on the position offset so that the position parameter of the root node of the virtual image finally outputted is close to the real value, thereby effectively avoiding the virtual image from sliding, floating or sinking.

[0100] Specifically, Figure 7 The implementation process of the method for determining the position offset of the root node of the virtual image at the second moment described in some embodiments of the present disclosure is shown. Figure 7 As shown, the method may include:

[0101] In step 702, the grounding state of both feet of the human body in the image at the second moment is determined.

[0102] If both feet of the human body in the image are in a grounded state at the second moment, step 704 is executed.

[0103] If both feet of the human body in the image are in a non-grounded state at the second moment, step 708 is executed.

[0104] If the first foot of the human body in the image is in a grounded state at the second moment, and the second foot is in a non-grounded state at the second moment, step 710 is executed.

[0105] If the second foot of the human body in the image is in a grounded state at the second moment, and the first foot is in a non-grounded state at the second moment, step 712 is executed.

[0106] In step 704 , the 3D coordinates of the midpoints of both feet at the first moment and the second moment are determined based on the 3D coordinates of the key points of the feet at the first moment and the second moment, respectively, and then step 706 is executed.

[0107] In an embodiment of the present disclosure, the 3D coordinates of the midpoints of the two feet at a certain moment can be obtained by averaging the 3D coordinates of the key points of the two feet at that moment.

[0108] In step 706, the difference between the 3D coordinates of the midpoints of both feet at the first moment and the second moment is used as the position offset of the root node of the virtual image at the second moment.

[0109] In step 708 , the position offset of the root node of the avatar at the second moment is determined based on the position offset of the root node of the avatar at the first moment.

[0110] Specifically, in an embodiment of the present disclosure, the product of the root node offset of the avatar at the first moment and the preset penalty coefficient can be used as the root node offset of the avatar at the second moment. In some specific examples, the penalty coefficient can be set to a real number less than 1, for example, 0.99.

[0111] In step 710, the difference between the 3D coordinates of the key point of the first foot at the first moment and the 3D coordinates of the second moment is used as the position offset of the root node of the virtual image at the second moment.

[0112] In step 712, the difference between the 3D coordinates of the key point of the second foot at the first moment and the 3D coordinates of the second moment is used as the position offset of the root node of the virtual image at the second moment.

[0113] It can be seen that the above method can determine the position offset of the root node of the avatar based on the 3D coordinates of the key points of the foot in the grounded state, so the accuracy of the position offset of the root node of the avatar can be guaranteed. The above method mainly calculates the position offset of the root node from the horizontal direction, that is, the X direction, and finally compensates the position offset to the root node, thereby effectively alleviating the situation where the avatar slips.

[0114] In addition, in the above method, when both feet are not touching the ground, the position offset of the root node of the virtual image at the second moment is determined based on the position offset of the root node of the virtual image at the first moment, so that the continuity of the predicted root node position can still be maintained when not touching the ground. Further, by increasing the penalty coefficient, the horizontal error accumulation of the position parameter of the root node of the virtual image can be prevented, thereby causing the virtual image to gradually move out of the screen.

[0115] In addition, when the human body's feet switch from an ungrounded state to a grounded state, if there is an error in the height information in the position parameter of the root node of the avatar, the plane where the avatar's feet are grounded will change in height, thus causing the avatar's feet to sink or float. Figure 7Based on the method shown above, we further add Figure 8 The method shown in FIG. 1 is to correct the position offset of the root node of the virtual image at the second moment. Figure 8 As shown, the method may include the following steps.

[0116] In step 802, the height of the virtual ground is determined.

[0117] In step 804, in response to determining that at least one of the feet of the human body is in a grounded state at the second moment, the position offset of the root node of the virtual image at the second moment is adjusted based on the height of the virtual ground and the 3D coordinates of the key points of the foot at the second moment.

[0118] Specifically, in an embodiment of the present disclosure, determining the height of the virtual ground in step 802 may include: first, obtaining a predetermined number of grounded image frames. Then, respectively obtaining the 3D coordinates of the key points of the feet of the feet in the predetermined number of grounded image frames, and using the vertical coordinate values ​​of the 3D coordinates as the ground heights corresponding to the respective grounded image frames. Finally, determining the height of the virtual ground based on the ground heights corresponding to the respective grounded image frames.

[0119] In the embodiment of the present disclosure, at least one of the two feet of the human body is in a grounded state in the grounded image frame. That is, in the embodiment of the present disclosure, the image frame in which at least one of the two feet of the human body is in a grounded state is called a grounded image frame.

[0120] In addition, in the embodiments of the present disclosure, the above-mentioned predetermined number can be set according to actual needs, for example, it can be set to 60 or other values.

[0121] In the above step 802, for the above predetermined number of grounded image frames, the coordinate value in the Y direction of the 3D coordinates of the key points of the feet in the grounded state, that is, the height value of the key points of the feet in the vertical direction, can be extracted as the ground height corresponding to each grounded image frame.

[0122] In an embodiment of the present disclosure, the height of the virtual ground can be determined by averaging or sliding the ground heights corresponding to the above-mentioned ground image frames. The sliding average method can specifically include: first, setting the sliding window width and the weight values ​​corresponding to each element in the sliding window; then, preventing the sliding window from reaching the beginning of a sequence consisting of the ground heights corresponding to a predetermined number of ground image frames; next, calculating the weighted average of each element in the sliding window, and replacing the last element in the sliding window with the calculated average, and then sliding the sliding window to the next group of elements; and repeating the above calculation and sliding steps until sliding to the last element of the sequence. The final weighted average value can be considered as the height of the virtual ground. For example, assuming that the width of the sliding window is 2, the weight values ​​of the two elements in the window are 0.7 and 0.3. Then, for a sequence {a1, a2, ..., an} consisting of the ground heights corresponding to a predetermined number of ground image frames, the sliding window is first set at the beginning of the sequence [a1, a2], and the weighted average value a2'=a1*0.7+a2*0.3 of the two elements a1 and a2 in the window is calculated first; then, the value of a2 in the sequence is replaced by the weighted average value a2', and the sliding window is moved to [a2', a3]; then, the weighted average value a3'=a2'*0.7+a3*0.3 of the two elements a2' and a3 in the window is calculated; then, the value of a3 in the sequence is replaced by the weighted average value a3', and the sliding window is moved to [a3', a4]; ...; repeat the above process until the sliding window moves to [a(n-1)', an]. At this time, the weighted average value an'=a(n-1)'*0.7+an*0.3 of the two elements a(n-1)' and an in the window is calculated. Finally, the calculated weighted average value an' is the height of the virtual ground.

[0123] With respect to the above step 804, in the embodiment of the present disclosure, Fig. 9 The method shown adjusts the position offset of the root node of the virtual image at the second moment based on the height of the virtual ground and the 3D coordinates of the key points of the feet at the second moment. The method may specifically include:

[0124] In step 902, the lowest height of both feet at the second moment is determined according to the 3D coordinates of the key points of the feet at the second moment.

[0125] In step 904, a correction amount of the position offset of the root node of the avatar at the second moment is determined based on the virtual ground height, the lowest height of both feet at the second moment, and the position offset of the root node of the avatar at the second moment.

[0126] In step 906, the position offset of the root node of the avatar at the second moment is adjusted based on the correction amount.

[0127] Specifically, the above step 904 may include: first, summing the component of the position offset of the root node of the virtual image at the second moment in the vertical direction and the lowest height of both feet at the second moment to obtain the height difference; then, taking the difference between the virtual ground height and the height difference as the correction amount of the position offset of the root node of the virtual image at the second moment.

[0128] Specifically, assuming that the virtual ground height is represented by h ground Indicates; the lowest height of the feet at the second moment is h t Indicates; the position offset of the root node of the virtual image at the second moment is represented by offset t Indicates. Among them, the position offset of the root node of the virtual image at the second moment is t Contains components in three directions (x, y, z), and offset t.y represents the component of the position offset of the root node of the avatar at the second moment in the vertical direction. In the embodiment of the present disclosure, the correction amount diff of the position offset of the root node of the avatar at the second moment is ground It can be determined by the following calculation expression (1).

[0129] diff ground =h ground -(h t +offset t.y ) (1)

[0130] Thus, in the above step 906, the position offset of the root node of the virtual image at the second moment can be adjusted by the following calculation expression (2).

[0131] offset t.y =offset t.y +diff ground (2)

[0132] In order to avoid jitter or discontinuity in the virtual image due to excessive correction, a correction speed coefficient v can be further set. ground At this time, the above step 904 may include: first, summing the vertical component of the position offset of the root node of the virtual image at the second moment and the lowest height of the two feet at the second moment to obtain the height difference; then, taking the product of the height difference and the preset correction speed coefficient as the single-frame correction; finally, taking the difference between the virtual ground height and the single-frame correction as the correction of the position offset of the root node of the virtual image at the second moment. That is, in the above embodiment, the correction of the position offset of the root node of the virtual image at the second moment is diff. groundIt can be determined by the following calculation expression (3).

[0133] diff ground =h ground -(h t +offset t.y )×v ground (3)

[0134] With respect to the above step 210, the position parameters of the root node of the virtual image at the second moment can be adjusted based on the above position offset by the following method. First, the position parameters of the root node of the human body at the second moment are obtained. As mentioned above, the position parameters of the root node of the human body at the second moment can be directly extracted from the 3D information of the human body key points output by the 3D human body key point detection model. Next, the position parameters of the root node of the human body at the second moment are summed with the above position offset to obtain the adjusted position parameters of the root node of the virtual image at the second moment.

[0135] It can be seen that the above method can determine the position offset of the human body root node based on the grounding state of the human body's feet, and adjust the position parameters of the human body root node based on the determined position offset, so as to avoid the virtual image from sliding, the feet floating up or the feet sinking.

[0136] Specifically, as mentioned above, the virtual image slides, the feet float up, or the feet sink, etc., mostly because the 3D information of the human body key points output by the 3D human body key point detection model is inaccurate or unstable. The inaccurate or unstable 3D information of the human body key points output by the 3D human body key point detection model is mainly reflected in the depth direction, that is, the Z direction. Therefore, in the embodiment of the present disclosure, the 3D coordinates of the foot key points are projected into more stable and accurate two-dimensional coordinates, that is, UV coordinates, which can effectively avoid disturbances in the Z direction, so that the grounding state of the human body's feet can be accurately determined. After determining the grounding state of the human body's feet, the position of the grounded foot can be kept stable by moving the root node, so as to achieve a real sense of stepping on the ground for the virtual image.

[0137] Furthermore, the above method can also introduce the error accumulation of the grounding height, by establishing a virtual ground and further correcting the position offset of the root node in the vertical direction, to further effectively prevent the foot from sinking or floating. The lateral error accumulation during continuous jumping can be solved by increasing the lateral penalty coefficient.

[0138] It can be seen that the above scheme is simple and direct. While achieving stable grounding of the feet, it can also effectively avoid phenomena such as discontinuity and jumps in leg postures.

[0139] Corresponding to the above method, the embodiment of the present disclosure further discloses a virtual image driving device, the internal structure of which can be as follows: Fig.10 As shown, including:

[0140] An acquisition module 1010 is used to acquire the three-dimensional coordinates of the key points of the feet of the human body in the image at the first moment and the second moment;

[0141] A projection module 1020, configured to project the three-dimensional coordinates of the key point of the foot at the first moment and the second moment onto a two-dimensional plane, respectively, to obtain the two-dimensional coordinates of the key point of the foot at the first moment and the second moment;

[0142] A grounding detection module 1030, configured to determine a grounding state of the foot of the human body at the second moment based on the two-dimensional coordinates of the key points of the foot at the first moment and the second moment;

[0143] a position offset determining module 1040, configured to determine a position offset of a root node of the avatar at the second moment based on a grounding state of the foot of the human body at the second moment and three-dimensional coordinates of key points of the foot at the first moment and the second moment; and

[0144] The position parameter adjustment module 1050 is used to adjust the position parameter of the root node of the virtual image at the second moment based on the position offset.

[0145] The specific implementation of each of the above modules can refer to the above methods and drawings, and will not be repeated here. For the convenience of description, the above devices are described in various modules according to their functions. Of course, when implementing the present disclosure, the functions of each module can be implemented in the same or more software and / or hardware. The device of the above embodiment is used to implement the corresponding virtual image driving method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0146] Fig.11 A more specific hardware structure diagram of an electronic device provided in this embodiment is shown, and the device may include: a processor 2010, a memory 2020, an input / output interface 2030, a communication interface 2040, and a bus 2050. The processor 2010, the memory 2020, the input / output interface 2030, and the communication interface 2040 are connected to each other in communication within the device through the bus 2050.

[0147] The processor 2010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0148] The memory 2020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 2020 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 2020 and are called and executed by the processor 2010.

[0149] The input / output interface 2030 is used to connect input / output devices to realize information input and output. The input / output devices can be configured in the device as components, or can be externally connected to the device to provide corresponding functions. The input devices can include microphones, various sensors, etc., and the output devices can include displays, speakers, vibrators, indicator lights, etc.

[0150] The communication interface 2040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0151] The bus 2050 includes a path that transmits information between the various components of the device (eg, the processor 2010, the memory 2020, the input / output interface 2030, and the communication interface 2040).

[0152] It should be noted that, although the above device only shows the processor 2010, the memory 2020, the input / output interface 2030, the communication interface 2040 and the bus 2050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0153] The electronic device of the above-mentioned embodiment is used to implement the corresponding virtual image driving method in any of the above-mentioned embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0154] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present disclosure also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the virtual image driving method described in any of the above embodiments.

[0155] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0156] The computer instructions stored in the storage medium of the above embodiments are used to enable the computer to execute the task processing method described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0157] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present disclosure (including the claims) is limited to these examples. Based on the concept of the present disclosure, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present disclosure as described above, which are not provided in detail for the sake of simplicity.

[0158] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present disclosure difficult to understand, the known power / ground connections to the integrated circuit (IC) chips and other components may or may not be shown in the provided figures. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present disclosure difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which the embodiments of the present disclosure will be implemented (that is, these details should be fully within the scope of understanding of those skilled in the art). Where specific details (e.g., circuits) are set forth to describe exemplary embodiments of the present disclosure, it is apparent to those skilled in the art that the embodiments of the present disclosure can be implemented without these specific details or with changes in these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0159] Although the present disclosure has been described in conjunction with specific embodiments of the present disclosure, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0160] The embodiments of the present disclosure are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. A method for driving a virtual image, comprising: Obtain the three-dimensional coordinates of the key points of the human foot in the image at the first moment and the second moment; Projecting the three-dimensional coordinates of the key points of the foot at the first moment and the second moment onto a two-dimensional plane respectively to obtain the two-dimensional coordinates of the key points of the foot at the first moment and the second moment; Determining a grounding state of the foot of the human body at the second moment based on the two-dimensional coordinates of the key points of the foot at the first moment and the second moment; Determine the position offset of the root node of the avatar at the second moment based on the grounding state of the foot of the human body at the second moment and the three-dimensional coordinates of the key points of the foot at the first moment and the second moment; and The position parameter of the root node of the virtual image at the second moment is adjusted based on the position offset.

2. The method according to claim 1, wherein: Projecting the three-dimensional coordinates of the foot key point at the first moment and the second moment onto a two-dimensional plane respectively comprises: The three-dimensional coordinates of the key points of the foot at the first moment and the second moment are projected onto a two-dimensional plane formed by a horizontal direction and a vertical direction respectively.

3. The method according to claim 1, wherein: Determining the grounding state of the foot of the human body at the second moment based on the two-dimensional coordinates of the key point of the foot at the first moment and the second moment includes: Determining a moving speed of the key point of the foot at the second moment based on the two-dimensional coordinates of the key point of the foot at the first moment and the second moment; and The ground contact state of the foot of the human body at the second moment is determined based on the moving speed.

4. The method according to claim 3, wherein: Determining the moving speed of the foot key point at the second moment includes: Determine, based on the two-dimensional coordinates of the key point of the foot at the first moment and the second moment, a distance moved by the key point of the foot on the two-dimensional plane from the first moment to the second moment; Determine the distance moved by the key point of the foot in a unit time based on the difference between the first moment and the second moment and the distance; and The distance moved in the unit time is taken as the moving speed of the foot key point.

5. The method according to claim 3, wherein: Determining the ground contact state of the foot of the human body at the second moment based on the moving speed includes: Compare the moving speed of the foot key point with the foot grounding threshold, In response to determining that the moving speed of the foot key point is greater than the foot ground contact threshold, determining that the foot where the foot key point is located is in a non-ground contact state at the second moment; and In response to determining that the moving speed of the foot key point is less than or equal to the foot ground contact threshold, it is determined that the foot where the foot key point is located is in a ground contact state at the second moment.

6. The method according to claim 5, wherein: Determining the ground contact state of the foot of the human body at the second moment based on the moving speed further includes: Obtaining a depth parameter of a human body in the image; Based on the depth parameter, projecting a preset average human height parameter onto the two-dimensional plane to obtain a projected height; and The product of the projected body height and a preset first ratio is used as the foot contact threshold.

7. The method according to claim 5, wherein: Determining the ground contact state of the foot of the human body at the second moment based on the moving speed further includes: Determine whether both feet of the human body in the image are in a grounded state or are both in a jumping state, In response to determining that both feet of the human body in the image are jumping, setting both feet of the human body in the image to be in a non-grounded state at the second moment; In response to determining that both feet of the human body in the image are not both grounded and not both jumped, the moving speeds of the key points of the two feet are compared, and the foot with the smaller moving speed of the key points of the feet is set to be in a grounded state at the second moment, while the other foot is set to be in a non-grounded state at the second moment.

8. The method according to claim 7, wherein: Determining whether the foot of the human body in the image jumps includes: Determine a moving speed of the key point of the foot in a vertical direction based on the two-dimensional coordinates of the key point of the foot at the first moment and the second moment; Compare the moving speed of the foot key point in the vertical direction with the foot jump threshold, In response to determining that the moving speed of the foot key point in the vertical direction is greater than the foot jump threshold, determining that the foot where the foot key point is located jumps; and In response to determining that the moving speed of the foot key point in the vertical direction is less than or equal to the foot jumping threshold, it is determined that the foot at which the foot key point is located has not jumped.

9. The method according to claim 8, wherein: Determining whether the foot of the human body in the image jumps further includes: Obtaining a depth parameter of a human body in the image; Based on the depth parameter, projecting a preset average human height parameter onto the two-dimensional plane to obtain a projected height; and The product of the projected body height and a preset second ratio is used as the foot jumping threshold.

10. The method according to claim 1, wherein: Determining the position offset of the root node of the virtual image at the second moment based on the grounding state of the foot of the human body at the second moment and the three-dimensional coordinates of the key points of the foot at the first moment and the second moment includes: In response to determining that both feet of the human body in the image are in a grounded state at the second moment, the three-dimensional coordinates of the midpoints of the two feet at the first moment and the second moment are determined based on the three-dimensional coordinates of the key points of the feet at the first moment and the second moment, respectively, and the difference between the three-dimensional coordinates of the midpoints of the two feet at the first moment and the second moment is used as the position offset of the root node of the virtual image at the second moment; In response to determining that both feet of the human body in the image are in a non-grounded state at the second moment, determining a position offset of the root node of the virtual image at the second moment based on a position offset of the root node of the virtual image at the first moment; In response to determining that a first foot of the human body in the image is in a grounded state at the second moment and a second foot is in a non-grounded state at the second moment, taking a difference between three-dimensional coordinates of a key point of the first foot at the first moment and the second moment as a position offset of a root node of the avatar at the second moment; and In response to determining that the second foot of the human body in the image is in a grounded state at the second moment and the first foot is in a non-grounded state at the second moment, the difference between the three-dimensional coordinates of the key points of the second foot at the first moment and the second moment is used as the position offset of the root node of the virtual image at the second moment.

11. The method according to claim 10, wherein: Determining the position offset of the root node of the avatar at the second moment based on the position offset of the root node of the avatar at the first moment includes: The product of the position offset of the root node of the virtual image at the first moment and a preset penalty coefficient is used as the position offset of the root node of the virtual image at the second moment.

12. The method according to claim 10, wherein: Determining the position offset of the root node of the virtual image at the second moment based on the grounding state of the foot of the human body at the second moment and the three-dimensional coordinates of the key points of the foot at the first moment and the second moment further includes: determining the height of the virtual ground; and In response to determining that at least one of the feet of the human body is in a grounded state at the second moment, the position offset of the root node of the virtual image at the second moment is adjusted based on the height of the virtual ground and the three-dimensional coordinates of the key points of the foot at the second moment.

13. The method according to claim 12, wherein: Determining the height of the virtual ground includes: Acquire a predetermined number of grounding image frames; wherein, in the grounding image frames, at least one of the two feet of the human body is in a grounding state; respectively acquiring the three-dimensional coordinates of the key points of the feet of the predetermined number of grounded image frames that are in a grounded state; Using the coordinate values ​​in the vertical direction of the three-dimensional coordinates as the ground height corresponding to each ground image frame; and The height of the virtual ground is determined based on the ground heights corresponding to each ground image frame.

14. The method according to claim 13, wherein: Determining the height of the virtual ground based on the ground heights corresponding to each ground image frame includes: Performing a sliding average on the ground heights corresponding to the ground image frames; and The obtained sliding average value is used as the height of the virtual ground.

15. The method according to claim 12, wherein: Adjusting the position offset of the root node of the virtual image at the second moment based on the height of the virtual ground and the three-dimensional coordinates of the key point of the foot at the second moment includes: Determine the lowest height of both feet at the second moment according to the three-dimensional coordinates of the key points of the feet at the second moment; Determining a correction amount of the position offset of the root node of the avatar at the second moment based on the virtual ground height, the lowest height of the two feet at the second moment, and the position offset of the root node of the avatar at the second moment; and The position offset of the root node of the avatar at the second moment is adjusted based on the correction amount.

16. The method according to claim 15, wherein: Determining a correction amount of the position offset of the root node of the avatar at the second moment based on the virtual ground height, the lowest height of the two feet at the second moment, and the position offset of the root node of the avatar at the second moment includes: Summing the component of the position offset of the root node of the avatar at the second moment in the vertical direction and the lowest height of both feet at the second moment to obtain a height difference; and The difference between the virtual ground height and the height difference is used as a correction amount of the position offset of the root node of the virtual image at the second moment.

17. The method according to claim 15, wherein: Determining a correction amount of the root node offset of the avatar at the second moment in the vertical direction based on the virtual ground height, the lowest height of the two feet at the second moment, and the root node offset of the avatar at the second moment includes: Summing the component of the position offset of the root node of the avatar at the second moment in the vertical direction and the lowest height of both feet at the second moment to obtain a height difference; The product of the height difference and a preset correction speed coefficient is used as a single-frame correction amount; and The difference between the virtual ground height and the single-frame correction amount is used as the correction amount of the position offset of the root node of the virtual image at the second moment.

18. The method according to claim 1, wherein: Adjusting the position parameter of the root node of the avatar at the second moment based on the position offset includes: Obtaining position parameters of the root node of the human body at the second moment; and The position parameter of the root node of the human body at the second moment is summed with the position offset to obtain the adjusted position parameter of the root node of the virtual image at the second moment.

19. A virtual image driving device, comprising: An acquisition module, used for acquiring the three-dimensional coordinates of the key points of the feet of the human body in the image at the first moment and the second moment; A projection module, used to project the three-dimensional coordinates of the key point of the foot at the first moment and the second moment onto a two-dimensional plane respectively, to obtain the two-dimensional coordinates of the key point of the foot at the first moment and the second moment; A grounding detection module, used to determine the grounding state of the foot of the human body at the second moment based on the two-dimensional coordinates of the key points of the foot at the first moment and the second moment; A position offset determination module, used to determine the position offset of the root node of the virtual image at the second moment based on the grounding state of the foot of the human body at the second moment and the three-dimensional coordinates of the key points of the foot at the first moment and the second moment; as well as A position parameter adjustment module is used to adjust the position parameter of the root node of the virtual image at the second moment based on the position offset.

20. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the virtual image driving method as described in any one of claims 1 to 18 when executing the computer program.

21. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the virtual image driving method according to any one of claims 1 to 18.

22. A computer program product, comprising computer program instructions, which, when executed on a computer, enable the computer to execute the virtual image driving method according to any one of claims 1 to 18.

Images