Virtual image driving method and related device

Abstract

Provided in the present disclosure is a virtual image driving method, comprising: acquiring three-dimensional coordinates of a foot key point of a human body in a picture at a first moment and a second moment (202); projecting the three-dimensional coordinates of the foot key point at the first moment and the second moment onto a two-dimensional plane, so as to obtain two-dimensional coordinates of the foot key point at the first moment and the second moment (204); on the basis of the two-dimensional coordinates of the foot key point at the first moment and the second moment, determining a ground contact state of a foot of the human body at the second moment (206); on the basis of the ground contact state of the foot of the human body at the second moment and the three-dimensional coordinates of the foot key point at the first moment and the second moment, determining a position offset of a root node of a virtual image at the second moment (208); and on the basis of the position offset of the root node of the virtual image at the second moment, adjusting a position parameter of the root node of the virtual image at the second moment (210). Further disclosed in the present disclosure are a virtual image driving apparatus, a storage medium and a program product.

Description

Virtual image driving method and related equipment

[0001] This application claims priority to the Chinese invention patent application entitled “Virtual image driving method and related equipment” and application number 202311277653.X, filed on September 28, 2023. The entire contents of that application are incorporated by reference into this application. Technical Field

[0002] The present disclosure relates to the field of Internet technology, and in particular to a virtual image driving method, device, storage medium, and program product. Background Art

[0003] In the three-dimensional (3D) human body full-body driving scenario, the 3D coordinates of the human body key points and the position parameters of the human body root node output by the 3D human body key point detection model are usually inaccurate and unstable. As a result, the virtual image driven by human body movements may slip, float above the ground, or sink into the ground during its movements, which greatly affects the user experience.

[0004] Summary of the Invention

[0005] The virtual image driving method described in the embodiment of the present disclosure may include: obtaining the three-dimensional coordinates of the key points of the feet of the human body in the image at the first moment and the second moment; projecting the three-dimensional coordinates of the key points of the feet at the first moment and the second moment onto a two-dimensional plane respectively to obtain the two-dimensional coordinates of the key points of the feet at the first moment and the second moment; determining the grounding state of the human body's feet at the second moment based on the two-dimensional coordinates of the key points of the feet at the first moment and the second moment; determining the position offset of the root node of the virtual image at the second moment based on the grounding state of the human body's feet at the second moment and the three-dimensional coordinates of the key points of the feet at the first moment and the second moment; and adjusting the position parameters of the root node of the virtual image at the second moment based on the position offset.

[0006] In an embodiment of the present disclosure, projecting the three-dimensional coordinates of the foot key points at the first moment and the second moment onto a two-dimensional plane includes: projecting the three-dimensional coordinates of the foot key points at the first moment and the second moment onto a two-dimensional plane composed of a horizontal direction and a vertical direction.

[0007] In an embodiment of the present disclosure, determining the ground contact state of the human body's foot at the second moment based on the two-dimensional coordinates of the key points of the foot at the first moment and the second moment includes: determining the moving speed of the key points of the foot at the second moment based on the two-dimensional coordinates of the key points of the foot at the first moment and the second moment; and determining the ground contact state of the human body's foot at the second moment based on the moving speed.

[0008] In an embodiment of the present disclosure, determining the moving speed of the foot key point at the second moment includes: determining the distance moved by the foot key point from the first moment to the second moment on the two-dimensional plane based on the two-dimensional coordinates of the foot key point at the first moment and the second moment; determining the distance moved by the foot key point in unit time based on the difference between the first moment and the second moment and the distance; and using the distance moved in the unit time as the moving speed of the foot key point.

[0009] In an embodiment of the present disclosure, determining the ground contact state of the foot of the human body at the second moment based on the movement speed includes: comparing the movement speed of the key point of the foot with a foot ground contact threshold, and in response to determining that the movement speed of the key point of the foot is greater than the foot ground contact threshold, determining that the foot where the key point of the foot is located is in a non-ground contact state at the second moment; and in response to determining that the movement speed of the key point of the foot is less than or equal to the foot ground contact threshold, determining that the foot where the key point of the foot is located is in a ground contact state at the second moment.

[0010] In an embodiment of the present disclosure, determining the ground contact state of the human body's foot at the second moment based on the movement speed further includes: obtaining a depth parameter of the human body in the image; based on the depth parameter, projecting a preset average height parameter of the human body onto the two-dimensional plane to obtain a projected height; and using the product of the projected height and a preset first ratio as the foot ground contact threshold.

[0011] In an embodiment of the present disclosure, determining the grounding state of the feet of the human body at the second moment based on the movement speed further includes: determining whether both feet of the human body in the image are in the grounding state or whether both are jumping, and in response to determining that both feet of the human body in the image are jumping, setting both feet of the human body in the image to be in the non-grounding state at the second moment; in response to determining that both feet of the human body in the image are not both grounded and not both jumping, comparing the movement speeds of the key points of the feet of the two feet, setting the foot with the smaller movement speed of the key points of the feet to be in the grounding state at the second moment, and setting the other foot to be in the non-grounding state at the second moment.

[0012] In an embodiment of the present disclosure, determining whether the foot of the human body in the image jumps includes: determining the moving speed of the foot key point in the vertical direction based on the two-dimensional coordinates of the foot key point at the first moment and the second moment; comparing the moving speed of the foot key point in the vertical direction with a foot jumping threshold, and in response to determining that the moving speed of the foot key point in the vertical direction is greater than the foot jumping threshold, determining that the foot where the foot key point is located has jumped; and in response to determining that the moving speed of the foot key point in the vertical direction is less than or equal to the foot jumping threshold, determining that the foot where the foot key point is located has not jumped.

[0013] In an embodiment of the present disclosure, determining whether the feet of the human body in the image jump up further includes: obtaining a depth parameter of the human body in the image; based on the depth parameter, projecting a preset average height parameter of the human body onto the two-dimensional plane to obtain a projected height; and using the product of the projected height and a preset second ratio as the foot jumping threshold.

[0014] In an embodiment of the present disclosure, determining the position offset of the root node of the virtual image at the second moment based on the grounding state of the human body's feet at the second moment and the three-dimensional coordinates of the key points of the feet at the first moment and the second moment includes: in response to determining that both feet of the human body in the image are in the grounding state at the second moment, determining the three-dimensional coordinates of the midpoints of the two feet at the first moment and the second moment based on the three-dimensional coordinates of the key points of the feet at the first moment and the second moment, and using the difference between the three-dimensional coordinates of the midpoints of the two feet at the first moment and the second moment as the position offset of the root node of the virtual image at the second moment; in response to determining that both feet of the human body in the image are in the non-grounding state at the second moment, determining the three-dimensional coordinates of the midpoints of the two feet at the first moment and the second moment based on the three-dimensional coordinates of the key points of the feet at the first moment and the second moment The position offset of the root node of the virtual image at the first moment determines the position offset of the root node of the virtual image at the second moment; in response to determining that the first foot of the human body in the image is in a grounded state at the second moment and the second foot is in a non-grounded state at the second moment, the difference between the three-dimensional coordinates of the foot key points of the first foot at the first moment and the second moment is used as the position offset of the root node of the virtual image at the second moment; and in response to determining that the second foot of the human body in the image is in a grounded state at the second moment and the first foot is in a non-grounded state at the second moment, the difference between the three-dimensional coordinates of the foot key points of the second foot at the first moment and the second moment is used as the position offset of the root node of the virtual image at the second moment.

[0015] In an embodiment of the present disclosure, determining the position offset of the root node of the virtual image at the second moment based on the position offset of the root node of the virtual image at the first moment includes: taking the product of the position offset of the root node of the virtual image at the first moment and a preset penalty coefficient as the position offset of the root node of the virtual image at the second moment.

[0016] In an embodiment of the present disclosure, determining the position offset of the root node of the virtual image at the second moment based on the grounding state of the human body's feet at the second moment and the three-dimensional coordinates of the key points of the feet at the first moment and the second moment further includes: determining the height of the virtual ground; and in response to determining that at least one of the two feet of the human body is in the grounding state at the second moment, adjusting the position offset of the root node of the virtual image at the second moment based on the height of the virtual ground and the three-dimensional coordinates of the key points of the feet at the second moment.

[0017] In an embodiment of the present disclosure, determining the height of the virtual ground includes: obtaining a predetermined number of ground image frames; wherein, in the ground image frames, at least one of the two feet of the human body is in a grounded state; respectively obtaining the three-dimensional coordinates of the key points of the feet that are in a grounded state in the predetermined number of ground image frames; using the vertical coordinate values ​​of the three-dimensional coordinates as the ground heights corresponding to each ground image frame; and determining the height of the virtual ground based on the ground heights corresponding to each ground image frame.

[0018] In an embodiment of the present disclosure, determining the height of the virtual ground based on the ground heights corresponding to each ground image frame includes: performing a sliding average on the ground heights corresponding to each ground image frame; and using the obtained sliding average as the height of the virtual ground.

[0019] In an embodiment of the present disclosure, adjusting the position offset of the root node of the virtual image at the second moment based on the height of the virtual ground and the three-dimensional coordinates of the foot key point at the second moment includes: determining the lowest height of both feet at the second moment according to the three-dimensional coordinates of the foot key point at the second moment; determining a correction amount of the position offset of the root node of the virtual image at the second moment based on the height of the virtual ground, the lowest height of both feet at the second moment and the position offset of the root node of the virtual image at the second moment; and adjusting the position offset of the root node of the virtual image at the second moment based on the correction amount.

[0020] In an embodiment of the present disclosure, determining a correction amount for the position offset of the root node of the virtual image at the second moment based on the virtual ground height, the lowest height of the feet at the second moment, and the position offset of the root node of the virtual image at the second moment includes: summing the component of the position offset of the root node of the virtual image at the second moment in the vertical direction and the lowest height of the feet at the second moment to obtain a height difference; and using the difference between the virtual ground height and the height difference as the correction amount for the position offset of the root node of the virtual image at the second moment.

[0021] In an embodiment of the present disclosure, determining the correction amount of the root node offset of the virtual image at the second moment in the vertical direction based on the virtual ground height, the lowest height of the feet at the second moment, and the root node offset of the virtual image at the second moment includes: summing the component of the position offset of the root node of the virtual image at the second moment in the vertical direction with the lowest height of the feet at the second moment to obtain a height difference; taking the product of the height difference and a preset correction speed coefficient as a single-frame correction amount; and taking the difference between the virtual ground height and the single-frame correction amount as the correction amount of the position offset of the root node of the virtual image at the second moment.

[0022] In an embodiment of the present disclosure, adjusting the position parameters of the root node of the virtual image at the second moment based on the position offset includes: obtaining the position parameters of the root node of the human body at the second moment; and summing the position parameters of the root node of the human body at the second moment with the position offset to obtain the adjusted position parameters of the root node of the virtual image at the second moment.

[0023] Based on the above method, an embodiment of the present disclosure further provides a device for driving a virtual image, comprising:

[0024] An acquisition module, configured to acquire the three-dimensional coordinates of key points of a foot of a human body in an image at a first moment and a second moment;

[0025] a projection module, configured to project the three-dimensional coordinates of the key points of the foot at the first moment and the second moment onto a two-dimensional plane, respectively, to obtain the two-dimensional coordinates of the key points of the foot at the first moment and the second moment;

[0026] a grounding detection module, configured to determine a grounding state of the foot of the human body at the second moment based on the two-dimensional coordinates of the key points of the foot at the first moment and the second moment;

[0027] a position offset determining module, configured to determine a position offset of a root node of the avatar at the second moment based on the ground contact state of the foot of the human body at the second moment and the three-dimensional coordinates of the key points of the foot at the first moment and the second moment; and

[0028] A position parameter adjustment module is used to adjust the position parameter of the root node of the virtual image at the second moment based on the position offset.

[0029] An embodiment of the present disclosure further provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned virtual image driving method when executing the computer program.

[0030] An embodiment of the present disclosure further provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the above-mentioned virtual image driving method.

[0031] An embodiment of the present disclosure further provides a computer program product, comprising computer program instructions, which, when executed on a computer, enable the computer to execute the aforementioned virtual image driving method. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the present disclosure or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0033] FIG1 shows a schematic diagram of a 3D human body full-body driving scene according to an embodiment of the present disclosure;

[0034] FIG2 shows an implementation process of a virtual image driving method according to some embodiments of the present disclosure;

[0035] FIG3 shows an implementation flow of a method for determining a ground contact state of a human foot at a second moment according to some embodiments of the present disclosure;

[0036] FIG4 shows an implementation flow of a method for determining a ground contact state of a human foot at a second moment based on the moving speed of a key point of the foot on a two-dimensional plane according to some embodiments of the present disclosure;

[0037] FIG5 shows an implementation process of a method for correcting a foot grounding state according to some embodiments of the present disclosure;

[0038] FIG6 shows an implementation flow of a method for determining whether a foot of a human body in an image has jumped, according to some embodiments of the present disclosure;

[0039] FIG7 shows an implementation flow of a method for determining a position offset of a root node of a virtual image at a second moment according to some embodiments of the present disclosure;

[0040] FIG8 shows an implementation flow of a method for correcting a position offset of a root node of a virtual image at a second moment according to some embodiments of the present disclosure;

[0041] FIG9 shows an implementation flow of a method for adjusting a position offset of a root node of an avatar at a second moment based on the height of a virtual ground and the 3D coordinates of a foot key point at the second moment, according to some embodiments of the present disclosure;

[0042] FIG10 shows a schematic diagram of the internal structure of a virtual image driven device according to some embodiments of the present disclosure; and

[0043] FIG11 shows a more specific schematic diagram of the hardware structure of an electronic device according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0044] In order to make the objectives, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0045] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should have the usual meanings understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the embodiments of the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative position relationships. When the absolute position of the described object changes, the relative position relationship may also change accordingly.

[0046] It is understandable that before using the technical solutions of each embodiment of the present disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved will be informed to the user in an appropriate manner, and the user's authorization will be obtained.

[0047] For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operation of the disclosed technical solution based on the prompt message.

[0048] As an optional but non-limiting implementation, in response to a user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. Furthermore, the pop-up window may also contain a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0049] It is understandable that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0050] FIG1 shows a schematic diagram of a 3D human body full-body driving scenario according to an embodiment of the present disclosure. In the scenario shown in FIG1 , a user can first capture an image containing a human body through an image acquisition device 110. The above-mentioned image generally refers to a video containing multiple image frames. Next, the above-mentioned image acquisition device 110 can transmit the captured image to a 3D human body key point detection device 120, which processes the received image to obtain 3D information of each key point of the human body at each moment. Furthermore, the above-mentioned 3D human body key point detection device 120 outputs the 3D information of each key point of the human body at each moment to a virtual image driving device 130, which generates a virtual image 140 in a right-handed coordinate system that can follow the movement of the human body based on the 3D information of each key point of the human body at each moment.

[0051] Typically, the 3D human key point detection device 120 may include a 3D human key point detection model. This 3D human key point detection model is primarily used to detect key points of the human body in an image and output 3D information about each key point at each moment. Subsequently, the avatar driving device 120 further drives various actions of the avatar based on the 3D information about each key point at each moment output by the 3D human key point detection model.

[0052] As mentioned above, in the above-mentioned 3D human body full-body driving scenario, since the 3D coordinates of each key point in the 3D information of the human body key points output by the 3D human body key point detection model and the position parameters of the root node are usually inaccurate and unstable, especially the coordinates in the depth direction are inaccurate or unstable, the driven virtual image will slip, the feet will float above the ground, or the feet will sink into the ground during the movement, which will seriously affect the user experience.

[0053] In order to solve the above problems, the embodiments of the present disclosure provide a virtual image driving method, which first projects the acquired 3D coordinates of the key points of the human foot onto a two-dimensional plane to obtain two-dimensional coordinates to eliminate the disturbance of the depth direction coordinates; then, the grounding detection of the human foot is performed based on the projected two-dimensional coordinates to determine the grounding state of the human foot; finally, the position of the root node of the virtual image driven by the human body is adjusted based on the grounding state of the human foot, thereby achieving the purpose of stably placing the grounded foot on the ground, avoiding the virtual image from sliding or sinking into the ground during the movement, and realizing a real sense of stepping on the ground for the virtual image.

[0054] FIG2 shows the implementation process of the virtual image driving method according to an embodiment of the present disclosure. In an embodiment of the present disclosure, the virtual image driving method can be executed by the virtual image driving device 130. As shown in FIG2, the virtual image driving method according to an embodiment of the present disclosure can include the following steps:

[0055] In step 202 , the 3D coordinates of the key points of the feet of the human body in the image at the first moment and the second moment are obtained.

[0056] In step 204 , the 3D coordinates of the key points of the foot at the first moment and the second moment are projected onto a two-dimensional plane respectively to obtain the two-dimensional coordinates of the key points of the foot at the first moment and the second moment.

[0057] In step 206 , the ground contact state of the foot of the human body at the second moment is determined based on the two-dimensional coordinates of the key points of the foot at the first moment and the second moment.

[0058] In step 208 , the position offset of the root node of the avatar at the second moment is determined based on the grounding state of the human foot at the second moment and the 3D coordinates of the key points of the foot at the first moment and the second moment.

[0059] In step 210 , the position parameter of the root node of the avatar at the second moment is adjusted based on the position offset of the root node of the avatar at the second moment.

[0060] The following further describes in detail each step of the above virtual image driving method with reference to the accompanying drawings and specific examples.

[0061] As mentioned above, in a 3D human body full-body driving scene, the 3D human body key point detection model in the above-mentioned 3D human body key point detection device 120 can perform human body key point detection on images containing the human body captured by the image acquisition device 110, and output 3D information of each human body key point at each moment.

[0062] Those skilled in the art will appreciate that the aforementioned human body key points may include key points on the head, neck, shoulders, arms, hands, hips, legs, feet, etc. The detected key points on each human body will correspond one-to-one with key points on the limbs of the avatar driven by the human body.

[0063] Furthermore, it is understood that the various key points of the human body can form a tree structure. Typically, one of the key points of the human body can be pre-defined as the root node of the tree. For example, the hip key point can be defined as the root node of the tree, while other key points can be defined as child nodes of the root node or child nodes of other child nodes. Typically, the position and rotational posture of each key point on the avatar's limbs in the right-handed coordinate system can be determined based on the 3D information of the root node.

[0064] In some embodiments, the 3D information of each key point of the human body at each moment output by the 3D human key point detection model may include: the quaternion and 3D coordinates of each key point at each moment. The above-mentioned 3D information of each key point of the human body at each moment may also include: the position parameters of the root node at each moment. Among them, the above-mentioned quaternion represents the quaternion of each key point relative to the root node; the above-mentioned 3D coordinates represent the relative coordinates of each key point relative to the root node; and the position parameters of the above-mentioned root node represent the absolute coordinates of the root node in the above-mentioned right-handed coordinate system. In this way, when the position parameters of the root node of the virtual image are determined, the absolute position of each key point in the above-mentioned right-handed coordinate system can be obtained based on the 3D coordinates of the above-mentioned key points. Based on this, it can be understood that when the 3D information of the key points of the human body at a certain moment output by the 3D human key point detection model is inaccurate or unstable, the above-mentioned virtual image may slide, the feet may float above the ground, or the feet may sink.

[0065] Based on the above, in step 202, the 3D coordinates of the key points of the human foot in the image are first obtained. As previously mentioned, the 3D coordinates of the key points of the human foot in the image are contained in the 3D information of each key point at each moment output by the 3D human key point detection model. Therefore, the 3D coordinates of the key points of the human foot can be directly extracted from the 3D information of each key point at each moment output by the 3D human key point detection model.

[0066] In addition, it can be understood that, under normal circumstances, the human body includes two feet: a left foot and a right foot. For the convenience of description, when only one of the two feet of the human body is distinguished from the other, and the specific direction of left or right is not distinguished, in the embodiments of the present disclosure, the left foot can be called the first foot; the right foot can be called the second foot. It should be noted that, without loss of generality, the right foot can also be called the first foot, and the left foot can be called the second foot. The embodiments of the present disclosure do not limit which specific foot is the first foot. In addition, it can be understood that in the embodiments of the present disclosure, one or more key points can usually be set on each foot of the human body. The one or more key points set on the foot mentioned above can be regarded as the key points of the foot described in the embodiments of the present disclosure, and the embodiments of the present disclosure do not limit the key points of the specific position of the foot.

[0067] In addition, it should be noted that in the embodiments of the present disclosure, the above-mentioned first moment and second moment may refer to the moments corresponding to the two consecutive images being processed. Normally, the first moment usually refers to the previous moment of the second moment. In some examples, the first moment may also be described as the t-1 moment or the previous moment; and the second moment may be described as the t+1 moment or the current moment. Moreover, since the method described in the embodiments of the present disclosure is repeatedly executed for each moment corresponding to the image frame, when the time moves to the next moment after the aforementioned second moment and the above-mentioned method needs to be executed again, the previous moment of the current moment (that is, the original second moment) can be used as the new first moment, and the current moment (that is, the next moment of the original second moment) can be used as the new second moment.

[0068] For the above step 204, specifically, the 3D coordinates of the above foot key points at the first moment and the second moment can be projected onto a two-dimensional plane composed of a horizontal direction and a vertical direction, respectively. In a right-handed coordinate system, the above two-dimensional plane is a plane composed of a coordinate axis in the X direction and a coordinate axis in the Y direction. In an embodiment of the present disclosure, the above two-dimensional plane composed of a horizontal direction and a vertical direction can also be referred to as a UV plane. As for how to project the 3D coordinates onto the UV plane, it can be determined based on the specific form of the 3D information of each human key point at each moment output by the above 3D human key point detection model, and the embodiment of the present disclosure does not limit this. In an embodiment of the present disclosure, the above two-dimensional coordinates obtained after projection can also be referred to as the UV coordinates of the key points.

[0069] With respect to the above step 206 , the ground contact state of the human foot at the second moment may be determined based on the two-dimensional coordinates of the key points of the foot at the first moment and the second moment using various methods.

[0070] FIG3 shows an implementation process of a method for determining the ground contact state of a foot of a human body at a second moment according to some embodiments of the present disclosure. As shown in FIG3 , the method may include the following steps:

[0071] In step 302 , a moving speed of the foot key point on a two-dimensional plane is determined based on the two-dimensional coordinates of the foot key point at the first moment and the second moment.

[0072] Specifically, in some embodiments, the distance moved by the foot key point on the above-mentioned two-dimensional plane from the first moment to the second moment can be first determined based on the two-dimensional coordinates of the foot key point at the first moment and the second moment; then, the distance moved by the foot key point per unit time can be determined based on the difference between the above-mentioned first moment and the second moment and the above-mentioned distance; finally, the distance moved per unit time can be used as the movement speed of the foot key point.

[0073] In step 304 , the ground contact state of the human foot at the second moment is determined based on the movement speed.

[0074] Specifically, in some embodiments, the specific implementation method of determining the ground contact state of the human foot at the second moment based on the moving speed of the key points of the foot on the two-dimensional plane described in step 304 may be as shown in FIG. 4 , and specifically includes:

[0075] In step 402 , the determined moving speed of the foot key points on the two-dimensional plane is compared with a foot contact threshold.

[0076] In step 404 , in response to determining that the movement speed of the key point of the foot is greater than the foot ground contact threshold, it is determined that the foot where the key point of the foot is located is in a non-ground contact state at a second moment.

[0077] In step 406 , in response to determining that the movement speed of the foot key point is less than or equal to the foot ground contact threshold, it is determined that the foot where the foot key point is located is in a ground contact state at a second moment.

[0078] In the above method, the foot grounding state can be detected based on the relationship between the foot movement speed and the foot grounding threshold. When the foot movement speed is high, the foot is determined to be in a non-grounding state, and when the foot movement speed is low, the foot is determined to be in a grounding state.

[0079] However, in the embodiments of the present disclosure, the foot movement speed is the speed of movement on the two-dimensional plane after being projected onto it. This speed is not only related to the actual movement speed of the human foot, but also to the image resolution and the proximity (i.e., depth) of the human body in the image. Therefore, it can be understood that in the embodiments of the present disclosure, using a fixed foot contact threshold to determine the foot contact state will result in significant errors. Considering this, in some embodiments of the present disclosure, before executing the method shown in FIG. 4 , a depth parameter of the human body in the image can be first obtained; then, the foot contact threshold can be determined based on this depth parameter. Specifically, the depth parameter generally refers to the coordinate value of the human body in the depth direction, i.e., the Z direction, of a right-handed coordinate system. In some embodiments, the depth parameter can be determined based on the Z direction coordinate value of the root node's position parameter. Furthermore, when determining the foot contact threshold based on the depth parameter, a preset average human height parameter can be first projected onto the two-dimensional plane based on the depth parameter to obtain a projected height. Then, the product of the projected height and a preset first ratio is used as the foot contact threshold. For example, the average height parameter of the human body can be set to 1.7m, and the first ratio can be set to 10%. In this way, the above-mentioned foot grounding threshold is 10% of the projected height obtained by projecting the height of 1.7m onto a two-dimensional plane based on the depth parameter of the human body in the acquired image. It should be noted that the specific values ​​of the above-mentioned average height parameter and the second ratio are only examples, and the embodiments of the present disclosure are not limited to the above-mentioned values. Through the flexible setting method of the above-mentioned foot grounding threshold based on the depth information of the human body in the image, the set foot grounding threshold can be adaptive to images of different sizes or resolutions, and can also be adaptive to the postures of human bodies at different distances. Therefore, the judgment of the foot grounding state based on the foot grounding threshold set in this way can obtain a higher accuracy rate.

[0080] In addition, it should be noted that, as mentioned above, each foot of the human body may have more than one key point. In this case, in the embodiments of the present disclosure, the above-mentioned grounding state detection can be performed separately based on multiple key points of each foot, and the grounding state of the foot can be determined based on the multiple grounding states obtained. For example, in some examples, if at least one of the multiple grounding states obtained is a grounding state, it can be determined that the foot is in a grounding state. Of course, other judgment strategies can also be adopted according to actual application conditions, and this disclosure is not limited to this.

[0081] Experiments have shown that the above grounding detection strategy is highly robust in most situations. However, stability is poor when the feet rapidly switch between grounded and ungrounded states, such as when the human body moves quickly and with small amplitudes. This is primarily because the feet move at a high speed during rapid movement. Using the above grounding detection method, the feet are often deemed ungrounded, making it virtually impossible to determine their grounding status. This ultimately causes the driven avatar to appear to be suspended.

[0082] To address this issue, other embodiments of the present disclosure further modify the foot grounding state based on the solution shown in Figure 4. Specifically, after executing steps 402-406, the following foot grounding state modification method shown in Figure 5 may be further executed.

[0083] In step 502 , it is determined whether both feet of the human body in the image are in a grounded state or both feet are in a jumping state at the second moment.

[0084] In step 504 , in response to determining that both feet of the human body in the image are in a grounded state at the second moment, both feet of the human body in the image are kept in a grounded state at the second moment.

[0085] In step 506 , in response to determining that both feet of the person in the image jump at the second moment, both feet of the person in the image are set to be in a non-grounded state at the second moment.

[0086] In step 508 , in response to determining that both feet of the human body in the image are not both grounded and not both jumping at the second moment, the moving speeds of the key points of the feet of the two feet are compared, and then step 510 is executed.

[0087] In step 510 , the foot with the slower moving speed of the foot key point is set to be in the ground contact state at the second moment, and the other foot is set to be in the non-ground contact state at the second moment.

[0088] In an embodiment of the present disclosure, the method for determining whether the foot of a person in an image jumps in step 502 may be as shown in FIG6 , including the following steps:

[0089] In step 602 , a moving speed of the foot key point in the vertical direction is determined based on the two-dimensional coordinates of the foot key point at the first moment and the second moment.

[0090] Specifically, the vertical movement speed of the foot key point refers to the distance the foot key point moves vertically per unit time. The movement distance can be determined based on the difference in the vertical direction, i.e., the Y direction, between the two-dimensional coordinates at the first moment and the second moment. The movement time can be determined based on the difference between the first moment and the second moment.

[0091] In step 604 , the vertical movement speed of the foot key points is compared with a foot jump threshold.

[0092] Similarly, similar to the aforementioned foot touchdown threshold, due to differences in the resolution of the input images, the position of the human body in the image can also vary. Therefore, using a fixed foot jump threshold to determine whether the foot has jumped will also result in significant errors. Considering this, in some embodiments of the present disclosure, a depth parameter of the human body in the image can be first obtained; then, the foot jump threshold can be dynamically determined based on this depth parameter. Similarly, when determining the foot jump threshold based on the depth parameter, a preset average human height parameter can be first projected onto the two-dimensional plane based on the depth parameter to obtain a projected height. Then, the product of this projected height and a preset second ratio can be used as the foot jump threshold. For example, the average human height parameter can be set to 1.7 meters, and the second ratio can be set to 6%. In this way, the foot jump threshold is 6% of the projected height obtained by projecting a 1.7-meter height onto the two-dimensional plane based on the depth parameter of the human body in the image. It should be noted that the specific values ​​of the average height parameter and the second ratio are merely examples, and the embodiments of the present disclosure are not limited to these values. It can be seen that through the flexible setting method of the above-mentioned foot jumping threshold, the set foot jumping threshold can adapt to images of different sizes or resolutions, and can also adapt to the postures of people at different distances. Therefore, the judgment of whether the foot jumps based on such a set foot jumping threshold can achieve a higher accuracy.

[0093] In step 606 , in response to determining that the moving speed of the foot key point in the vertical direction is greater than the foot jump threshold, it is determined that the foot where the foot key point is located jumps.

[0094] In step 608 , in response to determining that the moving speed of the foot key point in the vertical direction is less than or equal to the foot jump threshold, it is determined that the foot where the foot key point is located has not jumped.

[0095] Furthermore, it should be noted that, as previously mentioned, each foot of the human body may have more than one key point. In this case, in embodiments of the present disclosure, the aforementioned jump determination can be performed based on multiple key points of each foot, and the jump determination for that foot can be determined based on the resulting determination results. For example, in some examples, if all of the multiple jump determination results obtained based on multiple key points of a foot indicate a jump, then the jump determination for that foot can be determined. Of course, other determination strategies can also be employed based on actual application circumstances, and this disclosure does not limit this.

[0096] By using the method shown in FIG6 , it can be determined whether a certain foot of the human body jumps up. After performing the above judgment on both feet of the human body, it can be determined whether both feet of the human body jump up.

[0097] It can be seen that through the foot grounding state correction method shown in Figure 5, when the human body in the image does not have both feet jumping, at least one foot of the human body can be set to be in a grounding state, thereby generating an accurate, stable and continuous grounding state, thereby effectively avoiding the problem that when the human body moves quickly and slightly, the grounding state of the feet cannot be determined, resulting in the virtual image being visually in a suspended state.

[0098] Regarding the above step 208 , in an embodiment of the present disclosure, the position offset of the root node of the avatar at the second moment may be determined by various methods.

[0099] In the embodiments of the present disclosure, the position offset of the root node of the aforementioned avatar at the second moment refers to the offset between the position parameters of the root node of the avatar at the second moment, as output by the predicted 3D human key point detection model, and the actual position parameters of the root node of the avatar at the second moment. If the prediction of the aforementioned position offset is accurate, then compensating the position parameters of the root node of the avatar at the second moment based on the aforementioned position offset can make the position parameters of the root node of the avatar ultimately outputted close to the actual values, thereby effectively preventing the avatar from slipping, floating up, or sinking its feet.

[0100] Specifically, FIG7 shows the implementation process of the method for determining the position offset of the root node of the virtual image at the second moment according to some embodiments of the present disclosure. As shown in FIG7 , the method may include:

[0101] In step 702, the grounding state of both feet of the human body in the image at the second moment is determined.

[0102] If both feet of the human body in the image are in a grounded state at the second moment, step 704 is executed.

[0103] If both feet of the human body in the image are in a non-grounded state at the second moment, step 708 is executed.

[0104] If the first foot of the human body in the image is in a grounded state at the second moment, and the second foot is in a non-grounded state at the second moment, step 710 is executed.

[0105] If the second foot of the human body in the image is in the grounded state at the second moment and the first foot is in the non-grounded state at the second moment, step 712 is executed.

[0106] In step 704 , the 3D coordinates of the midpoints of both feet at the first moment and the second moment are determined based on the 3D coordinates of the key points of the feet at the first moment and the second moment, respectively, and then step 706 is executed.

[0107] In an embodiment of the present disclosure, the 3D coordinates of the midpoints of the two feet at a certain moment can be obtained by averaging the 3D coordinates of the key points of the two feet at that moment.

[0108] In step 706, the difference between the 3D coordinates of the midpoints of the two feet at the first moment and the second moment is used as the position offset of the root node of the virtual image at the second moment.

[0109] In step 708 , the position offset of the root node of the avatar at the second moment is determined based on the position offset of the root node of the avatar at the first moment.

[0110] Specifically, in an embodiment of the present disclosure, the root node offset of the avatar at the first moment multiplied by a preset penalty coefficient can be used as the root node offset of the avatar at the second moment. In some specific examples, the penalty coefficient can be set to a real number less than 1, for example, 0.99.

[0111] In step 710 , the difference between the 3D coordinates of the key point of the first foot at the first moment and the 3D coordinates of the key point of the first foot at the second moment is used as the position offset of the root node of the virtual image at the second moment.

[0112] In step 712, the difference between the 3D coordinates of the key point of the second foot at the first moment and the 3D coordinates of the second moment is used as the position offset of the root node of the virtual image at the second moment.

[0113] This method can determine the position offset of the avatar's root node based on the 3D coordinates of the grounded foot's key points, ensuring the accuracy of the avatar's root node position offset. This method primarily calculates the root node position offset in the horizontal, or X, direction and ultimately compensates this offset to the root node, effectively mitigating the avatar's slipping.

[0114] Furthermore, when both feet are not touching the ground, the above method determines the position offset of the avatar's root node at the second moment based on the position offset of the avatar's root node at the first moment. This ensures that the predicted root node position remains consistent even when the avatar is not touching the ground. Furthermore, by increasing the penalty coefficient, lateral error accumulation in the position parameters of the avatar's root node can be prevented, which could cause the avatar to gradually move out of the frame.

[0115] Furthermore, when a person's foot switches from an ungrounded state to a grounded state, if the height information in the position parameters of the avatar's root node contains errors, the height of the plane on which the avatar's foot touches the ground will change, causing the avatar's foot to sink or float. To address this issue, other embodiments of the present disclosure, based on the method shown in FIG7 , further include a method for correcting the position offset of the avatar's root node at the second moment, as shown in FIG8 . As shown in FIG8 , this method may include the following steps.

[0116] In step 802, the height of the virtual ground is determined.

[0117] In step 804, in response to determining that at least one of the human body's feet is in a grounded state at the second moment, the position offset of the root node of the virtual image at the second moment is adjusted based on the height of the virtual ground and the 3D coordinates of the key points of the foot at the second moment.

[0118] Specifically, in an embodiment of the present disclosure, determining the height of the virtual ground in step 802 may include: first, acquiring a predetermined number of ground-engaging image frames; then, acquiring the 3D coordinates of key points of the feet of the feet that are grounded in the predetermined number of ground-engaging image frames; and using the vertical coordinate values ​​of the 3D coordinates as the ground height corresponding to each ground-engaging image frame; and finally, determining the height of the virtual ground based on the ground height corresponding to each ground-engaging image frame.

[0119] In the embodiment of the present disclosure, at least one of the two feet of the human body is in a grounded state in the grounded image frame. That is, in the embodiment of the present disclosure, an image frame in which at least one of the two feet of the human body is in a grounded state is referred to as a grounded image frame.

[0120] In addition, in the embodiment of the present disclosure, the above-mentioned predetermined number can be set according to actual needs, for example, it can be set to 60 or other values.

[0121] In the above-mentioned step 802, for the above-mentioned predetermined number of grounding image frames, the Y-direction coordinate value of the 3D coordinate of the key point of the foot in the grounding state, that is, the height value of the key point of the foot in the vertical direction, can be extracted as the ground height corresponding to each grounding image frame.

[0122] In an embodiment of the present disclosure, the height of the virtual ground can be determined by averaging or sliding the ground heights corresponding to the above-mentioned ground image frames. The sliding average method can specifically include: first, setting the sliding window width and the weight value corresponding to each element in the sliding window; then, preventing the sliding window from reaching the beginning of a sequence consisting of the ground heights corresponding to a predetermined number of ground image frames; next, calculating the weighted average of each element in the sliding window, and replacing the last element in the sliding window with the calculated average, and then sliding the sliding window to the next group of elements; and repeating the above calculation and sliding steps until sliding to the last element of the sequence. The final weighted average value can be considered as the height of the virtual ground. For example, assuming that the width of the sliding window is 2, the weight values ​​of the two elements in the window are 0.7 and 0.3. For a sequence {a1, a2, ..., an} consisting of ground heights corresponding to a predetermined number of ground image frames, the sliding window is first set at the beginning of the sequence [a1, a2]. The weighted average value a2' = a1 * 0.7 + a2 * 0.3 of the two elements a1 and a2 in the window is first calculated. Then, the weighted average value a2' is used to replace the value of a2 in the sequence, and the sliding window is moved to [a2', a3]. The weighted average value a3' = a2' * 0.7 + a3 * 0.3 of the two elements a2' and a3 in the window is then calculated. Then, the weighted average value a3' is used to replace the value of a3 in the sequence, and the sliding window is moved to [a3', a4]. The above process is repeated until the sliding window moves to [a(n-1)', an]. At this time, the weighted average value an' = a(n-1)' * 0.7 + an * 0.3 of the two elements a(n-1)' and an in the window is calculated. Finally, the calculated weighted average value an' is the height of the virtual ground.

[0123] Regarding step 804 above, in an embodiment of the present disclosure, the position offset of the root node of the avatar at the second moment can be adjusted based on the height of the virtual ground and the 3D coordinates of the foot key points at the second moment using the method shown in FIG9 . The method may specifically include:

[0124] In step 902, the lowest height of both feet at the second moment is determined based on the 3D coordinates of the key points of the feet at the second moment.

[0125] In step 904 , a correction amount of the position offset of the root node of the avatar at the second moment is determined based on the virtual ground height, the lowest height of the feet at the second moment, and the position offset of the root node of the avatar at the second moment.

[0126] In step 906 , the position offset of the root node of the avatar at the second moment is adjusted based on the correction amount.

[0127] Specifically, the above step 904 may include: first, summing the component of the position offset of the root node of the virtual image at the second moment in the vertical direction and the lowest height of the two feet at the second moment to obtain the height difference; then, using the difference between the virtual ground height and the height difference as the correction amount of the position offset of the root node of the virtual image at the second moment.

[0128] Specifically, assuming that the virtual ground height is h ground Indicates the lowest height of the feet at the second moment by h t Indicates that the position offset of the root node of the virtual image at the second moment is represented by offset t Indicates that the root node of the virtual image has an offset at the second moment. t Contains components in three directions (x, y, z), and offset t.y The vertical component of the position offset of the root node of the avatar at the second moment is represented by: ground It can be determined by the following calculation expression (1). ground =h ground -(h t +offset t.y ) (1)

[0129] Thus, in step 906, the position offset of the root node of the avatar at the second moment can be adjusted by the following calculation expression (2). t.y =offset t.y +diff ground (2)

[0130] In order to avoid the virtual image from shaking or grounding discontinuity due to excessive correction, a correction speed coefficient v can be further set. ground At this time, the above step 904 may include: first, summing the vertical component of the position offset of the root node of the virtual image at the second moment and the lowest height of the two feet at the second moment to obtain the height difference; then, multiplying the height difference by the preset correction speed coefficient as the single-frame correction; finally, taking the difference between the virtual ground height and the single-frame correction as the correction of the position offset of the root node of the virtual image at the second moment. That is, in the above embodiment, the correction of the position offset of the root node of the virtual image at the second moment is diff ground It can be determined by the following calculation expression (3). ground =h ground -(h t+offset t.y )×v ground (3)

[0131] Regarding step 210, the position parameters of the root node of the avatar at the second moment can be adjusted based on the position offset using the following method. First, the position parameters of the root node of the human body at the second moment are obtained. As previously described, the position parameters of the root node of the human body at the second moment can be directly extracted from the 3D information of the human body key points output by the 3D human key point detection model. Next, the position parameters of the root node of the human body at the second moment are summed with the position offset to obtain the adjusted position parameters of the root node of the avatar at the second moment.

[0132] It can be seen that the above method can determine the position offset of the human body root node based on the grounding state of the human feet, and adjust the position parameters of the human body root node based on the determined position offset, so as to prevent the virtual image from sliding, the feet floating up or the feet sinking.

[0133] Specifically, as mentioned above, the virtual image slides, the feet float up, or the feet sink, etc., mostly because the 3D information of the human body key points output by the 3D human body key point detection model is inaccurate or unstable. The inaccurate or unstable 3D information of the human body key points output by the 3D human body key point detection model is mainly reflected in the depth direction, that is, the Z direction. Therefore, in the embodiment of the present disclosure, the 3D coordinates of the foot key points are projected into more stable and accurate two-dimensional coordinates, that is, UV coordinates, which can effectively avoid disturbances in the Z direction, so that the grounding state of the human foot can be accurately determined. After determining the grounding state of the human foot, the position of the grounded foot can be kept stable by moving the root node, so as to achieve a real sense of stepping on the ground for the virtual image.

[0134] Furthermore, the above method can also incorporate ground height error accumulation. By establishing a virtual ground and further correcting the vertical offset of the root node, this method effectively prevents the foot from sinking or floating. Accumulating lateral errors during consecutive jumps can be addressed by increasing the lateral penalty coefficient.

[0135] It can be seen that the above solution is simple and direct. While achieving stable ground contact of the feet, it can also effectively avoid phenomena such as discontinuity and jumps in leg posture.

[0136] Corresponding to the above method, an embodiment of the present disclosure further discloses a virtual image driving device, the internal structure of which may be shown in FIG10 , including:

[0137] An acquisition module 1010 is configured to acquire the three-dimensional coordinates of key points of a foot of a human body in an image at a first moment and a second moment;

[0138] A projection module 1020, configured to project the three-dimensional coordinates of the key points of the foot at the first moment and the second moment onto a two-dimensional plane, respectively, to obtain the two-dimensional coordinates of the key points of the foot at the first moment and the second moment;

[0139] a grounding detection module 1030, configured to determine a grounding state of the foot of the human body at the second moment based on the two-dimensional coordinates of the key points of the foot at the first moment and the second moment;

[0140] a position offset determining module 1040 for determining a position offset of a root node of the avatar at the second moment based on the ground contact state of the foot of the human body at the second moment and the three-dimensional coordinates of the key points of the foot at the first moment and the second moment; and

[0141] The position parameter adjustment module 1050 is configured to adjust the position parameter of the root node of the avatar at the second moment based on the position offset.

[0142] The specific implementation of each of the above modules can refer to the aforementioned methods and drawings, and will not be repeated here. For the convenience of description, the above devices are described separately by function and are divided into various modules. Of course, when implementing the present disclosure, the functions of each module can be implemented in the same or multiple software and / or hardware. The device of the above embodiment is used to implement the corresponding virtual image driving method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0143] FIG11 shows a schematic diagram of the hardware structure of a more specific electronic device provided in this embodiment. The device may include: a processor 2010, a memory 2020, an input / output interface 2030, a communication interface 2040, and a bus 2050. The processor 2010, the memory 2020, the input / output interface 2030, and the communication interface 2040 are connected to each other within the device via the bus 2050.

[0144] The processor 2010 can be implemented using a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0145] The memory 2020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage devices, dynamic storage devices, etc. The memory 2020 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 220 and is called and executed by the processor 2010.

[0146] The input / output interface 2030 is used to connect input / output devices to enable information input and output. Input / output devices can be configured as components within the device or externally connected to the device to provide corresponding functions. Input devices may include microphones and various sensors, while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0147] The communication interface 2040 is used to connect to a communication module (not shown) to enable communication between the device and other devices. The communication module can communicate via a wired method (such as USB, network cable, etc.) or a wireless method (such as mobile network, WiFi, Bluetooth, etc.).

[0148] The bus 2050 comprises a path for transmitting information between the various components of the device (eg, the processor 2010 , the memory 2020 , the input / output interface 2030 , and the communication interface 2040 ).

[0149] It should be noted that although the above device only shows the processor 2010, the memory 2020, the input / output interface 2030, the communication interface 2040, and the bus 2050, in a specific implementation, the device may also include other components necessary for normal operation. In addition, it will be understood by those skilled in the art that the above device may only include the components necessary to implement the embodiments of this specification, and does not necessarily include all the components shown in the figure.

[0150] The electronic device of the above embodiment is used to implement the corresponding virtual image driving method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0151] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present disclosure also provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the virtual image driving method described in any of the above embodiments.

[0152] The aforementioned avatar driving method and related equipment first project the 3D coordinates of key foot points into more stable and accurate 2D coordinates, effectively avoiding depth-direction disturbances and thus accurately determining the grounding state of the human foot. Next, the method further adjusts the position of the root node of the avatar driven by human motion based on the determined grounding state of the human foot, ensuring that the grounded foot is stably placed on the ground. This prevents the avatar from slipping or sinking into the ground during movement, ensuring a realistic sense of grounding for the avatar and enhancing the user experience.

[0153] The computer-readable media of this embodiment include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0154] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the task processing method described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0155] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present disclosure (including the claims) is limited to these examples. Within the scope of the present disclosure, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present disclosure as described above, which are not provided in detail for the sake of simplicity.

[0156] In addition, to simplify the description and discussion, and so as not to obscure the embodiments of the present disclosure, known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided figures. In addition, devices may be shown in the form of block diagrams to avoid obscuring the embodiments of the present disclosure, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which the embodiments of the present disclosure are to be implemented (i.e., these details should be fully within the purview of those skilled in the art). Where specific details (e.g., circuits) are set forth to describe exemplary embodiments of the present disclosure, it will be apparent to those skilled in the art that the embodiments of the present disclosure may be implemented without these specific details or with variations in these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0157] Although the present disclosure has been described in conjunction with specific embodiments thereof, many alternatives, modifications, and variations of these embodiments will be apparent to those skilled in the art based on the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0158] The embodiments of the present disclosure are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. A method for driving a virtual image, comprising: Obtain the three-dimensional coordinates of the key points of the human foot in the image at the first moment and the second moment; Projecting the three-dimensional coordinates of the key points of the foot at the first moment and the second moment onto a two-dimensional plane respectively to obtain the two-dimensional coordinates of the key points of the foot at the first moment and the second moment; Determining a grounding state of the foot of the human body at the second moment based on the two-dimensional coordinates of the key points of the foot at the first moment and the second moment; Determine the position offset of the root node of the avatar at the second moment based on the grounding state of the foot of the human body at the second moment and the three-dimensional coordinates of the key points of the foot at the first moment and the second moment; and The position parameter of the root node of the virtual image at the second moment is adjusted based on the position offset.

2. The method according to claim 1, wherein: Projecting the three-dimensional coordinates of the foot key point at the first moment and the second moment onto a two-dimensional plane respectively comprises: The three-dimensional coordinates of the key points of the foot at the first moment and the second moment are projected onto a two-dimensional plane formed by a horizontal direction and a vertical direction respectively.

3. The method according to claim 1, wherein: Determining the grounding state of the foot of the human body at the second moment based on the two-dimensional coordinates of the key point of the foot at the first moment and the second moment includes: Determining a moving speed of the key point of the foot at the second moment based on the two-dimensional coordinates of the key point of the foot at the first moment and the second moment; and The ground contact state of the foot of the human body at the second moment is determined based on the moving speed.

4. The method according to claim 3, wherein: Determining the moving speed of the foot key point at the second moment includes: Determine the position of the foot key point on the two-dimensional plane from the first moment to the second moment based on the two-dimensional coordinates of the foot key point at the first moment and the second moment The distance moved up; Determine the distance moved by the key point of the foot in a unit time based on the difference between the first moment and the second moment and the distance; and The distance moved in the unit time is taken as the moving speed of the foot key point.

5. The method according to claim 3, wherein: Determining the ground contact state of the foot of the human body at the second moment based on the moving speed includes: Compare the moving speed of the foot key point with the foot grounding threshold, In response to determining that the moving speed of the foot key point is greater than the foot ground contact threshold, determining that the foot where the foot key point is located is in a non-ground contact state at the second moment; and In response to determining that the moving speed of the foot key point is less than or equal to the foot ground contact threshold, it is determined that the foot where the foot key point is located is in a ground contact state at the second moment.

6. The method according to claim 5, wherein: Determining the ground contact state of the foot of the human body at the second moment based on the moving speed further includes: Obtaining a depth parameter of a human body in the image; Based on the depth parameter, projecting a preset average human height parameter onto the two-dimensional plane to obtain a projected height; and The product of the projected body height and a preset first ratio is used as the foot contact threshold.

7. The method according to claim 5, wherein: Determining the ground contact state of the foot of the human body at the second moment based on the moving speed further includes: Determine whether both feet of the human body in the image are in a grounded state or are both in a jumping state, In response to determining that both feet of the human body in the image are jumping, setting both feet of the human body in the image to be in a non-grounded state at the second moment; In response to determining that both feet of the human body in the image are not both grounded and not both jumped, the moving speeds of the key points of the two feet are compared, and the foot with the smaller moving speed of the key points of the feet is set to be in a grounded state at the second moment, while the other foot is set to be in a non-grounded state at the second moment.

8. The method according to claim 7, wherein: Determining whether the foot of the human body in the image jumps includes: Based on the two-dimensional coordinates of the foot key point at the first moment and the second moment Determine the moving speed of the key point of the foot in the vertical direction; Compare the moving speed of the foot key point in the vertical direction with the foot jump threshold, In response to determining that the moving speed of the foot key point in the vertical direction is greater than the foot jump threshold, determining that the foot where the foot key point is located jumps; and In response to determining that the moving speed of the foot key point in the vertical direction is less than or equal to the foot jumping threshold, it is determined that the foot at which the foot key point is located has not jumped.

9. The method according to claim 8, wherein: Determining whether the foot of the human body in the image jumps further includes: Obtaining a depth parameter of a human body in the image; Based on the depth parameter, projecting a preset average human height parameter onto the two-dimensional plane to obtain a projected height; and The product of the projected body height and a preset second ratio is used as the foot jumping threshold.

10. The method according to claim 1, wherein: Determining the position offset of the root node of the virtual image at the second moment based on the grounding state of the foot of the human body at the second moment and the three-dimensional coordinates of the key points of the foot at the first moment and the second moment includes: In response to determining that both feet of the human body in the image are in a grounded state at the second moment, the three-dimensional coordinates of the midpoints of the two feet at the first moment and the second moment are determined based on the three-dimensional coordinates of the key points of the feet at the first moment and the second moment, respectively, and the difference between the three-dimensional coordinates of the midpoints of the two feet at the first moment and the second moment is used as the position offset of the root node of the virtual image at the second moment; In response to determining that both feet of the human body in the image are in a non-grounded state at the second moment, determining a position offset of the root node of the virtual image at the second moment based on a position offset of the root node of the virtual image at the first moment; In response to determining that a first foot of the human body in the image is in a grounded state at the second moment and a second foot is in a non-grounded state at the second moment, taking a difference between three-dimensional coordinates of a key point of the first foot at the first moment and the second moment as a position offset of a root node of the avatar at the second moment; and In response to determining that the second foot of the human body in the image is in a grounded state at the second moment and the first foot is in a non-grounded state at the second moment, the difference between the three-dimensional coordinates of the key points of the second foot at the first moment and the second moment is used as the position offset of the root node of the virtual image at the second moment.

11. The method according to claim 10, wherein: Determining the position offset of the root node of the avatar at the second moment based on the position offset of the root node of the avatar at the first moment includes: The product of the position offset of the root node of the virtual image at the first moment and a preset penalty coefficient is used as the position offset of the root node of the virtual image at the second moment.

12. The method according to claim 10, wherein: Determining the position offset of the root node of the virtual image at the second moment based on the grounding state of the foot of the human body at the second moment and the three-dimensional coordinates of the key points of the foot at the first moment and the second moment further includes: determining the height of the virtual ground; and In response to determining that at least one of the feet of the human body is in a grounded state at the second moment, the position offset of the root node of the virtual image at the second moment is adjusted based on the height of the virtual ground and the three-dimensional coordinates of the key points of the foot at the second moment.

13. The method according to claim 12, wherein: Determining the height of the virtual ground includes: Acquire a predetermined number of grounding image frames; wherein, in the grounding image frames, at least one of the two feet of the human body is in a grounding state; respectively acquiring the three-dimensional coordinates of the key points of the feet of the predetermined number of grounded image frames that are in a grounded state; Using the coordinate values ​​in the vertical direction of the three-dimensional coordinates as the ground height corresponding to each ground image frame; and The height of the virtual ground is determined based on the ground heights corresponding to each ground image frame.

14. The method according to claim 13, wherein: Determining the height of the virtual ground based on the ground heights corresponding to each ground image frame includes: Performing a sliding average on the ground heights corresponding to the ground image frames; and The obtained sliding average value is used as the height of the virtual ground.

15. The method according to claim 12, wherein: Adjusting the position offset of the root node of the virtual image at the second moment based on the height of the virtual ground and the three-dimensional coordinates of the key point of the foot at the second moment includes: Determine the lowest height of both feet at the second moment according to the three-dimensional coordinates of the key points of the feet at the second moment; Determining a correction amount of the position offset of the root node of the avatar at the second moment based on the virtual ground height, the lowest height of the two feet at the second moment, and the position offset of the root node of the avatar at the second moment; and The position offset of the root node of the avatar at the second moment is adjusted based on the correction amount.

16. The method according to claim 15, wherein: Determining a correction amount of the position offset of the root node of the avatar at the second moment based on the virtual ground height, the lowest height of the two feet at the second moment, and the position offset of the root node of the avatar at the second moment includes: Summing the component of the position offset of the root node of the avatar at the second moment in the vertical direction and the lowest height of both feet at the second moment to obtain a height difference; and The difference between the virtual ground height and the height difference is used as a correction amount of the position offset of the root node of the virtual image at the second moment.

17. The method according to claim 15, wherein: Determining a correction amount of the root node offset of the avatar at the second moment in the vertical direction based on the virtual ground height, the lowest height of the two feet at the second moment, and the root node offset of the avatar at the second moment includes: Summing the component of the position offset of the root node of the avatar at the second moment in the vertical direction and the lowest height of both feet at the second moment to obtain a height difference; The product of the height difference and a preset correction speed coefficient is used as a single-frame correction amount; and The difference between the virtual ground height and the single frame correction value is used as the virtual image The correction amount of the root node position offset at the second moment.

18. The method according to claim 1, wherein: Adjusting the position parameter of the root node of the avatar at the second moment based on the position offset includes: Obtaining position parameters of the root node of the human body at the second moment; and The position parameter of the root node of the human body at the second moment is summed with the position offset to obtain the adjusted position parameter of the root node of the virtual image at the second moment.

19. A virtual image driving device, comprising: An acquisition module, used for acquiring the three-dimensional coordinates of the key points of the feet of the human body in the image at the first moment and the second moment; A projection module, used to project the three-dimensional coordinates of the key point of the foot at the first moment and the second moment onto a two-dimensional plane respectively, to obtain the two-dimensional coordinates of the key point of the foot at the first moment and the second moment; A grounding detection module, used to determine the grounding state of the foot of the human body at the second moment based on the two-dimensional coordinates of the key points of the foot at the first moment and the second moment; A position offset determination module, used to determine the position offset of the root node of the virtual image at the second moment based on the grounding state of the foot of the human body at the second moment and the three-dimensional coordinates of the key points of the foot at the first moment and the second moment; as well as A position parameter adjustment module is used to adjust the position parameter of the root node of the virtual image at the second moment based on the position offset.

20. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the virtual image driving method as described in any one of claims 1 to 18 when executing the computer program.

21. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the virtual image driving method according to any one of claims 1 to 18.

22. A computer program product, comprising computer program instructions, which, when executed on a computer, enable the computer to execute the virtual image driving method according to any one of claims 1 to 18.

Images