Virtual image driving method, device, electronic device, and medium

Abstract

The present disclosure provides a method, device, electronic device and medium for driving a virtual image, which relates to the field of artificial intelligence technology, specifically computer vision, augmented reality, virtual reality, deep learning and other technical fields, and can be applied to scenes such as metaverse and digital human. The method may include: obtaining human joint point positions corresponding to multiple joints of a target human body; obtaining human joint rotation data based on the human joint point positions, the human joint rotation data including the respective rotation angles of the multiple joints; determining the orientation of at least one key joint among the multiple joints based on the human joint rotation data; obtaining updated human joint rotation data by updating the rotation angle of at least one other joint among the multiple joints based on the orientation of the at least one key joint; and driving the virtual image of the target human body based on the updated human joint rotation data.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, specifically computer vision, augmented reality, virtual reality, deep learning and other technical fields, and can be applied to scenes such as the metaverse and digital humans, and specifically relates to a method, device, electronic device, computer-readable storage medium and computer program product for driving a virtual image. Background Art

[0002] In scenarios such as augmented reality, virtual reality, metaverse, and digital humans, it is often necessary to obtain or generate virtual images. There is an increasing demand for generating accurate, reasonable, and flexible virtual images.

[0003] The methods described in this section are not necessarily methods that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any method described in this section is considered to be prior art simply because it is included in this section. Similarly, unless otherwise indicated, the issues mentioned in this section should not be considered to have been recognized in any prior art. Summary of the invention

[0004] The present disclosure provides a method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product for driving a virtual image.

[0005] According to one aspect of the present disclosure, a method for driving a virtual image is provided, comprising: obtaining human joint point positions corresponding to multiple joints of a target human body; obtaining human joint rotation data based on the human joint point positions, the human joint rotation data including the respective rotation angles of the multiple joints; determining the orientation of at least one key joint among the multiple joints based on the human joint rotation data; obtaining updated human joint rotation data by updating the rotation angle of at least one other joint among the multiple joints based on the orientation of the at least one key joint; and driving the virtual image of the target human body based on the updated human joint rotation data.

[0006] According to another aspect of the present disclosure, a driving device for a virtual image is provided, comprising: a position obtaining unit for obtaining human body joint point positions corresponding to multiple joints of a target human body; a rotation data obtaining unit for obtaining human body joint rotation data based on the human body joint point positions, the human body joint rotation data including the respective rotation angles of the multiple joints; an orientation determining unit for determining the orientation of at least one key joint among the multiple joints based on the human body joint rotation data; an updating unit for obtaining updated human body joint rotation data by updating the rotation angle of at least one other joint among the multiple joints based on the orientation of the at least one key joint; and a virtual image driving unit for driving the virtual image of the target human body based on the updated human body joint rotation data.

[0007] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a method for driving a virtual image according to one or more embodiments of the present disclosure.

[0008] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method for driving a virtual image according to one or more embodiments of the present disclosure.

[0009] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein the computer program implements the method for driving a virtual image according to one or more embodiments of the present disclosure when executed by a processor.

[0010] According to one or more embodiments of the present disclosure, a reasonable human body posture can be obtained, thereby generating a realistic virtual image.

[0011] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0013] Figure 1 A schematic diagram showing an exemplary system in which the various methods described herein may be implemented according to an embodiment of the present disclosure;

[0014] Figure 2 A flowchart of a method for driving a virtual character according to an embodiment of the present disclosure is shown;

[0015] Figure 3 A schematic diagram of a human body node tree according to an embodiment of the present disclosure is shown;

[0016] Figure 4 A flowchart of a method for driving a virtual character according to another embodiment of the present disclosure is shown;

[0017] Figure 5An exemplary schematic diagram of an unreasonable posture is shown;

[0018] Figure 6 A flow chart of a whole body solution method according to an embodiment of the present disclosure is shown;

[0019] Figure 7 A schematic diagram showing a driven skeleton effect according to an embodiment of the present disclosure is shown;

[0020] Figure 8 A data flow diagram according to an exemplary embodiment of the present disclosure is shown;

[0021] Fig. 9 A structural block diagram of a driving device for a virtual image according to an embodiment of the present disclosure is shown;

[0022] Fig.10 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0023] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0024] In the present disclosure, unless otherwise specified, the use of the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, timing relationship, or importance relationship of these elements, and such terms are only used to distinguish one element from another element. In some examples, the first element and the second element may refer to the same instance of the element, and in some cases, based on the description of the context, they may also refer to different instances.

[0025] The terms used in the description of various examples in this disclosure are only for the purpose of describing specific examples and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element can be one or more. In addition, the term "and / or" used in this disclosure covers any one of the listed items and all possible combinations.

[0026] The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0027] Figure 1 FIG. 1 is a schematic diagram of an exemplary system 100 in which various methods and apparatuses described herein may be implemented according to an embodiment of the present disclosure. Figure 1, the system 100 includes one or more client devices 101, 102, 103, 104, 105, and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105, and 106 may be configured to execute one or more applications.

[0028] In an embodiment of the present disclosure, the server 120 may run one or more services or software applications that enable execution of the driving method of the avatar according to the present disclosure.

[0029] In some embodiments, server 120 may also provide other services or software applications, which may include non-virtualized environments and virtualized environments. In some embodiments, these services may be provided as web-based services or cloud services, such as provided to users of client devices 101, 102, 103, 104, 105, and / or 106 under a software as a service (SaaS) model.

[0030] exist Figure 1 In the configuration shown, the server 120 may include one or more components that implement the functions performed by the server 120. These components may include software components, hardware components, or a combination thereof that can be executed by one or more processors. Users operating client devices 101, 102, 103, 104, 105, and / or 106 may in turn utilize one or more client applications to interact with the server 120 to utilize the services provided by these components. It should be understood that a variety of different system configurations are possible, which may differ from the system 100. Therefore, Figure 1 is one example of a system for implementing the various methods described herein and is not intended to be limiting.

[0031] A user may use client devices 101, 102, 103, 104, 105, and / or 106 to capture video or images, obtain an avatar, view an avatar, etc. The client device may provide an interface that enables a user of the client device to interact with the client device. The client device may also output information to the user via the interface. Figure 1 Only six client devices are depicted, but one skilled in the art will appreciate that the present disclosure may support any number of client devices.

[0032] Client devices 101, 102, 103, 104, 105 and / or 106 may include various types of computer devices, such as portable handheld devices, general-purpose computers (such as personal computers and laptop computers), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, game systems, thin clients, various messaging devices, sensors or other sensing devices, etc. These computer devices may run various types and versions of software applications and operating systems, such as MICROSOFT Windows, APPLE iOS, UNIX-like operating systems, Linux or Linux-like operating systems (such as GOOGLE Chrome OS); or include various mobile operating systems, such as MICROSOFT Windows Mobile OS, iOS, Windows Phone, Android. Portable handheld devices may include cellular phones, smart phones, tablet computers, personal digital assistants (PDAs), etc. Wearable devices may include head-mounted displays (such as smart glasses) and other devices. Game systems may include various handheld game devices, Internet-enabled game devices, etc. Client devices are capable of executing various different applications, such as various Internet-related applications, communication applications (such as email applications), short message service (SMS) applications, and may use various communication protocols.

[0033] The network 110 may be any type of network known to those skilled in the art that may support data communications using any of a variety of available protocols, including but not limited to TCP / IP, SNA, IPX, etc. By way of example only, the one or more networks 110 may be a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, WIFI), and / or any combination of these and / or other networks.

[0034] Server 120 may include one or more general purpose computers, dedicated server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running virtual operating systems, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that may be virtualized to maintain a server's virtual storage device). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

[0035] The computing units in the server 120 may run one or more operating systems including any of the above operating systems and any commercially available server operating systems. The server 120 may also run any of a variety of additional server applications and / or middle-tier applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0036] In some implementations, server 120 may include one or more applications to analyze and consolidate data feeds and / or event updates received from users of client devices 101, 102, 103, 104, 105, and 106. Server 120 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105, and 106.

[0037] In some embodiments, the server 120 may be a server of a distributed system, or a server combined with a blockchain. The server 120 may also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. A cloud server is a host product in a cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private servers (VPS) services.

[0038] The system 100 may also include one or more databases 130. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 130 may be used to store information such as audio files and video files. The databases 130 may reside in various locations. For example, the database used by the server 120 may be local to the server 120, or may be remote from the server 120 and may communicate with the server 120 via a network-based or dedicated connection. The databases 130 may be of different types. In some embodiments, the databases used by the server 120 may be, for example, relational databases. One or more of these databases may store, update, and retrieve data to and from the databases in response to commands.

[0039] In some embodiments, one or more of the databases 130 may also be used by applications to store application data. The databases used by the applications may be different types of databases, such as a key-value store, an object store, or a conventional store backed by a file system.

[0040] Figure 1The system 100 may be configured and operated in various ways to enable the application of various methods and apparatuses described in the present disclosure.

[0041] Reference below Figure 2 A driving method 200 of an avatar according to an exemplary embodiment of the present disclosure is described.

[0042] In step S201, human body joint point positions corresponding to multiple joints of the target human body are obtained.

[0043] In step S202, human joint rotation data is obtained based on the human joint point positions, and the human joint rotation data includes respective rotation angles of the multiple joints.

[0044] In step S203, the orientation of at least one key joint among the multiple joints is determined based on the human joint rotation data.

[0045] At step S204, updated human joint rotation data is obtained by updating the rotation angle of at least one other joint among the plurality of joints based on the orientation of the at least one key joint.

[0046] In step S205, the virtual image of the target human body is driven based on the updated human body joint rotation data.

[0047] According to the method described in the embodiment of the present disclosure, a reasonable human body posture can be obtained, thereby generating a realistic virtual image.

[0048] It is understandable that the key joints may be relatively important joints in the human body, joints that have a greater impact on the posture and authenticity of the human body, joints that are prone to errors, etc. The orientation of the key joint may be the rotation angle of the joint, or may be the direction in which the joint points, such as the direction of the knee, the twisting direction of the hip, the direction in which the acromion faces, etc. Determining the orientation of at least one key joint among the multiple joints based on the human joint rotation data may be adjusting the orientation of the key joint based on the rotation angle and position of the related joint, such as updating it, so that the overall posture is more coordinated.

[0049] In the related art, it is hoped that a reasonable posture can be obtained based on the joints of the human body to drive a virtual image such as a digital human, and the problem of limited expressiveness of actions can be overcome. According to the embodiments of the present disclosure, the rotation angle of the joint can be obtained based on the position of the joints of the human body. In addition, the orientation of other joints can be updated through key joints, and constraints between joints can be added to obtain a more reasonable posture of the human body, thereby generating a more realistic virtual image.

[0050] According to some embodiments, obtaining the positions of human joint points corresponding to multiple joints of the human body includes perceiving and obtaining the positions of the human joint points from a video or image containing the target human body.

[0051] The joint positions can be determined based on the collected human body video or image, thereby generating a virtual image corresponding to the real human body image, for example, generating a digital human based on the perceived human body.

[0052] According to some embodiments, the plurality of joints follow a predetermined node tree, and the at least one other joint comprises a child node of the at least one key joint.

[0053] After determining the key node, the orientation position of the child node may be updated accordingly. For example, after updating the orientation of the knee, the rotation angle of the child node (eg, foot) may be modified accordingly.

[0054] Combine the following Figure 3 A node tree schematic diagram 300 according to an exemplary embodiment of the present disclosure is described, wherein a human skeleton (joint) node tree consisting of node 0 to node 24 is shown.

[0055] According to some embodiments, the at least one key joint is selected from the group consisting of: shoulder, upper arm, knee and hip.

[0056] Key joints can be shoulders, upper arms, knees and hips, and the orientation of such key nodes is related to the positions of other nodes. For example, the orientation of the knee position should be consistent with the orientation of the hip, and the direction of the shoulder should be consistent with the direction of the upper arm.

[0057] In such an example, for example, Figure 3 The direction of the plane formed by the nodes 12, 13, and 14, i.e., the joints of the hip, knee, and foot, is used as the direction of the leg and further as the orientation of the hip joint.

[0058] According to some embodiments, obtaining human joint rotation data based on the human joint point positions includes: obtaining a standard posture; obtaining the rotation angle of a root node among the multiple joints relative to the standard posture based on the human joint point positions; and based on the rotation angle of the root node, solving the rotation angle of a sub-node among the multiple joints relative to the standard posture.

[0059] For example, you can obtain a standard pose, such as the standard pose Tpose under the openpose specification, and then solve the pose of the current joint relative to Tpose from the parent node step by step to eliminate the torsion step by step. Figure 3For example, the pose matrix of node 12, i.e., the hip joint, can be multiplied by the inverse of the pose matrix of its parent node 8, i.e., the waist joint, so as to eliminate the torsion and obtain the pose of node 12 relative to the corresponding node in Tpose. For another example, the current vector obtained by subtracting the position of node 12 (hip joint) from the position of node 13 (knee joint) in the current pose, and the standard vector obtained by subtracting the position of node 12 from the position of node 13 in the standard pose, and the rotation matrix between these two vectors can be used as the joint angle of node 12.

[0060] According to some embodiments, the method 200 may further include: obtaining a joint rationality constraint condition; and adjusting the human body joint rotation data so that the adjusted human body joint rotation data satisfies the joint rationality constraint condition.

[0061] In such an embodiment, plausibility constraints can be added to reduce possible counterintuitive self-spins.

[0062] According to some embodiments, the joint rationality constraint condition includes at least one of the following: a constraint on the number of degrees of freedom of one or more joints among the multiple joints; a constraint on the direction of the rotation angle of one or more joints among the multiple joints; and a constraint on the size of the rotation angle of one or more joints among the multiple joints.

[0063] For example, returning a reference Figure 3 , where nodes 3 and 6 (elbow joints) and nodes 13 and 10 (knee joints) can only have one degree of freedom. For another example, some joints have specific rotation axes and rotation angles. In this way, unreasonable joint positions can be reduced.

[0064] According to some embodiments, method 200 may further include: performing smoothing filtering on the human body joint point positions to obtain smoothed human body joint point positions, and wherein obtaining human body joint rotation data based on the human body joint point positions includes: obtaining the human body joint rotation data based on the smoothed human body joint point positions.

[0065] Since the positions of human joints may be perceived or predicted by the network, smoothing filtering can reduce jitter.

[0066] According to some embodiments, obtaining human joint point positions corresponding to multiple joints of a target human body includes: obtaining corresponding at least two human joint point positions from at least two video frames containing the target human body, wherein obtaining human joint rotation data based on the human joint point positions includes: obtaining at least two human joint rotation data based on the at least two human joint point positions, the at least two video frames include an adjacent first frame and a second frame, and wherein the method further includes: in response to determining that a change in the position of a joint in the second frame relative to a corresponding joint in the first frame is greater than a threshold, fusing the rotation angle of the corresponding joint in the first frame with the rotation angle of the joint in the second frame to update the human joint rotation data corresponding to the second frame.

[0067] In such an embodiment, a nonlinear sliding window transition may be added to achieve transition continuity of the action.

[0068] According to some embodiments, method 200 may also include: for a first joint among the multiple joints, wherein the first joint has at least one sub-joint: decomposing the rotation angle of the first joint to obtain at least one decomposed sub-rotation angle; and adjusting the rotation angle of the at least one sub-joint based on the at least one decomposed sub-rotation angle.

[0069] Thus, a more reasonable expression of joint rotation can be achieved. As an example, the method may also include bone redirection, which is to map the rotation data of the defined skeleton to another bone structure, such as the bone structure of commercial data. The other bone structure may have additional intermediate sub-joints, such as several intermediate sub-joints between the shoulder and elbow joints. For a more reasonable expression, it is possible to consider transferring the joint rotation and decomposing the shoulder rotation data to the following sub-joints. For example, the shoulder rotation data can be converted to an axis angle, and then the angle can be divided equally or unequally, and then new rotation data can be composed and assigned to the sub-joints.

[0070] As the Metaverse continues to gain popularity, how users can achieve low-cost interactions in the Metaverse has become an important point of contention for Metaverse apps. Good human-computer interaction will greatly enhance the user's immersive experience. Motion capture technology is an important technology that enables human-computer interaction in the Metaverse, that is, real people make some movements and dances, and then digital people in the Metaverse make corresponding movements and dances. The interaction between the real world and the Metaverse brings new entertainment value.

[0071] The present disclosure proposes a driving method, device, electronic device and medium for a virtual image, for example, a full-body IK solution based on 3D joints of the human body can be realized. In the system architecture, according to one or more embodiments of the present disclosure, it is possible to combine 3D perception of the human body with full-body inverse kinematics, inversely calculate the joint posture of the human body, and perform reasonable constraints on the joints, and finally obtain a reasonable posture to drive the digital human.

[0072] Combine the following Figure 4 A method 400 for driving a virtual image based on 3D joints of a human body according to another exemplary embodiment of the present disclosure is described.

[0073] At step S401, 3D joint point perception of the human body is performed. The input may be a video of continuous frames captured by a camera in real time, and the output may be the 3D joint point position of each frame. With the development of deep learning, there are many methods for acquiring 3D joints of the human body based on captured images (HRNET, SPIN, MotionBERT, etc.). As an example, the specification of the key points used may be openpose. It is understandable that the whole-body IK solution may be applicable to different human joint point position perception modules, and the present disclosure is not limited thereto.

[0074] At step S402, smoothing filtering is performed. After the algorithm model is calculated, the 3D position of the human body joint can be obtained, but the point at this time may have some jitter because it is predicted by the network. Therefore, a smoothing filtering processing algorithm can be performed to improve the smooth transition between positions.

[0075] At step S403, full-body inverse kinematics (IK) back calculation is performed. After obtaining reliable human joint point output, the joint rotation angle of a specific virtual character needs to be obtained through a full-body IK algorithm, thereby driving a vivid virtual human.

[0076] At step S404, joint rationality constraints are performed. The rotation of each joint can be obtained through the whole body IK inverse calculation solution. Because there is no restriction on the 3D joint freedom, some counterintuitive self-rotations may occur as shown in the figure below. Therefore, in order to make the driven digital human effect more reasonable, joint rationality constraints can be added to limit it.

[0077] According to some embodiments, rationality constraints can be added based on prior information about the human body. For example, the knee and elbow joints can rotate with only one degree of freedom. Figure 5 The self-rotation produced by the small arm shown, etc.

[0078] According to some other embodiments, the rotation angle of the joint can be added. For example, the ankle can only rotate about 60 degrees around the z-axis (directly in front of itself).

[0079] At step S405, the virtual image is driven and rendered. For example, the optimized joint angle data can be mapped to the joints on the IP digital data to drive and render the IP digital human.

[0080] It is understood that although steps S401-S405 are described in order above, this does not require that they must be performed in accordance with the above steps. For example, the operation of step S404 can be performed at any time point, such as before steps S401-S403, or simultaneously with any of steps S401-S403 or S405. Alternatively, step S404 can be performed multiple times throughout the method to verify whether the joint rationality constraints are met, thereby ensuring the rationality of the joint.

[0081] Combine the following Figure 6 A flowchart of a method 600 for solving a full-body IK inverse calculation according to a non-limiting embodiment of the present disclosure is described. It can be understood that the method 600 here can correspond to step S403 in this article.

[0082] In step S601, a standard pose is defined, such as the standard pose Tpose of openpose. The subsequent rotation angles of the joints may be defined relative to Tpose.

[0083] At step S602, the pose of the current joint is solved. For example, the pose of the root node may be first obtained, and then the pose of the current joint relative to Tpose may be solved step by step from the parent node.

[0084] For example, return reference Figure 3 In step S602, the orientation of node 8 can be determined first, and then the torsion can be eliminated by the inverse of the transformation matrix from node 12 to node 8 to obtain the pose of node 12 relative to Tpose. As another example, the rotation matrix between the vector 13-12 in the current pose and the vector 13-12 in Tpose can be used as the joint angle of node 12. Then, the pose of each joint can be obtained by pushing down step by step.

[0085] At step S603, the orientation of the key point joints is calculated, for example, the orientation of the upper arm and hip joints may be calculated.

[0086] For example, return reference Figure 3 , the orientation of the faces of nodes 12, 13, and 14 can be used as the orientation of 12. For example, the orientation of the leg can be considered to be the orientation of the hip, and the orientation between the knee and the hip is consistent. Then, the calculated orientation is continued to adjust the angle of the key point joint. For example, the angle of the hip joint, shoulder joint, etc. can be adjusted.

[0087] Exemplarily, step S603 may mainly target four points: left and right shoulders, left and right hips, but the present disclosure is not limited thereto, and step S603 may also target other key joints, such as key joints whose orientation has a constraint relationship with the position or orientation of other sub-joints.

[0088] In step S604, the joint rotation of the child node is updated.

[0089] At step S605, the skeleton is driven. For example, the joint rotation of the whole body can be obtained to drive the Tpose skeleton forward. Exemplarily, the correctness of the back calculation can also be verified. Figure 7 , where an exemplary driven skeletal effect 700 is shown.

[0090] According to one or more exemplary embodiments of the present disclosure, a nonlinear sliding window transition operation and a joint rotation transfer operation may also be included.

[0091] For nonlinear sliding window transition, since the angles of normal adjacent frames have a certain continuity, in practice, if the angles of some joints in adjacent frames change too much, it will cause the motion to be non-smooth. For this situation, nonlinear sliding window transition can be added. When a joint in the kth frame changes too much, the rotation data of the corresponding joint of k-1 can be used for nonlinear fusion to obtain k', and then the data of the k+1 frame and k' are nonlinearly fused to obtain k+1'. In the case of more serious non-smoothness, about 3 to 5 sliding windows can be executed in this way to achieve the transition continuity of the motion.

[0092] According to one or more exemplary embodiments of the present disclosure, a skeleton redirection operation may also be included, so as to constitute a full-process solution for the entire monocular motion capture of commercial landing. In some example scenarios, after obtaining the rotation of the whole body joints, there is still a step of skeleton redirection before the actual landing. Skeleton redirection is to map the rotation data of the defined skeleton to the skeleton structure of the commercial digital resource, wherein the skeleton design of the general commercial digital resource may have more sub-joints relative to the standard pose (e.g., Tpose based on Openpose). For example, in the skeleton stock price that finally landed, several intermediate sub-joints will be added between the shoulder and elbow joints, while the skeleton defined by Tpose has only two joints, the shoulder and the elbow. In this case, in order to express more reasonably, it is possible to consider performing joint rotation transfer and disassembling the rotation data of the shoulder to the following sub-joints. As a specific non-limiting embodiment, the rotation data of the parent joint (e.g., shoulder) can be converted to the axis angle, and then the angle is divided equally or unequally, and then the new rotation data is composed and assigned to the sub-joints.

[0093] Figure 8FIG. 8 shows a data flow 800 according to an exemplary embodiment of the present disclosure. Figure 8 As shown, an image 801 can be obtained, which is input into a human joint point 3D perception module 810 to obtain a perceived human 8D joint point 802, which is then smoothed and filtered by a smoothing filter module 820, and then passed through a full body IK solution module 830 and a joint rationality constraint module 840 to obtain a driven digital human 803. It is to be understood that the above module division is only an example, and the present disclosure is not limited thereto.

[0094] According to one or more embodiments of the present disclosure, a full-body IK solution based on 3D joints of the human body is defined. In this system architecture, the three-dimensional posture information of the human body can be perceived and the joints can be determined, and then the human body can be three-dimensionally modeled using the joints to obtain a three-dimensional representation of human motion, and based on the three-dimensional representation, the inverse kinematics characteristics of human motion are obtained, thereby realizing the perception of human motion. By modeling the human body and solving inverse kinematics and constraining the rationality of joints, accurate perception of human motion can be achieved, and it has good generalization performance and robustness. Finally, the acquired human motion rotation data is input into the digital human, and it is enabled to perform corresponding tasks. The method described in this application is not only applicable to offline video content generation of virtual digital humans in the metaverse, but can also be applied to scenarios such as online live broadcasts.

[0095] Reference now Fig. 9 A driving device 900 of a virtual image according to an embodiment of the present disclosure is described. The driving device 900 of the virtual image may include a position obtaining unit 901, a rotation data obtaining unit 902, an orientation determining unit 903, an updating unit 904, and a virtual image driving unit 905. The position obtaining unit 901 may be used to obtain human joint point positions corresponding to multiple joints of a target human body. The rotation data obtaining unit 902 may be used to obtain human joint rotation data based on the human joint point positions, and the human joint rotation data includes the respective rotation angles of the multiple joints. The orientation determining unit 903 may be used to determine the orientation of at least one key joint among the multiple joints based on the human joint rotation data. The updating unit 904 may be used to obtain updated human joint rotation data by updating the rotation angle of at least one other joint among the multiple joints based on the orientation of the at least one key joint. The virtual image driving unit 905 may be used to drive the virtual image of the target human body based on the updated human joint rotation data.

[0096] According to the device described in the embodiment of the present disclosure, a reasonable human body posture can be obtained, thereby generating a realistic virtual image.

[0097] According to some embodiments, the joint point position obtaining unit may be configured to obtain the human body joint point positions from a video or image perception containing the target human body.

[0098] According to some embodiments, the at least one key joint may be selected from the group consisting of: shoulder, upper arm, knee and hip.

[0099] According to some embodiments, the plurality of joints follow a predetermined node tree, and wherein the at least one other joint comprises a child node of the at least one key joint.

[0100] According to some embodiments, the rotation data acquisition unit may include: a unit for obtaining a standard posture; a unit for obtaining the rotation angle of a root node among the multiple joints relative to the standard posture based on the position of the human body joint points; and a unit for solving the rotation angle of a child node among the multiple joints relative to the standard posture based on the rotation angle of the root node.

[0101] According to some embodiments, the apparatus may further include: a unit for obtaining joint rationality constraints; and a unit for adjusting the human joint rotation data so that the adjusted human joint rotation data satisfies the joint rationality constraints.

[0102] According to some embodiments, the joint rationality constraint condition may include at least one of the following: a constraint on the number of degrees of freedom of one or more joints among the multiple joints; a constraint on the direction of the rotation angle of one or more joints among the multiple joints; and a constraint on the size of the rotation angle of one or more joints among the multiple joints.

[0103] In the technical solution disclosed herein, the collection, acquisition, storage, use, processing, transmission, provision and public application of user personal information involved are in compliance with the relevant laws and regulations and do not violate public order and good morals.

[0104] According to an embodiment of the present disclosure, an electronic device, a readable storage medium and a computer program product are also provided.

[0105] refer to Fig.10, a block diagram of an electronic device 1000 that can be used as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0106] like Fig.10 As shown, the electronic device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the electronic device 1000 can also be stored. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0107] Multiple components in the electronic device 1000 are connected to the I / O interface 1005, including: an input unit 1006, an output unit 1007, a storage unit 1008, and a communication unit 1009. The input unit 1006 can be any type of device that can input information to the electronic device 1000. The input unit 1006 can receive input digital or character information, and generate key signal input related to user settings and / or function control of the electronic device, and can include but is not limited to a mouse, a keyboard, a touch screen, a track pad, a track ball, a joystick, a microphone, and / or a remote controller. The output unit 1007 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 1008 can include but is not limited to a disk and an optical disk. The communication unit 1009 allows the electronic device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and may include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver and / or a chipset, such as a Bluetooth device, an 802.11 device, a WiFi device, a WiMax device, a cellular communication device and / or the like.

[0108] The computing unit 1001 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1001 performs the various methods and processes described above, such as methods 200 and / or 400 and their variants, etc. For example, in some embodiments, methods 200 and / or 400 and their variants, etc. may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into the RAM 1003 and executed by the computing unit 1001, one or more steps of the methods 200 and / or 400 and their variants, etc. described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to execute the methods 200 and / or 400 and their variations, etc., in any other appropriate manner (eg, by means of firmware).

[0109] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0110] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0111] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0112] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0113] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0114] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0115] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0116] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but only by the claims after authorization and their equivalent scope. Various elements in the embodiments or examples can be omitted or replaced by their equivalent elements. In addition, each step can be performed in an order different from that described in the present disclosure. Further, the various elements in the embodiments or examples can be combined in various ways. It is important that with the evolution of technology, many elements described herein can be replaced by equivalent elements that appear after the present disclosure.

Claims

1. A method for driving a virtual image, comprising: Obtaining human body joint point positions corresponding to multiple joints of a target human body; Obtaining human joint rotation data based on the human joint point positions, wherein the human joint rotation data includes respective rotation angles of the plurality of joints; determining at least one key joint among the plurality of joints, the at least one key joint being a joint among the plurality of joints having an orientation related to positions of other nodes among the plurality of joints; Determining the orientation of at least one key joint among the plurality of joints based on the human joint rotation data; Obtaining updated human joint rotation data by updating a rotation angle of at least one other joint among the plurality of joints based on the orientation of the at least one key joint, wherein updating the rotation angle of at least one other joint among the plurality of joints comprises: For a key joint of the at least one key joint, the at least one key joint is selected from the group consisting of: shoulder, upper arm, knee and hip, based on the direction of the surface formed by one or more child nodes of the key joint, updating the orientation of the key joint so that the direction of the surface formed by the one or more child nodes of the key joint serves as the updated orientation of the key joint; adjusting angles of key joints based on the updated orientation; and modifying the rotation angle of the at least one other joint based on the adjusted angle of the key joint; and The virtual image of the target human body is driven based on the updated human body joint rotation data.

2. The method according to claim 1, wherein: Obtaining the positions of human joint points corresponding to multiple joints of the human body includes obtaining the positions of the human joint points from a video or image perception containing the target human body.

3. The method according to any one of claims 1 to 2, wherein: The plurality of joints follow a predetermined node tree, and wherein the at least one other joint comprises a child node of the at least one key joint.

4. The method according to any one of claims 1 to 3, wherein: Acquiring human joint rotation data based on the human joint point position includes: Obtain standard posture; Acquire the rotation angle of the root node of the plurality of joints relative to the standard posture based on the position of the human body joint point; and Based on the rotation angle of the root node, the rotation angles of the child nodes in the multiple joints relative to the standard pose are solved.

5. The method according to any one of claims 1 to 4, further comprising: Get joint rationality constraints; as well as The human body joint rotation data is adjusted so that the adjusted human body joint rotation data satisfies the joint rationality constraint condition.

6. The method according to claim 5, wherein: The joint rationality constraint condition includes at least one of the following: A constraint on the number of degrees of freedom of one or more joints among the plurality of joints; A rotational direction constraint of one or more joints among the plurality of joints; and A rotation angle size constraint of one or more joints among the plurality of joints.

7. The method according to any one of claims 1 to 6, further comprising: The human body joint point positions are smoothed and filtered to obtain smoothed human body joint point positions, and wherein, Acquiring human joint rotation data based on the human joint point position includes: The human body joint rotation data is obtained based on the smoothed human body joint point positions.

8. The method according to any one of claims 1 to 7, in, Obtaining the positions of human joint points corresponding to the multiple joints of the target human body comprises: obtaining corresponding positions of at least two human joint points from at least two video frames containing the target human body, Wherein, obtaining human joint rotation data based on the human joint point positions comprises: obtaining at least two human joint rotation data based on the at least two human joint point positions, the at least two video frames comprising an adjacent first frame and a second frame, and The method also includes: in response to determining that the change in the position of a joint in the second frame relative to the corresponding joint in the first frame is greater than a threshold, fusing the rotation angle of the corresponding joint in the first frame with the rotation angle of the joint in the second frame to update the human joint rotation data corresponding to the second frame.

9. The method according to any one of claims 1 to 8, further comprising: For a first joint of the plurality of joints, wherein the first joint has at least one sub-joint: Decomposing the rotation angle of the first joint to obtain at least one decomposed sub-rotation angle; as well as The rotation angle of the at least one sub-joint is adjusted based on the decomposed at least one sub-rotation angle.

10. A driving device for a virtual image, comprising: A position obtaining unit, used for obtaining the positions of human joint points corresponding to a plurality of joints of a target human body; A rotation data obtaining unit, used for obtaining human joint rotation data based on the human joint point positions, wherein the human joint rotation data includes respective rotation angles of the plurality of joints; an orientation determining unit, configured to determine at least one key joint among the plurality of joints, the at least one key joint being a joint among the plurality of joints having an orientation related to positions of other nodes among the plurality of joints, and determining the orientation of the at least one key joint among the plurality of joints based on the human body joint rotation data; An updating unit, configured to obtain updated human joint rotation data by updating the rotation angle of at least one other joint among the multiple joints based on the orientation of the at least one key joint, wherein updating the rotation angle of at least one other joint among the multiple joints comprises: for a key joint among the at least one key joint, the at least one key joint is selected from the group consisting of the following items: shoulder, upper arm, knee and hip, based on the direction of the surface formed by one or more subnodes of the key joint, updating the orientation of the key joint so that the direction of the surface formed by the one or more subnodes of the key joint serves as the updated orientation of the key joint; adjusting the angle of the key joint based on the updated orientation; and modifying the rotation angle of the at least one other joint based on the adjusted angle of the key joint; and The virtual image driving unit is used to drive the virtual image of the target human body based on the updated human body joint rotation data.

11. The device according to claim 10, wherein: The joint point position obtaining unit is configured to obtain the human body joint point positions from a video or image perception containing the target human body.

12. The device according to any one of claims 10-11, wherein: The plurality of joints follow a predetermined node tree, and wherein the at least one other joint comprises a child node of the at least one key joint.

13. The device according to any one of claims 10 to 12, wherein: The rotation data obtaining unit comprises: Unit for obtaining standard pose; A unit for acquiring a rotation angle of a root node of the plurality of joints relative to a standard posture based on the position of the human body joint point; and A unit for solving the rotation angles of the child nodes in the multiple joints relative to the standard pose based on the rotation angle of the root node.

14. The apparatus according to any one of claims 10 to 13, further comprising: Unit for obtaining joint rationality constraints; as well as A unit for adjusting the human body joint rotation data so that the adjusted human body joint rotation data satisfies the joint rationality constraint condition.

15. The device according to claim 14, wherein: The joint rationality constraint condition includes at least one of the following: A constraint on the number of degrees of freedom of one or more joints among the plurality of joints; A rotational direction constraint of one or more joints among the plurality of joints; and A rotation angle size constraint of one or more joints among the plurality of joints.

16. The device according to any one of claims 10 to 15, in, Obtaining the positions of human joint points corresponding to the multiple joints of the target human body comprises: obtaining corresponding positions of at least two human joint points from at least two video frames containing the target human body, Wherein, obtaining human joint rotation data based on the human joint point positions comprises: obtaining at least two human joint rotation data based on the at least two human joint point positions, the at least two video frames comprising an adjacent first frame and a second frame, and In which, the device also includes a unit for the following operations: in response to determining that the change in the position of a joint in the second frame relative to the corresponding joint in the first frame is greater than a threshold, the rotation angle of the corresponding joint in the first frame is merged with the rotation angle of the joint in the second frame to update the human joint rotation data corresponding to the second frame.

17. An electronic device comprising: at least one processor; as well as a memory communicatively coupled to the at least one processor; in The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 9.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-9.

19. A computer program product comprising a computer program, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images