Virtual character action generation method and device
Through key point detection and angle calculation of the benchmark skeleton architecture, the problem of insufficient unified position standards and computational performance in virtual character action generation is solved, and adaptive action generation and efficient computational performance to multiple bone standards is achieved.
Patent Information
- Application Number
- CN202411069541.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-05
- Publication Date
- 2024-11-12
AI Technical Summary
The prior art has problems with location standards uniformity and insufficient computing performance in the generation of virtual role actions, especially in real-time mapping scenarios, it is difficult to adapt to multiple bone standards and improve operational efficiency at the same time.
Through the key point detection of the reference skeleton architecture, the key point coordinates of the reference organism are obtained, the incremental rotation angle and rotation angle are calculated, and the virtual character bone structure is mapped according to these angles to generate actions. It is suitable for a variety of bone standards. Save computing resources and improve computing performance.
It realizes adaptive action generation to a variety of bone standards, improves computing performance and operation efficiency, and can generate virtual character actions at a processing efficiency of 20-25 frame rates.
Smart Images

Figure CN118941681A_ABST
Abstract
Description
Technical Field
[0001] One or more embodiments of the present specification relate to the field of computer technology, and more particularly, to a method and device for generating actions of a virtual character. Background Art
[0002] With the development of network technology, the application of virtual characters is becoming more and more extensive. For example, live videos displayed by digital people. In practice, in addition to digital people, virtual characters can also be other virtual creatures, such as digital cats, digital geese, etc. Virtual characters (also called 3D characters, or 3D characters, etc.) can be used as role models in games or animations and become carriers for action presentation. The video display of virtual characters can include the display and switching of sounds, actions, etc. Taking digital people as an example, the action generation of virtual characters can usually include the following methods: real-time monitoring of the action changes of real people and mapping them to digital people; determining the posture changes of real people under various actions through pre-collected images (continuous photos or videos), and mapping the posture changes under the corresponding actions to digital people under the action trigger of digital people; and so on.
[0003] In the process of using virtual characters, important technical issues may include at least one of the following: uniformity between part standards, computing performance during the mapping process (especially in the scenario of real-time action mapping). Among them, the restrictions on virtual character part standards are often very strict. If the naming used in the adopted technical solution modifies the original part, it is impossible to simultaneously apply multiple part standards (such as skeletal standards of humans of different categories and regions, etc.). On the other hand, computing performance will greatly limit operating efficiency, especially in the real-time mapping scenario, which requires very high computing performance. Summary of the invention
[0004] One or more embodiments of this specification describe a method and device for generating a knowledge graph based on a text corpus, so as to solve one or more problems mentioned in the background technology.
[0005] According to a first aspect, a method for generating an action of a virtual character is provided, which is used to generate corresponding actions for a virtual character based on an image captured for a reference organism, the method comprising: acquiring coordinates of multiple key points corresponding to a first part of a reference organism from a second image, wherein the second image is an image captured adjacent to a current first image, and the multiple key points are determined by key point detection of a reference skeletal structure; performing action calculation on the first part based on the coordinates of the multiple key points to obtain an incremental rotation angle of the first part as a whole, and a rotation angle of the first part along its own skeletal axis, wherein the incremental rotation angle is used to describe the difference between a target direction of the first part in the second image and a current direction of the first part in the first image; and mapping to a character skeletal structure corresponding to the virtual character according to the incremental rotation angle, the rotation angle, and pre-set constraints to generate the current action of the virtual character.
[0006] In one embodiment, the incremental rotation angle is determined in the following manner: the target vector corresponding to the target direction is determined by using the coordinate values of two key points corresponding to the skeletal axis of the first part in the second image; the incremental rotation angle is determined by the angle between the target vector and the current vector corresponding to the current direction, wherein the current vector is determined based on the coordinate values of two key points corresponding to the skeletal axis of the first part in the first image.
[0007] In one embodiment, the incremental rotation angle includes a rotation angle decomposed into each coordinate axis.
[0008] In one embodiment, the rotation angle is determined in the following manner: determining the target orientation of the first part based on three key points on the first part whose relative positions remain unchanged and are not collinear, the target orientation being in a direction perpendicular to the plane determined by the three key points; and comparing the current orientation of the first part in the first image with the target orientation to determine the rotation angle.
[0009] In one embodiment, when the first part belongs to a preset part set, the rotation angle is determined to be 0, and the part set is used to record various parts that cannot rotate along their own bone axis.
[0010] In one embodiment, the first part corresponds to a first part coordinate system, the first coordinate axis in the first part coordinate system is consistent with the skeletal axis of the first part, and the first part coordinate system has a strong correlation consistency with the first part; the mapping to the character skeletal structure corresponding to the virtual character according to the incremental rotation angle, the rotation angle and the pre-set constraints, so as to generate the current action of the virtual character includes: determining the first rotation action of the first part coordinate system when the relative positions of each coordinate axis remain unchanged according to the incremental rotation angle and the constraints; determining the second rotation action of the first part coordinate system with the first coordinate axis as the rotation axis according to the rotation angle and the constraints; mapping the first rotation action and the second rotation action to the character skeletal structure to generate the current action.
[0011] In one embodiment, the constraint conditions include the value ranges of the incremental rotation angle and the rotation angle of each part.
[0012] In one embodiment, obtaining the coordinates of multiple key points corresponding to the first part of the reference organism from the second image includes: detecting each key point of the reference organism from the second image through an open source reasoning framework for key point detection, each key point including multiple key points corresponding to the first part; determining the coordinates of the multiple key points based on a depth estimation algorithm for a monocular image.
[0013] According to a second aspect, a device for generating an action of a virtual character is provided, for generating corresponding actions for the virtual character according to an image collected for a reference organism, the device comprising:
[0014] an acquisition unit configured to acquire coordinates of a plurality of key points corresponding to a first part of a reference organism from a second image, wherein the second image is an image acquired adjacent to the current first image, and the plurality of key points are determined by key point detection of a reference skeleton architecture;
[0015] a calculation unit configured to perform motion calculation on the first part according to the coordinates of the plurality of key points, and obtain an incremental rotation angle of the first part as a whole, and a rotation angle along its own bone axis, wherein the incremental rotation angle is used to describe the difference between a target direction of the first part in the second image and a current direction of the first part in the first image;
[0016] The generating unit is configured to map the incremental rotation angle, the rotation angle and the preset constraint conditions to the character skeleton structure corresponding to the virtual character to generate the current action of the virtual character.
[0017] According to a third aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed in a computer, the computer is caused to execute the method of the first aspect.
[0018] According to a fourth aspect, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method of the first aspect is implemented.
[0019] Through the method and device provided by the embodiments of this specification, in order to generate corresponding actions for virtual characters using images collected for reference organisms, the actions of various parts of the reference organism in the image can be solved according to the detected key points, so as to determine the incremental rotation angle of the corresponding part as a whole, and the rotation angle along the axis of its own skeleton. Then, the current action of the virtual character is generated according to the incremental rotation angle and the rotation angle. Since the key points are determined based on the key point detection of the reference skeleton structure, after the action solution is completed, the incremental rotation angle and the rotation angle are mapped to the character skeleton structure corresponding to the virtual character, so as to adapt to the action generation of various character skeleton structures. In addition, the incremental rotation angle and the rotation angle are determined based on the heuristic determination of the key point coordinates, which can effectively save computing resources and improve computing performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.
[0021] Figure 1 A schematic diagram of a specific implementation architecture for generating actions for a virtual character in this specification is shown;
[0022] Figure 2 A schematic diagram showing a flow chart of action generation of a virtual character according to an embodiment of the present specification;
[0023] Figure 3 A schematic diagram of key point distribution determined by human key point detection of a specific example is shown;
[0024] Figure 4 A schematic diagram of the distribution of key points of a hand according to a specific example is shown;
[0025] Figure 5 A schematic diagram of the rotation of the coordinate system of a part of a specific example is shown;
[0026] Figure 6A schematic diagram of the rotation of the coordinate system of a specific example is shown;
[0027] Figure 7 A structural block diagram of a device for generating actions of a virtual character according to an embodiment of the present specification is shown. DETAILED DESCRIPTION
[0028] The solution provided in this specification is described below in conjunction with the accompanying drawings.
[0029] Figure 1 A schematic diagram of a specific implementation architecture of this specification is described. Figure 1 As shown, the implementation architecture includes a server (or cloud) and a terminal. Among them, the terminal can be an image acquisition device or a user terminal including an image acquisition module, such as a smart phone, a tablet computer, a smart watch, etc. Through the terminal, continuous images or videos of natural persons (or other organisms) can be collected and uploaded to the server. It is understandable that the server can also be located in the cloud. The server or cloud can be provided with a computing platform for parsing the image or video of the natural person frame by frame to determine the character's movements and map them to a digital person for rendering. The rendered digital person can be displayed on the terminal. The parsing process of the computing platform can be performed, for example, by Figure 1 The action parsing module shown is implemented. The technical solution provided in this specification is about the action parsing process.
[0030] In practice, there can be multiple terminals, natural persons can also be other organisms (such as animals, bionic robots, etc.), and digital people can also be other virtual characters, such as various virtual animals, etc., which are not limited here. The above implementation architecture can be used in processing scenarios where natural persons do not want to disclose privacy (such as not wanting to be seen in person), multi-platform display, or other processing scenarios with corresponding business requirements.
[0031] Depend on Figure 1 It can be seen that the core of the implementation architecture lies in the motion analysis process. The motion analysis process can be carried out in real time (analyzing and displaying images as they are collected or displayed) relative to the image acquisition or digital human display operation, or it can be carried out separately after image acquisition or ahead of digital human display, such as analyzing after all images are collected, storing them for backup after the analysis is completed, and directly using the analysis results when displaying the digital human.
[0032] The process of calculating motion can also be called motion capture. Motion capture can be divided into the detection of changes in various parts of the organism (also called joints in this field, which are different from the bone joints in daily life or in the medical field). The parts here can be understood as the part that exists as a whole in the action, for example Figure 1In the arm, the upper arm corresponds to one part and the lower arm corresponds to one part. Taking a person as an example, the parts included include: head, shoulder, cheekbone, chin, upper arm, lower arm, palm, five fingers, etc. A part can be determined by the line connecting two endpoints, or a single center point, and the action can be achieved by the rotation and rotation of the part (since the parts of the organism are usually connected and constrained, translation is not considered here). For example, the action of clapping can be achieved by the rotation of the upper arms of the left and right arms, the rotation of the lower arms and hands relative to the shoulders, and other changes in the position of the parts.
[0033] Usually, a single organism can contain multiple parts, such as a person's head and shoulders, and there may be a containment relationship between parts. For example, the head can be used as a part to perform actions relative to the body, such as shaking the head, turning the head, etc., and the eyes, cheeks, mouth and other parts of the head can also be used as separate parts to perform actions, such as the cheeks and mouth move when smiling, and the eyes move when blinking.
[0034] In conventional technology, Mediapipe Python can be used for monocular vision real-time motion capture. Figure 1 For example, we first detect key points of the person in the image through the official API interface of Mediapipe, apply smoothing to remove jitter on the detected key point coordinates, and then build an optimization-based driver solver. The optimization parameter is the rotation angle of each part of the 3D character. The optimization target is the direct difference between the position of each part and the detected key point coordinates and other constraint functions, so that the character gradually approaches the posture of the person in the video to achieve the effect of motion capture. Generally, the optimization-based method has high requirements for computing performance.
[0035] In view of this, this specification provides a kind of action generation technical scheme of virtual character, carries out key point detection and action detection on the image of organism under the reference skeleton framework, and then maps to the skeleton framework used by virtual character (for example, be recorded as character skeleton framework), and renders corresponding action.Here, the reference skeleton framework can be any skeleton framework selected in advance, and this skeleton framework can include the basic skeleton framework of reference organism, and can be transformed to various skeleton frameworks.In this way, by the key point detection of human body under the reference skeleton framework, action parsing and action migration to the part framework of virtual character, it can be applicable to 3D character motion capture scenes of multiple different skeleton standards, and no additional optimization is needed, and computing performance and motion capture effect can be better balanced.
[0036] Combine the following Figure 2 The embodiments are shown and the technical concept of the present specification is described in detail.
[0037] Figure 2The following is a schematic diagram of a virtual character action generation process according to an embodiment. The execution subject of the process may be a computer, device, or server with certain computing capabilities. More specifically, for example Figure 1 computing platform in .
[0038] Assuming that the first image is the current image, and the second image has a posture change relative to the first image, in order to determine the action from the first image to the second image, the changes in each part of the second image relative to the first image can be solved. Take any part (referred to as the first part) as an example, Figure 2 As shown, the action generation process of the virtual character provided in the embodiment of the present specification may include the following steps: Step 201, obtaining the coordinates of multiple key points corresponding to the first part of the reference organism from the second image, wherein the second image is an image captured adjacent to the current first image, and the multiple key points are determined by key point detection of the reference skeleton structure; Step 202, performing action solving on the first part according to the coordinates of the above multiple key points to obtain the incremental rotation angle of the first part as a whole, and the rotation angle of the first part along its own skeleton axis, wherein the incremental rotation angle is used to describe the difference between the target direction of the first part in the second image and the current direction of the first part in the first image; Step 203, mapping to the character skeleton structure corresponding to the virtual character according to the incremental rotation angle, the rotation angle and the pre-set constraints to generate the current action of the virtual character.
[0039] First, in step 201, coordinates of a plurality of key points corresponding to a first part of a reference organism are acquired from a second image.
[0040] The organism here can be any object with autonomous activity ability and corresponding skeletal structure, such as humans, animals, humanoid robots, etc. Among them, the corresponding skeletal structure can be a skeletal system with a certain commonality for the structure of a class of objects. For example, humans have a similar skeletal structure, usually with 206 bones in the trunk, limbs, and head. The bones are interconnected, and the corresponding parts of the upper limbs in the limbs can be divided into upper arms, lower arms, palms, etc., but each person's skeletal structure is not completely consistent. Different people may have different bone size ratios and slightly different positions. In this way, a single type of organism can correspond to at least one benchmark skeletal structure. The benchmark skeletal structure is used to describe the common skeletal structure of the corresponding organism, and pre-set the corresponding size ratios and relative positions between bones.
[0041] The reference organism can be any organism that is currently the subject of the source action, e.g. Figure 1The natural person shown, etc. In order to determine the action of the reference organism, a video or multiple images can be collected for the reference organism. These videos or images can be collected in real time or in advance, which is not limited here. The first image and the second image can be adjacent images acquired in the corresponding video or multiple images. Optionally, in the case of dense video frames (such as 240 frames per minute), images can be acquired as adjacent images at a predetermined frame interval (such as acquiring one frame every 4 frames). The first image and the second image are adjacent images acquired from the acquired video or multiple images.
[0042] For the acquired image, the key points corresponding to each part can be determined according to the key point detection of the benchmark skeleton architecture. Conventional key point detection methods include, for example, directly calling the corresponding closed source application interface (such as Mediapipe official API), etc. Those skilled in the art can understand that closed source applications usually contain a variety of pre-set functions. Once called, various functional modules may all run, thereby affecting the efficiency of key point detection. To this end, in an optional embodiment, key point detection can be performed with the help of a deep learning tool library (including inference library, model optimization, etc., a series of functions related to deep learning model deployment) optimized for the hardware platform in an open source reasoning framework (such as OpenVino reasoning framework, etc.). For example, data preprocessing and post-processing adapted to the open source reasoning framework are performed, and then combined with the call of the closed source framework interface, the key point detection of the organism is quickly performed. After testing, with the help of the open source reasoning framework, the accuracy can be almost consistent while the efficiency is higher than directly calling the corresponding closed source application interface, which can reduce the detection time by about 50%.
[0043] It can be understood that the reference organism can include multiple parts, and a single part can correspond to a corresponding key point. Generally, for a part extending in one direction, the key point detected is located at the end point of the corresponding part (such as the fingertip), or the junction of two parts (such as an end point of the upper arm or forearm at the elbow joint). For a block-shaped part extending in multiple directions (such as the head), the boundary points at both ends can be determined in a certain direction as key points.
[0044] like Figure 3 As shown in FIG. 1 , it is a distribution diagram of key points detected by mediapipe in a specific example. Among them, 10 key points are detected on the head, one key point for each of the left shoulder and the right shoulder, one key point for each of the elbow and wrist on the arm (such as key points 14, 16, etc.), three key points at the ends of the thumb, middle finger, and little finger on the hand (such as key points 18, 20, 22), two key points on the waist (such as key points 23, 24), and one key point each of the knee, ankle, heel, and toe (such as key points 28, 30, 32, etc.).
[0045] A reference organism can contain multiple parts, and there may be inclusion relationships between parts. Figure 3 In the figure, the eyes (corresponding to key points 1 to 6), cheeks (corresponding to key points 7 and 8), mouth (corresponding to key points 9 and 10) and other parts can move independently or as a part of the head and move with the head.
[0046] Here, it is assumed that the first part represents any part of the organism. It can be understood that for a single part, at least one key point can be detected through a single image. Among them, the key points in the first image can be pre-detected (for example, detected when solving the previous action), or can be detected and acquired in the current step 201. Therefore, in this step 201, at least a plurality of key points corresponding to the first part can be detected for the second image of the reference organism.
[0047] Understandably, Figure 3 This is just a specific example. In practice, the number of key points detected based on the open source reasoning framework can be much larger, for example Figure 4 In another specific example, the key points detected for the left hand are shown, corresponding to 21 key points from 0 to 20.
[0048] For the reference organism, the corresponding key points may be the points at the two ends, or the center point or the end point and the connection point with other parts. For example, the two key points corresponding to the forearm may be the elbow key point and the wrist key point, and the key points corresponding to the head may be the center point and the key point at the connection between the head and the neck. Figure 4 In the diagram of the distribution of left hand key points, 21 key points correspond to the joints and ends of the left hand, such as key point 0 is the key point at the wrist joint, key point 1 corresponds to the base joint of the thumb, key point 4 corresponds to the end of the thumb, and so on.
[0049] For each key point, its corresponding coordinates can also be determined. The coordinates can be coordinates in the world coordinate system or in the image coordinate system. Among them, the world coordinate system is an absolute coordinate system on the device system that is not affected by the user coordinate system. The image coordinate system can be a coordinate system constructed with a predetermined position point in the image (such as a center point or a boundary point, etc.) as the origin. It can be understood that the world coordinate system and the image coordinate system mentioned here can both be three-dimensional coordinate systems, such as a coordinate system composed of three coordinate axes of x, y, and z, or a spherical coordinate system. This specification describes the coordinate system composed of three coordinate axes as an example. For images captured by a monocular camera, they are usually two-dimensional. In order to obtain coordinates in a three-dimensional coordinate system, a conventional depth estimation algorithm (binocular stereo vision method, structured light, TOF, etc.) can be used for depth estimation, which will not be repeated here.
[0050] In this step 201, key points related to the first part and their coordinates in the second image can be obtained from the detected key points. It is worth noting that since some key points may be located at joints, a single key point may correspond to multiple parts.
[0051] Then, according to step 202, the motion of the first part is solved according to the coordinates of the above-mentioned multiple key points to obtain the incremental rotation angle of the first part as a whole and the rotation angle along its own bone axis.
[0052] Here, the so-called action solution is the calculation process of determining the operation required to change from the state displayed by the first image to the state displayed by the second image through comparison. Under the technical concept of this specification, the action solution of the first part can be divided into two parts, one part is the change of the overall position of the first part relative to other parts, which is expressed by the incremental rotation angle (the various parts of the organism are related and constitute related constraints, so the action of each part can be regarded as rotation without considering translation), and the other part is the possible rotation of the first part itself, which can also be understood as a change in the orientation state without changing the position.
[0053] On the one hand, the change of the overall position of the first part relative to other parts is solved. The action state in the first image is the current state, the orientation of the first part in the current state is the current direction, the second image is the target image, the action state in the second image is the target state, the orientation of the first part in the target state is the target direction, then the rotation angle of the first part from the current direction to the target direction can be called the incremental rotation angle. The incremental rotation angle can be determined based on the key points.
[0054] Usually, the rotation angle can be described by the difference in the vector direction of the corresponding part. It can be understood that two points in space determine a straight line, which corresponds to a direction. To this end, the three-dimensional coordinates of the key points can be obtained to determine the direction of the first part (usually the direction of the bone axis). In the case where the first part corresponds to two key points, the connecting line of the two key points can be used to determine the current direction and target direction of the first part. Taking the human forearm as an example, two key points, the wrist key point and the elbow key point, can correspond. A single key point can correspond to a coordinate in the world coordinate system (two-dimensional or three-dimensional coordinates), wherein the world coordinate system can be a coordinate system with a single position in the second image (such as the lower left corner) as the origin. The difference between the coordinates of the two key points can be used to express the target direction of the first part.
[0055] Taking the three-dimensional coordinate system as an example, if the coordinates of the wrist key point are marked as (a2, b2, c2), and the coordinates of the elbow key point are marked as (x2, y2, z2), then the direction of the forearm in the second image, i.e., the target direction, is, for example, (a2-x2, b2-y2, c2-z2). For ease of description, the target direction is, for example, recorded as P2=(r2, s2, t2). Similarly, the current direction of the first part in the current first image can also be determined in a similar manner, such as recorded as P1=(r1, s1, t1).
[0056] It is understandable that since two points determine a straight line, the angle between the two straight lines is determined. The angle describes the difference between the target direction of the first part in the second image and the current direction in the first image, and can be used as the incremental rotation angle. In one embodiment, the incremental rotation angle can be described by radians α. For example, the angle between the vectors (r2, s2, t2) and (r1, s1, t1) is θ=α·π / 180, then radians In another embodiment, the incremental rotation angle can be described by Euler angles, which are angles determined by decomposing the angle θ to each coordinate axis, for example, composed of nutation angle, precession angle and rotation angle, which will not be described in detail here.
[0057] In some optional implementations, a single key point corresponding to the first part (such as its center point) and a predetermined position point (such as a coordinate origin) can also be used as its corresponding key points to assist in determining the corresponding incremental rotation angle.
[0058] In other embodiments, the incremental rotation angle of the first part can be determined by other methods, which will not be described in detail here. Figure 5 For example, Figure 5 As shown, from left to right, the forearm drives the hand to rotate outward as a whole, and the incremental rotation angle is recorded as θ.
[0059] On the other hand, the rotation angle of the first part can be determined based on the orientation of the first part. Here, the rotation angle is used to describe the rotation or flipping of the first part around its own bone axis. Figure 6 As shown in FIG. 1 , the right hand turns from the palm facing left to the palm facing backward. During this change, the position of the center point of the hand may not change, but a rotation occurs.
[0060] It can be understood that the incremental rotation angle describes the overall position change of the first part. In practice, the first part may also rotate. For example, when the lower body remains stationary, the upper body turns 90° to the left or right. For another example, when the upper arm remains stationary, the palm turns from upward to left, and the forearm rotates. Generally, for natural organisms (such as humans or animals), the parts that may rotate are predetermined parts, such as the head, body, forearm, upper arm, thigh, calf, etc.
[0061] The rotation of a part is usually accompanied by the overall rotation of a plane of the part. Therefore, the rotation angle can be determined according to the orientation of the part. The orientation of the first part can be determined by the perpendicular direction of the plane corresponding to the first part. For example, if the upper body is regarded as a plane, the perpendicular direction of the plane corresponding to the upper body can be the front of the body. Usually, a plane can be determined by three non-collinear points. Therefore, in order to determine the orientation of the first part, three points whose relative position of the first part remains unchanged and which are not collinear can be selected to determine the plane corresponding to the first part. Taking the first part as the upper body as an example, two key points corresponding to the shoulders and a key point of the center line of the body can be selected, such as the key point of the navel position (such as Figure 3 The key points of the shoulders and the navel can determine a plane, and the direction perpendicular to the plane can be used to locate the direction of the upper body, such as forward, left, etc. In this way, the rotation angle of the upper body can be determined by detecting the direction of the upper body.
[0062] Similarly, for other parts, similar methods can be used to determine the plane orientation and rotation angle. For example, for the hand, the plane can be constructed using the key points of the wrist and the key points of the bases of two fingers (such as Figure 4 The key points in 0 and any two of the following: 1, 5, 9, 13, 17), which will not be listed here one by one.
[0063] It is understandable that for a specific organism, some parts can often rotate by themselves. For example, fingers cannot rotate relative to the palm, eyes cannot rotate relative to the head, and so on. Therefore, in some possible implementations, a part set may be pre-defined, and the part set includes parts that can rotate by themselves. When the first part belongs to the part set, the rotation angle is determined according to the plane orientation determined by the corresponding key point, otherwise, the rotation angle is determined to be 0. In an optional embodiment, the preset set may include parts that cannot rotate by themselves, and the above part set is the complement of the set. In this case, the complement may not exist, but it is equivalent to setting a part set that can rotate by itself.
[0064] Furthermore, in step 203, the incremental rotation angle, the rotation angle and the pre-set constraint conditions are mapped to the character skeleton structure corresponding to the virtual character to generate the current action of the virtual character.
[0065] It can be understood that the incremental rotation angle describes the change in the overall position of the first part, which usually revolves around a fulcrum. For example, the rotation of the forearm usually uses the elbow key point as the fulcrum, and the rotation of the upper arm often uses the shoulder as the fulcrum. The fulcrum here is also the point where the corresponding part is associated with other parts of the body. On the other hand, the rotation angle describes the change in the direction of the first part itself. For example, the direction of the palm, the direction of the body, etc., this change often depends on the skeletal axis of the corresponding part itself.
[0066] Therefore, in order to generate actions according to incremental rotation angles and rotation angles, in one embodiment, a part coordinate system can be established for each part, and the part coordinate system and the corresponding part can be associated together to keep the position and direction relatively unchanged between the part coordinate system and the corresponding part. Thereby, when the incremental rotation angle and rotation angle are applied to the part coordinate system, the corresponding action of the part can be achieved. For the convenience of calculation, the origin of the part coordinate system is usually the key point serving as the fulcrum of rotation, and one coordinate axis of the part coordinate system (such as the first coordinate axis, this specification takes the y-axis in the xyz coordinate system as an example) is the skeletal axis of the corresponding part. For example: refer to Figure 5 As shown in the figure, the coordinate origin corresponding to the forearm can be the elbow key point, and the corresponding bone axis is along the line connecting the elbow key point and the wrist key point; refer to Figure 6 As shown, the coordinate origin corresponding to the hand is the wrist key point, and the corresponding bone axis is along the line connecting the wrist key point and the middle finger tip key point; and so on.
[0067] The part coordinate system for a single part may have a strong correlation consistency with the single part, that is, the relative position remains constant. For example, the y-axis is always the bone axis, and the angle between the direction of the x-axis and the z-axis and the orientation of the first part remains constant.
[0068] In this way, the incremental rotation angle acts on the part coordinate system, and the entire part coordinate system rotates as a whole. Figure 5 As shown, taking the first part as the forearm as an example, the constructed coordinate system is recorded as the first part coordinate system, and the y-axis of the coordinate system (such as the first coordinate axis) remains consistent with the direction of the forearm bone axis. During the outward rotation of the forearm, the y-axis of the bone axis always remains consistent with the direction of the forearm bone axis, and each coordinate axis of the entire coordinate system rotates by an incremental rotation angle θ. When the rotation angle acts on the part coordinate system, the corresponding part coordinate system rotates along the bone axis. Figure 6 As shown, taking the first part as the right hand as an example, the y-axis is the bone axis, the twisting angle acts on the part coordinate system, the y-axis remains unchanged, and the x and z axes rotate along the xz plane according to the rotation angle.
[0069] Similarly, the coordinate system of the parts corresponding to the upper body can be based on the spine direction, front-to-back direction, and left-to-right direction as three coordinate axes. The spine is the bone axis, the navel key point is the origin, and the bone axis of the upper body is the rotation axis. The rotation of the upper body relative to the lower body can be realized. The rotation of the upper body can also be regarded as the rotation of the part coordinate system, such as the upper body part coordinate system. Assuming that the spine direction is the bone axis, the coordinate axes in the front-to-back direction and the left-to-right direction rotate around the bone axis (such as the first coordinate axis). Other parts are similar.
[0070] In this way, by mapping the first part of the virtual character to the character's skeletal structure according to the incremental rotation angle and the rotation angle, an action consistent with the reference organism in the image can be generated. The generated action can be described by the effect of the incremental rotation angle and the rotation angle on the part coordinate system. For example, rotating α degrees in the opposite direction along the y-axis, etc. In this way, it is only necessary to determine the part correspondence between the reference organism and the virtual character, without paying attention to the differences between the skeletal structures used. When the incremental rotation angle and the rotation angle are applied to the part coordinate system, the corresponding part coordinate system of the virtual character can be adjusted to the corresponding state according to the generated action, that is, the corresponding part is adjusted to the corresponding state.
[0071] In actual business, virtual characters can use skeleton structures determined in various ways, for example, the skeleton structure obtained by re-modifying the baseline skeleton of a person using a naming similar to SMPL-X, the skeleton structure determined using the role in Mixamo, the skeleton structure of a cat, the skeleton structure of a goose, etc. The technical solution of this specification can select a skeleton structure as the baseline skeleton structure, for example, using a single role in Mixamo (such as an XBot role) as the baseline skeleton structure, and the incremental rotation angle, rotation angle, etc. in the foregoing text can all correspond to the baseline skeleton structure. In this step 203, the action in the baseline skeleton structure can be mapped to the current role skeleton structure in combination with the action redirection method, which will not be repeated here. In an optional embodiment, an action sequence can also be generated based on the collected video or multiple images, and the action sequence can be used to migrate to any role skeleton structure.
[0072] It is worth noting that for real organisms, the linkage between various parts has certain rules (or action limits), and it is usually difficult to perform actions that break these rules. For example, the head cannot rotate more than 180 degrees at a time, the upper body cannot rotate more than 240 degrees relative to the lower body at a time, and the fingers will not rotate relative to the palm, etc. Therefore, in some possible designs, constraints can be pre-set for the rotation angles and incremental rotation angles of various parts of the reference organism, and when generating actions for the virtual character, the pre-set constraints must be met.
[0073] In this way, from the first image to the second image, the corresponding actions can be determined for each part, thereby generating an action set for the virtual character, and the action set includes the rotation and rotation angle information of the coordinate system of each part. In an optional embodiment, for a video or multiple images, the action sets between two adjacent images can constitute an action set sequence, and the action set sequence is applied to the virtual character to complete a series of actions, such as a dance.
[0074] Reviewing the above process, the method for generating the action of the virtual character provided under the technical conception of this specification, for adjacent images collected from the reference organism, can detect key points based on the reference skeleton structure as the standard, and determine the incremental rotation angle and rotation angle of each part according to the key points under the reference skeleton structure, and then map it to the character skeleton structure. Since the action of the reference organism corresponds to the reference skeleton structure, and then maps it to any character skeleton structure, there is no need to generate the character skeleton, which can greatly reduce the threshold for motion capture for virtual characters. In addition, the incremental rotation angle and rotation angle can be determined with the help of the key points of the corresponding parts, which belongs to heuristic solution, does not involve optimization ideas, and can use matrix operations, so that various tensor operators can be used to accelerate calculations, thereby improving operating efficiency. Experiments show that the technical solution provided in this specification can achieve a processing efficiency of 20-25 frames per second (FPS) on the CPU.
[0075] According to another embodiment, a device for generating actions of a virtual character is also provided. The device can be arranged in a computer, a terminal or a server with a certain computing capability. Figure 7 FIG. 7 shows a virtual character action generation device 700 according to an embodiment. Figure 7 As shown, the apparatus 700 may include:
[0076] An acquisition unit 701 is configured to acquire coordinates of a plurality of key points corresponding to a first part of a reference organism from a second image, wherein the second image is an image captured adjacent to the current first image, and the plurality of key points are determined by detecting key points of a reference skeletal structure;
[0077] A solving unit 702 is configured to solve the motion of the first part according to the coordinates of the multiple key points, and obtain an incremental rotation angle of the first part as a whole, and a rotation angle along its own bone axis, wherein the incremental rotation angle is used to describe the difference between the target direction of the first part in the second image and the current direction of the first part in the first image;
[0078] The generating unit 703 is configured to map the incremental rotation angle, the rotation angle and the preset constraint conditions to the character skeleton structure corresponding to the virtual character to generate the current action of the virtual character.
[0079] It is worth mentioning that Figure 7 The device 700 shown is Figure 2 The method described corresponds to Figure 2 The corresponding description in the illustrated method embodiment is also applicable to the device 700 and will not be repeated here.
[0080] According to another embodiment, there is also provided a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute a combination of Figure 2 The method described by et al.
[0081] According to another embodiment of the present invention, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the Figure 2 The method described in the above examples should be appreciated by those skilled in the art that the functions described in the embodiments of this specification can be implemented by hardware, software, firmware, or any combination thereof in one or more of the above examples. When implemented by software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.
[0082] The specific implementation methods described above further illustrate the purpose, technical solutions and beneficial effects of the technical concept of this specification. It should be understood that the above description is only the specific implementation method of the technical concept of this specification, and is not used to limit the protection scope of the technical concept of this specification. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of the embodiments of this specification should be included in the protection scope of the technical concept of this specification.
Claims
1. A method for generating an action of a virtual character, for generating a corresponding action for the virtual character according to an image collected from a reference organism, the method comprising: Acquire coordinates of a plurality of key points corresponding to a first part of a reference organism from a second image, wherein the second image is an image captured adjacent to the current first image, and the plurality of key points are determined by key point detection of a reference skeletal structure; Performing motion calculation on the first part according to the coordinates of the multiple key points to obtain an incremental rotation angle of the first part as a whole and a rotation angle along its own bone axis, wherein the incremental rotation angle is used to describe the difference between a target direction of the first part in the second image and a current direction of the first part in the first image; According to the incremental rotation angle, the rotation angle and the preset constraint conditions, the character skeleton structure corresponding to the virtual character is mapped to generate the current action of the virtual character.
2. The method of claim 1, wherein: The incremental rotation angle is determined by: Determine a target vector corresponding to the target direction using coordinate values of two key points corresponding to the skeletal axis of the first part in the second image; The incremental rotation angle is determined by an angle between the target vector and a current vector corresponding to the current direction, wherein the current vector is determined based on coordinate values of two key points corresponding to the skeletal axis of the first part in the first image.
3. The method of claim 1, wherein: The incremental rotation angle includes the rotation angle decomposed into each coordinate axis.
4. The method of claim 1, wherein: The rotation angle is determined by: Determine a target orientation of the first part according to three key points on the first part whose relative positions remain unchanged and are not collinear, wherein the target orientation is along a direction perpendicular to a plane determined by the three key points; The rotation angle is determined by comparing the current orientation of the first part in the first image with the target orientation.
5. The method of claim 1, wherein: In the case where the first part belongs to a preset part set, the rotation angle is determined to be 0, and the part set is used to record various parts that cannot rotate along their own bone axis.
6. The method of claim 1, wherein: The first part corresponds to a first part coordinate system, a first coordinate axis in the first part coordinate system is consistent with a skeletal axis of the first part, and the first part coordinate system has a strong correlation consistency with the first part; The step of mapping the incremental rotation angle, the rotation angle and the preset constraint condition to the character skeleton structure corresponding to the virtual character, thereby generating the current action of the virtual character includes: Determine, according to the incremental rotation angle and the constraint condition, a first rotation action of the first part coordinate system under the condition that the relative positions of the coordinate axes remain unchanged; Determine, according to the rotation angle and the constraint condition, a second rotation action in which the first part coordinate system rotates with the first coordinate axis as the rotation axis; The first rotation action and the second rotation action are mapped to the character skeleton structure to generate the current action.
7. The method of claim 1, wherein: The constraint conditions include the value ranges of the incremental rotation angle and the rotation angle of each part.
8. The method of claim 1, wherein: The step of acquiring the coordinates of a plurality of key points corresponding to the first part of the reference organism from the second image comprises: Detecting key points of the reference organism from the second image using an open source reasoning framework for key point detection, wherein the key points include a plurality of key points corresponding to the first part; The coordinates of the multiple key points are determined based on a depth estimation algorithm for the monocular image.
9. A device for generating an action of a virtual character, for generating corresponding actions for the virtual character according to an image collected from a reference organism, the device comprising: an acquisition unit configured to acquire coordinates of a plurality of key points corresponding to a first part of a reference organism from a second image, wherein the second image is an image acquired adjacent to the current first image, and the plurality of key points are determined by key point detection of a reference skeleton architecture; a calculation unit configured to perform motion calculation on the first part according to the coordinates of the plurality of key points, and obtain an incremental rotation angle of the first part as a whole, and a rotation angle along its own bone axis, wherein the incremental rotation angle is used to describe the difference between a target direction of the first part in the second image and a current direction of the first part in the first image; The generating unit is configured to map the incremental rotation angle, the rotation angle and the preset constraint conditions to the character skeleton structure corresponding to the virtual character to generate the current action of the virtual character.
10. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 8.
11. A computing device comprising a memory and a processor, characterized in that: The memory stores executable codes, and when the processor executes the executable codes, the method according to any one of claims 1 to 8 is implemented.
Citation Information
Cited By
Method and system for generating motion of virtual character
CN120599106B
