Posture mapping method, device, computer equipment and storage medium

Abstract

The present application relates to a posture mapping method, device, computer equipment, computer-readable storage medium and computer program product, which can be applied to various scenarios such as cloud technology, artificial intelligence, smart transportation, and assisted driving. The method includes: obtaining the relative position relationship between multiple positioning points of a physical object, and determining the posture mapping object of the physical object; matching each point of the posture mapping object with each positioning point, and determining the target point that fails to match among each point; based on the relative position relationship, performing position interpolation processing on the target point according to the spatial position constraints between each point, and obtaining the interpolation position information of the target point; based on the relative position relationship and the interpolation position information, performing posture configuration on the posture mapping object, and obtaining the posture mapping result of the physical object. The above method can be used to expand the application scenarios of the posture mapping method.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a posture mapping method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] With the rapid development of computer technology, a posture mapping method has emerged that uses computers to generate posture mapping results of physical objects. The posture mapping method can be applied to the posture configuration of virtual objects in virtual scenes to display virtual objects that match the posture of real people.

[0003] In traditional posture mapping methods, if the position data of a certain positioning point of a physical object at the target moment is missing, it is necessary to determine the theoretical position of the positioning point at the target moment based on the position data of the positioning point before and after the target moment, and then obtain the posture mapping result of the physical object at the target moment. In traditional technologies, since the posture mapping process needs to combine the posture information of the previous and next moments, it cannot be applied to the scene of real-time posture mapping, and has the disadvantage of limited scenes. Summary of the invention

[0004] Based on this, it is necessary to provide a posture mapping method, device, computer equipment, computer-readable storage medium and computer program product that can expand application scenarios in response to the above technical problems.

[0005] In a first aspect, the present application provides a posture mapping method. The method comprises:

[0006] Acquire the relative position relationship between multiple positioning points of the physical object, and determine the posture mapping object of the physical object;

[0007] Matching each point of the posture mapping object with each positioning point, and determining a target point among the points where the matching fails;

[0008] Based on the relative position relationship, performing position interpolation processing on the target point according to the spatial position constraint conditions between the points, to obtain interpolation position information of the target point;

[0009] Based on the relative position relationship and the interpolation position information, the posture mapping object is posture configured to obtain a posture mapping result of the physical object.

[0010] In a second aspect, the present application also provides a posture mapping device. The device comprises:

[0011] An acquisition module, used to acquire the relative position relationship between multiple positioning points of a physical object and determine a posture mapping object of the physical object;

[0012] A point configuration module, used for matching each point of the posture mapping object with each positioning point, and determining a target point among the points where the matching fails;

[0013] A position interpolation module, used for performing position interpolation processing on the target point based on the relative position relationship and in accordance with the spatial position constraint conditions between the points, to obtain interpolation position information of the target point;

[0014] The posture mapping module is used to perform posture configuration on the posture mapping object based on the relative position relationship and the interpolation position information to obtain a posture mapping result of the physical object.

[0015] In a third aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0016] In a fourth aspect, the present application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented. The present application also provides a video time-effect classification device.

[0017] In a fifth aspect, the present application further provides a computer program product, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0018] The above-mentioned posture mapping method, device, computer equipment, computer-readable storage medium and computer program product obtain the relative position relationship between multiple positioning points of a physical object and determine the posture mapping object of the physical object; match each point of the posture mapping object with each positioning point to determine the target point that fails to match among each point; based on the relative position relationship, perform position interpolation processing on the target point according to the spatial position constraint conditions between each point to obtain the interpolation position information of the target point; based on the relative position relationship and the interpolation position information, perform posture configuration on the posture mapping object to obtain the posture mapping result of the physical object. In the above-mentioned posture mapping process, based on the relative position relationship between multiple positioning points of the physical object, in accordance with the spatial position constraints between the various points of the posture mapping object, the position interpolation is performed on the target points that failed to match, so as to obtain the interpolated position information of the target points, and then obtain the posture mapping result of the physical object. It is not necessary to rely on the posture information of the physical object at the previous and next moments, and the position interpolation of the target point at the current moment can be completed. It can be suitable for real-time posture mapping scenarios. Moreover, the various positioning points of the physical object and the various points of the posture mapping object of the physical object do not have to correspond one to one, which is conducive to expanding the application scenarios of the posture mapping method. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A diagram of an application environment of a gesture mapping method in some embodiments;

[0020] Figure 2 is a schematic diagram of a posture mapping process in some embodiments;

[0021] Figure 3 is a flowchart of a posture mapping method in some embodiments;

[0022] Figure 4 is a schematic diagram of human body positioning points in one embodiment;

[0023] Figure 5 A schematic diagram of a mapping principle from a physical object to a gesture mapping object in some embodiments;

[0024] Figure 6 A schematic diagram of the high-pass filtering principle in some embodiments;

[0025] Figure 7 It is a schematic diagram of the principle of determining the interpolation position based on multiple straight line intersection points in some embodiments;

[0026] Figure 8 A schematic diagram of an elbow joint interpolation principle in one embodiment;

[0027] Fig. 9 A schematic diagram of an elbow joint interpolation principle in another embodiment;

[0028] Fig.10 is a schematic diagram of the low-pass filtering principle in some embodiments;

[0029] Fig.11 is a flowchart of a posture mapping method in some other embodiments;

[0030] Fig.12 is a schematic diagram of a posture mapping process in an embodiment;

[0031] Fig.13 is a schematic diagram of a mapping effect without correction and smoothing after interpolation in one embodiment;

[0032] Fig.14 is a schematic diagram of a mapping effect after correction and smoothing processing is performed after interpolation in one embodiment;

[0033] Fig.15 is a structural block diagram of a posture mapping device in some embodiments;

[0034] Fig.16 1 is a diagram of the internal structure of a computer device in some embodiments. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0036] The gesture mapping method provided in this application may be based on artificial intelligence. For example, the gesture recognition model in this application may be a neural network model. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science. It attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that the machines have the functions of perception, reasoning and decision-making. Artificial intelligence technology is a comprehensive discipline that involves a wide range of fields, including both hardware-level technology and software-level technology. Basic artificial intelligence technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, and mechatronics. Artificial intelligence software technology mainly includes several major directions such as computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0037] Machine learning is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Compared with data mining, which seeks mutual characteristics between big data, machine learning focuses more on algorithm design, allowing computers to automatically "learn" rules from data and use the rules to predict unknown data.

[0038] Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning and other technologies. Reinforcement learning (RL), also known as reinforcement learning, evaluation learning or enhanced learning, is one of the paradigms and methodologies of machine learning. It is used to describe and solve the problem of how an agent can maximize its rewards or achieve specific goals through learning strategies during its interaction with the environment.

[0039] The posture mapping method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. The application scenario may include a terminal 102 and a server 104, and the terminal 102 and the server 104 may communicate through a communication network. The communication network may be a wired network or a wireless network. Therefore, the terminal 102 and the server 104 may be directly or indirectly connected through a wired or wireless communication method. For example, the terminal 102 may be indirectly connected to the server 104 through a wireless access point, or the terminal 102 may be directly connected to the server 104 through the Internet, and this application does not limit this.

[0040] Among them, the terminal 102 includes but is not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals, aircraft, etc. The embodiment of the present invention can be applied to the virtual anchor live broadcast scene, and is also applicable to multiple other scenes related to posture mapping, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, audio and video, etc. A client related to posture mapping can be installed on the terminal 102, and the client can be software (such as a browser, video software, etc.), or a web page, a small program, etc. The server 104 is a background server corresponding to the software or web page, a small program, etc., or a server specifically used for posture mapping, which is not specifically limited in this application. Further, the server 104 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network, content distribution network), and basic cloud computing services such as big data and artificial intelligence platforms. The data storage system can store the data that the server 104 needs to process. The data storage system may be integrated on the server 104 or placed on the cloud or other servers.

[0041] It should be noted that the posture mapping method in the embodiment of the present application can be executed by the terminal 102 or the server 104 alone, or can be executed by the terminal 102 and the server 104 together.

[0042] In some embodiments, the above method can be performed jointly by the terminal 102 and the server 104. For example, the terminal 102 can obtain the relative position relationship between multiple positioning points of the physical object, determine the posture mapping object of the physical object, and send the relative position relationship and the posture mapping object to the server 104. Then, the server 104 matches each point of the posture mapping object with each positioning point, determines the target point that fails to match in each point, and performs position interpolation processing on the target point according to the spatial position constraint conditions between each point based on the relative position relationship, obtains the interpolation position information of the target point, and performs posture configuration on the posture mapping object based on the relative position relationship and the interpolation position information, and obtains the posture mapping result of the physical object. After obtaining the posture mapping result, the server 104 can distribute the posture mapping result to multiple different terminals for display.

[0043] In some embodiments, when the data processing capability of the terminal 102 meets the data processing requirements, the application environment of the posture mapping method provided in the embodiment of the present application may only involve the terminal 102.

[0044] In some embodiments, the above method may also be performed by the server 104 alone. Figure 2 As shown, during the posture mapping process, the server 104: obtains the relative position relationship between multiple positioning points of the physical object, and determines the posture mapping object of the physical object; matches each point of the posture mapping object with each positioning point, and determines the target point that fails to match among the points; based on the relative position relationship, performs position interpolation processing on the target point according to the spatial position constraint conditions between the points, and obtains the interpolation position information of the target point; based on the relative position relationship and the interpolation position information, performs posture configuration on the posture mapping object, and obtains the posture mapping result of the physical object. After obtaining the posture mapping result, the server 104 can also respond to the acquisition request of the terminal 102 and send the image information containing the posture mapping object to the terminal 102, so as to display the posture mapping object at the terminal 102.

[0045] In a possible implementation, the above-mentioned posture mapping method can be applied to the live broadcast application scenario of the virtual anchor. Specifically, the anchor or organizer of the live broadcast activity can select the posture mapping object, that is, the character image of the virtual anchor in the virtual scene, which can be, for example, a cartoon character or a game character. Then, the server 104 can obtain the image frame taken by the camera during the live broadcast of the anchor, and extract the relative position relationship between the various joints of the anchor in the image frame. Next, the server 104 matches the various virtual joints of the virtual anchor with the various joints of the anchor, and determines the target joints that fail to match in each virtual joint. The target joint can correspond to the missing joint point of the anchor that does not appear in the image frame. Then, based on the relative position relationship between the various joints of the anchor, the server 104 performs position interpolation processing on the target point according to the spatial position constraint conditions between the virtual joints, and obtains the interpolation position information of the target point. Finally, the server 104 performs posture configuration on the virtual anchor based on the relative position relationship and the interpolation position information, obtains the posture mapping result of the anchor, and then obtains the live broadcast image of the virtual anchor that can be displayed to the terminal user.

[0046] In a possible implementation, the above-mentioned posture mapping method can also be applied to the application scenario of robot motion control, and can be specifically applied to the posture configuration of the robot. Specifically, the server 104 can obtain the relative position relationship between multiple positioning points of the human body through the motion capture device, and determine the posture mapping object of the human body, that is, the robot to be posture configured. Each positioning point may include joint points, as well as key points of the human body such as chest points and palms. Then, the server 104 matches each point of the robot with each positioning point of the human body, and determines the target point where the match fails in each point. The target point can be a newly added point of the robot that is different from the human body positioning point, or it can correspond to the human body positioning point where the position data is missing. Among them, the reason for the missing position data may be, for example, the failure of the motion capture device of the human body positioning point, or the motion capture device is not set at the human body positioning point. Then, based on the relative position relationship between the various positioning points of the human body, the server 104 performs position interpolation processing on the target point according to the spatial position constraint conditions between the various points of the robot to obtain the interpolation position information of the target point. Finally, the server 104 performs posture configuration on the robot based on the relative position relationship and the interpolated position information to obtain a posture mapping result of the human body.

[0047] The following is an introduction to the posture mapping method in the present application. As mentioned above, the posture mapping method can be applied to a server, or to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In some possible implementations, if the data processing capability of the terminal meets the data processing requirements of the posture mapping process, the posture mapping method can also be applied to the terminal.

[0048] In some embodiments, Figure 3 As shown, a posture mapping method is provided, which can be executed by a computer device, and the computer device can be Figure 1 In some embodiments, the computer device is a server as an example, and the method includes the following steps:

[0049] Step S302: acquiring the relative position relationship between multiple positioning points of the physical object, and determining the posture mapping object of the physical object.

[0050] Among them, the object refers to an entity that has the ability to change posture and can present a variety of postures. The object can be, for example, a person, an animal, a robot, or other objects that can change posture autonomously, or it can be a vehicle, a toy, or other objects that can achieve posture change through external control. A physical object refers to an object that serves as a mapping basis in the posture mapping process, and a posture mapping object refers to an object that needs to be configured based on the mapping basis in the posture mapping process. The object categories of a physical object and the posture mapping object of the physical object can be the same or different. For example, when the physical object is a real person, the posture mapping object of the physical object can be a virtual person, a robot, or a cartoon animal, etc.

[0051] Further, the posture mapping in the present application may be posture mapping in a real scene, for example, based on the posture information of a real person in the real scene, the posture configuration of the robot in the real scene is performed; it may also be posture mapping from a real scene to a virtual scene, for example, based on the posture information of a real person in the real scene, the posture configuration of a virtual person in the virtual scene is performed; it may also be posture mapping from a virtual scene to a real scene, for example, based on the posture information of a virtual person in the virtual scene, the posture configuration of the robot in the real scene is performed. That is to say, the physical object in the present application and the posture mapping object of the physical object may be a real object in the real scene or a virtual object in the virtual scene. Among them, the real scene refers to a scene existing in the objective world, and the virtual scene refers to a scene created by computer technology that is different from the real scene. The virtual scene may be, for example, a virtual live broadcast scene, a game scene, an animated film scene, or a virtual image display scene. The virtual scene may be a simulation scene of a real scene, a semi-simulation and semi-fictitious scene, or a purely fictitious scene. A virtual object refers to an active object in a virtual scene, such as a virtual person, a virtual animal, or an anime character.

[0052] The positioning points of a physical object refer to the key points used to represent the object posture of the physical object. In other words, the relative position relationship between the positioning points can be used as the posture information of the physical object to characterize the object posture of the physical object. Taking the case where the physical object is a human body as an example, the positioning points can include the joint points of the human body, as well as the key points such as the chest point, the palm, and the midpoint of a specific bone. Figure 4As shown, there are 24 positioning points of the human body defined in an embodiment, wherein point 9 is the chest point, and points 22 and 23 are the palms. Furthermore, the relative position relationship between the positioning points can be characterized by the distance and direction angle between the positions of the points, or by the coordinate information of each positioning point in the same coordinate system. In a specific implementation, the server obtains the coordinate information and confidence of each joint point of the real human body by using a three-dimensional posture recognition model. Among them, the three-dimensional posture recognition model is trained based on a three-dimensional human data set and is a neural network model with a small number of parameters. The three-dimensional human data set includes a human body image carrying three-dimensional annotation information, and the annotation information may include the three-dimensional coordinates of the human body joint points.

[0053] Specifically, the server can obtain various positioning points that can characterize the object posture of the physical object. Among them, each positioning point can be set by the developer, or determined by the server according to the posture transformation characteristics of the physical object. Exemplarily, the server can determine the movable joints of the physical object as the positioning points of the physical object; or determine the skeletal key points (such as the midpoint of the skeleton) of the physical object as the positioning points of the physical object. Then, the server obtains the relative position relationship between multiple positioning points of the physical object and determines the posture mapping object of the physical object. The multiple positioning points refer to at least a part of each positioning point. Further, the relative position relationship is not unique. For example, the server can obtain the relative position relationship between multiple positioning points of the physical object located in the collection area by image acquisition; for another example, the server can obtain the relative position relationship between multiple positioning points of the physical object located in the collection area with the help of a motion capture device. In a specific application, the server obtains the image information of the physical object collected by the camera in real time, and obtains the relative position relationship between multiple positioning points in the image information by performing posture recognition on the image information.

[0054] Step S304, matching each point of the gesture mapping object with each positioning point, and determining a target point among each point that fails to match.

[0055] Among them, similar to the positioning points of the physical object, the key points of the posture mapping object can be used to represent the object posture of the posture mapping object. The points of the posture mapping object refer to the points of the posture mapping object to be configured for posture, including at least a part of the key points of the posture mapping object. For example, at least a part of the joints of the virtual person. Another example is at least a part of the key points of the movable parts such as the front wheels, rear wheels, steering wheels, windows, etc. of the virtual car. In other words, in the process of posture configuration, only a part of the key points of the posture mapping object can be configured for posture. For example, only the points of the upper body of the virtual person can be configured for posture, and the lower body remains stationary or moves according to the set path; for another example, the points of a part of a physical object can be configured based on the posture information of the part of the posture mapping object, and the points of another part of the posture mapping object can be configured based on the posture information of another part of another physical object. For example, in animation movie scenes, there are usually anthropomorphic cartoon animal images, such as anthropomorphic rabbits or turtles, etc. The server can configure the points of the cartoon animal's face based on the facial posture of the human body, and configure the points of the cartoon animal's torso based on the torso posture of the animal.

[0056] Furthermore, the positioning points of the physical object correspond to the points of the posture mapping object of the physical object. That is to say, there is a point mapping relationship between the physical object and the posture mapping object of the physical object. The point mapping relationship can be a one-to-one mapping relationship, such as the mapping relationship between the human joints of the real person and the virtual person; or it can be a many-to-one mapping relationship, for example, the two finger positioning points of the real person correspond to the one finger point of the four-fingered virtual animal.

[0057] Specifically, the server obtains the point mapping relationship between the physical object and the gesture mapping object, and based on the point mapping relationship, matches each point of the gesture mapping object with each positioning point, and determines the non-target points that are successfully matched and the target points that fail to match in each point. Among them, the matching failure means that there is no positioning point that matches the target point among the multiple positioning points whose relative position relationship has been obtained. In other words, based on the obtained relative position relationship, the position information of the target point cannot be determined. For example, in the process of obtaining the relative position relationship between multiple positioning points of a physical object based on image acquisition, the relative position relationship associated with some missing positioning points that are not in the image acquisition area cannot be obtained; for another example, in the process of obtaining the relative position relationship between multiple positioning points of a physical object based on a motion capture device, for the positioning points where the motion capture device fails or the positioning points where the motion capture device is not set, the corresponding relative position relationship cannot be obtained.

[0058] Furthermore, the specific method of performing point matching is not unique. For example, the server can determine the target positioning point where the motion capture device is not installed among the positioning points based on the installation position of the motion capture device, and then determine the target point corresponding to the target positioning point based on the point mapping relationship between the physical object and the posture mapping object. In a specific implementation, step S304 includes: obtaining the object images of the physical object and the posture mapping object respectively; determining the point mapping relationship between the physical object and the posture mapping object based on each object image; matching each point of the posture mapping object with each positioning point according to the point mapping relationship, and determining the target point among each point where the match fails.

[0059] Among them, the point mapping relationship refers to the mapping relationship between the positioning point of the physical object and the point of the posture mapping object of the physical object. Specifically, the server can obtain the object images of the physical object and the posture mapping object respectively, determine the relative position relationship of each key point of the physical object in the object image of the physical object, and the relative position relationship of each key point of the posture mapping object in the object image of the posture mapping object, and perform position matching through each relative position relationship, determine the key points that successfully match as the key points with a mapping relationship, and then obtain the point mapping relationship between the physical object and the posture mapping object. For example, there is a mapping relationship between the shoulder joint of the virtual person and the real person, and for another example, there is a mapping relationship between the palm center of the real person and the forelimb center of the virtual animal. Then, according to the point mapping relationship, the server matches each point of the posture mapping object with each positioning point, and determines the target point that fails to match among each point.

[0060] In practical applications, the server can provide the terminal with object images of multiple candidate posture mapping objects, and the terminal selects the posture mapping object of the physical object from each candidate posture mapping object, and then performs posture configuration on the posture mapping object based on the posture information of the physical object. The object image can be obtained based on image acquisition, or it can be obtained through image modeling. That is to say, in this embodiment, the point mapping relationship is determined based on the object image, and then the point matching is performed based on the point mapping relationship, which is equivalent to realizing posture mapping from one physical object to multiple posture mapping objects, which is conducive to further improving the flexibility of the posture mapping method.

[0061] Step S306: Based on the relative position relationship, position interpolation processing is performed on the target point according to the spatial position constraint conditions between the points to obtain interpolated position information of the target point.

[0062] The spatial position constraint conditions between the points refer to the spatial position constraint conditions between the points, which may specifically include at least one of the connection relationship, distance, direction angle and symmetry between the points. Taking a virtual person as an example, for example, the points on the left hand of the virtual person and the points on the right hand meet the symmetry condition; for another example, the distance between the left shoulder and the left elbow of the virtual person (i.e., the length of the left upper arm) is equal to the distance between the right shoulder and the right elbow (i.e., the length of the right upper arm); for another example, there is a connection relationship between the elbow joint of the virtual person and the shoulder joint and the wrist joint.

[0063] Specifically, based on the relative position relationship between the multiple positioning points of the acquired physical object, the server can determine the relative position relationship between the multiple non-target points matched with each positioning point, and then determine the position information of each non-target point in the mapping coordinate system where the posture mapping object is located. Then, the server uses the position information of each non-target point to perform position interpolation processing on the target point according to the spatial position constraint conditions between the points, and the interpolation position information of the target point can be obtained. Further, the server can determine the associated points of the target point from the non-target points that are successfully matched, and obtain the spatial position constraint conditions between the target point and the associated points, and determine the position information of the associated points in the mapping coordinate system where the posture mapping object is located according to the associated positioning points matched with the associated points in the positioning points of the physical object, and the relative position relationship between the positioning points, and perform position interpolation processing on the target point according to the spatial position constraint conditions based on the position information of the associated points, and obtain the interpolation position information of the target point.

[0064] For example, when there is a target point that has no mapping relationship with the positioning point of the physical object among the multiple points of the gesture mapping object, the server can determine the interpolated position information of the target point based on the position information of two associated points that have a connection relationship with the target point and the spatial position constraints of the target point and each associated point. Figure 5 As shown, take the posture mapping of a real person to a robot as an example. The physical object's positioning point A1 matches the point A2 of the posture mapping object, and the physical object's positioning point B1 matches the point B2 of the posture mapping object. The posture mapping object also includes a target point C2 that has no mapping relationship with the physical object's positioning point, and the spatial position constraints between C2 and A2 and B2 are: the distance between C2 and A2 is d1, and the distance between C2 and B2 is d2. Then Figure 5As shown, the server can determine the interpolated position information of the target point C2 according to the position information of the points A2 and B2 and the above-mentioned spatial position constraints. That is to say, in the process of position interpolation, the present application does not require that the positioning points of the physical object correspond to the points of the gesture mapping object one by one. Figure 5 Point C2 in the figure), if the spatial position constraints of the newly added point and other points can be obtained, the position interpolation processing of the newly added point can be completed.

[0065] Step S308: performing posture configuration on the posture mapping object based on the relative position relationship and the interpolated position information to obtain a posture mapping result of the physical object.

[0066] Among them, the process of performing posture configuration on the posture mapping object is also the process of determining the position information of each point of the posture mapping object. Specifically, based on the relative position relationship and the interpolated position information, the server can determine the position information of each point in the mapping coordinate system where the posture mapping object is located. Then, the server performs posture configuration on the posture mapping object based on each position information to obtain the posture mapping result of the physical object. Furthermore, the server can also obtain the position information of each set point in the posture mapping object that does not need to be posture configured in the mapping coordinate system, and based on the position information of each set point, as well as each relative position relationship and the interpolated position information, perform posture configuration on the posture mapping object to obtain the posture mapping result of the physical object.

[0067] The above-mentioned posture mapping method obtains the relative position relationship between multiple positioning points of the physical object and determines the posture mapping object of the physical object; matches each point of the posture mapping object with each positioning point to determine the target point that fails to match among each point; based on the relative position relationship, performs position interpolation processing on the target point according to the spatial position constraint conditions between each point to obtain the interpolation position information of the target point; based on the relative position relationship and the interpolation position information, performs posture configuration on the posture mapping object to obtain the posture mapping result of the physical object. In the above-mentioned posture mapping process, based on the relative position relationship between multiple positioning points of the physical object, according to the spatial position constraint conditions between each point of the posture mapping object, position interpolation is performed on the target point that fails to match to obtain the interpolation position information of the target point, and then the posture mapping result of the physical object is obtained. It is not necessary to rely on the posture information of the physical object at the previous and next moments, and the position interpolation of the target point at the current moment can be completed. It can be applicable to the scene of real-time posture mapping, and each positioning point of the physical object and each point of the posture mapping object of the physical object do not have to correspond one to one, which is conducive to expanding the application scenario of the posture mapping method.

[0068] In some embodiments, the relative position relationship between multiple positioning points of a physical object is obtained, including: obtaining current posture information of the physical object at a current sampling moment, and historical posture information of the physical object at a sampling moment before the current sampling moment; for each positioning point, correcting the current position of the positioning point using the historical position of the positioning point to obtain the corrected position of the positioning point at the current sampling moment; and determining the relative position relationship between each positioning point based on the corrected position of each positioning point.

[0069] The current posture information includes the current positions of multiple positioning points of the physical object in the physical coordinate system, and the historical posture information includes the historical positions of multiple positioning points of the physical object in the physical coordinate system. The physical coordinate system refers to the coordinate system of the space where the physical object is located.

[0070] Specifically, during the process of posture transformation, the same positioning point of the physical object will change its position before and after the moment. At the same time, the position drift of the sampling device or the physical object itself will also cause the position change of the same positioning point before and after the moment. Nevertheless, the posture of the physical object usually has the characteristic of maintaining a stable time sequence, that is, during the process of posture transformation, there is an inheritance relationship between the current posture information at the current sampling moment and the historical posture information at the previous sampling moment of the current sampling moment. That is, for the same positioning point, the distance between the current position at the current sampling moment and the historical position at the previous sampling moment will not exceed the set threshold. Based on this, the server can obtain the current posture information of the physical object at the current sampling moment and the historical posture information of the physical object at the previous sampling moment of the current sampling moment, and then, for each positioning point, use the historical position of the positioning point to correct the current position of the positioning point, obtain the corrected position of the positioning point at the current sampling moment, and finally, based on the corrected position of each positioning point, determine the relative position relationship between each positioning point.

[0071] It should be noted that, since the changes between the current position and the historical position of each positioning point are different, the correction processing process corresponding to each positioning point is also different. For a certain positioning point, the specific process of the correction processing can be, for example, keeping the current position unchanged, that is, the correction position is the current position; it can also be replacing at least a part of the position information in the current position with the corresponding position information in the historical position, for example, replacing the current position with the historical position, that is, the correction position is the historical position, or replacing the x-coordinate of the current position with the x-coordinate of the historical position, etc.; it can also be obtained by performing statistical calculations on the current position and the historical position, etc. The specific algorithm of the statistical calculation can be averaging, weighted summation, etc.

[0072] In this embodiment, it is equivalent to correcting the current posture information based on the historical posture information, which can ensure the stability of the posture information in time sequence, and further ensure the accuracy of the determined relative position relationship.

[0073] In one embodiment, obtaining the current posture information of the physical object at the current sampling moment and the historical posture information of the physical object at the sampling moment before the current sampling moment includes: using a posture recognition model to determine the current posture information of the physical object at the current sampling moment and the historical posture information of the physical object at the sampling moment before the current sampling moment. In the case of this embodiment, the current position of the positioning point is corrected using the historical position of the positioning point to obtain the corrected position of the positioning point at the current sampling moment, including: obtaining the position confidence of the positioning point at the current sampling moment and the relative distance between the positioning point at the current sampling moment and the previous sampling moment; based on the position confidence and the relative distance, the current position of the positioning point is corrected using the historical position of the positioning point to obtain the corrected position of the positioning point at the current sampling moment.

[0074] Among them, the posture recognition model is a neural network model for identifying the posture information of the physical object. The posture information may include the position information of multiple positioning points of the physical object in the physical coordinate system. The specific type of the posture recognition model is not unique, for example, it may be a convolutional neural network model (CNN), a recurrent neural network model (RNN) or a transformer model. The input of the posture recognition model is the image information of the physical object, and the output is the position information and position confidence of multiple positioning points of the physical object in the physical coordinate system. The position confidence is used to characterize the credibility of the position information of the corresponding positioning point.

[0075] Specifically, the server can obtain an image sample carrying three-dimensional annotation information of a physical object, use the image sample for model training, and obtain a posture recognition model for posture recognition of the physical object. In the process of posture mapping, the server uses the posture recognition model to determine the current posture information of the physical object at the current sampling moment and the historical posture information of the physical object at the previous sampling moment of the current sampling moment. Then, the server obtains the position confidence of the positioning point at the current sampling moment and the relative distance of the positioning point at the current sampling moment and the previous sampling moment. Among them, the relative distance of the same positioning point at the current sampling moment and the previous sampling moment can be calculated based on the position information of the positioning point at the current sampling moment and the position information of the positioning point at the previous sampling moment in the same coordinate system. It can be understood that if the position confidence of the position information of a certain positioning point is low, it means that the position information of the positioning point is unreliable. Similarly, if the relative distance of a certain positioning point at the current sampling moment and the previous sampling moment is too large, it means that the displacement of the positioning point is large, which may not meet the temporal stability characteristics in the posture transformation process, or there is a picture jitter caused by non-posture transformation. Based on this, the server can use the historical position of the positioning point to correct the current position of the positioning point based on the position confidence of the positioning point and the relative distance corresponding to the positioning point, and obtain the corrected position of the positioning point at the current sampling moment.

[0076] In a specific implementation, if the position confidence of the positioning point satisfies the confidence condition and the relative distance corresponding to the positioning point satisfies the distance condition, the position confidence and position information of the positioning point at the current sampling moment are replaced with the position confidence and position information of the positioning point at the previous sampling moment; if the position confidence of the positioning point satisfies the confidence condition and the relative distance corresponding to the positioning point does not satisfy the distance condition, the position information of the positioning point at the current sampling moment is replaced with the position information of the positioning point at the previous sampling moment.

[0077] Among them, the position confidence satisfies the confidence condition, which may mean that the position confidence is less than the confidence threshold, or the position confidence is less than or equal to the confidence threshold. The relative distance satisfies the distance condition, which may mean that the relative distance is less than the distance threshold, or the relative distance is less than or equal to the distance threshold. The distance threshold can be determined based on a set threshold determined to maintain time series stability. For example, the distance threshold can be expressed as D / 8, D / 9 or D / 10, etc., where D is the set threshold.

[0078] Specifically, if the position confidence of the positioning point meets the confidence condition, it means that from the perspective of confidence, the current position of the positioning point is not credible. On this basis, if the relative distance associated with the positioning point meets the distance condition, it means that the moving distance of the positioning point in the time range from the previous sampling moment to the current sampling moment is relatively small, then the position confidence and position information of the positioning point at the current sampling moment are replaced with the position confidence and position information of the positioning point at the previous sampling moment. If the relative distance associated with the positioning point does not meet the distance condition, it means that the moving distance of the positioning point in the time range from the previous sampling moment to the current sampling moment is relatively large, then the position information of the positioning point at the current sampling moment is replaced with the position information of the positioning point at the previous sampling moment without replacing the position confidence.

[0079] Furthermore, taking the case where the physical object is a human body and multiple positioning points include a chest point as an example, if the position confidence of the chest point at the current sampling time satisfies the confidence condition, it means that the output result of the posture recognition model at the current sampling time is unreliable, and the position information and position confidence of the key points need to be corrected. The key points may include, for example, a shoulder point (such as the highest point of the shoulder or a shoulder joint) and a chest point, that is, in this case, the shoulder point and the chest point use the position information and position confidence of the previous sampling time.

[0080] In this embodiment, a posture recognition model is used to obtain the position confidence of the positioning point at the current sampling moment, and based on the position confidence and the relative distance between the positioning point at the current sampling moment and the previous sampling moment, the current position of the positioning point is corrected, which is beneficial to improve the accuracy of the corrected position obtained, and thus improve the accuracy of the subsequent posture mapping results determined based on the corrected position.

[0081] In one embodiment, the relative position relationship between each positioning point is determined based on the respective corrected position of each positioning point, including: filtering and smoothing each corrected position to obtain a smoothed updated position; based on each updated position, determining the relative position relationship between each positioning point of the physical object at the current sampling moment.

[0082] Among them, filtering and smoothing is a process that uses spatial domain filtering technology to achieve the purpose of smoothing and noise reduction. The specific algorithm of filtering and smoothing is not unique, for example, it can be a linear filtering algorithm or a nonlinear filtering algorithm. In a specific implementation, considering that low amplitude offset points may appear after correction, the filtering and smoothing process can be a high-pass filtering process for suppressing low amplitude differences. Figure 6 As shown, during the high-pass filtering process, point 2 and point 6, which have low amplitude offsets relative to the standard line, will be smoothed onto the standard line to avoid jitter of point 2 and point 6.

[0083] Specifically, the server performs filtering and smoothing processing on the corrected position of each positioning point to obtain the updated position of the positioning point after smoothing, and then determines the relative position relationship between the positioning points of the physical object at the current sampling time based on the updated position of each positioning point. It can be understood that if the corrected position of the positioning point itself does not have an offset, there is no need to perform filtering and smoothing processing, then the updated position and the corrected position are the same position, for example Figure 6 Points 1 and 4 in the.

[0084] In this embodiment, filtering and smoothing processing is performed after the correction processing, and then based on the smoothed updated position, the relative position relationship between the positioning points of the physical object at the current sampling moment is determined, which is helpful to avoid the jitter of abnormal points and further improve the accuracy of the relative position relationship.

[0085] In a possible implementation, each corrected position is filtered and smoothed to obtain a smoothed updated position, including: for each positioning point, according to the corrected position of the positioning point and the historical position of the positioning point, determining the corrected relative distance of the positioning point at the current sampling moment and the previous sampling moment; according to the difference between the corrected relative distance and the jitter threshold, determining the respective weight coefficients of the corrected position of the positioning point and the historical position of the positioning point; based on each weight coefficient, performing weighted sum processing on the corrected position of the positioning point and the historical position of the positioning point to obtain the updated position of the positioning point.

[0086] For a certain positioning point, the sum of the weight coefficients of the corrected position of the positioning point and the historical position of the positioning point is 1.

[0087] Specifically, for each positioning point, the server can calculate the corrected relative distance between the positioning point at the current sampling time and the previous sampling time according to the corrected position of the positioning point and the historical position of the positioning point. For example, if the corrected position of a positioning point in the physical coordinate system is (x1, y1, z1) and the historical position of the positioning point is (x2, y2, z2), then the corrected relative distance M between the positioning point at the current sampling time and the previous sampling time can be expressed as It can be seen that the corrected relative distance of the positioning point can reflect the jitter of the positioning point at the current sampling moment relative to the previous sampling moment to a certain extent.

[0088] Then, the server determines the weight coefficients of the corrected position of the positioning point and the historical position of the positioning point according to the difference between the corrected relative distance and the jitter threshold. The jitter threshold may include at least one of an upper limit of the jitter threshold and a lower limit of the jitter threshold. The upper limit of the jitter threshold refers to the maximum value allowed for jitter, such as half the width of a human shoulder. If the corrected relative distance is greater than or equal to the upper limit of the jitter threshold, it is considered to be a real large-scale movement, and the smoothed updated position is the same as the corrected position. The lower limit of the jitter threshold refers to the minimum jitter value under normal circumstances, which is usually caused by the position drift of the sampling device or the physical object itself. If the corrected relative distance is less than or equal to the lower limit of the jitter threshold, the corrected position is considered to be unreliable, and the smoothed updated position is the same as the historical position. If the corrected relative distance is within the jitter allowable range represented by the jitter threshold, the greater the difference between the corrected relative distance and the lower limit of the jitter threshold, the greater the jitter of the positioning point at the current sampling moment relative to the previous sampling moment, the smaller the weight coefficient of the corrected position, and the larger the weight coefficient of the historical position; the greater the difference between the corrected relative distance and the upper limit of the jitter threshold, the smaller the jitter of the positioning point at the current sampling moment relative to the previous sampling moment, the larger the weight coefficient of the corrected position, and the smaller the weight coefficient of the historical position. Further, the difference between the corrected relative distance and the jitter threshold can be obtained by subtracting and / or dividing the two. For example, the difference between the corrected relative distance M and the upper limit of the jitter threshold c can be expressed as M / c, Mc, or (Mb) / (cb), etc., where b can be the lower limit of the jitter threshold or a set value.

[0089] Finally, the server performs weighted sum processing on the corrected position of the positioning point and the historical position of the positioning point based on each weight coefficient to obtain the updated position of the positioning point.

[0090] It should be noted that in actual application, for each current sampling moment, the historical position of the previous sampling moment will be used to correct and filter the current position. Therefore, for a certain positioning point, the historical position of the previous sampling moment used in the process of correcting and filtering the current sampling moment is also the updated position obtained after correction and filtering. In this way, the jitter of the entire image frame sequence can be reduced.

[0091] In this embodiment, the weight coefficients of the corrected position and the historical position of the positioning point are determined based on the difference between the corrected relative distance of the positioning point and the jitter threshold, and then the updated position of the positioning point is determined by weighted summation based on the weight coefficients. This can achieve high-pass filtering smoothing to a certain extent and avoid low-amplitude offsets.

[0092] In a specific application, the jitter threshold includes a jitter threshold lower limit and a jitter threshold upper limit. In the case of this embodiment, according to the difference between the corrected relative distance and the jitter threshold, the weight coefficients of the corrected position of the positioning point and the historical position of the positioning point are determined, including: determining a first difference between the corrected relative distance and the lower limit of the jitter threshold, and a second difference between the upper limit of the jitter threshold and the lower limit of the jitter threshold; when the ratio of the first difference and the second difference meets the ratio condition, determining the ratio as the weight coefficient of the corrected position of the positioning point; according to the weight coefficient of the corrected position of the positioning point, determining the weight coefficient of the historical position of the positioning point.

[0093] Among them, the range interval corresponding to the ratio condition can be any one of (0,1), (0,1], [0,1) or [0,1]. Specifically, the server performs a difference calculation between the corrected relative distance and the lower limit of the jitter threshold to determine the first difference between the corrected relative distance and the lower limit of the jitter threshold, and performs a difference calculation between the upper limit of the jitter threshold and the lower limit of the jitter threshold to determine the second difference between the upper limit of the jitter threshold and the lower limit of the jitter threshold. Then, when the ratio of the first difference and the second difference meets the ratio condition, the ratio is determined as the weight coefficient of the corrected position of the positioning point, and then the weight coefficient of the historical position of the positioning point is determined by the difference between the weight coefficient of the corrected position of the positioning point and 1. Taking the case where the corrected position of a certain positioning point is cur and the historical position is pre as an example, the updated position cur1 of the positioning point can be expressed as: cur1=pre*(1-alpha)+cur*alpha, where alpha is the weight coefficient of the corrected position, (1-alpha) is the weight coefficient of the historical position, and alpha can be expressed as:

[0094]

[0095] Wherein, A=(Mb) / (cb), M is the relative distance after correction, c is the upper limit of the jitter threshold, and b is the lower limit of the jitter threshold.

[0096] In this embodiment, the first difference between the corrected relative distance of the positioning point and the lower limit of the jitter threshold, and the second difference between the upper limit of the jitter threshold and the lower limit of the jitter threshold are determined, and then based on the ratio of the first difference to the second difference, the weight coefficients of the corrected position and the historical position of the positioning point are determined. The algorithm is simple and is conducive to improving data processing efficiency.

[0097] In some embodiments, step S306 includes: determining the associated points of the target point from the non-target points that are successfully matched, and obtaining the spatial position constraints between the target point and the associated points; determining the position information of the associated points in the mapping coordinate system where the posture mapping object is located according to the associated positioning points in each positioning point that match the associated points, and the relative position relationship between the positioning points; based on the position information of the associated points, performing position interpolation processing on the target point according to the spatial position constraints to obtain the interpolated position information of the target point.

[0098] Among them, the associated point of the target point refers to the non-target point that has a spatial position constraint relationship with the target point. That is to say, the associated point of the target point needs to meet two conditions: one is that there is a spatial position constraint relationship with the target point; the other is that among the multiple positioning points of the physical object, there is an associated positioning point that matches the associated point. The spatial position constraint condition between the target point and the associated point refers to the constraint condition between the target point and the associated point in the spatial position, which may specifically include at least one of the connection relationship, distance, direction angle and symmetry between the target point and the associated point. Further, the number of associated points of the target point may be one or more. Exemplarily, when the number of associated points of the target point is one, the spatial position constraint condition between the target point and the associated point may include the direction angle and distance between the target point and the associated point. Among them, the direction angle includes multiple direction angles with each coordinate plane as the projection surface.

[0099] Specifically, the server can determine the associated points of the target point from the non-target points that are successfully matched, and obtain the spatial position constraint conditions between the target point and the associated points. Then, the server can determine the target position relationship related to the associated positioning points in the relative position relationship based on the associated positioning points that match the associated points in the positioning points of the physical object and the relative position relationship between the positioning points of the physical object. The target position relationship is the relative position relationship related to the associated points in the posture mapping object. Then, based on the target position relationship, the server can determine the position information of the associated points in the mapping coordinate system where the posture mapping object is located, and then based on the position information of the associated points, perform position interpolation processing on the target points according to the spatial position constraint conditions to obtain the interpolation position information of the target points.

[0100] Take the case where the target point is the elbow joint of a human body as an example.

[0101] In a specific application, the associated points of the target point may include the wrist joint and the shoulder joint, and the spatial position constraints between the target point and the associated points may include the upper arm length (i.e., the distance between the elbow joint and the shoulder joint) and the forearm length (i.e., the distance between the elbow joint and the wrist joint). The server then performs position interpolation processing on the elbow joint according to the upper arm length and the forearm length based on the respective position information of the wrist joint and the shoulder joint, and thereby determines the differential position information of the elbow joint.

[0102] In another specific application, the associated points of the target point may include the shoulder joint, the upper arm point, the wrist joint and the forearm point, and the spatial position constraints between the target point and the associated points may include: the elbow joint, the shoulder joint and the upper arm point are collinear, and the elbow joint, the wrist joint and the forearm point are collinear. Then, the server can determine the two straight lines where the elbow joint is located based on the respective position information of the shoulder joint, the upper arm point, the wrist joint and the forearm point, and then determine the position information of the intersection of the straight lines as the interpolated position information of the elbow joint. Among them, the upper arm point can be any point in the upper arm, such as the midpoint of the upper arm in the length direction, or a point with a set distance from the shoulder joint; the forearm point can be any point in the forearm, such as the midpoint of the forearm in the length direction, or a point with a preset distance from the wrist joint.

[0103] In another specific application, the associated points of the target point may include the shoulder joint and the upper arm point, and the spatial position constraint between the target point and the associated points may include the upper arm length (i.e., the distance between the elbow joint and the shoulder joint). The server determines the extension direction of the upper arm based on the respective position information of the shoulder joint and the upper arm point, and then performs position interpolation processing on the elbow joint according to the upper arm length to determine the difference position information of the elbow joint.

[0104] In this embodiment, based on the position information of the associated points of the target point, the position interpolation processing is performed on the target point according to the spatial position constraint conditions between the target point and the associated points to obtain the interpolated position information of the target point, which can reduce the workload of the position interpolation processing process and help improve efficiency.

[0105] In one embodiment, there are multiple associated points, and the spatial position constraint condition includes the collinear relationship between the target point and each associated point. In the case of this embodiment, based on the position information of the associated points, the target point is subjected to position interpolation processing according to the spatial position constraint condition to obtain the interpolated position information of the target point, including: determining at least two target lines where the target point is located according to the spatial position constraint condition and the position information of each associated point; and determining the interpolated position information of the target point according to the intersection position between each target straight line.

[0106] Specifically, the server can determine at least two target lines where the target point is located in the mapping coordinate system where the posture mapping object is located, based on the collinear relationship between the target point and each associated point, and the position information of each associated point. Then, the server determines the interpolated position information of the target point based on the intersection positions between the target lines. Exemplarily, when the number of intersections of each target line is one, the server can determine the position information of the intersection of the line as the interpolated position information of the target point; when the number of intersections of each target line is multiple, the server can obtain the interpolated position information of the target point by clustering, averaging or medianizing the position information of each intersection of the lines. Furthermore, if Figure 7 As shown, in a specific implementation, the server may determine the position information of the center of gravity of the triangle formed by the intersection points of the target straight lines l1, l2 and l3 as the interpolation position information of the target point D.

[0107] Taking the case where the target point is the human knee joint as an example, the associated points of the target point may include the hip joint, thigh point, ankle joint and calf point. Among them, the thigh point can be the midpoint of the thigh bone in the length direction, and the calf point can be the midpoint of the calf bone in the length direction. Since the hip joint, thigh point and knee joint are collinear, and the ankle joint, calf point and knee joint are collinear, the server can determine the two target straight lines where the knee joint is located based on the respective position information of the hip joint, thigh point, ankle joint and calf point, and then determine the intersection position information of the two target straight lines as the interpolated position information of the knee joint.

[0108] In a specific application, the posture mapping object is a virtual person, the target point is the elbow joint of the virtual person, and the associated points include the shoulder joint, upper arm point, wrist joint and palm joint of the virtual person. In the case of this embodiment, at least two target straight lines where the target point is located are determined according to the spatial position constraint conditions and the position information of each associated point, including: determining the virtual person's palm extension direction according to the position information of the wrist joint and the palm joint; determining the virtual person's upper arm extension direction according to the position information of the shoulder joint and the upper arm point; determining the palm extension direction and the upper arm extension direction as the two target straight lines where the virtual person's elbow joint is located. According to the intersection position between each target straight line, the interpolation position information of the target point is determined, including: determining the position information of the intersection of the palm extension direction and the upper arm extension direction as the interpolation position information of the virtual person's elbow joint.

[0109] Specifically, during the human body movement in live broadcast scenes, the image often appears out of frame. Figure 8As shown in , the left picture is the image information of a real person, and the right picture is the image information of a virtual person (i.e., the anchor's posture mapping object). When a real person waves, the server cannot obtain the relative position relationship associated with the elbow joint of the real person because the elbow joint is not in the image acquisition area. However, it is necessary to determine the position information of the virtual person's elbow joint in the right picture to ensure that the virtual person's arm can be driven. Based on this, the server determines the respective position information of the virtual person's shoulder joint E, upper arm point F, wrist joint G and palm joint H in the mapping coordinate system according to the relative position relationship between the real person's shoulder joint, upper arm point, wrist joint and palm joint. Among them, the palm joint H may include multiple joint points in the palm, such as Figure 8 The five joint points are shown in the dotted box. Furthermore, the gesture recognition model can be used to obtain the relative position relationship between the joints in the palm part to improve the accuracy of the relative position relationship. Then, the server determines the virtual person's palm extension direction l4 according to the respective position information of the wrist joint G and the palm joint H, and determines the virtual person's upper arm extension direction l5 according to the respective position information of the shoulder joint E and the upper arm point F, and determines the palm extension direction and the upper arm extension direction as the two target straight lines where the virtual person's elbow joint is located. Next, the server determines the position information of the intersection of the palm extension direction and the upper arm extension direction as the interpolated position information of the virtual person's elbow joint I.

[0110] In the above embodiment, the position interpolation processing is performed according to the common relationship between the target point and each associated point. The algorithm is simple and is conducive to further improving the processing efficiency.

[0111] In another specific application, the posture mapping object is a virtual person, the target point is the elbow joint of the virtual person, the associated points include the wrist joint and the palm joint of the virtual person, and the spatial position constraint condition includes the forearm length. Based on the position information of the associated points, the position interpolation processing is performed on the target points according to the spatial position constraint conditions to obtain the interpolation position information of the target points, including: determining the extension direction of the palm of the virtual person according to the respective position information of the wrist joint and the palm joint; determining the interpolation position information of the elbow joint of the virtual person according to the palm extension direction and the forearm length.

[0112] Specifically, the server determines the extension direction of the virtual person's palm based on the position information of the wrist joint and the palm joint, and then extends it in the opposite direction of the palm extension direction. The position on the extension line where the distance from the wrist joint is the length of the forearm is the position of the virtual person's elbow joint, and then obtains the interpolated position information of the virtual person's elbow joint.

[0113] In this embodiment, the position interpolation of joints is performed in combination with the human body structure, which can improve efficiency while ensuring the accuracy of the interpolated position information.

[0114] In one embodiment, the associated points include selected points symmetrical to the target point in the posture mapping object, at least two selected adjacent points of the selected point, and target adjacent points symmetrical to the selected adjacent points; the spatial position constraint condition includes: the target distance between the target adjacent point corresponding to any selected adjacent point and the target point is equal to the distance between the any selected adjacent point and the selected point. In the case of this embodiment, based on the position information of the associated points, the target point is subjected to position interpolation processing according to the spatial position constraint condition to obtain the interpolated position information of the target point, including: based on the position information of each associated point, determining the distance between the selected point and each selected adjacent point; determining the target distance between the target point and each target adjacent point according to each distance and the spatial position constraint condition; based on each target distance, the target point is subjected to position interpolation processing to obtain the interpolated position information of the target point.

[0115] Among them, the symmetry in this embodiment refers to the symmetry of the posture mapping object in the composition. Specifically, the target point is symmetrical with the selected point, the selected adjacent point is symmetrical with the target adjacent point, and the target distance between the target adjacent point and the target point corresponding to any selected adjacent point is equal to the distance between any selected adjacent point and the selected point. Exemplarily, if the posture mapping object is a virtual person, the virtual person's left elbow joint is symmetrical with the right elbow joint, the left shoulder joint is symmetrical with the right shoulder joint, and the length of the left upper arm (i.e., the distance between the left elbow joint and the left shoulder joint) is equal to the length of the right upper arm (i.e., the distance between the right elbow joint and the right shoulder joint); if the posture mapping object is a virtual vehicle, the center of the left rearview mirror of the virtual vehicle is symmetrical with the center of the right rearview mirror, the center of the left taillight is symmetrical with the center of the right taillight, and the distance between the center of the left rearview mirror and the center of the left taillight is equal to the distance between the center of the right rearview mirror and the center of the right taillight.

[0116] Specifically, the server determines the distance between the selected point and each selected adjacent point based on the position information of each associated point, and then determines the target distance between the target point and each target adjacent point based on each distance and spatial position constraint. Finally, the server performs position interpolation processing on the target point based on each target distance to obtain the interpolated position information of the target point.

[0117] Take the case where the target point is the left elbow joint of the virtual person as an example. Fig. 9As shown, the associated points may include the right elbow joint J, right shoulder joint K, right wrist joint L, left shoulder joint E and left wrist joint G of the virtual person. The server may determine the distance between the right elbow joint J and the right shoulder joint K, i.e., the right upper arm length d3, and the distance between the right elbow joint and the right wrist joint, i.e., the right forearm length d4, based on the position information of each associated point. Then, according to the spatial position constraint condition that the right upper arm length is equal to the left upper arm length and the right forearm length is equal to the left forearm length, the left upper arm length and the left forearm length of the virtual person are determined. Then, in combination with the respective position information of the left shoulder joint E and the left wrist joint G, the position interpolation processing is performed on the left elbow joint I to obtain the interpolated position information of the left elbow joint I.

[0118] In this embodiment, it is equivalent to performing position interpolation processing on the target point based on the symmetry of the posture mapping object, which can be applied to various types of posture mapping objects with symmetrical structures, and is conducive to further expanding the application scenarios of the posture mapping method.

[0119] In some embodiments, step S380 includes: determining the current mapping position of each point in the mapping coordinate system where the posture mapping object is located at the current mapping moment based on the relative position relationship and the interpolation position information; obtaining the historical mapping position of each point of the posture mapping object at the mapping moment before the current mapping moment; for each point, based on the historical mapping position of the point, correcting the current mapping position of the point to obtain the corrected mapping position of the point; using each corrected mapping position to perform posture configuration on the posture mapping object to obtain a posture mapping result of the physical object.

[0120] Specifically, after the interpolation position information of the target point is obtained by the sampling interpolation method, there may be a situation where the amplitude of the target point at the current mapping moment is larger than that of the previous mapping moment. If the interpolation position information is used directly for posture mapping, the image may be jittered due to the large amplitude, resulting in a poor visual experience of stuttering. Based on this, after obtaining the interpolation position information of the target point, the server determines the current mapping position of each point in the mapping coordinate system where the posture mapping object is located at the current mapping moment according to the relative position relationship and the interpolation position information, and obtains the historical mapping position of each point of the posture mapping object at the previous mapping moment of the current mapping moment. Then, the server corrects the current mapping position of each point based on the historical mapping position of the point to obtain the corrected mapping position of the point. Finally, the server uses each corrected mapping position to perform posture configuration on the posture mapping object to obtain the posture mapping result of the physical object.

[0121] Similar to the correction process for the positioning point, since the changes between the current mapping position and the historical mapping position of each point are different, the correction process corresponding to different points is also different. For a certain point, the specific process of the correction process can be, for example, to keep the current mapping position unchanged, that is, the correction mapping position is the current mapping position; it can also be to replace at least part of the position information in the current mapping position with the corresponding position information in the historical mapping position, such as replacing the current mapping position with the historical mapping position, that is, the correction mapping position is the historical mapping position, or replacing the x-coordinate of the current mapping position with the x-coordinate of the historical mapping position, etc.; it can also be to obtain the correction mapping position by performing statistical calculations on the current mapping position and the historical mapping position, etc. The specific algorithm of the statistical calculation can be averaging, weighted summation, etc.

[0122] Exemplarily, when the posture recognition model is applied to determine the current posture information and historical posture information of the physical object, the position confidence of each non-target point of the posture mapping object can inherit the position confidence of the positioning point in the physical object that matches the non-target point, and the position confidence of each target point of the posture mapping object can be the position confidence of the non-target point in the posture mapping object that has a spatial position constraint condition with the target point. In the case where there are multiple non-target points with spatial position constraints with the target point, the position confidence of the target point can be the average or median of the position confidence of each non-target point.

[0123] Based on this, for a certain point, the server can obtain the position confidence of the point at the current mapping moment, and the relative distance of the point at the current mapping moment and the previous mapping moment, and then determine the fusion weights of the current mapping position and the historical mapping position based on the relative distance and position confidence, and then perform weighted summation on the current mapping position and the historical mapping position based on each fusion weight to obtain the corrected mapping position. Among them, the fusion weight is inversely correlated with the position confidence, the fusion weight of the current mapping position is inversely correlated with the relative distance, and the sum of the fusion weight of the current mapping position and the fusion weight of the historical mapping position is 1. In a specific application, the fusion weight of the current mapping position can be dynamically adjusted according to the relative distance and the position confidence. For example, the initial fusion weight of the current mapping position can be obtained according to the relative distance and the corresponding relationship between the relative distance and the initial fusion weight. If the position confidence of the previous mapping position is less than the threshold (or less than or equal to the threshold), the fusion weight of the current mapping position is the product of the set coefficient and the initial fusion weight, wherein the set coefficient is less than 1. Otherwise, the fusion weight of the current mapping position is the initial fusion weight.

[0124] In this embodiment, before performing posture mapping based on the interpolated position information and the relative position relationship, a correction process is also performed to further ensure the accuracy of the posture mapping result.

[0125] In some embodiments, each correction mapping position is used to configure the posture of the posture mapping object to obtain the posture mapping result of the physical object, including: filtering and smoothing each correction mapping position to obtain an updated mapping position; using each updated mapping position to configure the posture of the posture mapping object to obtain the posture mapping result of the physical object.

[0126] Among them, filtering and smoothing is a process that uses spatial domain filtering technology to achieve the purpose of smoothing and noise reduction. The specific algorithm of filtering and smoothing is not unique, for example, it can be a linear filtering algorithm or a nonlinear filtering algorithm. In a specific implementation, considering that high amplitude offset points may appear after interpolation, the filtering and smoothing process can be a low-pass filtering process for suppressing high amplitude differences. Fig.10 As shown, during the low-pass filtering process, point 3, which has a high amplitude offset relative to the standard line, will be smoothed onto the standard line to avoid jitter of point 3.

[0127] Specifically, the server performs filtering and smoothing processing on each correction mapping position to obtain a smoothed update mapping position, and then uses each update mapping position to perform posture configuration on the posture mapping object to obtain a posture mapping result of the physical object. It can be understood that if the correction mapping position of a certain point does not have an offset, there is no need to perform filtering and smoothing processing, then the update mapping position and the correction mapping position are the same position, for example Fig.10 Point 1, point 2, point 4 and point 6.

[0128] In this embodiment, filtering and smoothing processing is performed after correction processing, and posture configuration is performed based on the smoothed update mapping position, which can avoid the jitter of abnormal points and further improve the accuracy of posture mapping results.

[0129] As mentioned above, the gesture mapping method of the present application can be executed by a computer device, which can be Figure 1 In some embodiments, the computer device is a server as an example. Fig.11 As shown, the posture mapping method includes the following steps:

[0130] Step S1101, using a posture recognition model, determining current posture information of a physical object at a current sampling moment, and historical posture information of the physical object at a sampling moment before the current sampling moment;

[0131] The current posture information includes the current positions of the multiple positioning points of the physical object in the physical coordinate system; the historical posture information includes the historical positions of the multiple positioning points of the physical object in the physical coordinate system;

[0132] Step S1102: for each positioning point, obtain the position confidence of the positioning point at the current sampling time, and the relative distance between the positioning point at the current sampling time and the previous sampling time; based on the position confidence and the relative distance, use the historical position of the positioning point to correct the current position of the positioning point, and obtain the corrected position of the positioning point at the current sampling time;

[0133] Step S1103, determining the corrected relative distance between the positioning point at the current sampling moment and the previous sampling moment according to the corrected position of the positioning point and the historical position of the positioning point;

[0134] Step S1104, determining the weight coefficients of the corrected position of the positioning point and the historical position of the positioning point according to the difference between the corrected relative distance and the jitter threshold;

[0135] Step S1105, based on each weight coefficient, weighted sum processing is performed on the corrected position of the positioning point and the historical position of the positioning point to obtain an updated position of the positioning point;

[0136] Step S1106, determining the relative position relationship between the positioning points of the physical object at the current sampling time based on the updated positions;

[0137] Step S1107, determining a posture mapping object of the physical object, and acquiring object images of the physical object and the posture mapping object respectively;

[0138] Step S1108, determining a point mapping relationship between a physical object and a gesture mapping object based on each object image;

[0139] Step S1109, matching each point of the posture mapping object with each positioning point according to the point mapping relationship, and determining the target point that fails to match among the points;

[0140] Step S1110, determining the associated points of the target point from the non-target points that are successfully matched, and obtaining the spatial position constraint conditions between the target point and the associated points;

[0141] Step S1111, determining the position information of the associated point in the mapping coordinate system where the gesture mapping object is located according to the associated positioning points among the positioning points of the physical object that match the associated point and the relative position relationship between the positioning points;

[0142] Step S1112, based on the position information of the associated points, performing position interpolation processing on the target points according to the spatial position constraint conditions to obtain the interpolated position information of the target points;

[0143] Step S1113, determining the current mapping position of each point in the mapping coordinate system where the posture mapping object is located at the current mapping moment according to the relative position relationship and the interpolation position information;

[0144] Step S1114, obtaining the historical mapping position of each point of the gesture mapping object at the previous mapping moment before the current mapping moment;

[0145] Step S1115, for each point, based on the historical mapping position of the point, correct the current mapping position of the point to obtain the corrected mapping position of the point;

[0146] Step S1116, filtering and smoothing are performed on each corrected mapping position to obtain an updated mapping position, and each updated mapping position is used to perform posture configuration on the posture mapping object to obtain a posture mapping result of the physical object.

[0147] The following takes the posture mapping process from real people to virtual people as an example to explain the posture mapping method in detail. Among them, the posture mapping method from real people to virtual people can be applied to the live broadcast field or the game field. Based on computer vision understanding technology, the posture information of each part of the human body can be obtained from the video of the real person, and then the virtual person can be driven according to this posture information, that is, the posture of the virtual person is configured to obtain the posture mapping result of the real person, so as to achieve the same effect of the real person and the virtual person's movements, realize the synchronous live broadcast of the virtual person, and related functions of human-computer interaction. Figure 8 As shown, the trained human 3D posture recognition model is input Figure 8 The real person image in the middle left picture can obtain the coordinate information of the corresponding positioning points of the human body, and then complete the subsequent driving of the virtual person and realize posture mapping.

[0148] In a specific implementation, such as Fig.12 As shown in the figure, the whole posture mapping process includes the following six steps:

[0149] Step 1: Get human body posture information.

[0150] Among them, the human body posture information may include the human body's torso posture information and palm posture information. Specifically, the coordinate information and position confidence of each positioning point of the human body can be obtained through a trained torso posture recognition model, and the coordinate information and position confidence of multiple positioning points in the human palm can be obtained through a trained palm posture recognition model. Since the torso posture is coarse-grained posture information relative to the palm posture, using different models to perform torso posture recognition and palm posture recognition respectively is conducive to improving the accuracy of the posture information obtained. Furthermore, each positioning point may include a joint point, as well as key points such as a chest point and an eyeball center.

[0151] Step 2: Correct the trunk posture information and palm posture information according to the human body prior to obtain the corrected posture information.

[0152] There are two prior assumptions about human motion here: one is that human motion has the characteristic of maintaining a smooth time sequence; the other is that during normal motion, the distance between the same positioning point in the previous and next frames will not exceed a certain threshold value D. Among them, the frame rate for image sampling can be, for example, 30fps. Based on this, the server can use the distance M of the same positioning point between the current frame and the previous frame, and the position confidence of the positioning point in the current frame to correct the current coordinate position of the positioning point in the current frame to obtain the corrected position. For example, when the position confidence of the positioning point in the current frame is lower than the threshold, if M is less than D / 10, the current coordinate position of the current frame is directly replaced by the historical coordinate position of the previous frame, and the position confidence of the current frame is replaced by the position confidence of the previous frame; if M is greater than or equal to D / 10, only the coordinate position is replaced, and the position confidence is not replaced. For another example, when the chest point confidence of the current frame is lower than the threshold, the current coordinate position and position confidence of the shoulder and chest point of the current frame are replaced by the historical coordinate position and position confidence of the corresponding positioning point in the previous frame.

[0153] Step 3: Use high-pass filtering to smooth low-amplitude positioning points.

[0154] Assuming that the coordinates of a certain positioning point in the previous frame are pre, and the coordinates of the positioning point in the current frame after correction are cur, the updated position cur1 of the positioning point can be expressed as: cur1 = pre*(1-alpha)+cur*alpha, where alpha is the weight coefficient of the correction position, (1-alpha) is the weight coefficient of the historical position, and alpha can be expressed as:

[0155]

[0156] Among them, A=(M1-b) / (cb), M1 is the corrected position of the positioning point in the current frame after correction, and the relative distance to the historical position of the positioning point in the previous frame; b is the lower limit of the jitter threshold, which refers to the minimum jitter value under normal circumstances; c is the upper limit of the jitter threshold, for example, it can be half of the shoulder width. If it exceeds c, it is considered to be a large-scale movement of a real person.

[0157] It should be noted that in actual applications, since each frame will use the historical position of the previous frame for correction and smoothing, for a certain positioning point, the historical position of the previous frame used in the process of correcting and smoothing the current frame is also the updated position obtained after correction and smoothing. High-pass filtering suppresses low-amplitude differences, but does not suppress high-amplitude differences, and considers them all to be real movements. Figure 6 As shown, points 2 and 6 are low-amplitude offsets relative to the standard line. After high-pass filtering, they will be smoothed to the standard line to avoid jitter at points 2 and 6.

[0158] Step 4: Interpolate the torso positioning points based on the palm posture information.

[0159] In actual applications, anchors usually use terminals to collect their own image information. During human motion, images often appear out of the picture. Figure 8 In the left picture, when the anchor is waving, the server cannot obtain the elbow joint position information of the real person because the elbow joint is not in the image acquisition area. However, it is necessary to determine the elbow joint position information of the virtual person in the right picture to ensure that the virtual person's arm can be driven. Based on this, the server combines the palm extension direction and forearm length to interpolate the elbow joint position information.

[0160] The fifth step is to perform secondary correction on the position information based on human body priors.

[0161] Since the sampling interpolation method in the fourth step obtains the interpolated position information, and the interpolated position information often has a large amplitude, and the insufficient stability of the posture recognition model may also cause the same positioning point to send a large offset in the positions of the previous and next frames, based on this, after obtaining the interpolated position information, the server performs a secondary correction on the position information, that is, weighted fusion of the coordinate information of the current frame and the previous frame, and the fusion weight is dynamically adjusted according to the relative distance and position confidence between the current frame and the previous frame.

[0162] Step 6: Use low-pass filtering for smoothing.

[0163] Specifically, although the fifth step corrects the interpolated position information to avoid large amplitude deviation, the amplitude change will still cause jitter and smoothing is still required. Since the interpolated points will have a larger amplitude than the model input points, the filtering here considers using low-pass filtering and only smoothing the points whose distance between the front and back is greater than a certain threshold, for example Fig.10 In the figure, after low-pass filtering, point 3 will be smoothed to the standard line.

[0164] After the second smoothing process, the updated mapping position of each point of the virtual person can be obtained. Then, the server uses each updated mapping position to perform posture configuration on the virtual person to obtain the posture mapping result of the real person.

[0165] like Fig.13 This is a schematic diagram of the mapping effect after interpolation without correction and smoothing. Fig.14 The following is a diagram of the mapping effect after correction and smoothing after interpolation. It can be found that if the right elbow is out of the picture, there will be obvious joint jitter if no secondary correction and smoothing are performed, which is manifested as the virtual person's right hand shaking, that is, there is a significant position difference between the right elbow joint of the previous and next frames. After the secondary correction and smoothing after interpolation, the jitter phenomenon has been significantly improved.

[0166] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0167] Based on the same inventive concept, the embodiment of the present application also provides a posture mapping device for implementing the posture mapping method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more posture mapping device embodiments provided below can refer to the limitations on the posture mapping method above, and will not be repeated here.

[0168] In some embodiments, Fig.15As shown, a posture mapping device 1500 is provided, including an acquisition module 1502, a point configuration module 1504, a position interpolation module 1506 and a posture mapping module 1508.

[0169] An acquisition module 1502 is used to acquire the relative position relationship between multiple positioning points of a physical object and determine a posture mapping object of the physical object;

[0170] The point configuration module 1504 is used to match each point of the gesture mapping object with each positioning point, and determine the target point that fails to match among the points;

[0171] The position interpolation module 1506 is used to perform position interpolation processing on the target point based on the relative position relationship and in accordance with the spatial position constraint conditions between the points to obtain the interpolation position information of the target point;

[0172] The posture mapping module 1508 is used to perform posture configuration on the posture mapping object based on the relative position relationship and the interpolated position information to obtain the posture mapping result of the physical object.

[0173] In some embodiments, the acquisition module 1502 includes: a posture information acquisition unit, which is used to acquire the current posture information of the physical object at the current sampling moment, and the historical posture information of the physical object at the sampling moment before the current sampling moment; the current posture information includes the current position of each of the multiple positioning points of the physical object in the physical coordinate system; the historical posture information includes the historical position of each of the multiple positioning points of the physical object in the physical coordinate system; a positioning point correction unit, which is used to correct the current position of each positioning point using the historical position of the positioning point to obtain the corrected position of the positioning point at the current sampling moment; a relative position relationship determination unit, which is used to determine the relative position relationship between each positioning point based on the corrected position of each positioning point.

[0174] In some embodiments, the posture information acquisition unit is specifically used to: use the posture recognition model to determine the current posture information of the physical object at the current sampling moment, and the historical posture information of the physical object at the previous sampling moment of the current sampling moment. In the case of this embodiment, the positioning point correction unit is specifically used to: obtain the position confidence of the positioning point at the current sampling moment, and the relative distance between the positioning point at the current sampling moment and the previous sampling moment; based on the position confidence and the relative distance, use the historical position of the positioning point to correct the current position of the positioning point, and obtain the corrected position of the positioning point at the current sampling moment.

[0175] In some embodiments, the relative position relationship determination unit includes: an update position determination component, which is used to filter and smooth each corrected position to obtain a smoothed update position; and a relative position relationship determination component, which is used to determine the relative position relationship between each positioning point of the physical object at the current sampling moment based on each updated position.

[0176] In one embodiment, the update position determination component is specifically used to: for each positioning point, determine the corrected relative distance between the positioning point at the current sampling moment and the previous sampling moment according to the corrected position of the positioning point and the historical position of the positioning point; determine the weight coefficients of the corrected position of the positioning point and the historical position of the positioning point according to the difference between the corrected relative distance and the jitter threshold; based on each weight coefficient, perform weighted sum processing on the corrected position of the positioning point and the historical position of the positioning point to obtain the updated position of the positioning point.

[0177] In a specific application, the jitter threshold includes a jitter threshold lower limit and a jitter threshold upper limit. In the case of this embodiment, the position determination component is updated, specifically for: determining a first difference between the corrected relative distance and the jitter threshold lower limit, and a second difference between the jitter threshold upper limit and the jitter threshold lower limit; when the ratio of the first difference to the second difference meets the ratio condition, determining the ratio as the weight coefficient of the corrected position of the positioning point; and determining the weight coefficient of the historical position of the positioning point according to the weight coefficient of the corrected position of the positioning point.

[0178] In some embodiments, the point configuration module 1504 is specifically used to: obtain the object images of the physical object and the posture mapping object respectively; determine the point mapping relationship between the physical object and the posture mapping object based on each object image; according to the point mapping relationship, match each point of the posture mapping object with each positioning point, and determine the target point among each point where the match fails.

[0179] In some embodiments, the position interpolation module 1506 includes: an associated point determination unit, which is used to determine the associated points of the target point from the non-target points that have been successfully matched, and obtain the spatial position constraints between the target point and the associated points; a position information determination unit, which is used to determine the position information of the associated points in the mapping coordinate system where the posture mapping object is located according to the associated positioning points in each positioning point that match the associated points, and the relative position relationship between each positioning point; a position interpolation unit, which is used to perform position interpolation processing on the target point according to the spatial position constraints based on the position information of the associated points, and obtain the interpolated position information of the target point.

[0180] In one embodiment, there are multiple associated points, and the spatial position constraint condition includes a collinear relationship between the target point and each associated point. In the case of this embodiment, the position interpolation unit is specifically used to: determine at least two target lines where the target point is located according to the spatial position constraint condition and the position information of each associated point; and determine the interpolation position information of the target point according to the intersection position between each target straight line.

[0181] In a specific application, the posture mapping object is a virtual person, the target point is the elbow joint of the virtual person, the associated points include the wrist joint and the palm joint of the virtual person, and the spatial position constraint condition includes the forearm length. In the case of this embodiment, the position interpolation unit is specifically used to: determine the extension direction of the palm of the virtual person according to the position information of the wrist joint and the palm joint; determine the interpolation position information of the elbow joint of the virtual person according to the palm extension direction and the forearm length.

[0182] In one embodiment, the associated points include a selected point symmetrical to the target point in the posture mapping object, at least two selected adjacent points of the selected point, and a target adjacent point symmetrical to the selected adjacent point; the spatial position constraint condition includes: the target distance between the target adjacent point corresponding to any selected adjacent point and the target point is equal to the distance between the selected adjacent point and the selected point. In the case of this embodiment, the position interpolation unit is specifically used to: determine the distance between the selected point and each selected adjacent point based on the position information of each associated point; determine the target distance between the target point and each target adjacent point according to each distance and the spatial position constraint condition; perform position interpolation processing on the target point based on each target distance to obtain the interpolated position information of the target point.

[0183] In some embodiments, the posture mapping module 1508 includes: a current mapping position determination unit, which is used to determine the current mapping position of each point in the mapping coordinate system where the posture mapping object is located at the current mapping moment based on the relative position relationship and the interpolation position information; a historical mapping position acquisition unit, which is used to obtain the historical mapping position of each point of the posture mapping object at the mapping moment before the current mapping moment; a mapping position correction unit, which is used to correct the current mapping position of each point based on the historical mapping position of the point to obtain the corrected mapping position of the point; a posture mapping unit, which is used to use each corrected mapping position to perform posture configuration on the posture mapping object to obtain the posture mapping result of the physical object.

[0184] In some embodiments, the gesture mapping unit is specifically used to: perform filtering and smoothing processing on each corrected mapping position to obtain an updated mapping position; and perform gesture configuration on the gesture mapping object using each updated mapping position to obtain a gesture mapping result of the physical object.

[0185] The above-mentioned posture mapping device, during the posture mapping process, based on the relative position relationship between multiple positioning points of the physical object and in accordance with the spatial position constraints between the various points of the posture mapping object, performs position interpolation on the target points that fail to match, so as to obtain the interpolated position information of the target points, and then obtain the posture mapping result of the physical object. It does not need to rely on the posture information of the physical object at the previous and next moments, and can complete the position interpolation of the target point at the current moment. It can be suitable for real-time posture mapping scenarios. In addition, the various positioning points of the physical object and the various points of the posture mapping object of the physical object do not have to correspond one-to-one, which is conducive to expanding the application scenarios of the posture mapping method.

[0186] Each module in the above-mentioned posture mapping device can be implemented in whole or in part by software, hardware and a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module above.

[0187] In some embodiments, a computer device is provided. The computer device may be a terminal or a server. In this embodiment, the computer device is described as a server. The internal structure diagram thereof may be as follows: Fig.16 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data involved in the above method. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a posture mapping method is implemented.

[0188] Those skilled in the art will understand that Fig.16 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0189] In some embodiments, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned gesture mapping method when executing the computer program.

[0190] In some embodiments, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned gesture mapping method are implemented.

[0191] In some embodiments, a computer program product is provided, including a computer program, which implements the steps in the above-mentioned gesture mapping method when executed by a processor.

[0192] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0193] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0194] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0195] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A posture mapping method, characterized in that: The method comprises: Acquire current posture information of the physical object at a current sampling moment, and historical posture information of the physical object at a sampling moment before the current sampling moment; the current posture information includes the current positions of each of a plurality of positioning points of the physical object in a physical coordinate system; the historical posture information includes the historical positions of each of a plurality of positioning points of the physical object in the physical coordinate system; For each of the positioning points, correct the current position of the positioning point using the historical position of the positioning point to obtain the corrected position of the positioning point at the current sampling moment; For each of the positioning points, determining a corrected relative distance between the positioning point at the current sampling moment and the previous sampling moment according to the corrected position of the positioning point and the historical position of the positioning point; Determine a first difference between the corrected relative distance and a lower limit of a jitter threshold, and a second difference between an upper limit of a jitter threshold and the lower limit of the jitter threshold; When a ratio of the first difference to the second difference satisfies a ratio condition, determining the ratio as a weight coefficient of the corrected position of the positioning point; Determining a weight coefficient of a historical position of the positioning point according to a weight coefficient of the corrected position of the positioning point; Based on each of the weight coefficients, a weighted sum process is performed on the corrected position of the positioning point and the historical position of the positioning point to obtain an updated position of the positioning point; Based on each of the updated positions, determining the relative position relationship between each of the positioning points of the physical object at the current sampling moment; Determining a posture mapping object of the physical object; Matching each point of the posture mapping object with each positioning point, and determining a target point among the points where the matching fails; Based on the relative position relationship, the target point is subjected to position interpolation processing according to the spatial position constraint conditions between the points to obtain the interpolation position information of the target point; the spatial position constraint conditions include at least one of the connection relationship, distance, direction angle and symmetry between the points; Based on the relative position relationship and the interpolation position information, the posture mapping object is posture configured to obtain a posture mapping result of the physical object.

2. The method according to claim 1, characterized in that The acquiring of current posture information of the physical object at the current sampling moment and historical posture information of the physical object at a sampling moment before the current sampling moment includes: Using a posture recognition model, determining current posture information of a physical object at a current sampling moment and historical posture information of the physical object at a sampling moment before the current sampling moment; The correcting the current position of the positioning point by using the historical position of the positioning point to obtain the corrected position of the positioning point at the current sampling moment includes: Obtaining the position confidence of the positioning point at the current sampling moment, and the relative distance between the positioning point at the current sampling moment and the previous sampling moment; Based on the position confidence and the relative distance, the current position of the positioning point is corrected using the historical position of the positioning point to obtain the corrected position of the positioning point at the current sampling moment.

3. The method according to claim 1, characterized in that The step of matching each point of the posture mapping object with each positioning point and determining a target point among the points where the matching fails comprises: Acquire respective object images of the physical object and the gesture mapping object; Determining a point mapping relationship between the physical object and the gesture mapping object based on each of the object images; According to the point mapping relationship, each point of the posture mapping object is matched with each positioning point, and a target point that fails to match among the points is determined.

4. The method according to claim 1, characterized in that: The step of performing position interpolation processing on the target point based on the relative position relationship and in accordance with the spatial position constraint conditions between the points to obtain the interpolation position information of the target point includes: Determine the associated points of the target point from the non-target points that are successfully matched, and obtain the spatial position constraint conditions between the target point and the associated points; Determine the position information of the associated point in the mapping coordinate system where the posture mapping object is located according to the associated positioning point among the positioning points that matches the associated point and the relative position relationship between the positioning points; Based on the position information of the associated point, position interpolation processing is performed on the target point according to the spatial position constraint condition to obtain interpolated position information of the target point.

5. The method according to claim 4, characterized in that The number of the associated points is multiple, and the spatial position constraint condition includes a collinear relationship between the target point and each of the associated points; The step of performing position interpolation processing on the target point based on the position information of the associated point according to the spatial position constraint condition to obtain the interpolated position information of the target point includes: Determine at least two target straight lines where the target point is located according to the spatial position constraint condition and the position information of each of the associated points; The interpolation position information of the target point is determined according to the intersection positions between the target straight lines.

6. The method according to claim 4, characterized in that The associated points include a selected point in the posture mapping object that is symmetrical to the target point, at least two selected adjacent points of the selected point, and a target adjacent point that is symmetrical to the selected adjacent point; The spatial position constraint condition includes: the target distance between the target adjacent point corresponding to any selected adjacent point and the target point is equal to the distance between any selected adjacent point and the selected point; The step of performing position interpolation processing on the target point based on the position information of the associated point according to the spatial position constraint condition to obtain the interpolated position information of the target point includes: Based on the position information of each of the associated points, determining the distance between the selected point and each of the selected adjacent points; Determining a target distance between the target point and each of the target adjacent points according to each of the distances and the spatial position constraint condition; Based on each of the target distances, position interpolation processing is performed on the target point to obtain interpolation position information of the target point.

7. The method according to claim 4, characterized in that The posture mapping object is a virtual person, the target point is the elbow joint of the virtual person, the associated points include the wrist joint and the palm joint of the virtual person; the spatial position constraint condition includes the forearm length; The step of performing position interpolation processing on the target point based on the position information of the associated point according to the spatial position constraint condition to obtain the interpolated position information of the target point includes: Determine the extension direction of the virtual person's palm according to the position information of the wrist joint and the palm joint; The interpolation position information of the elbow joint of the virtual person is determined according to the palm extension direction and the forearm length.

8. The method according to any one of claims 1 to 7, characterized in that The step of performing posture configuration on the posture mapping object based on the relative position relationship and the interpolation position information to obtain a posture mapping result of the physical object includes: Determine the current mapping position of each of the points in the mapping coordinate system where the posture mapping object is located at the current mapping moment according to the relative position relationship and the interpolation position information; Acquire the historical mapping positions of the points of the gesture mapping object at a mapping moment before the current mapping moment; For each of the points, based on the historical mapping position of the point, correct the current mapping position of the point to obtain the corrected mapping position of the point; The posture mapping object is posture-configured using each of the corrected mapping positions to obtain a posture mapping result of the physical object.

9. The method according to claim 8, characterized in that The step of performing posture configuration on the posture mapping object using each of the correction mapping positions to obtain a posture mapping result of the physical object includes: Performing filtering and smoothing processing on each of the corrected mapping positions to obtain an updated mapping position; The gesture mapping object is gesture configured using each of the updated mapping positions to obtain a gesture mapping result of the physical object.

10. A posture mapping device, characterized in that: The device comprises: A posture information acquisition unit, used to acquire current posture information of the physical object at a current sampling moment, and historical posture information of the physical object at a sampling moment before the current sampling moment; the current posture information includes the current positions of each of a plurality of positioning points of the physical object in a physical coordinate system; the historical posture information includes the historical positions of each of a plurality of positioning points of the physical object in the physical coordinate system; A positioning point correction unit, used for correcting the current position of each positioning point by using the historical position of the positioning point to obtain the corrected position of the positioning point at the current sampling moment; An update position determination component is used to determine, for each of the positioning points, the corrected relative distance of the positioning point at the current sampling moment and the previous sampling moment according to the corrected position of the positioning point and the historical position of the positioning point; determine a first difference between the corrected relative distance and the lower limit of the jitter threshold, and a second difference between the upper limit of the jitter threshold and the lower limit of the jitter threshold; when the ratio of the first difference to the second difference meets the ratio condition, determine the ratio as the weight coefficient of the corrected position of the positioning point; determine the weight coefficient of the historical position of the positioning point according to the weight coefficient of the corrected position of the positioning point; based on each of the weight coefficients, perform weighted sum processing on the corrected position of the positioning point and the historical position of the positioning point to obtain the updated position of the positioning point; A relative position relationship determination component, used to determine the relative position relationship between each of the positioning points of the physical object at the current sampling moment based on each of the updated positions; A point configuration module, used for matching each point of the posture mapping object with each positioning point, and determining a target point among the points that fails to match; A position interpolation module, used for performing position interpolation processing on the target point according to the spatial position constraint conditions between the points based on the relative position relationship to obtain interpolation position information of the target point; the spatial position constraint conditions include at least one of the connection relationship, distance, direction angle and symmetry between the points; The posture mapping module is used to perform posture configuration on the posture mapping object based on the relative position relationship and the interpolation position information to obtain a posture mapping result of the physical object.

11. The device according to claim 10, characterized in that The posture information acquisition unit is specifically used for: Using a posture recognition model, determining current posture information of a physical object at a current sampling moment and historical posture information of the physical object at a sampling moment before the current sampling moment; The positioning point correction unit is specifically used for: Obtaining the position confidence of the positioning point at the current sampling moment, and the relative distance between the positioning point at the current sampling moment and the previous sampling moment; Based on the position confidence and the relative distance, the current position of the positioning point is corrected using the historical position of the positioning point to obtain the corrected position of the positioning point at the current sampling moment.

12. The device according to claim 10, characterized in that The point configuration module is specifically used for: Acquire respective object images of the physical object and the gesture mapping object; Determining a point mapping relationship between the physical object and the gesture mapping object based on each of the object images; According to the point mapping relationship, each point of the posture mapping object is matched with each positioning point, and a target point that fails to match among the points is determined.

13. The device according to claim 10, characterized in that The position interpolation module comprises: An associated point determination unit, used to determine an associated point of the target point from each successfully matched non-target point, and obtain a spatial position constraint condition between the target point and the associated point; A position information determining unit, configured to determine the position information of the associated point in the mapping coordinate system where the posture mapping object is located, according to the associated positioning points among the positioning points that match the associated point and the relative position relationship between the positioning points; A position interpolation unit is used to perform position interpolation processing on the target point based on the position information of the associated point and in accordance with the spatial position constraint condition to obtain the interpolated position information of the target point.

14. The device according to claim 13, characterized in that The number of the associated points is multiple, and the spatial position constraint condition includes a collinear relationship between the target point and each of the associated points; The position interpolation unit is specifically used for: Determine at least two target straight lines where the target point is located according to the spatial position constraint condition and the position information of each of the associated points; The interpolation position information of the target point is determined according to the intersection positions between the target straight lines.

15. The device according to claim 13, characterized in that The associated points include a selected point in the posture mapping object that is symmetrical to the target point, at least two selected adjacent points of the selected point, and a target adjacent point that is symmetrical to the selected adjacent point; The spatial position constraint condition includes: the target distance between the target adjacent point corresponding to any selected adjacent point and the target point is equal to the distance between any selected adjacent point and the selected point; The position interpolation unit is specifically used for: Based on the position information of each of the associated points, determining the distance between the selected point and each of the selected adjacent points; Determining a target distance between the target point and each of the target adjacent points according to each of the distances and the spatial position constraint condition; Based on each of the target distances, position interpolation processing is performed on the target point to obtain interpolation position information of the target point.

16. The device according to claim 13, characterized in that The posture mapping object is a virtual person, the target point is the elbow joint of the virtual person, the associated points include the wrist joint and the palm joint of the virtual person; the spatial position constraint condition includes the forearm length; The position interpolation unit is specifically used for: Determine the extension direction of the virtual person's palm according to the position information of the wrist joint and the palm joint; The interpolation position information of the elbow joint of the virtual person is determined according to the palm extension direction and the forearm length.

17. The device according to any one of claims 10 to 16, characterized in that The posture mapping module comprises: A current mapping position determining unit, used to determine the current mapping position of each of the points in the mapping coordinate system where the posture mapping object is located at the current mapping moment according to the relative position relationship and the interpolation position information; A historical mapping position acquisition unit, used for acquiring the historical mapping position of each point of the gesture mapping object at a mapping moment before the current mapping moment; A mapping position correction unit, configured to correct the current mapping position of each point based on the historical mapping position of the point to obtain a corrected mapping position of the point; The posture mapping unit is used to perform posture configuration on the posture mapping object using each of the correction mapping positions to obtain a posture mapping result of the physical object.

18. The device according to claim 17, characterized in that The posture mapping unit is specifically used for: Performing filtering and smoothing processing on each of the corrected mapping positions to obtain an updated mapping position; The gesture mapping object is gesture configured using each of the updated mapping positions to obtain a gesture mapping result of the physical object.

19. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 9 are implemented.

20. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

21. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images