Data processing method, device, data processing system, electronic device and computer program
The data processing method improves virtual object posture accuracy by correcting joint point coordinates using a three-dimensional pose estimation model, addressing the limitations of wearable sensors in existing technologies and enhancing movement realism in virtual and augmented reality applications.
Patent Information
- Application Number
- JP2023572039
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2021-10-27
- Filing Date
- 2022-10-24
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2042-10-24
AI Technical Summary
Existing methods for determining the posture of a virtual person based on a real person using wearable sensors are inconvenient and affect accuracy, leading to issues such as unnatural movement and high implementation costs in applications like immersive virtual and augmented reality.
A data processing method that utilizes a three-dimensional pose estimation model to determine the posture of a virtual object by correcting the three-dimensional coordinates of joint points based on history coordinates and displacement offsets, eliminating the need for wearable sensors and improving accuracy.
The method enhances the accuracy and naturalness of virtual object movements by correcting joint point coordinates, reducing the complexity and cost of motion capture systems, and enabling precise control of virtual object postures.
Smart Images

Figure 0007585600000004 
Figure 0007585600000005 
Figure 0007585600000006
Abstract
Description
[Technical field]
[0001] This application claims priority to a Chinese patent application filed on October 27, 2021, bearing application number 202111256971.9 and entitled "Data Processing Method, Apparatus, Electronic Device, and Computer-Readable Storage Medium," the entire contents of which are incorporated herein by reference.
[0002] The present invention relates to the technical field of data processing, and in particular to a data processing method, apparatus, electronic device, computer-readable storage medium and computer program product. [Background technology]
[0003] With the development of science and technology, how to determine the posture of a virtual character based on the posture of a real person has become an important part of the development of modern industry, such as smart factories, visual effects applications in movies and games, and is of great value to reality, immersive virtual and augmented reality, immersive and free-viewpoint three-dimensional videos.
[0004] Currently, wearable sensors are mainly used to perform a certain action in front of an infrared light source, and based on the infrared light reflected by the reflective landmarks on the wearable sensor, the three-dimensional spatial coordinates of each reflective landmark are calculated, and the posture of a virtual person is identified based on the three-dimensional spatial coordinates of each reflective landmark.
[0005] However, this method relies on a wearable sensor, which can cause inconvenience to the living body wearing the wearable sensor and affect the accuracy of the execution of the movement. Summary of the Invention
[0006] The embodiments of the present invention provide a data processing method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product, which can effectively improve the accuracy of the posture of a virtual object.
[0007] In one aspect of an embodiment of the present invention, there is provided a data processing method executed by an electronic device, the method comprising: acquiring a target video of a target object, the target object including at least one frame image; and performing processing for each frame image to acquire a three-dimensional posture corresponding to each frame image of a virtual object related to the target object, the processing including: determining, based on the frame images, a three-dimensional posture angle in the frame image of each target joint point of the target object, and first three-dimensional coordinates corresponding to the frame image in a first coordinate system of each of a first joint point and a second joint point, the first joint point and the second joint point being designated joint points of the each target joint point, and the first coordinate system being a coordinate system of a previous coordinate system. a coordinate system corresponding to the virtual object, the first joint point being a root node among the target joint points; a step of determining a displacement offset of the second joint point based on a first three-dimensional coordinate of the second joint point corresponding to the frame image and a historical three-dimensional coordinate of the second joint point corresponding to a frame image immediately before the frame image; if the displacement offset is equal to or less than a set value, a step of correcting the first three-dimensional coordinate of the first joint point based on the first three-dimensional coordinate and the historical three-dimensional coordinate of the second joint point to acquire target three-dimensional coordinates of the first joint point; and a step of determining a three-dimensional posture of the virtual object corresponding to the frame image based on the target three-dimensional coordinate of the first joint point and a three-dimensional posture angle of each target joint point in the frame image.
[0008] In another aspect of an embodiment of the present invention, a data processing device includes a video acquisition module that acquires a target video of a target object, the target object including at least one frame image, and a pose determination module that performs processing on each frame image to acquire a three-dimensional pose corresponding to each frame image of a virtual object related to the target object, wherein the pose determination module, when performing the processing, determines, based on the frame images, a three-dimensional pose angle of each target joint point of the target object in the frame image, and first three-dimensional coordinates corresponding to the frame image in a first coordinate system for each of a first joint point and a second joint point, and the first joint point and the second joint point are each a three-dimensional pose angle of each target joint point in the frame image in a first coordinate system for each of the first joint point and the second joint point. the first coordinate system is a coordinate system corresponding to the virtual object, the first joint point is a root node among the target joint points, a displacement offset of the second joint point is determined based on a first three-dimensional coordinate of the second joint point corresponding to the frame image and a historical three-dimensional coordinate corresponding to a frame image immediately preceding the frame image, and if the displacement offset is equal to or less than a set value, the first three-dimensional coordinate of the first joint point is corrected based on the first three-dimensional coordinate and the historical three-dimensional coordinate of the second joint point to obtain target three-dimensional coordinates of the first joint point, and a three-dimensional posture of the virtual object corresponding to the frame image is determined based on the target three-dimensional coordinate of the first joint point and a three-dimensional posture angle of each target joint point in the frame image.
[0009] Another aspect of an embodiment of the present invention provides a data processing system including an image capturing device, a data processing device, and a terminal device, wherein the image capturing device captures a target video of a target object, the target video including at least one frame image, and the data processing device acquires a three-dimensional pose of a virtual object related to the target object corresponding to each frame image of the target video by executing the above method based on the target video captured by the image capturing device, and transmits the acquired three-dimensional pose to the terminal device, and the terminal device drives the virtual object to perform a corresponding action based on the acquired three-dimensional pose.
[0010] In another aspect of an embodiment of the present invention, there is provided an electronic device including a memory having a computer program stored therein and a processor, the processor executing the computer program to realize the above-mentioned data processing method.
[0011] Another aspect of an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, the computer program implementing the above-described data processing method when executed by a processor.
[0012] In another aspect of an embodiment of the present invention there is provided a computer program product comprising a computer program which, when executed by a processor, implements the data processing method described above. [Brief description of the drawings]
[0013] In order to describe the technical solutions of the embodiments of the present invention more clearly, the following briefly introduces the drawings required for describing the embodiments. [Figure 1] 1 is a flowchart of an example of a data processing method according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a schematic diagram showing one frame image in a target video in an example of the present invention; [Figure 3a] 4 is a schematic diagram illustrating a pose of a target object according to an embodiment of the present invention; [Figure 3b] FIG. 2 is a schematic diagram illustrating a posture of a virtual object according to an embodiment of the present invention. [Figure 4a] FIG. 11 is a schematic diagram illustrating a posture of a virtual object determined when a first 3D coordinate of a first joint point is not corrected based on a displacement offset of a second joint point according to an embodiment of the present invention; [Figure 4b] FIG. 11 is a schematic diagram illustrating a posture of a virtual object determined when a first 3D coordinate of a first joint point is corrected based on a displacement offset of a second joint point according to an embodiment of the present invention; [Diagram 5] FIG. 2 is a schematic diagram illustrating 3D pose estimation for one frame image according to an embodiment of the present invention. [Figure 6] 1 is a schematic diagram of an image capture device according to an embodiment of the present invention; [Figure 7a] 4 is a schematic diagram showing the determination of 3D pose angles in the frame images of each target joint point according to an embodiment of the present invention. FIG. [Figure 7b] FIG. 2 is a schematic diagram showing an area where simulated missing joint points are located according to an embodiment of the present invention; [Figure 7c] FIG. 4 is a schematic diagram showing a three-dimensional attitude angle after attitude calibration according to an embodiment of the present invention. [Figure 8] FIG. 13 is a schematic diagram illustrating a comparison of one-dimensional attitude angle sequence of one target joint point before and after filtering processing according to an embodiment of the present invention. [Figure 9a] 4 is a schematic diagram showing a three-dimensional posture angle of a target object in one frame image before filtering processing in an embodiment of the present invention. FIG. [Figure 9b] FIG. 2 is a schematic diagram showing a three-dimensional posture angle of a target object in one frame image after filtering processing in an embodiment of the present invention. [Figure 10]1 is a schematic diagram showing an example of a configuration of a data processing system to which the present invention is applied; [Figure 11] 1 is a flow chart illustrating a data processing method according to one scenario embodiment of the present invention. [Figure 12] 1 is a schematic diagram showing an example of a configuration of a data processing device according to an embodiment of the present invention; [Figure 13] 1 is a schematic diagram illustrating an example of a configuration of a data processing system according to an embodiment of the present invention. [Figure 14] FIG. 1 is a schematic diagram illustrating an example of a configuration of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0014] Hereinafter, the embodiments of the present invention will be described with reference to the drawings in this specification. Note that the embodiments described with reference to the drawings below are illustrative for explaining the technical means of the embodiments of the present invention, and do not limit the technical means of the embodiments of the present invention.
[0015] In addition, unless otherwise specified, the singular forms "a", "one", "the" and "the" used in this specification may include the plural forms. In addition, the terms "including" and "comprising" used in this specification mean that the corresponding features may be realized as the presented features, information, data, steps, operations, components, and / or assemblies, but do not exclude that they may be implemented as other features, information, data, steps, operations, components, assemblies, and / or combinations thereof supported in the art. In addition, when one element is "coupled" or "connected" to another element, it may mean that the one element is directly coupled or coupled to the other element, or that the one element and the other element establish a connection relationship through an intermediate element. In addition, "coupled" or "connected" as used in this specification may include wireless coupling or wireless connection. The term "and / or" used in this specification indicates at least one of the items specified by the term, for example, "A and / or B" may be realized as "A", "B", or "A and B".
[0016] In order to make the objectives, technical means and advantages of the present invention clearer, the following describes in more detail the embodiments of the present invention with reference to the drawings.
[0017] The data processing method according to the embodiment of the present invention may be realized based on artificial intelligence (AI) technology. For example, for each frame image of a target video, a 3D posture estimation model may be invoked to obtain the 3D posture angle of each target joint point of a target object in the frame image. AI is a theory, method, technology and application system that uses a digital computer or a device controlled by a digital computer to simulate, develop and extend human intelligence, sense the environment, obtain knowledge, and use the knowledge to obtain optimal results.
[0018] Here, the 3D pose estimation model may be a neural network model based on computer vision technology (CV). Computer vision refers to using a camera and a computer instead of human eyes to recognize, track, measure, etc. a target, and further performing graphic processing, so that the image processed by the computer is more suitable for observation by human eyes or is sent to an equipment for detection.
[0019] Preferably, the data processing method according to the embodiment of the present invention may be implemented based on cloud technology. For example, cloud technology may be adopted when training a neural network model to obtain the above 3D pose estimation model, and data calculation in the training process may adopt a cloud computing manner.
[0020] Cloud technology refers to hosting technology that integrates resources such as hardware, software, and networks within a wide area network or local area network to realize data calculation, storage, processing, and sharing. Cloud computing is a model of IT infrastructure provision and use that refers to obtaining required resources over a network in an on-demand, scalable manner.
[0021] In order to better describe and understand the solution according to the embodiments of the present invention, some technical terms related to the embodiments of the present invention will be explained first.
[0022] Motion capture (Mocap): Also called dynamic capture, abbreviated as "Mocap", is a technology for recording and processing the movements of people or other objects, mainly applied to animation / film production, large-scale games, and virtual reality. Motion capture equipment is divided into several categories, such as mechanical, acoustic, optical, electromagnetic, and inertial, according to the working principle. Currently, based on the method of optical motion capture, the actor performing the action wears a wearable sensor with several markers, and the positions of these markers are captured by a dense camera array, and the positions of these markers are restored and rendered on a virtual object, thereby realizing the mapping of the real actor's movements to skeletal animation.
[0023] 3D pose estimation: Based on a deep neural network, the coordinates of each joint point in the 3D space of the human body are predicted from an image containing the body's pose, and the actual pose of the human body is restored.
[0024] Video Frame Interpolation: A technique for increasing the total number of frames in a video and improving the frame rate of a video by generating transition frames between two consecutive frames in the video, without changing the total duration of the video.
[0025] Monocular depth estimation: A deep learning method that uses a single image to predict the distance of each pixel in the image relative to the camera.
[0026] Aircraft attitude angle: Determined by the relationship between the aircraft coordinate system and the geographic coordinate system, it is expressed by three Euler angles, namely, yawing angle, pitching angle and rolling angle. Different rotation orders of the aircraft form different coordinate transformation matrices, and usually express the spatial rotation of the aircraft coordinate system relative to the geographic coordinate system according to the order of yawing angle, pitching angle and rolling angle. In 3D human body attitude estimation, the aircraft attitude angle is used to express the rotation attitude of each joint of the human body, hereinafter also referred to as the 3D attitude angle of the joint point.
[0027] Savitzky-Golay filter: A filter that fits polynomial data based on the local polynomial least squares method in the time domain. This filter can remove noise while keeping the shape and width of the signal constant.
[0028] SMPL-X model (Skinned Multi-Person Linear eXpressive model) or SMPL model (Skinned Multi-Person Linear model): refers to a 3D model that calculates body posture, hand posture and facial expression from a single frame RGB image, usually using three parameters to control body joint posture, body shape and facial expression separately.
[0029] In the motion capture, the conventional method requires the performer to wear a designated wearable device, such as various sensors, which makes the performer's actions inconvenient and affects the accuracy with which the performer performs the predetermined actions, the demand for capture locations is relatively high, the implementation cost is relatively high, and the condition for using the motion capture system is relatively high, since the technician needs to undergo specialized training to use such a motion capture system. In addition, this method requires capturing information of multiple markers in the wearable sensor worn by the performer to realize the final motion capture and identify the posture of the virtual object corresponding to the performer. Therefore, this motion capture method is highly complex and the accuracy of the posture of the determined virtual object is low, and it cannot fully meet practical requirements.
[0030] In contrast, the embodiment of the present invention provides a data processing method that corrects the first 3D coordinates of the first joint point capable of controlling the overall position of the virtual object, thereby avoiding the problem of unnatural behavior of the virtual object due to erroneous movement of the target object, such as the feet of the virtual object floating in the air or slipping during movement. In addition, based on the target 3D coordinates of the first joint point, the history 3D coordinates, and the 3D attitude angles of each target joint point in the frame image, the 3D attitude of the virtual object corresponding to the frame image can be more accurately determined, the motion of the virtual object can be made more accurate, the movement can be made more natural and smooth, the realism of the action of the virtual object can be improved, and the virtual object can be effectively driven to restore the real movement trajectory of the target object.
[0031] The method according to the embodiment of the present invention can be applied to any application scenario that requires controlling the 3D attitude of a virtual object based on the 3D attitude (e.g., 3D attitude angle, expressed as 3D coordinates) of a target object. For example, the method may be applied to driving the movement of game characters in various games, creating fantasy movies / dramas / animations, virtual reality, etc. based on the movements performed by real actors.
[0032] The following describes the technical means according to the embodiments of the present invention and the technical effects of the technical means of the present invention by describing several exemplary embodiments. Note that the following embodiments may be mutually referenced, referred to, or combined, and the same terms, similar features, and similar implementation steps in different embodiments will not be described.
[0033] FIG. 1 is a flowchart of an example of a data processing method according to an embodiment of the present invention. The execution subject of the data processing method may be an electronic device. In one possible aspect, the electronic device may include, but is not limited to, a terminal device or a server, and preferably, the server may be a cloud server. Here, the terminal device may be a user equipment (UE), a mobile device, a mobile phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, or a wearable device. Here, the data processing method may be realized by a processor calling a computer-readable instruction stored in a memory. Preferably, the data processing method may be executed by a user terminal, for example, the user terminal includes, but is not limited to, a mobile phone, a computer, an intelligent voice dialogue device, a smart home appliance, an in-vehicle terminal, an electronic device, an AR / VR device, a virtual robot, and the like.
[0034] As shown in FIG. 1, the data processing method may include steps S11 to S12.
[0035] Step S11: Obtain a target video of a target object, where the target object includes at least one frame image.
[0036] Preferably, the target object may be any operable object, such as a person, an animal, etc., that can move autonomously by itself. In the present invention, the data processing method may be described by taking a person that moves autonomously by itself as the target object.
[0037] The method of acquiring the target video according to the embodiment of the present invention is not limited. For example, the target video may be an initial video directly acquired by an image capturing device, or may be a video obtained by performing frame extraction processing on the initial video according to a preset frame extraction rule in order to reduce the amount of data processing and improve data processing efficiency. Here, the image capturing device includes, but is not limited to, a surveillance camera, a video camera, a camera, or other devices having an image capturing function (such as a mobile phone, a tablet computer, etc.). Here, the image capturing device may be a monocular image capturing device, a binocular image capturing device, or preferably an RGB image capturing device.
[0038] Preferably, the target video may be a video obtained by performing a frame extraction process on the initial video at a predetermined time interval. Here, the predetermined time interval may be determined according to the total time length of the initial video or the number of images included in the initial video, and the present invention is not limited thereto. For example, the predetermined time interval may have a positive correlation with the total time length of the initial video or the number of images included in the initial video. For example, if the total time length of the initial video is 1 hour, the predetermined time interval may be set to 1 minute, and if the total time length of the initial video is 15 minutes, the predetermined time interval may be set to 5 seconds. The predetermined time intervals corresponding to different target videos may be the same, and the frame extraction rules may be different according to different application scenes.
[0039] In practical applications, to complete the motion capture of the target object, the target object may perform a predetermined action, such as moving, according to practical application needs, and the target video may be a video of the target object performing a predetermined action captured by an image capture device, where the present invention is not limited to a specific form of the predetermined action, which is determined based on practical application needs.
[0040] Step S12: For each frame image of the target video, the following steps S121 to S124 are executed, respectively, to obtain a three-dimensional posture corresponding to each frame image of a virtual object related to the target object.
[0041] Step S121: Based on the frame image, a three-dimensional posture angle in the frame image of each target joint point of the target object, and first three-dimensional coordinates corresponding to the frame image in a first coordinate system for each of the first joint point and the second joint point are determined.
[0042] Here, the first joint point and the second joint point are designated joint points among a plurality of target joint points, the first coordinate system is a coordinate system corresponding to a virtual object, and the first joint point is a root node (which may also be referred to as a central root node) among the target joint points.
[0043] Here, since each target joint point has a parent-child relationship with the other target joint points, and the parent-child relationship between the waist joint point among the target joint points and the other target joint points is not complicated, the waist joint point may be selected as the first joint point in order to reduce the amount of data processing and improve the efficiency of data processing.
[0044] The second joint point may be called a reference joint point, which means a target joint point whose relative positional relationship with a reference plane is fixed, and may be the same as the first joint point or may be different from the first joint point. For example, the second joint point may be a point on the reference plane, which may be the ground. For example, in a practical application, the second joint point of a virtual object is fixed relatively to a specified direction in a first coordinate system, and one foot always contacts the ground during the walking of the virtual object (the foot that contacts the ground may be called a foot that performs a gravity support function, and is abbreviated as a gravity support foot), so the second joint point may be a toe joint point.
[0045] 2 is a schematic diagram showing one frame image in a target video in an example of the present invention. As shown in FIG. 2, the first joint point A may be the hip joint point of the target object, and the second joint point B may be the toe joint point of the target object.
[0046] In this embodiment, each target joint point of the target object may be all joint points of the target object, or a part of joint points selected according to application needs. For example, the target object is a person, and the target joint points may be all joint points of the person, or a part of specific joint points may be selected from all joint points of the person. The number of selected joint points and the specific target joint points may be set according to actual needs. For example, when the target object is a person, 33 target joint points may be set, and the 33 target joint points include target joint points on the torso (which may be referred to as torso joint points), wrist joint points of both hands, and ankle joint points of both feet.
[0047] Usually, the movement of each wrist joint point based on the 3D posture angle is relatively fine, while the movement of each ankle joint point based on the 3D posture angle is relatively coarse. The positional relationship of the ankle joint points is more constant than that of the torso joint points, so the torso joint points may include the ankle joint points of both feet. That is, when the torso joint points and wrist joint points are identified, the target joint points can be obtained.
[0048] Preferably, the torso joint points and the wrist joint points may have joint points that partially overlap (but may not overlap). For example, the torso joint points may include a left wrist joint point and a right wrist joint point, the left hand joint points may also include a left wrist joint point, and the right hand joint points may include a right wrist joint point. As an example, among the above 33 joint points, the number of target joint points on the torso is 23, the number of joint points on the left hand is 6, and the number of joint points on the right hand is 6.
[0049] Preferably, the step of determining first three-dimensional coordinates corresponding to the frame image in the first coordinate system of each of the first articulation point and the second articulation point may be realized by the following steps.
[0050] Based on the frame images, a three-dimensional posture angle of each target joint point in the frame images is determined by a three-dimensional posture estimation model.
[0051] A second three-dimensional coordinate corresponding to the frame image in a second coordinate system of the first articulation point is determined, the second coordinate system being an image coordinate system corresponding to the target video.
[0052] A first three-dimensional coordinate of the first joint point is determined based on the second three-dimensional coordinate of the first joint point and a coordinate transformation relationship, where the coordinate transformation relationship is a transformation relationship between the first coordinate system and the second coordinate system.
[0053] A first three-dimensional coordinate of a second joint point is determined based on the first three-dimensional coordinate of the first joint point and a three-dimensional pose angle associated with the second joint point among the three-dimensional pose angles of each target joint point.
[0054] Preferably, the first three-dimensional coordinates of the first articulation point may be determined as the three-dimensional coordinates of the first articulation point of the virtual object.
[0055] Here, the 3D posture angle associated with the second joint point may be the 3D posture angle of the target joint point required in the process of tracing from the first joint point to the second joint point based on the parent-child relationship of each target joint point.
[0056] Step S122: A displacement offset of the second joint point is determined based on the first three-dimensional coordinates of the second joint point corresponding to the frame image and the historical three-dimensional coordinates corresponding to the frame image immediately before the frame image.
[0057] Preferably, the displacement offset may be a Euclidean distance between the first three-dimensional coordinates corresponding to the frame image of the second joint point and the history coordinates. By determining the displacement offset, it is possible to determine a moving distance between the second joint point of the target object in the frame image and the second joint point in a frame image immediately before the frame image.
[0058] Step S123: If the displacement offset is less than or equal to a set value, correct the first three-dimensional coordinate of the first joint point based on the first three-dimensional coordinate and the history three-dimensional coordinate of the second joint point, and obtain the target three-dimensional coordinate of the first joint point.
[0059] Preferably, the method may further include the step of setting the first three-dimensional coordinates of the first joint point as target three-dimensional coordinates of the first joint point when the displacement offset is greater than a set value.
[0060] Preferably, the set value may be a predetermined fixed value, or may be determined based on the amplitude between adjacent movements among the predetermined movements performed by the target object, but the present invention is not limited thereto. For example, the set value may be set to 2 to 3 cm.
[0061] In one possible embodiment, step S123 may include the following steps.
[0062] The difference in each dimension between the first 3D coordinate and the historical 3D coordinate of the second articulation point is determined.
[0063] Based on the difference corresponding to each dimension, the coordinate of the corresponding dimension among the first three-dimensional coordinates of the first joint point is corrected to obtain the target three-dimensional coordinates of the first joint point.
[0064] Preferably, the target three-dimensional coordinates of the first joint point may be obtained by subtracting the difference corresponding to the corresponding dimension from the coordinates of each dimension in the first three-dimensional coordinates of the first joint point. Alternatively, a weight corresponding to each dimension may be determined based on the relationship between the differences corresponding to each dimension, and a weighting process may be performed on the coordinates of the corresponding dimension in the first three-dimensional coordinates of the first joint point based on the weight corresponding to each dimension to obtain the target three-dimensional coordinates of the first joint point. In this way, the larger the difference corresponding to the dimension, the larger the range by which the coordinate value in that dimension in the first three-dimensional coordinates of the first joint point is adjusted.
[0065] Step S124: Based on the target three-dimensional coordinates of the first joint point and the three-dimensional posture angle of each target joint point in the frame image, the three-dimensional posture of the virtual object corresponding to the frame image is determined.
[0066] Preferably, target three-dimensional coordinates in the first coordinate system of the other joint points in the frame image are determined based on the target three-dimensional coordinates of the first joint point and the three-dimensional attitude angles of the other joint points, where the other joint points are joint points other than the first joint point and the second joint point among the target joint points.
[0067] A three-dimensional posture of a virtual object corresponding to the frame image is determined based on the target three-dimensional coordinates of each target joint point and the three-dimensional posture angle of each target joint point in the frame image.
[0068] Here, as described above, when the displacement offset is greater than the set value, the target three-dimensional coordinates of the first joint point are determined as the first three-dimensional coordinates of the first joint point, and the target three-dimensional coordinates of the second joint point become the history three-dimensional coordinates of the second joint point. In step S123, when the displacement offset is less than the set value and the first three-dimensional coordinates of the first joint point are corrected to obtain the target three-dimensional coordinates of the first joint point, the target three-dimensional coordinates of the second joint point become the first three-dimensional coordinates of the second joint point.
[0069] When determining the three-dimensional posture of a virtual object related to a target object based on the motion of the target object, the three-dimensional posture of the virtual object may be determined based on a target video captured by an image capture device, which is simple to implement, does not require the target object to wear a complex or heavy wearable device, makes the operation of the entire system simple, low-cost, and has minimal physical limitations on the target object (e.g., an actor), effectively solving various limitations of the traditional motion capture system and further meeting the needs of practical applications.
[0070] Furthermore, in the embodiment of the present invention, in determining the target 3D coordinates of the root node among each target joint point, the magnitude of the displacement offset corresponding to two adjacent frame images of the second joint point may be specified, and it may be determined whether or not the first 3D coordinates corresponding to the current frame of the root node need to be corrected based on the displacement offset. Here, the second joint point may be a target joint point whose relative positional relationship with the ground in the specified first coordinate system is fixed, for example, a toe joint point. This can effectively avoid the problem of unnatural behavior of the virtual object, such as the feet of the virtual object floating in the air or slipping during movement, due to erroneous movement of the target object.
[0071] According to an embodiment of the present invention, the target 3D coordinates of each target joint point and the 3D posture angle of each target joint point in the frame image are determined based on the video of the target object, so that the 3D posture of the virtual object can be more accurately determined, the realism of the performed movement of the virtual object can be improved, and the virtual object can be effectively driven to restore the actual movement trajectory of the human body model in the 3D space, so that the movement of the virtual object can be more accurate and the movement can be more natural and smooth.
[0072] FIG. 3a is a schematic diagram showing the posture of a target object according to an embodiment of the present invention. FIG. 3b is a schematic diagram showing the posture of a virtual object according to an embodiment of the present invention. As shown in FIG. 3a, FIG. 3a is a human body model of a target object obtained by inputting the three-dimensional posture angles of each target joint point into the SMPL-X model, and the human body model shows various postures. As shown in FIG. 3b, based on the posture of the target object in the frame image, the three-dimensional posture of the virtual object corresponding to the frame image can be accurately determined. According to the data processing method according to the embodiment of the present invention, the virtual object can be effectively driven to restore the three-dimensional posture of the target object.
[0073] FIG. 4a is a schematic diagram showing the posture of a virtual object determined when the first three-dimensional coordinate of the first joint point according to an embodiment of the present invention is not corrected based on the displacement offset of the second joint point. FIG. 4b is a schematic diagram showing the posture of a virtual object determined when the first three-dimensional coordinate of the first joint point according to an embodiment of the present invention is corrected based on the displacement offset of the second joint point. As shown in FIG. 4a, when the displacement offset is smaller than a set value, if the posture of the virtual object is determined based on the first three-dimensional coordinate of the first joint point and the three-dimensional posture angle of each target joint point in the frame image without correction, a phenomenon occurs in which the feet of the virtual person sink into the ground. As shown in FIG. 4b, the first three-dimensional coordinate of the first joint point is corrected, the three-dimensional coordinate after correction is set as the target three-dimensional coordinate of the first joint point, and the three-dimensional posture corresponding to the frame image of the virtual object is determined based on the target three-dimensional coordinate of the first joint point in the frame image and the three-dimensional posture angle of each target joint point in the frame image, so that the feet of the virtual person can be lifted off the ground and "pulled out" from the ground.
[0074] Preferably, when determining the 3D posture angles of each target joint point in the frame image, a torso region image and a hand region (left hand region, right hand region) image in the image are first determined, and a 3D posture estimation model is called up to determine the 3D posture angles of the torso joint points in the torso region image and the 3D posture angles of the wrist joint points in the hand region image.
[0075] Preferably, the torso region image and the hand region image in the frame image may be specified based on the detection result of a two-dimensional joint point detection model. Specifically, a target joint point in the frame image may be detected by a two-dimensional joint point detection model, and the frame image may be cut out based on the position of the target joint point in the frame image to cut out the torso region image and the hand region image.
[0076] Here, the 2D joint point detection model may be obtained by training a neural network model based on a training data set including a large number of training samples. Here, the embodiment of the present invention is not limited to a specific network structure of the 2D joint point detection model, and may be configured according to actual needs.
[0077] Preferably, the 3D pose estimation model may be a model based on a convolutional neural network, for example, the 3D pose estimation model may include, but is not limited to, a neural network model based on a model structure such as ResNet (residual network), InceptionNet (deep learning neural network), DenseNet (high density neural network), VggNet (deep convolutional neural network), SENet (Squeeze-and-Excitation Networks), etc. Here, the input of the model is one frame image or image data obtained by vectorizing the frame image, and the output of the model is a joint point tree in the frame image or each label of a plurality of joint points, where the output of the model may be the two-dimensional coordinates of each joint point in the frame image. Here, the label may be a number, an alphabet, etc., and the present invention is not limited thereto.
[0078] Preferably, the 2D joint point detection model may be a neural network model based on ResNet, and a large amount of human body joint point detection data set may be used to train the neural network model to obtain a 2D keypoint detection model that meets the training end conditions (e.g., convergence of the loss function of the model, detection accuracy meets the demand, etc.).
[0079] Fig. 5 is a schematic diagram showing estimation of a 3D pose for one frame image according to an embodiment of the present invention. As shown in Fig. 5, a first rectangular area (indicated by a dashed frame 501 in Fig. 5) of a minimum bounding rectangle formed by each target joint point may be specified, a first image corresponding to the first rectangular area in the frame image may be cut out, and the first image may be used as an image of a body area of the target object in the frame image.
[0080] Preferably, at least one second rectangular area (indicated by solid-line frames 502, 503 in FIG. 5) of a minimum bounding rectangle of a predetermined multiple is identified with the wrist joint points of both hands of the target object in the frame image as the center, and at least one second image corresponding to the at least one second rectangular area in the frame image is cut out, and the at least one second image is used as an image of the hand area in the frame image of the target object.
[0081] In one possible embodiment, for each frame image of the target video, a 3D pose estimation model may be invoked to obtain the 3D pose angle of each target joint point of the target object in that frame image.
[0082] Preferably, after obtaining the 3D posture angle of each target joint point of the target object in the frame image, the 3D posture angle of each target joint point in the frame image may be input into an SMPL model to obtain a human body model of the target object, i.e., the human body model of the target object shown in FIG. 3a.
[0083] Here, the 3D pose estimation model may be obtained by training a neural network model based on a training data set including a large number of training samples. Here, the embodiment of the present invention is not limited to a specific network structure of the 3D pose estimation model, and may be configured according to actual needs. Preferably, the 3D pose estimation model may be a model based on a convolutional neural network, for example, the 3D pose estimation model may include, but is not limited to, neural network models based on model structures such as ResNet, InceptionNet, DenseNet, VggNet, and SENet. Here, the input of the model is one frame image or image data obtained by vectorizing the image of the frame, and the output of the model is the 3D pose angle or 3D coordinates of the joint point in the image of the frame.
[0084] In a preferred embodiment of the present invention, the 3D pose estimation model may be obtained by training a ResNet-based neural network using the Frankmocap algorithm based on multiple 3D pose datasets, such as Human 3.6M, where the Human3.6M 3D pose dataset is the largest dataset for 3D human pose estimation, and is composed of 3.6 million poses and corresponding video frames, which include the process of 11 actors performing 15 daily activities from four camera viewing angles.
[0085] As shown in Fig. 5, for example, taking the input of the 3D pose estimation model as a current frame image in a target video, after inputting the determined first image and at least one second image into the 3D pose estimation model, a 3D pose estimation 504 of the torso may be performed using the first image and the 3D pose estimation model of the torso to determine a 3D pose angle of a target joint point on the torso of the target object in the frame image, and a 3D pose estimation 505 of the hand may be performed using the at least one second image and the 3D pose estimation model of the hand to determine a 3D pose angle of a target joint point of both hands of the target object in the frame image.
[0086] After obtaining the 3D posture angles of the target joint points on the torso of the target object in the frame image and the 3D posture angles of the target joint points of both hands of the target object in the frame image, the part integration model 506 may perform an integration process on the 3D posture angles of the wrist joint points of both hands (i.e., the wrist joint point of the left hand and the wrist joint point of the right hand) determined by the 3D posture estimation model of the torso and the 3D posture angles of the target joint points on the torso of the target object in the frame image, and perform an integration process on the 3D posture angles of the wrist joint points of both hands determined by the 3D posture estimation model of the hands and the 3D posture angles of the target joint points of both hands in the frame image to obtain the whole body 3D posture 507, i.e., the 3D posture angles of each target joint point of the target object in the frame image. The integration process includes connection between the 3D posture angles, smoothing, etc.
[0087] In a single-view target video, some areas of the target object may be occluded, i.e., the 3D pose angles or 3D coordinates of some target joint points (i.e., missing joint points) may not be accurately obtained, resulting in a decrease in the accuracy of the determined pose of the virtual object. Therefore, the initial 3D pose angles of each target joint point determined by the target video may be calibrated by determining the 3D pose angles or 3D coordinates of the missing joint points based on a non-target video related to the target video.
[0088] In one possible embodiment, at least two videos may be obtained that are captured by an image capture device having at least two capture viewing angles, where the at least two capture viewing angles include a first viewing angle and at least a second viewing angle, and the target video is a video that corresponds to the first viewing angle.
[0089] Here, each image capture device may be configured to capture images of the target object simultaneously, or each image capture device may be configured to capture images of the target object within different time periods, but the present invention is not limited thereto.
[0090] Fig. 6 is a schematic diagram of an image capturing device according to an embodiment of the present invention. As shown in Fig. 6, three same types of image capturing devices 601, 602, 603 may simultaneously capture images of a target object 604. Here, the capturing viewing angle of each image capturing device 601, 602, 603 is 120°, each image capturing device 601, 602, 603 is installed around the target object, each image capturing device 601, 602, 603 is equally distant from the target object 604, and each image capturing device 601, 602, 603 is equally distant from the ground. In this way, the target object 604 is kept within the capturing viewing angle of all image capturing devices 601, 602, 603.
[0091] Specifically, in this aspect, the above-mentioned step S121, i.e., the step of determining, based on the frame image, the three-dimensional posture angle in the frame image of each target joint point of the target object, and the first three-dimensional coordinates corresponding to the frame image in the first coordinate system of each of the first joint point and the second joint point, may be realized by the following operations.
[0092] Based on the frame images, initial joint points in the frame images of a target object are detected, where each target joint point includes each initial joint point.
[0093] Based on the frame images, an initial three-dimensional posture angle of each target joint point in the frame images is determined.
[0094] If the number of initial joint points is equal to the total number of each target joint point, it is determined that there is no missing joint point, and the initial 3D posture angle of each target joint point in the frame image is determined as the 3D posture angle of each target joint point in the frame image, where the missing joint point is a joint point other than each initial node among each target joint point.
[0095] If the number of the initial joint points is smaller than the total number of the target joint points, it is determined that there is a missing joint point, and a 3D pose angle of the missing joint point is determined based on an associated image corresponding to the frame image in the non-target video, and the determined 3D pose angle of the missing joint point and the initial 3D pose angle of each initial joint point are determined as the 3D pose angle of each target joint point in the frame image, where the non-target video is at least one video other than the target video among the at least two videos.
[0096] Here, for each frame image of the target video, the initial articulation points are target articulation points that can be determined based on the frame image. After determining the initial articulation points in each frame image of the target video by invoking the above two-dimensional articulation point detection model, the number of initial articulation points may be determined.
[0097] Here, the missing joint points are joint points among the target joint points other than the initial joint points. Whether or not a missing joint point exists may be determined based on a relationship between the number of initial joint points and the total number of the target joint points. If the number of initial joint points is smaller than the total number of the target joint points, it is determined that a missing joint point exists. If the number of initial joint points is equal to the total number of the target joint points, it is determined that a missing joint point does not exist.
[0098] Preferably, the related image may be an image having the same time stamp corresponding to a frame image in the non-target video. When it is determined that a missing joint point exists, the above-mentioned 3D posture estimation model may be invoked to determine an initial 3D posture angle of each target joint point in the frame image and a 3D posture angle of each target joint point in the related image. Then, the 3D posture angle of the initial joint point is determined based on the initial 3D posture angle of each target joint point in the frame image. The 3D posture angle of the missing joint point is determined based on the 3D posture angle of each target joint point in the related image. Therefore, the 3D posture angle of the initial joint point and the 3D posture angle of the missing joint point are determined as the 3D posture angle of each target joint point in the frame image.
[0099] Fig. 7a is a schematic diagram showing the determination of the 3D pose angle in the frame image of each target joint point according to an embodiment of the present invention, Fig. 7b is a schematic diagram showing the area where the simulated missing joint point is located according to an embodiment of the present invention, and Fig. 7c is a schematic diagram showing the 3D pose angle after pose calibration according to an embodiment of the present invention.
[0100] As shown in Fig. 7a, in one frame image of the target video, the left arm of the target object is occluded by other parts of the body as shown in the solid line frame 701, so when performing 2D joint point detection using the frame image, the target joint points in the left arm region of the target object cannot be detected. Note that even if prediction is performed based on the 3D pose estimation model, initial 3D pose angles of all target joint points are obtained, and the human body model of the target object can be restored using the SMPL-X model, the 3D pose angles of the target joint points in the left arm region of the target object cannot be accurately identified.
[0101] In order to show the missing joint points more clearly, the present invention performs a simulation based on Fig. 7a and obtains the image shown in Fig. 7b. As can be seen from Fig. 7b, the occluded left arm region of the target object cannot be completely identified based on the frame image, that is, the target joint points of the left arm region of the target object cannot be detected based on the frame image. For example, as shown in the solid line frame 702, the wrist and hand are missing.
[0102] By performing 2D joint point detection on the related images, the 3D pose angles of the missing joint points can be accurately determined, and for example, as shown in the solid line frame 703, the 3D pose angles of the wrist and hand can also be accurately obtained, so that the 3D pose angles of each target joint point of the target object can be obtained, and the human body model of the target object, i.e., Fig. 7c, can be restored by the SMPL-X model.
[0103] By capturing videos of the target object by image capture devices with different viewing angles, and replacing the initial 3D posture angles of the missing joint points in the initial 3D posture angles of the target joint points determined based on the target video with the 3D posture angles of the missing joint points determined based on the non-target video, the initial 3D posture angles of each target joint point determined by the target video can be calibrated, and the posture of the virtual object can be more accurately controlled. The data processing method according to the present invention can be realized by at least two image capture devices, does not need to rely on wearable sensors, is simple in operation, and has low cost.
[0104] Preferably, the step of determining a second three-dimensional coordinate of the first joint point corresponding to the frame image in the second coordinate system includes a step of determining the second three-dimensional coordinate of the first joint point based on a three-dimensional posture angle of the first joint point in the frame image.
[0105] Alternatively, preferably, the step of determining second three-dimensional coordinates of the first joint point corresponding to the frame image in the second coordinate system includes the following steps.
[0106] The two-dimensional coordinates of the first joint point in the second coordinate system are determined based on the image position of the first joint point in the frame image.
[0107] The depth coordinate of the first joint point is estimated based on the frame image and a correlation image of the frame image.
[0108] A second three-dimensional coordinate of the first joint point is determined based on the two-dimensional coordinate and the depth coordinate.
[0109] Here, the target video is a video captured by a monocular image capture device and the correlation image is a frame image immediately preceding the frame image, or the target video is one of two videos captured by a binocular image capture device and the correlation image is an image corresponding to the frame image in the other of the two videos.
[0110] Here, the 3D posture angle of the first joint point in the frame image may be input to the SMPL-X model to determine the second 3D coordinate of the first joint point. Here, the SMPL-X model can output a human body model of a motion corresponding to the 3D posture angle of each target joint point and the second 3D coordinate of each target joint point based on the input 3D posture angle of each target joint point.
[0111] In order to determine the second three-dimensional coordinate of the first joint point more accurately, the two-dimensional coordinate of the first joint point in the second coordinate system may be determined based on the image position of the first joint point in the frame image. Also, the depth coordinate of the first joint point of the target object in the frame image may be obtained by estimating depth information of the first joint point, and the second three-dimensional coordinate of the first joint point in the frame image may be obtained based on the two-dimensional coordinate and the depth coordinate.
[0112] Preferably, when the image capture device is a monocular image capture device, the depth information of the target object in the current frame image may be estimated based on an image captured by the image capture device corresponding to the current time (i.e., the current frame image) and an image captured by the image capture device corresponding to the time immediately before the current time (the frame image immediately before the current frame image) based on a monocular depth estimation algorithm. Here, the monocular depth estimation algorithm may be a codec-based monocular depth estimation algorithm, i.e., may be realized using a trained neural network model. When the image capture device is a binocular image capture device, the depth information of the target object in the current frame image may be determined based on two images captured by the image capture device at the current time based on a binocular depth estimation algorithm.
[0113] As shown in FIG. 2, assuming that the image capture device is a monocular image capture device, based on the two-dimensional coordinate (x, y) of the first joint point A and the above depth information z of the estimated target object, a second three-dimensional coordinate path of the first joint point is calculated. pri (x,y,z) may be determined.
[0114] In one possible aspect, the coordinate transformation relationship may be obtained by the following steps.
[0115] Determine a first distance between the two associated joint points based on the second three-dimensional coordinates of the two associated joint points, where the two associated joint points are two designated joint points having a parent-child relationship among each of the target joint points.
[0116] A reference distance in a first coordinate system of the two related joint points is obtained.
[0117] A coordinate transformation relationship is determined based on the first distance and the reference distance.
[0118] Preferably, any two of the target joint points may be selected as the above two joint points having a parent-child relationship. In order to reduce the amount of data and improve the efficiency of data processing, the two related joint points may be two joint points having a direct parent-child relationship, for example, the two related joint points may be a knee joint point and an ankle joint point.
[0119] Preferably, the second three-dimensional coordinates of the two associated joint points are respectively determined according to the above method for determining the second three-dimensional coordinates of the first joint point, and the first distance h is calculated based on the second three-dimensional coordinates of each associated joint point. 1 may be determined.
[0120] Preferably, the virtual object is constructed by any programmable image construction model, for example, the virtual object may be constructed by a virtual engine. After the virtual object is constructed, the root node of the virtual object is matched with the origin of the first coordinate system, and the reference distance h 0 Determine the coordinate transformation relationship h 1 / h 0 may be determined.
[0121] In this manner, the first three-dimensional coordinate of the first articulation point may be determined according to the following equation (1):
[0122]
number
[0123] In the process of acquiring the target video, due to the imaging error of the optical camera, etc., the final generated target video has obvious noise. When determining the posture of the virtual object based on the noisy target video, the virtual object may exhibit unnatural or abnormal body movements, and the target joint points of the virtual object may exhibit shaking, abnormal rotation, etc.
[0124] In contrast, the method may further perform a filtering process on the attitude angle sequence in each dimension of each target gate joint point, specifically according to the following steps:
[0125] Get the frame rate of the target video.
[0126] Based on the frame rate of the target video, the filtering sliding window width of the 2D pose angles of the joint points is determined.
[0127] In this aspect, the step of determining a filtering sliding window width of the two-dimensional posture angle of the joint point based on the frame rate of the target video includes the following steps.
[0128] Based on the frame rate, an adjustment factor for the sliding window width is determined.
[0129] An initial sliding window width is determined based on the product of the frame rate and an adjustment factor.
[0130] determining a larger value of the initial sliding window width and the predetermined sliding window width as a filtering sliding window width;
[0131] Here, the adjustment coefficient of the sliding width is a nonlinear coefficient, and the predetermined sliding window width is a minimum sliding window width.
[0132] For example, the sliding window width may be calculated according to the following equation (2).
[0133]
number
[0134] According to the filtering sliding window width, a filtering process is performed on the posture angle sequence in each dimension of each target joint point to obtain a filtered posture angle sequence, where the posture angle sequence in one dimension of one target joint point includes a two-dimensional posture angle in the one dimension of each frame image of the target video of the target joint point.
[0135] FIG. 8 is a schematic diagram showing a comparison of a one-dimensional posture angle sequence of one target joint point before and after filtering processing according to an embodiment of the present invention. As shown in FIG. 8, the horizontal axis represents a frame index (frame_idx), i.e., the total number of frame images included in the video, and the vertical axis represents a two-dimensional posture angle (θ) in degrees. A dashed line 801 indicates a curve of the one-dimensional posture angle sequence of the target joint point before filtering processing, and a solid line 802 indicates a curve of the one-dimensional posture angle sequence of the target joint point after filtering processing. As shown in the dashed line 801 in FIG. 8, due to the presence of the noise, burrs are repeatedly generated in the smooth curve formed by the posture angle sequence of each dimension of each target joint point, while a curve 802 formed by the posture angle sequence after filtering processing has almost no burrs, and the burrs can be efficiently removed.
[0136] In this aspect, the step of determining a 3D posture of a virtual object corresponding to the frame image based on the target 3D coordinates of the first joint point and the 3D posture angles of each target joint point in the frame image may include the following steps.
[0137] Based on the filtered posture angle sequence in each dimension, a filtered 3D posture angle of each target joint point in the frame image is obtained.
[0138] A three-dimensional posture of a virtual object corresponding to the frame image is determined based on the target three-dimensional coordinates of the first joint point and the filtered three-dimensional posture angle of each target joint point in the frame image.
[0139] 9a and 9b are schematic diagrams showing the 3D posture angles of a target object in one frame image before and after filtering according to an embodiment of the present invention;
[0140] As shown in frame 9a1 of FIG. 9a, it can be seen that an abnormal rotation posture occurs in the right wrist, that is, the left wrist is inverted.
[0141] As shown in box 9b1 of Figure 9b, after filtering the posture angle sequence in each dimension for each target joint point, the right wrist is restored to rotate normally, which is an ergonomically correct posture.
[0142] By performing a filtering process on the posture angle sequence in each dimension of each target joint point, the 3D posture angle of each target joint point can be calibrated, the burr noise can be removed while the original rotation range of each joint point can be retained, and the posture of the virtual object can be controlled based on the target joint points.
[0143] If the camera performance is limited, the frame rate of the obtained video will be low, which will not only cause an obvious time lag in the obtained 3D pose sequence, but also cause high-frequency rotation amounts to be erroneously smoothed as noise during the joint angle smoothing process, resulting in reduced pose accuracy.
[0144] Therefore, in one possible embodiment, the data processing method may further include the following steps:
[0145] For each adjacent frame in the target video, optical flow information of the target object between the adjacent frames is determined.
[0146] Based on the optical flow information, a frame interpolation process is performed between adjacent frames to obtain at least one frame interpolated image between the adjacent frames.
[0147] A frame-interpolated target video is obtained based on the target video and each frame-interpolated image.
[0148] In this aspect, executing steps S121 to S124 for each image frame of the target video may include executing the above step S121 (i.e., determining, based on the frame image, a 3D posture angle in the frame image of each target joint point of the target object, and first 3D coordinates in the first coordinate system of each of the first joint point and the second joint point corresponding to the frame image) to step S124 (determining the 3D posture of the virtual object corresponding to the frame image, based on the target 3D coordinates of the first joint point and the 3D posture angle in the frame image of each target joint point) for each image frame of the target video after frame interpolation.
[0149] Preferably, the optical flow information of the target object may include the optical flow trajectory and optical flow direction of the target object. Based on the video frame complementation algorithm of the optical flow, the trajectory and direction of the optical flow between two adjacent frame images of the target object in the target video may be calculated, and the trajectory and direction of the intermediate optical flow between these two frame images of the target object may be predicted to obtain the intermediate frame between these two frame images, and the target video after frame complementation may be obtained. That is, without changing the original time length of the target video, the total number of frames of the target video can be effectively increased, the frame rate of the target video can be improved, and the consistency of the pose between two adjacent frame images of the target object can be improved. This can effectively solve the obvious time lag appearing in the pose angle sequence of each dimension of each target joint point, and the erroneous smoothing problem of the 3D pose angle of the target joint point due to the high frequency rotation of the target joint point of the target object.
[0150] Preferably, when it is determined that a missing joint point exists, the above-mentioned frame interpolation process is performed on the non-target video to obtain the non-target video after frame interpolation, and then the three-dimensional posture angle of the missing joint point needs to be determined based on an associated image corresponding to the frame image in the non-target video after frame interpolation.
[0151] By performing frame interpolation on the target video, and determining target 3D coordinates of the first joint point and 3D posture angles of each target joint point in the frame images based on each frame image of the target video after frame interpolation, and further determining the 3D posture of the virtual object corresponding to the frame images, it is possible to improve the consistency between adjacent frame images of the virtual object.
[0152] In order to better explain the practicality of the scheme according to the embodiment of the present invention, the embodiment of the present invention will be described below with reference to a specific application scenario of the data processing method. The application scenario is various games, the production process of fantasy movies / dramas, etc. In the scenario, when an actor (i.e., the above-mentioned target object) performs a performance, the actor's action video can be captured by three cameras, and the movement of the virtual character in the game can be driven according to the actor's performance action. FIG. 10 is a schematic diagram showing an example of the configuration of a data processing system 1000 to which the present invention is applied. As shown in FIG. 10, the data processing system 1000 may include an image capture device 10, a data processing device 20, a terminal device 30, and a network 40.
[0153] Here, the image capture device 10 is any set of electronic devices capable of image capture. Referring to the image capture scheme shown in Fig. 6 above, the image capture device 10 includes three image capture devices 601, 602, 603 corresponding to different capture viewing angles, the image capture devices 601, 602, 603 surround a target object 604 and are evenly arranged around the target object 604. Here, the three image capture devices include one first device corresponding to a first viewing angle and two second devices corresponding to a second viewing angle, the target video is the video captured by the first device, and the video captured by the second devices is referred to as non-target video. Then, the image capture device 10 transmits the captured target video and non-target video to the data processing device 20.
[0154] According to the method of the embodiment of the present invention, the data processing device 20 obtains a 3D pose corresponding to each frame image of a virtual object related to a target object based on each frame image of a target video, where the data processing device 20 performs the following process for each frame image of the target video to determine a 3D pose angle of each target joint point of the target object in the frame image:
[0155] Based on the frame images of the target video, initial articulation points in the frame images of the target object are detected, the target articulation points including the initial articulation points.
[0156] Based on the frame images of the target video, an initial 3D pose angle of each of the target joint points in the frame images is determined.
[0157] If the number of the initial joint points is equal to the total number of the target joint points, it is determined that there are no missing joint points, and the initial 3D pose angle of each of the target joint points in the frame image is determined as the 3D pose angle of each of the target joint points in the frame image.
[0158] If the number of the initial joint points is smaller than the total number of the target joint points, it is determined that a missing joint point exists, and a 3D posture angle of the missing joint point is determined based on an associated image corresponding to the frame image in a non-target video, and the determined 3D posture angle of the missing joint point and the initial 3D posture angle of each initial joint point are determined as the 3D posture angle of each target joint point in the frame image. The non-target video is at least one video other than the target video among the at least two videos.
[0159] Then, the data processing device 20 transmits to the terminal device 30 the obtained three-dimensional posture of the virtual object corresponding to each frame image.
[0160] An application program capable of implementing virtual reality (VR) conversion is installed in the terminal device 30, and drives the virtual object to execute a corresponding action based on the three-dimensional posture of the virtual object in each acquired frame image. A user can transmit a control command to the virtual object via this application program.
[0161] The terminal device 30 is connected to the data processing device 20 via a network 40. The data processing device 20 may be a server or a group of servers equipped with a database, in which three-dimensional attitude angles and three-dimensional coordinates of a large number of target joint points are stored.
[0162] The following describes a preferred embodiment of the present invention with reference to the system. Figure 11 is a flowchart showing a data processing method according to one scenario embodiment of the present invention. The method is executed by an electronic device, for example, the data processing device 20 in Figure 10 above. As shown in Figure 11, the method may include steps S901 to S915.
[0163] Step S901: Perform image capture on an actor simultaneously by three image capture devices of the same type, and obtain videos captured by each image capture device, where the at least two captured viewing angles include a first viewing angle and at least a second viewing angle, determine the first viewing angle as a main viewing angle, determine the second viewing angle as an auxiliary viewing angle, determine a video corresponding to the first viewing angle as a target video, and determine a video corresponding to the second viewing angle as a non-target video.
[0164] Step S902: Perform frame interpolation on each video to obtain a target video after frame interpolation and a non-target video after frame interpolation.
[0165] The following steps S903 to S915 are executed for the image of each frame of the target video after frame interpolation.
[0166] Step S903: The frame image is input to a two-dimensional joint point detection model, and two-dimensional joint point detection is performed to obtain initial joint points in the frame image. Here, the total number of target joint points is 24.
[0167] Step S904: The frame image is input to a 3D posture estimation model, and an initial 3D posture angle in the frame image of each target joint point is determined.
[0168] Step S905: Determine whether the number of initial joint points is equal to 24.
[0169] Step S906: If the number of initial joint points is not equal to 24, i.e., is less than 24, it is determined that there are missing joint points, and an associated image having the same timestamp as the frame image in the non-target video after frame interpolation is input into the 3D pose estimation model to determine the 3D pose angle of each target joint point in the associated image. The 3D pose angle of the missing joint point is determined based on the 3D pose angle of each target joint point in the associated image, and the initial 3D pose angle of the initial joint point and the 3D pose angle of the missing joint point are further determined as the 3D pose angle of each target joint point.
[0170] Step S907: If the number of initial joint points is equal to 24, it is determined that there are no missing joint points, and the initial 3D posture angle of each target joint point in the frame image is determined as the 3D posture angle of each target joint point.
[0171] Step S908: According to the above formula (2), a filtering process is performed on the posture angle sequence in each dimension of each target joint point to obtain a posture angle sequence after the filtering process.
[0172] Step S909: Determine the knee joint point and the ankle joint point as two related joint points, input the three-dimensional posture angle of the knee joint point and the three-dimensional posture angle of the ankle joint point into the SMPL-X model, respectively, determine a second three-dimensional coordinate of the knee joint point in the second coordinate system corresponding to the frame image, and a second three-dimensional coordinate of the ankle joint point in the second coordinate system, and calculate a first distance h between the second three-dimensional coordinate of the knee joint point and the second three-dimensional coordinate of the ankle joint point. 1 Calculate.
[0173] Step S910: A virtual object is constructed by the above-mentioned virtual engine, and the hip joint point of the virtual object is matched with the origin of the first coordinate system in which the virtual object is located, and the distance h between the knee joint point and the ankle joint point of the virtual object is calculated. 0 , i.e., determine the above reference distance and define the coordinate transformation relationship h 1 / h 0 Determine.
[0174] Step S911: Calculate the second two-dimensional coordinate (x, y) in the second coordinate system of the root node, calculate the distance z between the image capturing device for capturing the target video and the real actor according to the monocular depth estimation algorithm, and calculate the second two-dimensional coordinate path of the root node. pri Determine (x, y, z) and calculate the first three-dimensional coordinate path of the root node according to equation (1). norm Determine (x,y,z), where the root node is the first node above.
[0175] Step S912: The first 3D coordinate of the root node and the filtered 3D posture angles of each target joint point are input to the virtual engine to determine the first 3D coordinate of the toe joint point corresponding to the frame image. Here, the toe joint point is set as the above-mentioned second joint point.
[0176] Here, the first three-dimensional coordinates of each joint point corresponding to each frame image may be stored in the virtual engine, but the present invention is not limited to a specific storage method.
[0177] Step S913: Based on the first three-dimensional coordinates corresponding to the frame image of the toe joint point and the historical three-dimensional coordinates, the Euclidean distance between them (i.e., the above-mentioned displacement offset) and the difference between them in each dimension are determined, and it is determined whether the distance between them is equal to or smaller than a set value, for example, 2 cm. Here, the historical three-dimensional coordinates are the first three-dimensional coordinates corresponding to the frame image immediately before the frame image of the toe joint point.
[0178] Step S914: If the distance between them is equal to or smaller than a set value, add the difference corresponding to each dimension to the coordinate of each dimension among the first three-dimensional coordinates of the root node to obtain the target three-dimensional coordinates of the root node, determine the target three-dimensional coordinates of the root node as the three-dimensional coordinates of the root node of the virtual object, drive the virtual engine again so that the virtual engine determines the target three-dimensional coordinates of the other joint points based on the three-dimensional attitude angles of the other joint points, and further determine the three-dimensional attitude of the virtual object corresponding to the frame image based on the target three-dimensional coordinates of the root node, the history three-dimensional coordinates of the second joint point, the target three-dimensional coordinates of the other joint points, and the three-dimensional attitude angles of each target joint point in the frame image. Here, the other joint points are joint points other than the root node and the toe joint point among the target joint points.
[0179] Step S915: If the distance between the two is greater than the set value, directly determine the first 3D coordinate of the root node as the target 3D coordinate of the root node of the virtual object, and drive the virtual engine again so that the virtual engine determines target 3D coordinates of other joint points based on the first 3D coordinate of the root node and the 3D attitude angles of the other joint points, and further determine the 3D attitude of the virtual object corresponding to the frame image based on the target 3D coordinates of each target joint point and the 3D attitude angles of each target joint point in the frame image.
[0180] The embodiment of the present invention further provides a data processing device based on the same principle as the data processing method according to the embodiment of the present invention. Figure 12 is a schematic diagram showing an example of the configuration of the data processing device according to the embodiment of the present invention. As shown in Figure 12, the data processing device 50 includes a video acquisition module 510 and an attitude determination module 520.
[0181] The video capture module 510 captures a target video of a target object, where the target object includes at least one frame image.
[0182] The pose determination module 520 performs processing on each frame image to obtain a corresponding 3D pose for each frame image of a virtual object associated with the target object.
[0183] When performing the process, the posture determination module 520 determines, based on the frame image, a three-dimensional posture angle in the frame image of each target joint point of the target object, and first three-dimensional coordinates in a first coordinate system of each of a first joint point and a second joint point corresponding to the frame image, wherein the first joint point and the second joint point are designated joint points among the target joint points, the first coordinate system is a coordinate system corresponding to a virtual object, the first joint point is a root node among the target joint points, and the frame image of the second joint point is a first three-dimensional coordinate in a first coordinate system of each of the target joint points. a displacement offset of the second joint point is determined based on the first three-dimensional coordinates corresponding to the frame image and the historical three-dimensional coordinates corresponding to the frame image immediately before the frame image, and if the displacement offset is equal to or less than a set value, the first three-dimensional coordinates of the first joint point are corrected based on the first three-dimensional coordinates and the historical three-dimensional coordinates of the second joint point to obtain target three-dimensional coordinates of the first joint point, and a three-dimensional posture of the virtual object corresponding to the frame image is determined based on the target three-dimensional coordinates of the first joint point and the three-dimensional posture angle of each target joint point in the frame image.
[0184] Preferably, the attitude determination module 520 sets the first 3D coordinate of the first joint point as the target 3D coordinate when the displacement offset is greater than a set value.
[0185] Preferably, when determining the three-dimensional posture of the virtual object corresponding to the frame image, the posture determination module 520 determines target three-dimensional coordinates in the first coordinate system of other joint points in the frame image based on the target three-dimensional coordinates of the first joint point and the three-dimensional posture angles of the other joint points, where the other joint points are joint points other than the first joint point and the second joint point among the target joint points, and determines the three-dimensional posture of the virtual object corresponding to the frame image based on the target three-dimensional coordinates of each target joint point and the three-dimensional posture angle of each target joint point in the frame image.
[0186] Preferably, when correcting the first three-dimensional coordinate of the first joint point, the attitude determination module 520 may determine a difference between the first three-dimensional coordinate and the historical three-dimensional coordinate of the second joint point in each dimension of the first coordinate system, and correct the coordinate of the corresponding dimension among the first three-dimensional coordinates of the first joint point based on the difference corresponding to each dimension, to obtain the target three-dimensional coordinate of the first joint point.
[0187] Preferably, the data processing device 50 further includes a filter module. The filter module acquires a frame rate of a target video, determines a filtering sliding window width of a two-dimensional posture angle of a joint point based on the frame rate of the target video, and performs a filtering process on a posture angle sequence in each dimension of each target joint point based on the filtering sliding window width to acquire a posture angle sequence after the filtering process. Here, the posture angle sequence in a dimension of a target joint point includes a two-dimensional posture angle in the dimension in each frame image of a target video of the target joint point.
[0188] In this aspect, when determining the 3D posture of the virtual object corresponding to the frame image, the posture determination module 520 may determine the 3D posture of the virtual object corresponding to the frame image based on the target 3D coordinates of the first joint point and the filtered 3D posture angle of each target joint point in the frame image.
[0189] Preferably, when determining the filtering sliding window width of the two-dimensional posture angle of the joint point based on the frame rate of the target video, the filter module may determine an adjustment coefficient of the sliding window width based on the frame rate, determine an initial sliding window width based on a product of the frame rate and the adjustment coefficient, and determine the larger value of the initial sliding window width and the predetermined sliding window width as the filtering sliding window width.
[0190] Preferably, when determining, based on the frame image, a 3D posture angle of each target joint point in the frame image of the target object and a first three-dimensional coordinate corresponding to the frame image in a first coordinate system for each of the first joint point and the second joint point, the posture determination module 520 may: determine, based on the frame image, a 3D posture angle of each target joint point in the frame image by a 3D posture estimation model; determine a second three-dimensional coordinate of the first joint point corresponding to the frame image in a second coordinate system, where the second coordinate system is an image coordinate system corresponding to the target video; determine the first three-dimensional coordinate of the first joint point based on the second three-dimensional coordinate and a coordinate transformation relationship, where the coordinate transformation relationship is a transformation relationship between the first coordinate system and the second coordinate system; and determine the first three-dimensional coordinate of the second joint point based on the first three-dimensional coordinate of the first joint point and a 3D posture angle associated with the second joint point among the 3D posture angles of each target joint point.
[0191] Preferably, when determining the second three-dimensional coordinate of the first joint point corresponding to the frame image in the second coordinate system, the attitude determination module 520 may determine the second three-dimensional coordinate of the first joint point based on the three-dimensional attitude angle of the first joint point in the frame image.
[0192] Alternatively, the pose determination module 520 may determine a two-dimensional coordinate of the first articular point in a second coordinate system based on an image position of the first articular point in the frame image, estimate a depth coordinate of the first articular point based on the frame image and a correlation image of the frame image, and determine a second three-dimensional coordinate of the first articular point based on the two-dimensional coordinate and the depth coordinate, where the target video is a video captured by a monocular image capture device and the correlation image is a frame image immediately before the frame image, or the target video is one of two videos captured by a binocular image capture device and the correlation image is an image corresponding to the frame image in the other of the two videos.
[0193] Preferably, when determining the coordinate transformation relationship, the posture determination module 520 may determine a first distance between two associated joint points based on second three-dimensional coordinates of the two associated joint points, where the two associated joint points are two designated joint points having a parent-child relationship among each target joint point, obtain a reference distance in a first coordinate system between the two associated joint points, and determine the coordinate transformation relationship based on the first distance and the reference distance.
[0194] Preferably, the video capture module 510 may capture at least two videos captured by an image capture device having at least two capture viewing angles, the at least two capture viewing angles including a first viewing angle and at least one second viewing angle, and the target video is a video corresponding to the first viewing angle.
[0195] In this aspect, when determining a 3D posture angle of each target joint point of a target object in the frame image based on a frame image, the posture determination module 520 detects each initial joint point in the frame image of the target object based on the frame image, and each target joint point includes each initial joint point, determines an initial 3D posture angle in the frame image of each target joint point based on the frame image, determines that no missing joint point exists if the number of initial joint points is equal to the total number of each target joint point, determines the initial 3D posture angle in the frame image of each target joint point as the 3D posture angle in the frame image of each target joint point, and determines that a missing joint point exists if the number of initial joint points is smaller than the total number of each target joint point, determines a 3D posture angle of the missing joint point based on an associated image corresponding to the frame image in a non-target video, and determines the determined 3D posture angle of the missing joint point and the initial 3D posture angle of each initial joint point as the 3D posture angle of each target joint point in the frame image, and the non-target video is at least one video other than the target video among the at least two videos.
[0196] Preferably, the data processing device 50 may further include a video frame complementation module. The video frame complementation module determines, for each adjacent frame in the target video, optical flow information of the target object between the adjacent frames, the optical flow information includes the optical flow trajectory and optical flow direction of the target object, and performs frame complementation processing between the adjacent frames based on the optical flow information to obtain at least one frame complemented image between the adjacent frames, and obtains the target video after frame complementation based on the target video and each frame complemented image. Here, the pose determination module 520 performs processing on each frame image of the target video after frame complementation to obtain a three-dimensional pose corresponding to each frame image of the virtual object. The apparatus of the embodiment of the present invention can execute the method according to the embodiment of the present invention, and the realization principle is similar, and the operations performed by each module of the apparatus of the embodiment of the present invention correspond to the steps of the method of the embodiment of the present invention, and the detailed function description of each module of the apparatus may refer to the description of the corresponding method above, and the description thereof will be omitted here.
[0197] 13 is a schematic diagram showing an example of the configuration of a data processing system according to an embodiment of the present invention. As shown in FIG. 13, a data processing system 1300 includes an image capture device 1310, a data processing device 1320, and a terminal device 1330.
[0198] The image capture device 1310 captures a target video of a target object, where the target video includes at least one frame image.
[0199] The data processing device 1320 executes the method of the above embodiment based on the target video captured by the image capture device, thereby obtaining a 3D pose corresponding to each frame image of the target video of a virtual object related to the target object, and sends the obtained 3D pose to the terminal device.
[0200] The terminal device 1330 drives the virtual object to perform a corresponding action based on the acquired three-dimensional posture. Preferably, the image capture device 1310 includes three image capture devices corresponding to different capture viewing angles, the three image capture devices surround the target object and are evenly arranged around the target object. Here, the three image capture devices include one first device corresponding to a first viewing angle and two second devices corresponding to a second viewing angle, and the target video is a video captured by the first device.
[0201] Preferably, as described above, three identical types of image capture devices, each having a capture field of view of 120°, may be used to simultaneously capture images of the target object, with each image capture device positioned around the target object, with each image capture device being at an equal distance from the target object and each image capture device being at an equal distance from the ground, thereby keeping the target object within the capture field of view of all image capture devices, with the three image capture devices surrounding the subject and evenly positioned around the target object.
[0202] When determining, for each frame image of the target video, a three-dimensional posture angle of each target joint point of the target object in the frame image, the data processing device 1320 detects each initial joint point in the frame image of the target object based on the frame image of the target video, and each target joint point includes each initial joint point, determines an initial three-dimensional posture angle of each target joint point in the frame image based on the frame image of the target video, and if the number of initial joint points is equal to the total number of target joint points, determines that there are no missing joint points, and determines the initial three-dimensional posture angle of each target joint point in the frame image. The non-target video is at least one video other than the target video among at least two videos corresponding to at least two viewing angles captured by the first device and the second device, the non-target video being ....
[0203] An embodiment of the present invention further provides an electronic device, the electronic device including a memory in which a computer program is stored and a processor, the processor executing the computer program to implement the above data processing method.
[0204] FIG. 14 is a schematic diagram showing an example of the configuration of an electronic device according to an embodiment of the present invention. In one preferred embodiment, an electronic device is provided. As shown in FIG. 14, the electronic device 4000 includes a processor 4001 and a memory 4003. The processor 4001 is connected to the memory 4003 via, for example, a bus 4002. Preferably, the electronic device 4000 may further include a transceiver 4004 that can be used for data interaction between the electronic device and other electronic devices, such as transmitting data and / or receiving data. Note that in practical applications, the transceiver 4004 is not limited to one, and the configuration of the electronic device 4000 does not limit the embodiment of the present invention.
[0205] The processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic device, transistor logic device, hardware component, or any combination thereof. It may implement or execute various example logic blocks, modules, and circuits described with reference to this specification. The processor 4001 may also be a combination that performs computational functions, including, for example, a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.
[0206] The bus 4002 may include paths for communicating information between the above components. The bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus 4002 may be divided into an address bus, a data bus, a control bus, etc. Although the buses are shown in bold lines in FIG. 14 for the sake of convenience, this does not mean that there is only one bus or that there is only one type of bus.
[0207] The memory 4003 may be, but is not limited to, a ROM (Read Only Memory) or other type of static storage device capable of storing static information and instructions, a RAM (Random Access Memory) or other type of dynamic storage device capable of storing information and instructions, an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory), or other optical disk storage device, optical disk storage device (including compressed disks, laser disks, optical disks, digital versatile disks, Blu-ray disks, etc.), a magnetic disk storage medium, other magnetic storage device, or any other medium that can be used to carry or store a computer program and that can be read by a computer.
[0208] The memory 4003 is for storing a computer program for carrying out an embodiment of the present invention, and the execution of which is controlled by the processor 4001. The processor 4001 is configured to execute the computer program stored in the memory 4003 to perform the steps shown in the above method embodiments.
[0209] An embodiment of the present invention provides a computer readable storage medium having a computer program stored thereon, the computer program implementing the above data processing method when executed by a processor.
[0210] An embodiment of the present invention further provides a computer program product including a computer program which, when executed by a processor, implements the above data processing method.
[0211] Based on the same principle as the method according to the embodiment of the present invention, the embodiment of the present invention further provides a computer program product or a computer program, comprising computer instructions stored in a computer-readable storage medium, wherein a processor of a computing device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions so that the computing device performs the method provided in any embodiment of the present invention above.
[0212] In the flowcharts of the embodiments of the present invention, each operation step is indicated by an arrow, but the execution order of these steps is not limited to the order indicated by the arrow. Unless explicitly stated herein, in some implementation scenarios of the embodiments of the present invention, the implementation steps in each flowchart may be executed in other orders as necessary. Furthermore, some or all of the steps in each flowchart may include multiple sub-steps or multiple stages based on the actual implementation scenario. Some or all of these sub-steps or phases may be executed at the same time, and each sub-step or phase of these sub-steps or phases may be executed at different times. In scenarios with different execution times, the execution order of these sub-steps or phases may be flexibly configured according to requirements, and the embodiments of the present invention are not limited thereto.
[0213] It should be noted that the above are merely preferred embodiments of some implementation scenarios of the present invention, and other similar implementation means based on the technical idea of the present invention without departing from the technical idea of the present invention are within the scope of the embodiments of the present invention.
Claims
1. A data processing method executed by an electronic device, comprising: acquiring a target video of a target object, the target object including at least one frame image; performing processing for each frame image to obtain a three-dimensional pose corresponding to each frame image of a virtual object related to the target object; The process comprises: determining, based on the frame image, a three-dimensional posture angle in the frame image of each target joint point of the target object, and first three-dimensional coordinates corresponding to the frame image in a first coordinate system of each of a first joint point and a second joint point, wherein the first joint point and the second joint point are designated joint points among the each target joint point, the first coordinate system is a coordinate system corresponding to the virtual object, and the first joint point is a root node among the each target joint point; determining a displacement offset of the second joint point based on a first three-dimensional coordinate of the second joint point corresponding to the frame image and a historical three-dimensional coordinate corresponding to a frame image immediately before the first frame image; if the displacement offset is equal to or smaller than a set value, correcting the first three-dimensional coordinate of the first joint point based on the first three-dimensional coordinate and the history three-dimensional coordinate of the second joint point to obtain a target three-dimensional coordinate of the first joint point; determining a three-dimensional posture of the virtual object corresponding to the frame image based on the target three-dimensional coordinates of the first joint point and a three-dimensional posture angle in the frame image of each of the target joint points.
2. determining a three-dimensional posture of the virtual object corresponding to the frame image based on the target three-dimensional coordinates of the first joint point and a three-dimensional posture angle of each of the target joint points in the frame image, a step of determining target three-dimensional coordinates in a first coordinate system of another joint point in the frame image based on the target three-dimensional coordinates of the first joint point and a three-dimensional posture angle of the other joint point, the other joint point being a joint point other than the first joint point and the second joint point among the target joint points; and determining a three-dimensional posture of the virtual object corresponding to the frame image based on the target three-dimensional coordinates of each of the target joint points and a three-dimensional posture angle of each of the target joint points in the frame image.
3. The step of correcting the first three-dimensional coordinates of the first joint point based on the first three-dimensional coordinates and the history three-dimensional coordinates of the second joint point includes: determining a difference between the first three-dimensional coordinate and the historical three-dimensional coordinate of the second articulation point in each dimension of the first coordinate system; and correcting a corresponding dimension of the first three-dimensional coordinates of the first joint point based on the difference corresponding to each dimension to obtain a target three-dimensional coordinate of the first joint point.
4. obtaining a frame rate of the target video; determining a filtering sliding window width of the two-dimensional pose angle of the joint point based on a frame rate of the target video; and performing a filtering process on the posture angle sequence in each dimension of each target joint point based on the filtering sliding window width to obtain a posture angle sequence after the filtering process, wherein the posture angle sequence in the dimension of the target joint point includes a two-dimensional posture angle in the dimension of each frame image of the target video of the target joint point, determining a three-dimensional posture of the virtual object corresponding to the frame image based on the target three-dimensional coordinates of the first joint point and a three-dimensional posture angle of each of the target joint points in the frame image, obtaining a filtered 3D pose angle of each target joint point in the frame image based on the filtered pose angle sequence in each dimension; and determining a three-dimensional posture of the virtual object corresponding to the frame image based on the target three-dimensional coordinates of the first joint point and a filtered three-dimensional posture angle of each of the target joint points in the frame image.
5. The step of determining a filtering sliding window width of the two-dimensional posture angle of the joint point based on the frame rate of the target video includes: determining an adjustment factor for a sliding window width based on the frame rate; determining an initial sliding window width based on the product of the frame rate and the adjustment factor; and determining a larger value of the initial sliding window width and a predetermined sliding window width as the filtering sliding window width.
6. The step of determining, based on the frame images, a three-dimensional posture angle in the frame images of each target joint point of the target object and first three-dimensional coordinates in a first coordinate system of each of a first joint point and a second joint point corresponding to the frame images, includes: determining a three-dimensional posture angle of each of the target joint points in the frame images using a three-dimensional posture estimation model based on the frame images; determining a second three-dimensional coordinate of the first articulation point in a second coordinate system corresponding to the frame image, the second coordinate system being an image coordinate system corresponding to the target video; determining a first three-dimensional coordinate of the first joint point based on the second three-dimensional coordinate and a coordinate transformation relationship, the coordinate transformation relationship being a transformation relationship between the first coordinate system and the second coordinate system; determining a first three-dimensional coordinate of the second joint point based on the first three-dimensional coordinate of the first joint point and a three-dimensional pose angle associated with the second joint point among the three-dimensional pose angles of each of the target joint points.
7. The step of determining a second three-dimensional coordinate of the first joint point in a second coordinate system corresponding to the frame image includes: The method according to claim 6 , further comprising: determining a second three-dimensional coordinate of the first joint point based on a three-dimensional pose angle of the first joint point in the frame image.
8. The step of determining a second three-dimensional coordinate of the first joint point in a second coordinate system corresponding to the frame image includes: determining a two-dimensional coordinate of the first joint point in the second coordinate system based on an image position of the first joint point in the frame image; estimating a coordinate in a depth direction of the first joint point based on the frame image and a correlation image of the frame image; determining a second three-dimensional coordinate of the first joint point based on the two-dimensional coordinate and the depth coordinate; 7. The method of claim 6, wherein the target video is a video captured by a monocular image capture device and the correlation image is a frame image immediately preceding the frame image, or the target video is one of two videos captured by a binocular image capture device and the correlation image is an image corresponding to the frame image in the other of the two videos.
9. determining a first distance between two associated joint points based on second three-dimensional coordinates of the two associated joint points, the two associated joint points being two designated joint points having a parent-child relationship among the target joint points; obtaining a reference distance in the first coordinate system of the two related joint points; The method of claim 6 , further comprising: determining the coordinate transformation relationship based on the first distance and the reference distance.
10. The step of obtaining a target video of the target object includes: obtaining at least two videos captured by an image capture device having at least two capture viewing angles, the at least two capture viewing angles including a first viewing angle and at least one second viewing angle, the target video being a video corresponding to the first viewing angle; The step of determining a three-dimensional posture angle of each target joint point of the target object in the frame image based on the frame image includes: detecting initial joint points in the frame images of the target object based on the frame images, the target joint points including the initial joint points; determining an initial three-dimensional posture angle of each of the target joint points in the frame image based on the frame image; When the number of the initial joint points is equal to the total number of the target joint points, determining that there is no missing joint point, and determining the initial three-dimensional posture angle of each of the target joint points in the frame image as the three-dimensional posture angle of each of the target joint points in the frame image; 2. The method of claim 1, further comprising: determining that a missing joint point exists when the number of the initial joint points is smaller than the total number of the target joint points; determining a three-dimensional posture angle of the missing joint point based on an associated image corresponding to the frame image in a non-target video; and determining the determined three-dimensional posture angle of the missing joint point and the initial three-dimensional posture angle of each initial joint point as the three-dimensional posture angle in the frame image of each target joint point, wherein the non-target video is at least one video other than the target video among the at least two videos.
11. before performing processing for each frame image to obtain a three-dimensional pose corresponding to each frame image of a virtual object related to the target object; determining, for each adjacent frame in the target video, optical flow information of the target object between the adjacent frames, the optical flow information including an optical flow trajectory and an optical flow direction of the target object; performing a frame interpolation process between the adjacent frames based on the optical flow information to obtain at least one frame interpolated image between the adjacent frames; and obtaining a frame-interpolated target video based on the target video and each frame-interpolated image. The method of claim 1 , further comprising: performing processing for each frame image of the frame-interpolated target video to obtain a three-dimensional pose of the virtual object corresponding to each frame image.
12. 1. A data processing device, comprising: a video capture module for capturing a target video of a target object, the target object including at least one frame image; a pose determination module that performs processing on each frame image to obtain a three-dimensional pose corresponding to each frame image of a virtual object related to the target object; The attitude determination module, in performing the process, determining, based on the frame image, a three-dimensional posture angle in the frame image of each target joint point of the target object, and first three-dimensional coordinates corresponding to the frame image in a first coordinate system for each of a first joint point and a second joint point, wherein the first joint point and the second joint point are designated joint points among the each target joint point, the first coordinate system is a coordinate system corresponding to the virtual object, and the first joint point is a root node among the each target joint point; determining a displacement offset of the second joint point based on a first three-dimensional coordinate of the second joint point corresponding to the frame image and a history three-dimensional coordinate corresponding to a frame image immediately before the first frame image; If the displacement offset is equal to or smaller than a set value, correcting the first three-dimensional coordinate of the first joint point based on the first three-dimensional coordinate and the history three-dimensional coordinate of the second joint point to obtain a target three-dimensional coordinate of the first joint point; determining a three-dimensional posture of the virtual object corresponding to the frame image based on target three-dimensional coordinates of the first joint point and a three-dimensional posture angle of each of the target joint points in the frame image.
13. A data processing system including an image capture device, a data processing device, and a terminal device, the image capture device captures a target video of a target object, the target video including at least one frame image; The data processing device executes the method according to any one of claims 1 to 11 based on the target video captured by the image capture device to obtain a three-dimensional pose of a virtual object related to the target object corresponding to each frame image of the target video, and transmits the obtained three-dimensional pose to the terminal device; The terminal device drives the virtual object to perform a corresponding action based on the acquired three-dimensional posture.
14. the image capture apparatus includes three image capture devices corresponding to different capture viewing angles, the three image capture devices surround the target object and are evenly spaced around the target object, the three image capture devices include one first device corresponding to a first viewing angle and two second devices corresponding to a second viewing angle, the target video is a video captured by the first devices; The data processing device determines, for each frame image of the target video, a three-dimensional posture angle of each target joint point of the target object in the frame image, Detecting initial joint points in the frame images of the target object based on the frame images of the target video, the target joint points including the initial joint points; determining an initial three-dimensional posture angle of each of the target joint points in the frame image based on the frame image of the target video; If the number of the initial joint points is equal to the total number of the target joint points, it is determined that there are no missing joint points, and the initial three-dimensional posture angles of the target joint points in the frame image are determined as the three-dimensional posture angles of the target joint points in the frame image; 14. The data processing system of claim 13, further comprising: determining that a missing joint point exists when the number of the initial joint points is smaller than the total number of the target joint points; determining a three-dimensional posture angle of the missing joint point based on an associated image corresponding to the frame image in a non-target video; determining the determined three-dimensional posture angle of the missing joint point and the initial three-dimensional posture angle of each initial joint point as the three-dimensional posture angle in the frame image of each target joint point; and determining the non-target video as at least one video other than the target video among at least two videos corresponding to at least two viewing angles captured by the first device and the second device.
15. 12. An electronic device comprising: a memory having a computer program stored therein; and a processor, the processor executing the computer program to implement a method according to any one of claims 1 to 11.
16. A computer program product causing a computer to carry out the method according to any one of claims 1 to 11.
Citation Information
Patent Citations
Movement converter for three-dimensional skeleton structure
JP1997330424A
Information terminal device, information processing method, system, and program
JP2016053788A
Information processing apparatus and method
JP2017097578A
Method for generating data for estimating three-dimensional pose of object included in input image, computer system, and method for constructing prediction model
JP2021111380A
Image processing method and apparatus, image device, and storage medium
JP2021525431A
