Motion capture system, motion capture method, and readable storage medium

Abstract

The invention discloses a motion capture system, a motion capture method and a readable storage medium, and belongs to the technical field of motion capture. The motion capture system comprises a motion capture sensor group and a motion capture processing unit, the motion capture sensor group comprises at least one invisible light image sensor and at least one visible light image sensor, the invisible light image sensor is used for sensing a detected target in a near-infrared or infrared band, and the visible light image sensor is used for sensing the detected target in a near-infrared or infrared band. The visible light image sensor is used for sensing a detected target in a visible light wave band and outputting a visible light image; and the motion capture processing unit is connected with the motion capture sensor group and is used for determining motion posture data of the detected target based on the visible light image and the near-infrared or infrared band image. According to the invention, the motion attitude data of the measured target is obtained based on the sensing data of different modes, and the precision and robustness of motion capture of the measured target by the motion capture system are improved.

Description

Technical Field

[0001] The present application belongs to the field of motion capture technology, and in particular, relates to a motion capture system, a motion capture method and a readable storage medium. Background Art

[0002] Optical motion capture technology is widely used in many fields due to its high precision and fast response. This technology mainly relies on near-infrared motion capture cameras to monitor and track specific markers installed on the target object. In theory, as long as the marker is captured by two cameras at the same time, the three-dimensional spatial position of the point can be determined. Through continuous shooting, the movement trajectory of the marker over time can be tracked. In the traditional optical motion capture system, multiple near-infrared cameras are arranged around the target to be measured, and their overlapping field of view constitutes the activity range of the target to be measured. In order to facilitate identification and processing, special luminous or reflective markers are usually attached to the surface of the target to be measured.

[0003] However, when optical motion capture technology is applied to deformable objects such as humans and animals, the geometric invariance may fail due to the movement of muscles, skin, hair, etc. on the surface of the object. This will cause the ID of the landmark point to be unable to be re-identified or to be misidentified after target tracking fails, resulting in inaccurate and non-robust results such as abnormal motion posture data or solution failure. Therefore, how to capture the motion of the target to be measured to meet the requirements of high precision and high robustness has become a major challenge in the field of motion capture. Summary of the invention

[0004] The present application aims to solve at least one of the technical problems existing in the related art. To this end, the present application proposes a motion capture system, a motion capture method and a readable storage medium, which obtain motion posture data of a measured target based on perception data of different modalities, thereby improving the accuracy and robustness of the motion capture system in capturing motion of the measured target.

[0005] In a first aspect, the present application provides a motion capture system, the system comprising:

[0006] Motion capture sensor group and motion capture processing unit,

[0007] The motion capture sensor group includes at least one non-visible light image sensor and at least one visible light image sensor, wherein the non-visible light image sensor is used to sense the target to be measured in the near infrared or infrared band and output the image of the near infrared or infrared band, and the visible light image sensor is used to sense the target to be measured in the visible light band and output the visible light image;

[0008] The motion capture processing unit is connected to the motion capture sensor group and is used to determine the motion posture data of the measured target based on the visible light image and the image in the near infrared or infrared band.

[0009] In the above technical scheme, the motion capture system includes a motion capture sensor group and a motion capture processing unit, the motion capture sensor group includes at least one non-visible light image sensor and at least one visible light image sensor, wherein the non-visible light image sensor is used to sense the target to be measured in the near-infrared or infrared band and output an image in the near-infrared or infrared band, and the visible light image sensor is used to sense the target to be measured in the visible light band and output a visible light image. The motion capture processing unit is connected to the motion capture sensor group and is used to determine the motion posture data of the target to be measured based on the visible light image and the image in the near-infrared or infrared band, thereby improving the accuracy and robustness of the motion capture system for motion capture of the target to be measured.

[0010] According to one embodiment of the present application, the motion capture system further includes at least one fill light source in the near-infrared or infrared band for illuminating the target to be measured;

[0011] At least one marking point is arranged on the measured target, and under the illumination of the supplementary light source, the marking point has a contrast with the background in the imaging of the non-visible light image sensor.

[0012] In the above technical solution, the motion capture system includes at least one fill light source in the near-infrared or infrared band, which is used to illuminate the target to be measured, and at least one marker point is arranged on the target to be measured. Under the illumination of the fill light source, the marker point has a contrast with the background in the imaging of the non-visible light image sensor, making the marker point more prominent, thereby improving the accuracy of identifying and locating the marker point in the imaging of the non-visible light image sensor, thereby improving the accuracy of the motion capture system in capturing the motion of the target to be measured.

[0013] According to one embodiment of the present application, the motion capture system also includes: a first image processing unit, connected to the non-visible light image sensor, for tracking and spatially positioning the marker points in the near-infrared or infrared band image, and outputting the coordinates of the marker points to the motion capture processing unit.

[0014] In the above technical solution, the motion capture system also includes a first image processing unit, which is connected to the non-visible light image sensor and is used to track and spatially locate marker points in images in the near-infrared or infrared bands, and output the coordinates of the marker points to the motion capture processing unit, thereby improving the flexibility of the motion capture system.

[0015] According to one embodiment of the present application, the motion capture processing unit is used to perform at least one of the following:

[0016] Performing three-dimensional reconstruction of the marker points according to the marker point coordinates;

[0017] Performing ID tracking on the marker point to obtain a marker point ID tracking result;

[0018] Performing key point detection on the visible light image to obtain key point information, wherein the key point information includes coordinates and IDs of the key points;

[0019] Performing motion posture estimation on the target to be measured in the visible light image to obtain a posture estimation result;

[0020] Reprojecting the marker point onto the visible light image and comparing it with the ID of the key point to verify the correctness of the marker point ID tracking result or to perform ID identification on the marker point that has lost tracking;

[0021] For image frames that have lost landmark information, the pose estimation result is used to fill in the missing landmark information to obtain motion pose data of the measured target.

[0022] In the above technical scheme, the motion capture processing unit is used to perform three-dimensional reconstruction of the marker points according to the coordinates of the marker points, and to perform ID tracking of the marker points to obtain the marker point ID tracking results, perform key point detection on the visible light image to obtain the coordinates and IDs of the key points, estimate the motion posture of the target under test in the visible light image to obtain the pose estimation results, reproject the marker points onto the visible light image, and compare them with the IDs of the key points to verify the correctness of the marker point ID tracking results or to perform ID identification on the marker points that have lost tracking, and fill in the image frames that have lost the marker point information with the pose estimation results, thereby realizing the motion posture data of the target under test based on the perception data of different modalities, and in the case of the loss of marker point information, the motion posture data of the target under test can also be obtained by filling in the pose estimation results obtained by performing motion posture estimation on the visible light image, thereby improving the accuracy and robustness of the motion capture system for motion capture of the target under test.

[0023] According to one embodiment of the present application, the motion capture system also includes: a second image processing unit, connected to the visible light image sensor, for performing one or more of the following processing on the visible light image: feature extraction, feature recognition, feature space positioning calculation, motion data solution, and sending the processing results to the motion capture processing unit.

[0024] In the above technical solution, the second image processing unit can be used to perform feature extraction, feature recognition, feature space positioning calculation or motion data solution on the visible image, and send the processing results to the motion capture processing unit. The second image processing unit can be an embedded motion capture processing unit, or it can exist in the motion capture system in the form of a host computer, thereby improving the flexibility of the motion capture system.

[0025] According to one embodiment of the present application, the second image processing unit is specifically configured to:

[0026] Performing key point detection on the visible light image to obtain key point information, wherein the key point information includes coordinates and IDs of the key points;

[0027] The motion posture of the measured target in the visible light image is estimated to obtain a posture estimation result.

[0028] In the above technical scheme, the second image processing unit is used to perform key point detection on the visible light image, obtain the coordinates and IDs of the key points, estimate the motion posture of the target under test in the visible light image, and obtain the pose estimation result, which helps to obtain the motion posture data of the target under test based on the perception data of different modalities, thereby improving the accuracy and robustness of the motion capture system for motion capture of the target under test.

[0029] According to one embodiment of the present application, the motion capture processing unit is used to perform at least one of the following:

[0030] Performing three-dimensional reconstruction of the marker points according to the marker point coordinates;

[0031] Perform ID tracking on the landmark points to obtain the landmark point ID tracking results;

[0032] Reprojecting the marker point onto the visible light image and comparing it with the ID of the key point to verify the correctness of the marker point ID tracking result or to perform ID identification on the marker point that has lost tracking;

[0033] For image frames that have lost landmark information, the pose estimation result is used to fill in the missing landmark information to obtain motion pose data of the measured target.

[0034] In the above technical scheme, the motion capture processing unit is used to perform three-dimensional reconstruction of the marker points according to the coordinates of the marker points, perform ID tracking on the marker points, obtain the marker point ID tracking results, reproject the marker points onto the visible light image, and compare them with the IDs of the key points to verify the correctness of the marker point ID tracking results or perform ID identification on the marker points that have lost tracking. For image frames that have lost marker point information, the pose estimation results are used to fill in the gaps, thereby achieving motion posture data of the target under test based on perception data of different modalities. In the event that the marker point information is lost, the pose estimation results obtained by performing motion posture estimation on the visible light image can also be used to fill in the gaps, thereby obtaining the motion posture data of the target under test, thereby improving the accuracy and robustness of the motion capture system in capturing the motion of the target under test.

[0035] In a second aspect, the present application provides a motion capture method, the method comprising:

[0036] Sensing the target in a near-infrared or infrared band, and outputting an image in the near-infrared or infrared band, wherein the image in the near-infrared or infrared band includes at least one marker point;

[0037] Tracking and spatially locating the marker points in the image of the near-infrared or infrared band to obtain the coordinates of the marker points;

[0038] Sensing the measured target in the visible light band and outputting a visible light image;

[0039] Based on the marker point coordinates and the visible light image, the motion posture data of the measured object is determined.

[0040] In the above technical scheme, the target to be measured is sensed in the near-infrared or infrared band, and an image of the near-infrared or infrared band is output, wherein the image of the near-infrared or infrared band includes at least one marker point, the marker point in the image of the near-infrared or infrared band is tracked and spatially positioned to obtain the coordinates of the marker point, the target to be measured is sensed in the visible light band, and a visible light image is output, and based on the marker point coordinates and the visible light image, the motion posture data of the target to be measured is obtained, thereby realizing the fusion of perception data of different modalities and improving the accuracy and robustness of the motion capture method.

[0041] According to an embodiment of the present application, determining the motion posture data of the measured target based on the marker point coordinates and the visible light image includes:

[0042] Performing three-dimensional reconstruction of the marker points according to the marker point coordinates;

[0043] Performing ID tracking on the marker point to obtain a marker point ID tracking result;

[0044] Performing key point detection on the visible light image to obtain key point information, wherein the key point information includes coordinates and IDs of the key points;

[0045] Performing motion posture estimation on the target to be measured in the visible light image to obtain a posture estimation result;

[0046] Reprojecting the marker point onto the visible light image and comparing it with the ID of the key point to verify the correctness of the marker point ID tracking result or to perform ID identification on the marker point that has lost tracking;

[0047] For image frames that have lost landmark information, the pose estimation result is used to fill in the missing landmark information to obtain motion pose data of the measured target.

[0048] In the above technical scheme, the marker points are three-dimensionally reconstructed according to the coordinates of the marker points, the marker points are ID tracked to obtain the marker point ID tracking results, the key point detection is performed on the visible light image to obtain the coordinates and IDs of the key points, the motion posture of the target to be measured in the visible light image is estimated to obtain the pose estimation result, the marker points are reprojected onto the visible light image, and compared with the IDs of the key points to verify the correctness of the marker point ID tracking results or to perform ID identification on the marker points that have lost tracking, and the image frames that have lost the marker point information are filled with the pose estimation results, so that the motion posture data of the target to be measured can be obtained based on the perception data of different modalities. In the case of loss of marker point information, it can also be filled based on the pose estimation result obtained by motion posture estimation of the visible light image, so as to obtain the motion posture data of the target to be measured, thereby improving the accuracy and robustness of the motion capture system in capturing the motion of the target to be measured.

[0049] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the motion capture method as described in the second aspect above when executing the computer program.

[0050] In a fourth aspect, the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the motion capture method as described in the second aspect above.

[0051] In a fifth aspect, the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the motion capture method as described in the second aspect.

[0052] In a sixth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the motion capture method as described in the second aspect above.

[0053] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0055] Figure 1 is one of the structural schematic diagrams of the motion capture system provided in some embodiments of the present application;

[0056] Figure 2 This is the second structural schematic diagram of the motion capture system provided by some embodiments of the present application;

[0057] Figure 3 This is the third structural schematic diagram of the motion capture system provided by some embodiments of the present application;

[0058] Figure 4 This is a fourth structural diagram of a motion capture system provided by some embodiments of the present application;

[0059] Figure 5 is a flowchart of a motion capture method provided by some embodiments of the present application;

[0060] Figure 6 It is a schematic diagram of the structure of an electronic device provided in some embodiments of the present application.

[0061] Description of reference numerals:

[0062] 101: motion capture sensor group; 102: motion capture processing unit; 103: non-visible light image sensor;

[0063] 104: visible light image sensor; 105: fill light source; 106: first image processing unit;

[0064] 107: second image processing unit; 600: electronic device; 601: processor; 602: memory. DETAILED DESCRIPTION

[0065] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0066] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0067] The motion capture system, motion capture method and readable storage medium provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0068] Figure 1 FIG. 1 is one of the structural diagrams of the motion capture system provided in some embodiments of the present application. Figure 1 As shown, the motion capture system includes:

[0069] Motion capture sensor group 101 and motion capture processing unit 102,

[0070] The motion capture sensor group 101 includes at least one non-visible light image sensor 103 and at least one visible light image sensor 104, wherein the non-visible light image sensor 103 is used to sense the target in the near-infrared or infrared band and output the image in the near-infrared or infrared band, and the visible light image sensor 104 is used to sense the target in the visible light band and output the visible light image;

[0071] The motion capture processing unit 102 is connected to the motion capture sensor group 101, and is used to determine the motion posture data of the measured target based on the visible light image and the image in the near infrared or infrared band.

[0072] It can be understood that the non-visible light image sensor 103 is used to sense the target to be measured in the near-infrared or infrared band and output the image of the near-infrared or infrared band. Since the light in the near-infrared and infrared bands is invisible to the human eye, the non-visible light image sensor is not limited by the visible light conditions and is more suitable for environments with undesirable visible light conditions. The motion capture sensor group includes at least one non-visible light image sensor 103, which improves the applicability of the motion capture system under different lighting conditions; the visible light image sensor 104 is used to sense the target to be measured in the visible light band and output the visible light image. Under good lighting conditions, the visible light image sensor 104 can output a high-resolution color image. Optionally, the Zhang Zhengyou calibration method is used to calibrate the sensors of each modality.

[0073] The motion capture sensor group 101 includes at least one non-visible light image sensor 103 and at least one visible light image sensor 104, so that the motion capture sensor group 101 can perceive the target from both optical and visual dual modes. The optical mode corresponds to the image in the near-infrared or infrared band output by the non-visible light image sensor 103, and the visual mode corresponds to the visible light image output by the visible light image sensor 104, thereby improving the comprehensiveness and accuracy of the perception of the target, obtaining motion posture data based on perception data of different modes, and also improving the accuracy of the motion capture system in capturing the motion of the target.

[0074] The motion capture processing unit 102 is used to process the optical and visual dual-modal perception data, i.e., images in the near-infrared or infrared bands and visible light images, such as data matching, fusion and analysis, to obtain the motion posture data of the measured target, and the motion posture data of the measured target includes the position data and posture data of the measured target in three-dimensional space, thereby realizing the motion capture of the measured target. It can be understood that the motion posture data is obtained based on the perception data of different modes, and the fusion of the perception data of different modes is realized, which improves the accuracy of the motion capture system in capturing the motion of the measured target. The sensor of a single mode may be affected by occlusion or other factors, resulting in the output perception data being inaccurate. The fusion of the perception data of sensors of different modes can improve the robustness of the motion capture system.

[0075] In the above technical scheme, the motion capture system includes a motion capture sensor group and a motion capture processing unit, the motion capture sensor group includes at least one non-visible light image sensor and at least one visible light image sensor, wherein the non-visible light image sensor is used to sense the target to be measured in the near-infrared or infrared band and output an image in the near-infrared or infrared band, and the visible light image sensor is used to sense the target to be measured in the visible light band and output a visible light image. The motion capture processing unit is connected to the motion capture sensor group and is used to determine the motion posture data of the target to be measured based on the visible light image and the image in the near-infrared or infrared band, thereby improving the accuracy and robustness of the motion capture system for motion capture of the target to be measured.

[0076] Figure 2 This is the second structural diagram of the motion capture system provided by some embodiments of the present application. Figure 2 As shown, in some embodiments, the motion capture system further includes at least one near-infrared or infrared fill light source 105 for illuminating the target;

[0077] At least one marking point is arranged on the measured target. Under the illumination of the supplementary light source 105 , the marking point has a contrast with the background in the imaging of the non-visible light image sensor 103 .

[0078] The near-infrared or infrared band fill light source 105 is used to provide near-infrared or infrared band light to illuminate the target to be measured, so that the non-visible light image sensor 103 can perceive the target to be measured more clearly, thereby improving the quality of the near-infrared or infrared band image output by the non-visible light image sensor 103, and also enabling the non-visible light image sensor 103 to still perceive the target to be measured when the near-infrared or infrared band lighting conditions are not ideal.

[0079] At least one mark point is arranged on the target to be measured. The mark point is a special mark or light-emitting point, also called a marker. Under the illumination of the fill light source 105, the mark point will form a contrast with the background in the imaging of the non-visible light image sensor, making the mark point more prominent in the image of the near-infrared or infrared band output by the non-visible light image sensor 103, and forming a contrast with the surrounding background, making the mark point easier to identify and track in subsequent image processing and analysis, thereby improving the accuracy of identifying and locating the mark point.

[0080] In the above technical solution, the motion capture system includes at least one fill light source in the near-infrared or infrared band, which is used to illuminate the target to be measured, and at least one marker point is arranged on the target to be measured. Under the illumination of the fill light source, the marker point has a contrast with the background in the imaging of the non-visible light image sensor, making the marker point more prominent, thereby improving the accuracy of identifying and locating the marker point in the imaging of the non-visible light image sensor, thereby improving the accuracy of the motion capture system in capturing the motion of the target to be measured.

[0081] Figure 3 This is the third structural diagram of the motion capture system provided by some embodiments of the present application. Figure 3 As shown, in some embodiments, the motion capture system also includes: a first image processing unit 106, connected to the non-visible light image sensor 103, for tracking and spatially positioning the marker points in the near-infrared or infrared band image, and outputting the coordinates of the marker points to the motion capture processing unit 102.

[0082] It is understandable that the motion capture sensor group 101 in the motion capture system includes at least one non-visible light image sensor 103 and at least one visible light image sensor 104, which are used to sense the target to be measured and obtain images in the near-infrared or infrared band and visible light images. In some embodiments, the motion capture processing unit 102 processes the images in the near-infrared or infrared band and the visible light images to obtain motion posture data of the target to be measured.

[0083] In some embodiments, the motion capture system also includes a first image processing unit 106, which uses image processing technology to track marker points in images in the near-infrared or infrared bands, and obtains the coordinates of the marker points through spatial positioning calculations. The motion capture processing unit 102 receives the coordinates of the marker points output by the first image processing unit 106, thereby improving the flexibility of the motion capture system.

[0084] Optionally, the first image processing unit 106 is an embedded motion capture processing unit, such as a Field Programmable Gate Array (FPGA), etc., which is embedded in the motion capture sensor group 101 and connected to the non-visible light image sensor 103 .

[0085] In the above technical solution, the motion capture system also includes a first image processing unit, which is connected to the non-visible light image sensor and is used to track and spatially locate marker points in images in the near-infrared or infrared bands, and output the coordinates of the marker points to the motion capture processing unit, thereby improving the flexibility of the motion capture system.

[0086] like Figure 3 As shown, when the motion capture system further includes a first image processing unit 106, the motion capture processing unit 102 is configured to perform at least one of the following:

[0087] Performing three-dimensional reconstruction of the marker points according to the marker point coordinates;

[0088] Performing ID tracking on the marker point to obtain a marker point ID tracking result;

[0089] Performing key point detection on the visible light image to obtain key point information, wherein the key point information includes coordinates and IDs of the key points;

[0090] Performing motion posture estimation on the target to be measured in the visible light image to obtain a posture estimation result;

[0091] Reprojecting the marker point onto the visible light image and comparing it with the ID of the key point to verify the correctness of the marker point ID tracking result or to perform ID identification on the marker point that has lost tracking;

[0092] For image frames that have lost landmark information, the pose estimation result is used to fill in the missing landmark information to obtain motion pose data of the measured target.

[0093] Optionally, the motion capture processing unit 102 is an embedded processing unit, such as a field programmable gate array (FPGA), etc., which is embedded in the motion capture sensor group 101, and can also exist in the motion capture system in the form of a host computer.

[0094] In some embodiments, the motion capture sensor group 101 includes at least one non-visible light image sensor 103, which can sense the object under test from at least one perspective to obtain images in the near-infrared or infrared bands, and the first image processing unit 106 tracks the marker points in these near-infrared or infrared band images through image processing technology, obtains the coordinates of the marker points through spatial positioning calculation and outputs them to the motion capture processing unit 102, and the motion capture processing unit 102 receives the coordinates of the marker points.

[0095] The motion capture processing unit 102 uses a triangulation method to perform multi-viewpoint three-dimensional reconstruction on the object under test according to the coordinates of the marker points to obtain the three-dimensional coordinates of the marker points.

[0096] The motion capture processing unit 102 performs ID tracking on each marker point to determine its position change in continuous image frames or time series, and assigns a unique ID to each marker point and tracks the ID of each marker point to achieve the ability to identify and associate the same marker point during the dynamic capture of the target being measured. Optionally, the motion capture processing unit 102 uses methods such as Kalman filtering to perform ID tracking on the marker point to obtain a marker point ID tracking result, which includes the three-dimensional coordinates of each marker point and its corresponding image frame sequence or time series.

[0097] Optionally, the motion capture processing unit 102 uses a deep learning algorithm, such as YOLOv8, to perform key point detection on the visible light image to obtain the coordinates and ID of the key point, where the ID is a unique identifier of the key point.

[0098] Optionally, the motion capture processing unit 102 uses algorithms such as MediaPipe to estimate the motion posture of the target under test in the visible light image to obtain a pose estimation result, which is an estimation of the motion posture data. The motion posture data of the target under test includes position data and posture data of the target under test in three-dimensional space.

[0099] Furthermore, the motion capture processing unit 102 reprojects the marker points onto the visible light image according to the three-dimensional coordinates of the marker points, converts the three-dimensional spatial coordinates of the marker points into two-dimensional reprojected coordinates, and compares the projection of the marker points in the visible light image with the key points detected in the visible light image, including comparing their coordinate positions and IDs, so as to verify the correctness of marker ID tracking or perform ID identification on marker points that have lost tracking.

[0100] For image frames with missing marker information, the missing marker information in the image frame is filled by using the pose estimation result to obtain the motion pose data of the measured target. The image frame here refers to a non-visible light image. Optionally, the filling method can be direct filling or filtering, such as Kalman filtering.

[0101] In the above technical scheme, the motion capture processing unit is used to perform three-dimensional reconstruction of the marker points according to the coordinates of the marker points, and to perform ID tracking of the marker points to obtain the marker point ID tracking results, perform key point detection on the visible light image to obtain the coordinates and IDs of the key points, estimate the motion posture of the target under test in the visible light image to obtain the pose estimation results, reproject the marker points onto the visible light image, and compare them with the IDs of the key points to verify the correctness of the marker point ID tracking results or to perform ID identification on the marker points that have lost tracking, and fill in the image frames that have lost the marker point information with the pose estimation results, thereby realizing the motion posture data of the target under test based on the perception data of different modalities, and in the case of the loss of marker point information, the motion posture data of the target under test can also be obtained by filling in the pose estimation results obtained by performing motion posture estimation on the visible light image, thereby improving the accuracy and robustness of the motion capture system for motion capture of the target under test.

[0102] Figure 4 FIG. 4 is a schematic diagram of the structure of the motion capture system provided in some embodiments of the present application. Figure 4 As shown, in some embodiments, the motion capture system also includes: a second image processing unit 107, connected to the visible light image sensor 104, for performing one or more of the following processing on the visible light image: feature extraction, feature recognition, feature space positioning calculation, motion data resolution, and sending the processing results to the motion capture processing unit 102.

[0103] It can be understood that feature extraction refers to extracting features from visible light images that can be used for subsequent motion capture of the target under test. Features may include edges, corners, textures, etc. in the image; feature recognition refers to the process of further determining the specific identity or category of these features based on feature extraction; feature space positioning calculation refers to estimating the spatial position and posture of the target under test by identifying and using features in the image; motion data solution refers to the process of determining the motion posture data of the target under test.

[0104] The second image processing unit 107 processes the visible light image, and the motion capture processing unit 102 receives the processing result sent by the second image processing unit 107 .

[0105] Optionally, the second image processing unit 107 is an embedded motion capture processing unit, such as a field programmable gate array (FPGA), etc., which is embedded in the motion capture sensor group 101 and can also exist in the motion capture system in the form of a host computer.

[0106] In the above technical solution, the second image processing unit can be used to perform feature extraction, feature recognition, feature space positioning calculation or motion data solution on the visible image, and send the processing results to the motion capture processing unit. The second image processing unit can be an embedded motion capture processing unit, or it can exist in the motion capture system in the form of a host computer, thereby improving the flexibility of the motion capture system.

[0107] like Figure 4 As shown, in some embodiments, the second image processing unit 107 is specifically used for:

[0108] Performing key point detection on the visible light image to obtain key point information, wherein the key point information includes coordinates and IDs of the key points;

[0109] The motion posture of the measured target in the visible light image is estimated to obtain a posture estimation result.

[0110] Optionally, the second image processing unit 107 uses a deep learning algorithm, such as YOLOv8, to detect key points in the visible light image to obtain the coordinates and IDs of the key points, where the ID is a unique identifier of the key points. Optionally, the second image processing unit 107 uses an algorithm such as MediaPipe to estimate the motion posture of the target under test in the visible light image to obtain a pose estimation result, which is an estimation of the motion posture data, and the motion posture data of the target under test includes the position data and posture data of the target under test in three-dimensional space.

[0111] In the above technical scheme, the second image processing unit is used to perform key point detection on the visible light image, obtain the coordinates and IDs of the key points, estimate the motion posture of the target under test in the visible light image, and obtain the pose estimation result, which helps to obtain the motion posture data of the target under test based on the perception data of different modalities, thereby improving the accuracy and robustness of the motion capture system for motion capture of the target under test.

[0112] like Figure 4 As shown, when the motion capture system further includes a first image processing unit 106 and a second image processing unit 107, the motion capture processing unit 102 is used to perform at least one of the following:

[0113] Performing three-dimensional reconstruction of the marker points according to the marker point coordinates;

[0114] Perform ID tracking on the landmark points to obtain the landmark point ID tracking results;

[0115] Reprojecting the marker point onto the visible light image and comparing it with the ID of the key point to verify the correctness of the marker point ID tracking result or to perform ID identification on the marker point that has lost tracking;

[0116] For image frames that have lost landmark information, the pose estimation result is used to fill in the missing landmark information to obtain motion pose data of the measured target.

[0117] In some embodiments, the motion capture sensor group 101 includes at least one non-visible light image sensor 103, which can sense the object to be measured from at least one viewing angle to obtain images in the near-infrared or infrared bands, and the first image processing unit 106 tracks the marker points in these near-infrared or infrared band images through image processing technology, obtains the coordinates of the marker points through spatial positioning calculation and outputs them to the motion capture processing unit 102, that is, the motion capture processing unit 102 receives the coordinates of the marker points. The motion capture processing unit 102 uses triangulation to perform multi-viewpoint three-dimensional reconstruction of the object to be measured based on the coordinates of the marker points to obtain the three-dimensional coordinates of the marker points.

[0118] It is understandable that the motion capture processing unit 102 tracks each marker point to determine its position change in continuous image frames or time series, and assigns a unique ID to each marker point and tracks the ID of each marker point to achieve the ability to identify and associate the same marker point during the dynamic capture of the target being measured. Optionally, the motion capture processing unit 102 uses methods such as Kalman filtering to track the marker point ID to obtain the marker point ID tracking result, which includes the three-dimensional coordinates of each marker point and its corresponding image frame sequence or time series.

[0119] Furthermore, the motion capture processing unit 102 reprojects the marker points onto the visible light image according to the three-dimensional coordinates of the marker points, converts the three-dimensional spatial coordinates of the marker points into two-dimensional reprojected coordinates, and compares the projection of the marker points in the visible light image with the key points detected in the visible light image, including comparing their coordinate positions and IDs, so as to verify the correctness of marker ID tracking or perform ID identification on marker points that have lost tracking.

[0120] For image frames with missing marker information, the missing marker information in the image frame is filled by using the pose estimation result to obtain the motion posture data of the measured target. The image frame here refers to a non-visible light image. Optionally, the filling method can be direct filling or filtering, such as Kalman filtering.

[0121] In the above technical scheme, the motion capture processing unit is used to perform three-dimensional reconstruction of the marker points according to the coordinates of the marker points, perform ID tracking on the marker points, obtain the marker point ID tracking results, reproject the marker points onto the visible light image, and compare them with the IDs of the key points to verify the correctness of the marker point ID tracking results or perform ID identification on the marker points that have lost tracking. For image frames that have lost marker point information, the pose estimation results are used to fill in the gaps, thereby achieving motion posture data of the target under test based on perception data of different modalities. In the event that the marker point information is lost, the pose estimation results obtained by performing motion posture estimation on the visible light image can also be used to fill in the gaps, thereby obtaining the motion posture data of the target under test, thereby improving the accuracy and robustness of the motion capture system in capturing the motion of the target under test.

[0122] The motion capture method provided in the embodiment of the present application may be executed by a motion capture system or a functional module or functional entity in the motion capture system that can implement the motion capture method. The motion capture method provided in the embodiment of the present application is described below using the motion capture system as an example of the execution body.

[0123] Figure 5 is a flow chart of a motion capture method provided by some embodiments of the present application. Figure 5 As shown, the motion capture method includes: step 510 and step 520.

[0124] Step 510: sensing the target in a near infrared or infrared band, and outputting an image in the near infrared or infrared band, wherein the image in the near infrared or infrared band includes at least one marker point;

[0125] Step 520: Track and perform spatial positioning calculation on the marker points in the near-infrared or infrared band image to obtain the coordinates of the marker points;

[0126] Step 530: sense the target in the visible light band and output a visible light image.

[0127] Step 540: Determine the motion posture data of the measured target based on the marker point coordinates and the visible light image.

[0128] Optionally, determining the motion posture data of the measured target based on the marker point coordinates and the visible light image includes:

[0129] Performing three-dimensional reconstruction of the marker points according to the marker point coordinates;

[0130] Performing ID tracking on the marker point to obtain a marker point ID tracking result;

[0131] Performing key point detection on the visible light image to obtain key point information, wherein the key point information includes coordinates and IDs of the key points;

[0132] Performing motion posture estimation on the target to be measured in the visible light image to obtain a posture estimation result;

[0133] Reprojecting the marker point onto the visible light image and comparing it with the ID of the key point to verify the correctness of the marker point ID tracking result or to perform ID identification on the marker point that has lost tracking;

[0134] For image frames that have lost landmark information, the pose estimation result is used to fill in the missing landmark information to obtain motion pose data of the measured target.

[0135] According to the coordinates of the marker points, the marker points are three-dimensionally reconstructed, the marker points are ID tracked to obtain the marker point ID tracking results, the key point detection is performed on the visible light image to obtain the coordinates and IDs of the key points, the motion posture of the target under test in the visible light image is estimated to obtain the pose estimation result, the marker points are reprojected onto the visible light image, and compared with the IDs of the key points to verify the correctness of the marker point ID tracking results or to perform ID identification on the marker points that have lost tracking. For image frames with lost marker point information, the pose estimation results are used to fill in the gaps, thereby realizing the motion posture data of the target under test based on the perception data of different modalities. In the case of lost marker point information, the pose estimation results can also be used to fill in the gaps, thereby obtaining the motion posture data of the target under test, thereby improving the accuracy and robustness of the motion capture system in capturing the motion of the target under test.

[0136] In the above technical scheme, the target to be measured is sensed in the near-infrared or infrared band, and an image of the near-infrared or infrared band is output, wherein the image of the near-infrared or infrared band includes at least one marker point, the marker point in the image of the near-infrared or infrared band is tracked and spatially positioned to obtain the coordinates of the marker point, the target to be measured is sensed in the visible light band, and a visible light image is output, and based on the marker point coordinates and the visible light image, the motion posture data of the target to be measured is obtained, thereby realizing the fusion of perception data of different modalities and improving the accuracy and robustness of the motion capture method.

[0137] For understanding of the motion capture method provided in the embodiment of the present application, one can refer to the aforementioned description of the motion capture system, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0138] In some embodiments, Figure 6As shown, an embodiment of the present application also provides an electronic device 600, including a processor 601, a memory 602, and a computer program stored in the memory 602 and executable on the processor 601. When the program is executed by the processor 601, each process of the above-mentioned motion capture method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0139] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0140] An embodiment of the present application also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned motion capture method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0141] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0142] An embodiment of the present application also provides a computer program product, including a computer program, which implements the above-mentioned motion capture method when executed by a processor.

[0143] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0144] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned motion capture method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0145] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0146] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0147] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the relevant technology, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0148] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

[0149] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0150] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

Claims

1. A motion capture system, characterized in that: include: Motion capture sensor group and motion capture processing unit, The motion capture sensor group includes at least one non-visible light image sensor and at least one visible light image sensor, wherein the non-visible light image sensor is used to sense the target to be measured in the near infrared or infrared band and output the image of the near infrared or infrared band, and the visible light image sensor is used to sense the target to be measured in the visible light band and output the visible light image; The motion capture processing unit is connected to the motion capture sensor group and is used to determine the motion posture data of the measured target based on the visible light image and the image in the near infrared or infrared band.

2. The motion capture system according to claim 1, characterized in that: The motion capture system further comprises at least one fill light source in the near infrared or infrared band for illuminating the target to be measured; At least one marking point is arranged on the measured target, and under the illumination of the supplementary light source, the marking point has a contrast with the background in the imaging of the non-visible light image sensor.

3. The motion capture system according to claim 2, characterized in that: The motion capture system also includes: a first image processing unit, connected to the non-visible light image sensor, for tracking and spatially locating the marker points in the image of the near-infrared or infrared band, and outputting the coordinates of the marker points to the motion capture processing unit.

4. The motion capture system according to claim 3, characterized in that: The motion capture processing unit is configured to perform at least one of the following: Performing three-dimensional reconstruction of the marker points according to the marker point coordinates; Performing ID tracking on the marker point to obtain a marker point ID tracking result; Performing key point detection on the visible light image to obtain key point information, wherein the key point information includes coordinates and IDs of the key points; Performing motion posture estimation on the target to be measured in the visible light image to obtain a posture estimation result; Reprojecting the marker point onto the visible light image and comparing it with the ID of the key point to verify the correctness of the marker point ID tracking result or to perform ID identification on the marker point that has lost tracking; For image frames that have lost landmark information, the pose estimation result is used to fill in the missing landmark information to obtain motion pose data of the measured target.

5. The motion capture system according to claim 3, characterized in that: The motion capture system also includes: a second image processing unit, connected to the visible light image sensor, for performing one or more of the following processing on the visible light image: feature extraction, feature recognition, feature space positioning calculation, motion data resolution, and sending the processing results to the motion capture processing unit.

6. The motion capture system according to claim 5, characterized in that: The second image processing unit is specifically used for: Performing key point detection on the visible light image to obtain key point information, wherein the key point information includes coordinates and IDs of the key points; The motion posture of the measured target in the visible light image is estimated to obtain a posture estimation result.

7. The motion capture system according to claim 6, characterized in that: The motion capture processing unit is configured to perform at least one of the following: Performing three-dimensional reconstruction of the marker points according to the marker point coordinates; Perform ID tracking on the landmark points to obtain the landmark point ID tracking results; Reprojecting the marker point onto the visible light image and comparing it with the ID of the key point to verify the correctness of the marker point ID tracking result or to perform ID identification on the marker point that has lost tracking; For image frames that have lost landmark information, the pose estimation result is used to fill in the missing landmark information to obtain motion pose data of the measured target.

8. A motion capture method, based on the motion capture system according to any one of claims 1 to 7, characterized in that: include: Sensing the measured target in a near-infrared or infrared band, and outputting an image of the near-infrared or infrared band, wherein the image of the near-infrared or infrared band includes at least one marking point; Tracking and spatially locating the marker points in the image of the near-infrared or infrared band to obtain the coordinates of the marker points; Sensing the measured target in the visible light band and outputting a visible light image; Based on the marker point coordinates and the visible light image, the motion posture data of the measured object is determined.

9. The motion capture method according to claim 8, characterized in that: The step of determining the motion posture data of the measured target based on the marker point coordinates and the visible light image includes: Performing three-dimensional reconstruction of the marker points according to the marker point coordinates; Performing ID tracking on the marker point to obtain a marker point ID tracking result; Performing key point detection on the visible light image to obtain key point information, wherein the key point information includes coordinates and IDs of the key points; Performing motion posture estimation on the target to be measured in the visible light image to obtain a posture estimation result; Reprojecting the marker point onto the visible light image and comparing it with the ID of the key point to verify the correctness of the marker point ID tracking result or to perform ID identification on the marker point that has lost tracking; For image frames that have lost landmark information, the pose estimation result is used to fill in the missing landmark information to obtain motion pose data of the measured target.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the motion capture method as described in any one of claims 8 to 9 is implemented.

Images