A markerless human three-dimensional posture capturing method, system and device based on fusion of two-dimensional images and three-dimensional spatial information
By fusing two-dimensional images with three-dimensional spatial information, the problem of unstable three-dimensional posture in markerless human posture capture is solved, achieving stable and continuous three-dimensional posture output, which is applicable to various sensor types.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Filing Date
- 2026-05-07
- Publication Date
- 2026-07-14
AI Technical Summary
In existing technologies for unlabeled human pose capture, single two-dimensional images lack depth information, and single three-dimensional spatial information lacks stable human semantic keypoints, resulting in unstable three-dimensional poses, especially when there is occlusion, lack of depth, and motion, leading to joint mismatch and jumps.
By fusing two-dimensional images and three-dimensional spatial information, and using two-dimensional image acquisition devices and three-dimensional spatial information acquisition devices for calibration, the mapping relationship between two-dimensional images and three-dimensional reference coordinate systems is obtained. Combined with local search, neighborhood averaging and timestamp synchronization, stable three-dimensional joint coordinates are output.
It improves the stability and continuity of 3D pose without wearable markers, reduces the impact of depth holes and occlusion on joint localization, and is applicable to various sensor types.
Smart Images
Figure CN122391285A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of human posture capture, computer vision, 3D reconstruction and multi-sensor fusion technology, and in particular to a label-free human 3D posture capture method, system and device based on the fusion of 2D images and 3D spatial information. Background Technology
[0002] Human 3D pose capture can be used in scenarios such as sports training, rehabilitation assessment, fitness analysis, motion capture, and human-computer interaction. Traditional optical motion capture solutions usually require the person being captured to wear reflective markers, inertial sensors, or special capture suits, which results in high system deployment costs, long preparation time, and may affect the natural movement state of the person being captured.
[0003] Among existing label-free human pose estimation schemes, those based on single 2D images can identify human semantic keypoints relatively well. However, 2D images lack true depth information, making it difficult to directly obtain stable and measurable 3D keypoint coordinates. Schemes based on depth cameras, point clouds, or other 3D spatial information can provide spatial location, but they are prone to keypoint mismatches, jumps, or short-term breakpoints under conditions such as human semantic keypoint recognition, rapid movement, occlusion, depth holes, and edge noise.
[0004] Therefore, there is a need for a markerless 3D pose capture scheme that can combine the ability to identify key human body points in 2D images with the ability to capture real spatial positions in 3D spatial information, so as to improve the stability, continuity and applicability of human body 3D pose output without wearing sensors or markers. Summary of the Invention
[0005] The purpose of this invention is to provide a label-free human 3D pose capture method, system, and device based on the fusion of 2D images and 3D spatial information, so as to solve the problems of lack of depth information in a single 2D image, lack of stable human semantic key points in a single 3D spatial information, and instability of 3D pose caused by depth loss, occlusion, time asynchrony, and joint jump.
[0006] To achieve the above objectives, the present invention provides a markerless human three-dimensional pose capture method, comprising the following steps:
[0007] The system acquires two-dimensional image data and three-dimensional spatial information data of the target human body, wherein the two-dimensional image data is acquired by a two-dimensional image acquisition device and the three-dimensional spatial information data is acquired by a three-dimensional spatial information acquisition device.
[0008] The two-dimensional image acquisition device and the three-dimensional spatial information acquisition device are calibrated to obtain the mapping relationship between the two-dimensional image coordinate system and the three-dimensional reference coordinate system.
[0009] The two-dimensional image data is input into the two-dimensional human posture recognition model to obtain the coordinates of multiple two-dimensional human key points of the target human body and their corresponding confidence scores.
[0010] Based on the mapping relationship, the two-dimensional human body key points are matched with the depth values, point cloud points or three-dimensional spatial coordinates in the three-dimensional spatial information data to obtain the corresponding three-dimensional joint coordinates;
[0011] When 3D spatial information near a joint is missing or abnormal, perform local search, neighborhood averaging, and / or backtracking to the previous stable 3D joint.
[0012] Based on the timestamp, the nearest frame matching is performed between the two-dimensional image frame and the three-dimensional spatial information frame, and the three-dimensional joint points are smoothed, abnormal jumps are processed, and skeleton constraints are applied.
[0013] Output the three-dimensional human skeleton, the three-dimensional coordinates of each joint, and / or the joint angles.
[0014] The three-dimensional spatial information acquisition device can be a depth camera, structured light camera, time-of-flight camera, binocular camera, multi-view camera, lidar, or other spatial sensors capable of outputting depth maps, point clouds, or three-dimensional spatial coordinates. The two-dimensional image acquisition device can be an RGB camera, color camera, industrial camera, USB camera, visible light image acquisition device, or two-dimensional camera array.
[0015] In one embodiment, a two-dimensional image acquisition device and a three-dimensional spatial information acquisition device are combined and calibrated into a fusion camera unit, with a fusion camera unit set on each of the left and right sides of the target human body. The installation distance of each fusion camera unit can be adjusted according to the scene to ensure that the target human body is fully presented within the acquisition field of view; the camera installation height can be set near the waist of the human body; the target human body moves within a fixed acquisition area.
[0016] In one embodiment, the two-dimensional image acquisition device is a RER-USBGS1200P01 RGB camera, and the three-dimensional spatial information acquisition device is an Intel D430 module. The resolution during acquisition and calibration is 640×480, and the frame rate is 90Hz. The above-mentioned models, resolutions, and frame rates are merely specific embodiments and do not constitute a limitation on the scope of protection of this invention.
[0017] In one embodiment, the system adopts a soft synchronization method: when a two-dimensional image acquisition device on one side obtains a new frame, the system selects the frame whose timestamp is closest to the new frame from the two-dimensional image acquisition device on the other side and each three-dimensional spatial information acquisition device to form a synchronization frame group for fusion calculation.
[0018] This invention also provides a markerless human 3D pose capture system, comprising a 2D image acquisition device, a 3D spatial information acquisition device, a synchronization unit, a calibration unit, a 2D pose recognition unit, a 3D fusion unit, a temporal stabilization unit, and an output unit. The 2D pose recognition unit outputs the coordinates and confidence levels of 2D human key points; the 3D fusion unit matches the 2D human key points with the 3D spatial information according to the calibration mapping relationship to obtain the coordinates of 3D joint points; the temporal stabilization unit handles low confidence, depth loss, occlusion, and depth jumps; and the output unit outputs the 3D human skeleton, the 3D coordinates of joint points, and / or joint angles.
[0019] The present invention also provides a device comprising at least one set of fusion camera units, a processor and a memory, wherein the memory stores program instructions that, when executed by the processor, implement the above-described markerless human three-dimensional pose capture method.
[0020] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-described markerless human three-dimensional pose capture method.
[0021] Compared with existing technologies, the present invention has at least the following beneficial effects: by fusing the key point recognition results of two-dimensional images with three-dimensional spatial information, measurable three-dimensional human posture can be output without wearable markers; through local depth search, neighborhood averaging, and missing depth backoff, the impact of depth holes, noise, and short-term occlusion on key point localization can be reduced; through timestamp nearest frame matching, smoothing processing, previous stable frame preservation, and skeleton constraints, the continuity and stability of three-dimensional posture output can be improved; by using a combination of two-dimensional image acquisition devices and three-dimensional spatial information acquisition devices, it can be adapted to various sensors such as RGB cameras, depth cameras, structured light cameras, TOF cameras, binocular cameras, multi-view cameras, or LiDAR. Attached Figure Description
[0022] Figure 1 This is a schematic diagram of the overall structure of a markerless human three-dimensional pose capture system provided in an embodiment of the present invention;
[0023] Figure 2 This is a schematic diagram showing the arrangement of the left and right fusion cameras relative to the target human body according to an embodiment of the present invention;
[0024] Figure 3 This is a schematic diagram of the calibration process for a two-dimensional image acquisition device and a three-dimensional spatial information acquisition device provided in an embodiment of the present invention;
[0025] Figure 4 This is a schematic diagram of the mapping and fusion process from two-dimensional key points to three-dimensional joints provided in an embodiment of the present invention;
[0026] Figure 5 A schematic diagram of the depth missing, occlusion, and temporal stabilization processing flow provided in an embodiment of the present invention;
[0027] Figure 6 This is a schematic diagram of a three-dimensional human skeleton and its output results provided in an embodiment of the present invention. Detailed Implementation
[0028] The present invention will be further described below with reference to the accompanying drawings and embodiments. It should be understood that the following embodiments are for illustrative purposes only and are not intended to limit the scope of protection of the present invention. Without departing from the concept of the present invention, those skilled in the art can replace or adjust the specific sensor model, calibration object, attitude recognition model, output interface, and application scenario.
[0029] I. System Structure
[0030] like Figure 1 As shown, the markerless human three-dimensional posture capture system 100 includes a two-dimensional image acquisition device 110, a three-dimensional spatial information acquisition device 120, a fusion camera unit 130, a synchronization unit 140, a calibration unit 150, a two-dimensional posture recognition unit 160, a three-dimensional fusion unit 170, a timing stabilization unit 180, and an output unit 190.
[0031] The two-dimensional image acquisition device 110 is used to acquire two-dimensional image frames of the target human body 200. The three-dimensional spatial information acquisition device 120 is used to acquire the depth map, point cloud, or three-dimensional spatial coordinates of the space where the target human body 200 is located. The two-dimensional image acquisition device 110 and the three-dimensional spatial information acquisition device 120 are combined and calibrated to form a fusion camera unit 130.
[0032] Synchronization unit 140 performs soft synchronization based on the timestamps of the acquired frames, selecting other 2D image frames and 3D spatial information frames whose timestamps are closest to the reference 2D image frame. Calibration unit 150 establishes the mapping relationship between the 2D image coordinate system and the 3D reference coordinate system. 2D pose recognition unit 160 identifies key human body points in the 2D image. 3D fusion unit 170 matches 2D human body key points with 3D spatial information. Timing stabilization unit 180 handles missing, abnormal, and jittering joints. Output unit 190 outputs the 3D human skeleton 300, 3D joint coordinates, and joint angles.
[0033] II. Camera Setup
[0034] like Figure 2As shown, in one specific embodiment, the target human body 200 is located within a fixed acquisition area 210, with a set of fusion camera units 130 respectively arranged on the left and right sides. Each set of fusion camera units 130 includes a two-dimensional image acquisition device 110 and a three-dimensional spatial information acquisition device 120. The installation distance of the fusion camera units 130 can be adjusted according to the scene to ensure that the target human body 200 is fully presented within the acquisition field of view; the installation height of the fusion camera units 130 can be located near the waist of the human body.
[0035] In one specific embodiment, the two-dimensional image acquisition device 110 uses a RER-USBGS1200P01 RGB camera, and the three-dimensional spatial information acquisition device 120 uses an Intel D430 module. The two are combined and calibrated into a fusion camera unit 130, with an acquisition and calibration resolution of 640×480 and a frame rate of 90Hz.
[0036] In other embodiments, the two-dimensional image acquisition device 110 can be replaced by a color camera, industrial camera, USB camera, monocular camera or two-dimensional camera array; the three-dimensional spatial information acquisition device 120 can be replaced by a depth camera, structured light camera, TOF camera, binocular camera, multi-view camera, lidar or other device that outputs three-dimensional spatial information.
[0037] III. Calibration Method
[0038] like Figure 3 As shown, in one embodiment, a checkerboard pattern is used as the calibration object 220 to calibrate each set of two-dimensional image acquisition devices 110 and three-dimensional spatial information acquisition devices 120. Specifically, multiple sets of two-dimensional images and three-dimensional spatial information images containing the calibration object 220 are acquired, corner points on the calibration object 220 are detected, and the intrinsic parameter matrices and distortion parameters of the two-dimensional image acquisition device 110 and the three-dimensional spatial information acquisition device 120 are solved respectively. Furthermore, the rotation matrix and translation vector between the two are solved.
[0039] In one specific embodiment, the checkerboard grid consists of 9×6 interior corner points, with a side length of 0.089m. Corner coordinates are obtained through corner detection and sub-pixel optimization. Then, based on the camera calibration algorithm, the intrinsic parameters K_rgb and distortion parameter dist_rgb of the 2D image acquisition device 110, the intrinsic parameters K_3d and distortion parameter dist_3d of the 3D spatial information acquisition device 120, and the rotation matrix R and translation vector T from the 3D reference coordinate system to the 2D image coordinate system are calculated.
[0040] In other embodiments, the calibration object 220 is not limited to a checkerboard pattern, but can also be an AprilTag calibration board, a dot array calibration board, a preset calibration action, or other objects or methods that can be used to establish a mapping relationship between a two-dimensional image coordinate system and a three-dimensional reference coordinate system.
[0041] IV. Two-dimensional pose recognition
[0042] The two-dimensional pose recognition unit 160 inputs a two-dimensional image frame into the two-dimensional human pose recognition model and outputs multiple two-dimensional human key points of the target human body 200. Each two-dimensional human key point can be represented as p_i=(u_i, v_i, s_i), where u_i and v_i are two-dimensional image coordinates, and s_i is the recognition confidence of the corresponding key point.
[0043] In one specific embodiment, the two-dimensional human pose recognition model adopts an RTMPose-type model, which can output full-body key points in the COCO human key point format. For kayaking paddling training scenarios, key points of the upper limbs and torso, such as shoulders, elbows, wrists, and hips, can be used as the main solution of the present invention. However, the main solution of the present invention is not limited to upper limb key points, and can be used for three-dimensional pose capture of full-body key points such as the head, shoulders, elbows, wrists, hips, knees, and ankles.
[0044] V. Mapping and Fusion of 2D Keypoints to 3D Joints
[0045] like Figure 4 As shown, the 3D fusion unit 170 projects the 3D points in the 3D spatial information data onto the 2D image plane according to the mapping relationship obtained by the calibration unit 150, and matches them with the 2D human body key points. For a point X=(X,Y,Z) in the 3D reference coordinate system, it can be projected onto the 2D image plane according to the rotation matrix R, the translation vector T and the intrinsic parameter K_rgb of the 2D image acquisition device to obtain the projected pixel coordinates.
[0046] For 3D spatial information in the form of depth maps, depth pixels can also be back-projected into 3D spatial points based on the intrinsic parameters of the 3D spatial information acquisition device. For example, for depth pixels (u_d, v_d) and depth value Z_d, the points in the 3D reference coordinate system can be obtained by X_d=(u_d-c_x) / f_x·Z_d, Y_d=(v_d-c_y) / f_y·Z_d, and Z_d=Z_d, where f_x, f_y, c_x, and c_y are the intrinsic parameters of the 3D spatial information acquisition device.
[0047] Once the two-dimensional human body key point p_i is determined, the three-dimensional fusion unit 170 searches for three-dimensional candidate points whose projection distance meets the threshold condition within a preset search window near the two-dimensional human body key point, and determines the coordinates of the target three-dimensional joint point based on the depth, projection distance, neighborhood validity and key point confidence of the candidate point.
[0048] In one embodiment, when there are multiple effective depth values or multiple 3D candidate points near the joint, the 3D fusion unit 170 performs neighborhood averaging on the candidate points and can preferentially select foreground candidate points that are closer to the camera to reduce mismatches caused by background points, depth holes and edge noise.
[0049] VI. Handling of depth missing, occlusion and temporal stability
[0050] like Figure 5 As shown, when there are no effective depth values near the two-dimensional human body key points, the three-dimensional candidate points are insufficient, the recognition confidence is low, or the depth value changes abnormally, the timing stabilization unit 180 performs abnormal processing.
[0051] In one embodiment, if there is no effective depth value near the joint point, an effective depth value is searched within a preset window around the two-dimensional human key point; if multiple effective depth values exist, the multiple effective depth values in the vicinity are averaged; if an effective three-dimensional joint point still cannot be obtained, the spatial position or depth information of the previous stable three-dimensional joint point is used for backtracking.
[0052] In one embodiment, when the confidence level of a key point is low or the depth value of a 3D joint changes abnormally, the timing stabilization unit 180 retains the result of the previous stable frame to avoid directly outputting the abnormal point. The system can also perform smoothing processing on the 3D joint sequence and correct unreasonable skeletal deformations by combining constraints on human skeleton length, shoulder width, or hip width.
[0053] In one embodiment, the synchronization unit 140 maintains a three-dimensional spatial information frame buffer and selects the three-dimensional spatial information frame with the closest time according to the timestamp of the two-dimensional image frame. When the time difference of the most recent frame exceeds a preset threshold, the use of that frame for fusion can be stopped, or the previous stable three-dimensional joint point can be used for maintenance.
[0054] VII. Output Method
[0055] like Figure 6 As shown, output unit 190 outputs a three-dimensional human skeleton 300, the three-dimensional coordinates of each human joint, and / or joint angles. The three-dimensional human skeleton 300 can be composed of multiple three-dimensional joints and their skeletal connections.
[0056] In one specific embodiment, the output unit 190 outputs the three-dimensional human skeleton, the three-dimensional coordinates of the joints, and the joint angles in real time or near real time via WebSocket. In other embodiments, the output unit 190 may also output via a local display interface, file, database, network interface, SDK interface, or other data interface.
[0057] VIII. Application Examples
[0058] This invention can be applied to scenarios such as sports training, rehabilitation assessment, fitness analysis, motion capture, and human-computer interaction. In a kayaking paddling training scenario, the target human body is located within a fixed acquisition area. Two sets of fusion camera units on the left and right sides acquire two-dimensional images and three-dimensional spatial information of the athlete's upper limbs and torso. The system outputs the three-dimensional coordinates and joint angles of joints such as the shoulder, elbow, wrist, and hip to support paddling motion analysis.
[0059] The above-described kayak paddling training is merely one application example of the present invention. The present invention is also applicable to full-body human posture capture, and can select all or part of the key human body points for 3D reconstruction, joint angle calculation, and motion analysis according to application needs.
[0060] IX. Multi-person Extended Implementation Example
[0061] In another embodiment, when there are multiple human targets in the captured image, the system can first detect or track each human target to obtain a set of two-dimensional key points for each human target, and then perform matching of two-dimensional key points and three-dimensional spatial information, depth missing processing, temporal stabilization processing and three-dimensional pose output on each human target to obtain the three-dimensional human skeleton of multiple human targets.
Claims
1. A label-free human 3D pose capture method based on the fusion of 2D images and 3D spatial information, characterized in that, include: Two-dimensional image data and three-dimensional spatial information data of the target human body are acquired, wherein the two-dimensional image data is acquired by a two-dimensional image acquisition device and the three-dimensional spatial information data is acquired by a three-dimensional spatial information acquisition device; The two-dimensional image acquisition device and the three-dimensional spatial information acquisition device are calibrated to obtain the mapping relationship between the two-dimensional image coordinate system and the three-dimensional reference coordinate system. The two-dimensional image data is input into a two-dimensional human pose recognition model to obtain the coordinates of multiple two-dimensional human key points of the target human body and their corresponding confidence scores; according to the mapping relationship, the two-dimensional human key points are matched with the depth values, point cloud points or three-dimensional spatial coordinates in the three-dimensional spatial information data to obtain the coordinates of three-dimensional joint points corresponding to the two-dimensional human key points; depth loss processing and temporal stabilization processing are performed on the three-dimensional joint point coordinates. Output the three-dimensional human skeleton, the three-dimensional coordinates of each joint, and / or the joint angles.
2. The method according to claim 1, characterized in that, The two-dimensional image acquisition device includes at least one of an RGB camera, a color camera, an industrial camera, a USB camera, a visible light image acquisition device, a monocular camera, or a two-dimensional camera array; the three-dimensional spatial information acquisition device includes at least one of a depth camera, a structured light camera, a TOF camera, a time-of-flight camera, a binocular camera, a multi-view camera, a lidar, or a spatial sensor capable of outputting depth maps, point clouds, or three-dimensional spatial coordinates.
3. The method according to claim 1, characterized in that, The two-dimensional image acquisition device and the three-dimensional spatial information acquisition device are combined and calibrated as a fusion camera unit. At least two sets of the fusion camera units are respectively set on the left and right sides of the target human body, so that the target human body is fully presented within the acquisition field of view of the two-dimensional image acquisition device and the three-dimensional spatial information acquisition device. In one embodiment, a set of the fusion camera units includes a RER-USBGS1200P01 RGB camera and an Intel D430 module, with a capture resolution and calibration resolution of 640×480 and a capture frame rate of 90Hz.
4. The method according to claim 1, characterized in that, Acquiring two-dimensional image data and three-dimensional spatial information data of the target human body includes soft synchronization processing: when a two-dimensional image acquisition device on one side obtains a new two-dimensional image frame, the frame with the closest time to the new two-dimensional image frame from other two-dimensional image acquisition devices and / or three-dimensional spatial information acquisition devices is selected according to the timestamp to form a synchronization frame group for fusion calculation.
5. The method according to claim 1, characterized in that, The calibration of the two-dimensional image acquisition device and the three-dimensional spatial information acquisition device includes: acquiring a two-dimensional image and a three-dimensional spatial information image containing the calibration object; detecting feature points on the calibration object; solving for the intrinsic parameters and distortion parameters of the two-dimensional image acquisition device and the three-dimensional spatial information acquisition device based on the feature points; and solving for the rotation matrix and translation vector between the two-dimensional image acquisition device and the three-dimensional spatial information acquisition device. The calibration object is a checkerboard, an AprilTag calibration board, a dot array calibration board, or a calibration object formed by a preset calibration action.
6. The method according to claim 1, characterized in that, Each two-dimensional human key point output by the two-dimensional human pose recognition model includes two-dimensional image coordinates and a corresponding confidence score. The two-dimensional human pose recognition model includes a two-dimensional pose recognition model that can output the coordinates and confidence scores of human key points. In one embodiment, the two-dimensional human pose recognition model is an RTMPose-type model.
7. The method according to claim 1, characterized in that, Matching the two-dimensional human key points with the three-dimensional spatial information data according to the mapping relationship includes: projecting the three-dimensional points in the three-dimensional spatial information data onto the two-dimensional image plane; searching for three-dimensional candidate points whose projection distance meets a threshold condition within a preset search window near the two-dimensional human key points; determining the coordinates of the three-dimensional key points based on the depth, projection distance, neighborhood validity, and / or the confidence level of the two-dimensional human key points; when the three-dimensional spatial information data is a depth map, back-projecting depth pixels into three-dimensional spatial points according to the intrinsic parameters of the three-dimensional spatial information acquisition device, and projecting the three-dimensional spatial points onto the two-dimensional image coordinate system to match the two-dimensional human key points.
8. The method according to claim 1, characterized in that, The depth loss processing and temporal stabilization processing for the three-dimensional joint coordinates includes: when there are no effective depth values or effective three-dimensional candidate points near the two-dimensional human key points, searching for effective depth values or effective three-dimensional candidate points within a preset window around the two-dimensional human key points; when there are multiple effective depth values or multiple effective three-dimensional candidate points, performing neighborhood averaging on the multiple effective depth values or multiple effective three-dimensional candidate points, and preferentially selecting foreground candidate points that are closer to the acquisition device; when effective three-dimensional joint coordinates still cannot be obtained, the confidence level of the two-dimensional human key points is lower than a preset threshold, or the depth value of the three-dimensional joint points undergoes abnormal jumps, using the three-dimensional joint coordinates or depth information of the previous stable frame for backtracking processing; smoothing the three-dimensional joint sequence, and correcting unreasonable skeleton deformations according to constraints such as human skeleton length, shoulder width, or hip width.
9. A markerless human 3D pose capture system based on the fusion of 2D images and 3D spatial information, characterized in that, include: Two-dimensional image acquisition device, used to acquire two-dimensional image data of a target human body; A three-dimensional spatial information acquisition device is used to acquire three-dimensional spatial information data of the space where the target human body is located; The calibration unit is used to calibrate the two-dimensional image acquisition device and the three-dimensional spatial information acquisition device to obtain the mapping relationship between the two-dimensional image coordinate system and the three-dimensional reference coordinate system. A two-dimensional pose recognition unit is used to obtain the coordinates of multiple two-dimensional human body key points and their corresponding confidence levels based on the two-dimensional image data. The three-dimensional fusion unit is used to match the two-dimensional human key points with the depth values, point cloud points or three-dimensional spatial coordinates in the three-dimensional spatial information data according to the mapping relationship, so as to obtain the corresponding three-dimensional joint coordinates; the temporal stabilization unit is used to perform depth missing processing and temporal stabilization processing on the three-dimensional joint coordinates. The output unit is used to output the three-dimensional human skeleton, the three-dimensional coordinates of each human joint point and / or joint angles through a real-time display interface, file, database, network interface, SDK interface or WebSocket interface.
10. A markerless human three-dimensional posture capture device, characterized in that, The method includes at least one set of fusion camera units, a processor, and a memory. The fusion camera units include a two-dimensional image acquisition device and a three-dimensional spatial information acquisition device. The memory stores program instructions, which, when executed by the processor, implement the method according to any one of claims 1 to 8.