A method for real-time reconstruction of a dynamic human body

Abstract

The application discloses a real-time reconstruction method of a dynamic human body, which comprises the following steps: connecting a human body measuring device; solving camera internal parameters and relative poses between two cameras; collecting a reference state image; dividing a speckle area; three-dimensionally reconstructing the reference state; collecting images on a time sequence; three-dimensionally reconstructing human body speckles on the time sequence; and quickly triangulating a dynamic three-dimension point cloud. On the premise of ensuring reconstruction accuracy, the point cloud density of the reconstructed human body data can be greatly improved, the reconstructed point cloud data can more fully express local details of the human body, real-time calculation of the dynamic human body can be completed, and online measurement and reconstruction of the human body can be realized.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, image processing technology and three-dimensional measurement, and in particular to a method for real-time reconstruction of a dynamic human body. Background Technology

[0002] With the continuous advancement of technology, 3D measurement techniques are constantly evolving. Existing methods include stereo vision, close-range photogrammetry, structured light projection, and digital image correlation. Stereo vision utilizes two or more digital images to obtain depth information by matching pixel information of feature points on the object's surface, and then calculates a 3D point cloud. This method relies on the structure and texture features of the object's surface, resulting in a relatively sparse reconstructed point cloud, a large computational load, and often low point cloud quality.

[0003] Close-range photogrammetry uses one or more cameras to capture multiple images of an object from different angles. Digital image processing techniques are used to calculate the image coordinates of corresponding points in the images from different perspectives, and optimization algorithms are used to solve for the 3D coordinates of these points. This method differs from stereo vision in two ways: first, stereo vision requires pre-calibration of the camera, while photogrammetry does not; second, it directly solves for 3D data information using optimization algorithms after obtaining the 2D images. This method has a simple hardware structure, flexible shooting methods, and is widely used in large-scene reconstruction and measurement. Structured light projection technology, based on stereo vision, projects a light spot pattern with a specific structure (such as random speckle or dotted spots) onto the object surface through a speckle projector. A camera captures the speckle pattern on the object surface, and the 3D information of the object surface is then calculated from it. Compared to stereo vision and photogrammetry, this method has higher efficiency in acquiring 3D data and higher measurement accuracy, and has been widely used in industrial measurement, human body scanning, and other fields.

[0004] Structured light projection technology, combined with digital image correlation, is used to reconstruct dynamic human bodies in three dimensions. To reconstruct high-density point clouds, a large amount of computational resources are required to calculate the correlation of image sub-regions, which is computationally difficult. Summary of the Invention

[0005] The purpose of this invention is to solve the problem that completing the three-dimensional reconstruction of a dynamic human body requires a large amount of computing resources and is difficult to perform, and to propose a real-time reconstruction method for a dynamic human body.

[0006] The technical problem of this invention is solved by the following technical solution:

[0007] This invention discloses a real-time reconstruction method for a dynamic human body, comprising the following steps:

[0008] S1. Connect to the human body measurement device, which includes a first camera and a second camera;

[0009] S2. Calibrate the first and second cameras, and solve for the camera intrinsic parameters and the relative attitude (R,T) between the two cameras;

[0010] Where R and T are the rotation matrix and translation vector from the camera coordinate system of the second camera to the camera coordinate system of the first camera;

[0011] S3. Reference State Image Acquisition: The first image acquired by the camera and used to divide the calculation area is considered as the reference state image; the speckle projector is turned on to project speckles onto the human body surface, while simultaneously acquiring human body speckle images;

[0012] S4. Speckle region division: Based on the speckle distribution area and human body distribution area on the left image, the region to be calculated is divided to determine the range of left and right image matching and human body reconstruction.

[0013] S5. 3D Reconstruction of Reference State: First, select seed points on the left image, then search for matching points of the seed points within the range of the strips set above and below the epipolar line, then complete the matching of all data points in the sub-regions divided in S4, and finally use (R,T) calibrated in S2 to reconstruct the 3D point cloud of the human body in the reference state.

[0014] S6. Image acquisition over time: Simultaneously acquire human speckle image streams via hardware triggering;

[0015] S7. 3D Reconstruction of Human Speckle Data over Time Series: Based on the image stream acquired in step S6, and considering the continuity of human dynamic motion, the deformation parameters of the speckle data points from the previous moment relative to the reference state are directly used as the initial values ​​of the corresponding data points at the current moment. The fine matching process for each data point is completed on the GPU, and the matching results are verified on the CPU to repair points that failed to be finely matched. Finally, the 3D point cloud of the human body in the current state is reconstructed according to the reconstruction method in step S5.

[0016] S8. Real-time output of the reconstructed 3D point cloud of the human body over time.

[0017] In some embodiments, in step S1, the human body measuring device further includes a workstation, a speckle projector, a camera, and a speckle projector control box; the LED light source in the speckle projector is controlled to work by a command pulse, the LED light source generates short-wavelength blue light, and the speckle projector causes the light source to be projected onto the human body surface through the speckle sheet according to the optical path design requirements to produce a speckle pattern.

[0018] In some embodiments, the calibration steps for the first camera and the second camera in step S2 are as follows:

[0019] S21. For the calibration plates placed in different positions within the camera's depth of field, the first camera and the second camera simultaneously and sequentially acquire images of the calibration plates at these multiple positions, generating multiple images in total.

[0020] S22. Perform image processing on the acquired images, calculate the pixel coordinates of coded and non-coded points on the calibration board, and use the coordinates on each image and the three-dimensional coordinates on the calibration board to solve the camera's intrinsic parameters and the relative attitude between cameras according to the bundle adjustment algorithm.

[0021] In some embodiments, the speckle region segmentation step S4 is detailed as follows:

[0022] Using three basic shape tools—rectangle, ellipse, and polygon—as well as Boolean operations for intersection and union between these three shapes, complex speckle regions to be calculated are selected on the left camera image in the reference state, serving as the calculation range for subsequent image matching.

[0023] In some embodiments, step S5, the step of three-dimensional reconstruction of the reference state, is as follows:

[0024] S51. Seed point selection: Based on the interval step size parameter of the seed point distribution, a seed point array is automatically selected on the image acquired by the first camera. The seed point array is represented as (x Lij ,y Lij ), where i = 0, 1…n; j = 0, 1…n;

[0025] S52. Coarse matching of seed points: Based on the relative poses (R,T) of the first and second cameras as determined in step S2, the seed points (x,t) are solved using epipolar constraints in two-view geometry. Lij ,y Lij The estimated position (x) on the image acquired by the second camera (right image) Rij ,y Rij This position corresponds to a line in the right image;

[0026] A correlation strategy for the left and right image sub-regions, such as the normalized cross-correlation function (NCC), is employed to perform an integer-pixel search within the epipolar stripe range. The point with the highest correlation coefficient is selected as the matching point, thus obtaining the integer-pixel coordinates (x') of the seed point matching. Rij ,y' Rij );

[0027] S53. Fine matching of seed points: Based on step S52, the obtained (x' Rij ,y' Rij Using ) as the initial value, fine matching is performed. First, the sub-pixel interpolation algorithm is used to perform grayscale interpolation on the search domain to obtain sub-pixel level coordinates and corresponding grayscale values;

[0028] After subpixel interpolation is completed, a fine-matching correlation coefficient calculation function is used for subpixel matching. Through subpixel search, the center of the search region with the highest correlation coefficient on the right image is selected as the matching point, thus obtaining the subpixel matching coordinates (x, y). Rij ,y Rij );

[0029] S54. Seed point diffusion: Based on the coordinates of the corresponding points in the seed point array on the right image relative to the coordinates of the seed points on the left image, as determined in step S53, calculate the coordinate increment (Δu) of the corresponding points in the seed point array on the right image. ij ,Δv ij );

[0030] Centered on the seed point, expand outwards in four directions (up, down, left, right) with a certain step size to obtain four image points. Then, divide the seed point (Δu)... ij ,Δv ij The coarse matching coordinate increments of these four image points are assigned, and then the fine matching process in step S53 is directly performed to obtain the fine matching coordinates and coordinate increments of these four points on the right image. Then, using these four points as the center, the same approach is used to complete the matching of all data points in the speckle area in step S54.

[0031] S55. Three-dimensional reconstruction of data points within the speckle area: Complete the three-dimensional reconstruction of all matching point pairs obtained in step S54 on the GPU.

[0032] In some embodiments, the detailed steps of the three-dimensional reconstruction over time series in step S7 are as follows:

[0033] S71. The time series of images acquired by the camera is represented by the sequence t0, t1, t2, t3…tn;

[0034] S72. Complete the data point matching and deformation parameter calculation for the image from time t0 to time t1;

[0035] S73. Initial value data acquisition of image data points at time ti+1: The positions of all calculated nodes on the image at time ti are used as the initial values ​​of the fine matching of the corresponding nodes on the image at time ti+1. At the same time, the deformation parameters of all data nodes on the image at time ti are used as the initial values ​​of the iteration of the deformation parameters of the corresponding nodes on the image at time ti+1.

[0036] S74. Data copying: Copy the initial value data from step S72 to the GPU's global memory, copy the left and right image data to the GPU's texture memory, and copy the relative pose (R,T) of the first camera and the second camera, as well as the distortion model coefficients, to the GPU's constant memory.

[0037] S75. Execute the kernel function. The fine matching process of data points is completed in the GPU kernel function. The corresponding points after fine matching are reconstructed. After the GPU completes the matching, the data points that are correctly matched and those that are incorrectly matched are distinguished by verification. The correct data points after fine matching and the reconstructed 3D data points are output.

[0038] S76. Data copy: Copy the data point pairs, deformation parameters of the point pairs, and three-dimensional point coordinates obtained in step S74 back to the CPU memory.

[0039] S77. For the data point pairs copied back to the CPU in step S75, the CPU determines the matching failure points based on the matching failure flag bit copied back from the GPU and repairs some node data that failed to match precisely, and reconstructs the coordinates of the repair points on the CPU.

[0040] S78. Assign the repaired point-pair data from step S77 to the initial values ​​of the data points in the image at time ti+2, and repeat steps S73 to S77 until the user stops the program.

[0041] Further, in step S75, the verification distinguishes between data points with correct and incorrect matches, specifically including: verifying the verification item, assigning a value of 1 to the flag bit of a correctly matched data point, and assigning a value of -1 to the flag bit of a failed match; the verification item includes:

[0042] a) The iteration fails to converge due to the equation having no solution or too many iterations.

[0043] b. The grayscale residual of the corresponding sub-region of the image is too large;

[0044] c. The reprojection error of the reconstructed 3D points is greater than the threshold.

[0045] d. The depth of the reconstructed 3D point coordinates is greater than the depth of field.

[0046] In some embodiments, in step S8, the three-dimensional point cloud data of the human body is output in real time.

[0047] In some embodiments, the intrinsic parameters in step S2 include focal length, principal point deviation, and distortion model coefficients. The calibration of the first camera and the second camera adopts a multi-step method and a bundle adjustment method. The multi-step method specifically involves placing the calibration plate at multiple different positions in the common field of view of the first camera and the second camera, and the two cameras simultaneously acquiring multiple pairs of images.

[0048] In some embodiments, in step S5, the defined strip range is a range of 3 pixels above and below the epipolar line.

[0049] Furthermore, the calibration plate mentioned in step S21 is a planar plate with a regular distribution of ring-shaped coded markers and circular non-coded markers. The size of the calibration plate used in calibration is consistent with the measurement area of ​​the camera, that is, within the depth of field of the camera, the image of the calibration plate occupies nearly 80% of the image area in the camera.

[0050] Further, in step S53, the zero-mean normalized minimum distance sum of squares (ZNSSD) coefficient is used to perform sub-pixel search and matching in the sub-region; in step S55, the matching point pair data is copied to the GPU global memory, the relative pose (R,T) of the first camera and the second camera and the distortion model coefficients are copied to the GPU constant memory, the kernel function used by the GPU for distortion removal and reconstruction calculation is used to complete the three-dimensional reconstruction of the data points on the GPU, and finally the three-dimensional point data is copied back to the CPU.

[0051] The beneficial effects of this invention compared to the prior art include:

[0052] The real-time reconstruction method for dynamic human bodies proposed in this invention avoids the following shortcomings of existing technologies: relying solely on CPU computation cannot achieve the goal of real-time human body reconstruction. This invention leverages the advantages of GPUs, which contain numerous computing cores and can allocate a large number of threads to perform the same computations simultaneously, significantly improving the efficiency of 3D reconstruction of dynamic human bodies. This invention introduces GPUs to handle a large amount of data computation and addresses the following problems brought about by the introduction of GPUs: excessive judgments lead to reduced computational efficiency, while computations without judgments can easily lead to mismatches of some data points, resulting in missing data points. This is solved by performing the verification of GPU matching results and repairing points that failed to achieve precise matching on the CPU. This method can significantly increase the point cloud density of the reconstructed human body data while maintaining reconstruction accuracy, allowing the reconstructed point cloud data to more fully express the local details of the human body, thereby enabling real-time computation of dynamic human bodies and realizing online human body measurement. The real-time reconstruction method for dynamic human bodies proposed in this invention, by projecting speckle patterns and matching speckle patterns over time, can further achieve real-time computation of dynamic human bodies, thereby realizing online human body measurement. Attached Figure Description

[0053] Figure 1 This is a flowchart of the real-time reconstruction method for dynamic human body in an embodiment of the present invention;

[0054] Figure 2 This is a schematic diagram of the experimental scenario in an embodiment of the present invention;

[0055] Figure 3 This is a schematic diagram of the speckle projector in an embodiment of the present invention;

[0056] Figure 4a1 This is a schematic diagram of the first camera calibration state 1 in an embodiment of the present invention;

[0057] Figure 4a2 This is a schematic diagram of the second camera calibration state 1 in an embodiment of the present invention;

[0058] Figure 4b1 This is a schematic diagram of the first camera calibration state 2 in an embodiment of the present invention;

[0059] Figure 4b2 This is a schematic diagram of the second camera calibration state 2 in an embodiment of the present invention;

[0060] Figure 4c1 This is a schematic diagram of the first camera calibration state 8 in an embodiment of the present invention;

[0061] Figure 4c2 This is a schematic diagram of the second camera calibration state 8 in an embodiment of the present invention;

[0062] Figure 5a This is a schematic diagram of the human body in the reference state of the first camera in an embodiment of the present invention;

[0063] Figure 5b This is a schematic diagram of the human body in the reference state of the second camera in an embodiment of the present invention;

[0064] Figure 6a This is a schematic diagram of the sub-region division for calculating the reference state of the first camera in an embodiment of the present invention;

[0065] Figure 6b This is a schematic diagram of the sub-region division for calculating the reference state of the second camera in an embodiment of the present invention;

[0066] Figure 7a This is a schematic diagram of the seed point diffusion process in an embodiment of the present invention;

[0067] Figure 7b This is a schematic diagram showing the shape and position of successfully matched sub-regions in the matched image in an embodiment of the present invention;

[0068] Figure 8 This is a schematic diagram of data exchange between the CPU and GPU in an embodiment of the present invention;

[0069] Figure 9This is a schematic diagram of GPU and CPU collaborative matching and reconstruction in an embodiment of the present invention;

[0070] Figure 10 This is a schematic diagram of the matching and repair approach in an embodiment of the present invention;

[0071] Figure 11 This is a comparison chart of camera frame rate and CPU and GPU computing speed in an embodiment of the present invention;

[0072] Figure 12 These are the reconstructed point cloud maps of the three states output sequentially in this embodiment of the invention;

[0073] The annotations in the attached figures are explained as follows:

[0074] 1-First camera, 2-Speckle projector, 3-Second camera, 4-Human body, 5-Speckle, 6-Human body 3D point cloud data in reference state, 7-Human body 3D point cloud data in t1 state, 8-Human body 3D point cloud data in t2 state, 21-PCB board, 22-LED light source, 23-Collimating lens, 24-Speckle plate, 25-Fixed focal length lens. Detailed Implementation

[0075] The present invention will be further described below with reference to the accompanying drawings and preferred embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other.

[0076] It should be noted that the directional terms such as left, right, up, down, top, and bottom used in this embodiment are only relative concepts or are based on the normal use of the product, and should not be considered as restrictive.

[0077] This embodiment addresses the problem of the inability to measure high-density point clouds of the human body reconstructed using structured light projection technology combined with digital image correlation (DIR). Reconstructing high-density point clouds using DIR requires extensive correlation calculations across numerous image sub-regions. Using only a CPU (Central Processing Unit) allows for offline measurement only, failing to achieve real-time human body reconstruction. GPUs (Graphics Processing Units), with their numerous computing cores and parallel processing capabilities, significantly improve the efficiency of 3D reconstruction of dynamic human bodies by incorporating them into the human body reconstruction process. The main approach in this embodiment is as follows: first, calibrating the intrinsic and extrinsic parameters of the first and second cameras used for reconstruction; acquiring images; dividing the speckle calculation sub-regions; performing 3D reconstruction of human speckles in a reference state; tracking, matching, and displaying human speckles over time; and displaying the dynamic 3D point cloud. In this embodiment, the GPU primarily performs 3D reconstruction of data points and tracking and matching human speckles over time. Finally, by comparing the image acquisition rate of the cameras, the GPU's frame rate, and the average frame rate of the CPU, the effectiveness of the experimental scheme in this embodiment is demonstrated.

[0078] Traditional digital image correlation methods are offline measurements, and calculating large numbers of data points requires significant time investment. Even with multi-threaded CPUs, the precise matching of speckle data points and the extensive computations involved in 3D reconstruction cannot be performed in real-time using only a CPU. In time-series image matching between consecutive frames, each data point operates independently, making the intensive computations highly suitable for GPUs. GPUs contain numerous computing cores, allowing for the simultaneous execution of multiple threads, greatly improving the efficiency of 3D reconstruction of dynamic human figures. Therefore, this invention leverages the high-density computing capabilities of GPUs to accelerate the reconstruction process of dynamic human figures.

[0079] The challenge of using GPUs for reconstruction lies in the fact that while GPUs excel in computation, excessive judgments can reduce computational efficiency. Conversely, computation without judgments can easily lead to mismatches of some data points, resulting in missing data points. This embodiment addresses this by using a CPU to repair missing data points based on data from four neighboring points. Compared to existing technologies, the reconstruction process in this embodiment can perform real-time calculations of the dynamic human body, enabling online measurement of the human body.

[0080] Before describing the embodiments, the following is an overview of the embodiments:

[0081] refer to Figure 1 As shown, the steps of the real-time reconstruction method for dynamic human body in this embodiment are as follows:

[0082] S1. Connect the human body measurement device: Complete the connection of the workstation, two cameras, speckle projector, camera and speckle projector control box, cables, etc.; Figure 2 As shown; the two cameras specifically refer to the first camera 1 and the second camera 3.

[0083] The speckle projector 2 used in step S1 has the following structure: Figure 3 As shown, the host computer sends on / off commands to the camera and speckle projector control box. PCB board 21 supplies power to LED light source 22, and controls the operation of LED light source 22 through command pulse signals. Collimating lens 23 provides focusing, ensuring that the light source hits the speckle plate 24 according to the optical path design requirements. Fixed-focus lens 25 images the light transmitted through speckle plate 24 onto the surface of human body 4, thereby producing speckle pattern 5. The light source is a short-wavelength blue light source, avoiding infrared absorption by human body 4, and is harmless to human eyes and skin.

[0084] The working principle of the speckle projector 2 is as follows: the control box supplies power to the speckle projector 2 and controls the LED light source 22 to work through the command pulse. The LED light source 22 generates short-wavelength blue light, and the speckle projector 2 makes the light source project onto the surface of the human body 4 through the speckle sheet 24 according to the optical path design requirements to produce a speckle pattern 5.

[0085] S2. Calibration of the first camera 1 and the second camera 3: Using the typical multi-step method and bundle adjustment method, solve for the camera intrinsic parameters (including focal length, principal point deviation and distortion model coefficients) and the phase pose matrix (R,T) between the two cameras, where R and T are the rotation matrix and translation vector from the camera coordinate system of the second camera 3 to the camera coordinate system of the first camera 1.

[0086] In this embodiment, the multi-step method is an 8-step method. The 8-step method involves placing the calibration plate at 8 different positions in the common field of view of the first camera 1 and the second camera 3, and the two cameras simultaneously acquiring 8 pairs of images for camera calibration.

[0087] In step S2, the calibration steps for the first camera 1 and the second camera 3 are as follows: the calibration plate used for calibration is placed in 8 different positions within the depth of field of the camera, and the first camera 1 and the second camera 3 acquire images of the calibration plate at these 8 positions in sequence, generating a total of 16 images.

[0088] The calibration plate is a planar plate with a regular distribution of ring-shaped coded markers and circular non-coded markers. The size of the calibration plate used in calibration needs to match the measurement area of ​​the camera, that is, within the depth of field of the camera, the image of the calibration plate in the camera occupies nearly 80% of the image area.

[0089] Image processing was performed on the 16 collected calibration board images to calculate the center position of the non-coded points on the calibration board, as well as the center position and number of the coded points.

[0090] The bundle adjustment algorithm utilizes the projection relationship between pixel coordinates on the image and the three-dimensional coordinates of marked points on the calibration board. This projection relationship can be described as... Solve for the camera's intrinsic parameter matrix K and the relative pose [R|T] between the cameras; where (u,v) are the pixel coordinates of the center of the marker point on the image, and (X,Y,Z) are the three-dimensional coordinates of the marker point.

[0091] After the calibration of the first camera 1 and the second camera 3 is completed, the human body part 4 that needs to be reconstructed is placed in the common field of view of the first camera 1 and the second camera 3, and within the depth of field of the camera.

[0092] S3. Reference state image acquisition: The first image acquired by the camera and used to divide the calculation area is regarded as the reference state image; turn on the speckle projector 2 and project artificial speckle 5 onto the surface of the human body 4. Simultaneously acquire the human body speckle 5 image captured by the first camera 1 and the second camera 3 through hardware triggering.

[0093] S4. Speckle Region Division: Based on the speckle distribution area and human body distribution area on the left image, the region to be calculated is divided to determine the range of left-right image matching and human body reconstruction; the speckle calculation area in the reference state is divided; in the human-computer interaction interface, the selection tools of three basic shapes, rectangle, ellipse and polygon, are used to divide the image into sub-regions, and Boolean operations such as intersection and union between the divided regions and the newly created regions are performed; on the left camera image in the reference state, the complex-shaped speckle region to be calculated is selected for subsequent image matching and point diffusion calculation range.

[0094] S5. 3D Reconstruction of Reference State: First, seed points are selected on the left image. Then, using the idea of ​​epipolar constraint, matching points of the seed points are searched within a certain range above and below the epipolar line. After that, the seed point diffusion approach is used to quickly complete the matching of all data points in the sub-regions divided in S4. Finally, using (R,T) calibrated in S2, the 3D point cloud of the human body in the reference state is reconstructed.

[0095] The three-dimensional reconstruction of nodes within the speckle division region in the reference state is completed by using the idea of ​​seed point matching and diffusion. The speckle division region is the "speckle region to be calculated" divided in step S4.

[0096] S6. Image acquisition in time series: Simultaneously acquire human speckle image streams captured by the first and second cameras via hardware triggering.

[0097] S7. 3D Reconstruction in Time Series: Based on the image stream captured in step S6, and taking into account the continuity of human motion, the deformation parameters of the speckle data points from the previous moment relative to the reference state are directly used as the initial values ​​for the corresponding data points at the current moment. The fine matching process for each data point is completed on the GPU, and the matching results are verified on the CPU, with points failing the fine matching process being repaired. Finally, according to the reconstruction method in step S5, the 3D point cloud of the human body in the current state is reconstructed.

[0098] Based on the continuity of deformation, the approach of tracking data points within a speckle-speckle region is adopted to complete the 3D reconstruction of the human body over time. The detailed steps are as follows:

[0099] S71, using t0, t1, t2, ... t n The sequence represents the time series of images acquired by the camera, where the image corresponding to time t0 is the image of the reference state described in step S3; the images corresponding to time t1 and thereafter are images of human motion continuously acquired in the time series in step S6.

[0100] S72. Complete the data point matching and deformation parameter calculation from the image at time t0 to the image at time t1. The algorithm flow is as described in steps S52 to S55.

[0101] S73, t i+1 Initial value data acquisition of image data points at time t i The positions of all computed nodes on the time-mapping image are taken as t. i+1 The initial value for fine matching of the corresponding node on the time-time image, and at the same time, t i The deformation parameters of all data nodes on the time-mapping image are used as t i+1 The initial values ​​of the deformation parameters of the corresponding nodes on the image at each time step;

[0102] S74. Data copying: Copy the initial value data from step S73 to the GPU's global memory, copy the left and right image data to the GPU's texture memory, and copy the relative pose (R,T) of the first camera and the second camera, as well as the distortion model coefficients, to the GPU's constant memory.

[0103] S75. Execute the kernel function to complete the fine matching process of data points in the GPU kernel function, and output the finely matched data point pairs. After the GPU completes the matching, it verifies four verification items: a) the equation in the iteration has no solution, or the iteration does not converge due to too many iterations; b) the grayscale residual of the corresponding sub-region of the image is too large; c) the reprojection error of the reconstructed 3D points is greater than the threshold; d) the coordinate depth of the reconstructed 3D points is greater than the depth of field; and assigns 1 to the flag bit of the data point that matches correctly and -1 to the flag bit of the data point that fails to match, and outputs the finely matched data points and the reconstructed 3D data points.

[0104] S76. Data copy: Copy the data point pairs, deformation parameters of the point pairs, and three-dimensional point coordinates obtained in step S75 back to the CPU memory.

[0105] S77. For the data point pairs copied back to the CPU in step S76, the CPU determines the matching failure point based on the matching failure flag bit copied back by the GPU, repairs some of the node data that failed to match precisely, and reconstructs the three-dimensional point coordinates of the repaired point.

[0106] S78. Assign the repaired point-to-point data from step S77 to t. i+2 The initial values ​​of the data points in the image at each time point are used to repeatedly execute steps S73 to S77 until the user stops the program.

[0107] S8. Output the three-dimensional point cloud data of the human body.

[0108] Example 1:

[0109] The process of dynamic real-time human reconstruction in this embodiment is as follows: Figure 1 As shown, the specific steps include:

[0110] S1. After connecting the two cameras, the speckle projector, the camera and speckle projector control box, cables, etc., as follows: Figure 2 As shown.

[0111] The example uses two cameras to form a stereo imaging system. The communication interface between the cameras and the workpiece is a Basler USB 3.0 interface. The image resolution is 2 megapixels (1920 pixels × 1200 pixels), the pixel size of the image sensor is 5.86 μm / pixel, and it is equipped with a 12mm RICOH lens.

[0112] S2. Calibration of the first camera 1 and the second camera 3: The typical 8-step method and bundle adjustment method are used to calibrate the intrinsic and extrinsic parameters of the cameras. The calibration board is placed in 8 different positions within the depth of field of the camera. The first camera 1 and the second camera 3 simultaneously and sequentially acquire images of the calibration board at these 8 positions, generating a total of 16 images. In this embodiment, camera a is the first camera 1 and camera b is the second camera 3.

[0113] Image processing was performed on the 16 collected calibration board images to calculate the center positions of non-coded points, as well as the center positions and numbers of coded points. The calculation results for coded and non-coded points are as follows: Figures 4a1-4c2 As shown, during the camera calibration process, the calibration board is placed in different positions, and the images (partial positions) of the calibration board captured by the first camera 1 and the second camera 3, as well as the identification results of the marker points, are shown. The coded marker points are numbered, and the non-coded marker points are represented by crosshairs.

[0114] In this embodiment, image processing was performed on the 16 acquired calibration board images to calculate the center positions of non-coded points, as well as the center positions and numbers of coded points. The calculation results for coded and non-coded points are as follows: Figure 4a1 , 4a2 As shown in 4b1, 4b2, 4c1, and 4c2, Figure 4a1 , 4a2 These are schematic diagrams of the calibration states of the first camera 1 and the second camera 3, respectively. Figure 4b1 , 4b2 These are schematic diagrams showing the calibration states of the first camera 1 and the second camera 3, respectively. Figure 4c1 , 4c2 These are schematic diagrams showing the calibration states of the first camera 1 and the second camera 3, respectively.

[0115] A bundle adjustment algorithm is employed, utilizing the projection relationship between pixel coordinates on the image and the three-dimensional coordinates of marked points on the calibration board, to achieve this. The projection model and the constructed error correction function are shown in the formula.

[0116]

[0117] in These are the projection points of three-dimensional points (coded and non-coded points on the calibration board) onto a two-dimensional image. Here are the image point coordinates for coded and non-coded points, m is the camera number, and k is the k-th marker point among all marker points captured by the m-th camera.

[0118] The calibration results for camera a and camera b are shown in the table below:

[0119]

[0120]

[0121] Where f0 is the focal length of the selected lens, f is the focal length after camera calibration, and c x c y K1, K2, and K3 are the principal point deviations of the camera in the x and y directions, respectively. K1, K2, and K3 are the three parameters in the radial distortion model, B1 and B2 are the two parameters in the tangential distortion model, and E1 and E2 are the two parameters in the thin prism distortion model.

[0122] Taking the camera coordinate system of camera a as the world coordinate system, the coordinate transformation matrix from camera b to camera a is:

[0123]

[0124] After the calibration of the first camera 1 and the second camera 3 is completed, the human body parts that need to be reconstructed are placed in the common field of view of the first camera 1 and the second camera 3, and within the depth of field of the cameras.

[0125] S3. Reference State Image Acquisition: Turn on speckle projector 2 to project artificial speckle 5 onto the human body surface, and simultaneously acquire images of the human body speckle 5 captured by cameras a and b. The acquired image in the reference state is as follows: Figure 5a , 5b As shown; where the left image Figure 5a The image captured by camera a, right image Figure 5b The image is captured by camera b, where F0 is the frame rate of the image captured by the camera; F1 is the frame rate of the GPU computation; and F2 is the number of reconstructed points.

[0126] S4. Divide the reference state speckle calculation region; select the calculation region using basic graphic tools such as polygons and rectangles. The result of selecting the calculation region in this embodiment is as follows: Figure 6a , 6b As shown, the left image Figure 6a The image captured by camera a, right image Figure 6b The image captured by camera b.

[0127] S5. The 3D reconstruction of nodes within the speckle gradation region in the reference state is completed using a seed point matching and diffusion approach. Seed points are automatically determined in the computational region selected in S4 using a strategy of equal row and column intervals. The seed point matching and diffusion process on the left and right images is completed on the CPU. The seed point matching and diffusion process is as follows: Figure 7a , 7b As shown, where Figure 7a This is a schematic diagram of the diffusion process of seed points in the image from camera a. Figure 7bThis is a schematic diagram showing the location and shape of the successfully matched sub-region on the camera b image during the diffusion process. Due to solid angle and perspective transformation, the shape of the corresponding sub-region on the camera b image is not rectangular; because a first-order deformation coefficient is used, the shape is a parallelogram.

[0128] The seed points mentioned above are selected manually. The matching and diffusion process of seed points is as follows: Figure 7a , 7b As shown, the process begins with coarse-fine matching of the seed points. After seed point matching is complete, the deformation coefficients of the seed points are output, containing six parameters. These deformation coefficients describe the shape and position of the successfully matched sub-regions on the camera b image. The diffusion process is described as utilizing the continuity of speckle patterns in adjacent regions, assigning the deformation coefficients of the seed points to four neighboring nodes with a certain step size in all directions (up, down, left, and right), and using these as the initial values ​​for fine matching. These steps are repeated to complete the node matching for the defined speckle region.

[0129] After seed point matching and diffusion are completed in the reference state, the matched point pairs are copied to the GPU's global memory. The relative poses (R,T) of camera a and camera b, as well as the distortion model coefficients, are copied to the GPU's constant memory. Kernel functions for GPU distortion correction and reconstruction calculations are written to perform 3D reconstruction of the data points on the GPU. Finally, the 3D point data is copied back to the CPU. The data transfer process between the CPU and GPU is as follows: Figure 8 As shown, the CPU transfers the matched image coordinates and deformation coefficients at time t to the GPU's global memory as initial values ​​and deformation coefficients for image matching at time t+1. Image data from cameras a and b at time t+1 are transferred to the GPU's texture memory, including grayscale and gradient information. The camera pose (R, T) (12 values) and the two cameras' intrinsic parameters (10 values ​​each), totaling 32 double-precision floating-point values, are transferred to the GPU's constant memory. After parallel matching of data points on the GPU, 3D reconstruction of the matched points is performed based on the camera pose and intrinsic parameters, and the 3D point coordinates are copied back to the CPU. Simultaneously, the matched node image coordinates from the two images at time t+1 are copied back to the CPU, and defect checks and repairs are performed on the CPU, serving as initial values ​​for time t+2. The terms Grid, Block, Registers, Thread, and Local M are used in GPU computation.

[0130] In step S5, the 3D reconstruction of nodes within the speckle area in the reference state is completed using a seed point matching and diffusion approach. The speckle area is the "speckle area to be calculated" defined in step S4. The detailed steps are described as follows:

[0131] S51. Seed point selection: Based on the interval step size parameter of the seed point distribution, a seed point array is automatically selected on the image acquired by the first camera. The seed point array is represented as (x Lij y Lij ), where i = 0, 1…n; j = 0, 1…n; the seed point array is automatically selected on the first camera image based on the interval step size parameter of the seed point distribution;

[0132] Step S52: Coarse matching of seed points. Based on the relative poses (R,T) of the first and second cameras calibrated in step S2, the seed points (x,t) are solved using epipolar constraints in two-view geometry. Lij ,y Lij The estimated position (x) on the image acquired by the second camera (right image) Rij ,y Rij This position corresponds to a line on the right image, where the second camera specifically refers to the right camera, and the image captured by the second camera is the right image;

[0133] Based on the mapping relationship of polar geometry, the following formula can be obtained:

[0134]

[0135] ax R0 +by R0 +c=0 (2)

[0136] Where a, b, and c are the simplified values ​​of the formula, and T in the formula... Λ This represents the antisymmetric matrix corresponding to the translation vector. For example, the antisymmetric matrix corresponding to the vector [t1 t2 t3] is...

[0137] Considering the existence of systematic errors and random noise, the integer pixel coordinates of the corresponding points of the selected seed points on the right image are searched within a certain strip range above and below the epipolar line determined by the formula.

[0138] A correlation strategy for the left and right image sub-regions, such as the normalized cross-correlation function (NCC), is employed to perform an integer pixel search within the epipolar stripe range. The point with the highest correlation coefficient is selected as the matching point, thus obtaining the coarse matching pixel coordinates (x') of the seed point array matching. Rij ,y' Rij ).

[0139] S53. Fine matching of seed points. Based on step S52, the obtained (x' Rij ,y' RijUsing the initial value (i,j), fine matching is performed. First, a sub-pixel interpolation algorithm is used to interpolate the grayscale of the search domain to obtain sub-pixel level coordinates and corresponding grayscale values. Commonly used interpolation algorithms include bilinear interpolation and bicubic spline interpolation. The following formula is the bilinear interpolation algorithm, which interpolates the sub-pixel grayscale values ​​between four points based on the grayscale values ​​of the four surrounding integer pixels (i,j), (i+1,j), (i,j+1), and (i+1,j+1). Sub-pixels are pixels that exist between two actual physical pixels.

[0140] g(x,y)=a 11 (xj)(yi)+a 21 (xj)(i+1-y)+

[0141] a 12 (j+1-x)(yi)+a 22 (j+1-x)(i+1-y)

[0142] j < x < j+1, i < y < i+1 (3)

[0143] Where g(x,y) is the gray value corresponding to the point (x,y) on the right image, and a 11 a 21 a 12 a 22 Let g(i,j), g(i+1,j), g(i,j+1), and g(i+1,j+1) be the grayscale values ​​of the four integer pixels surrounding (x,y). Bilinear interpolation has low computational cost and high accuracy, which can meet the accuracy requirements of dynamic human body reconstruction.

[0144] After subpixel interpolation is completed, the zero-mean normalized minimum distance sum of squares (ZNSSD) coefficient is used as the formula for calculating the fine-matching correlation coefficient to perform sub-region subpixel search and matching.

[0145]

[0146] In the formula f m g m Δf and Δg represent the mean gray values ​​of the matching sub-regions in the left and right images, respectively; Δf and Δg represent the sum of squares of the differences between the local gray values ​​and the mean gray values ​​of the matching sub-regions in the left and right images, respectively.

[0147] By searching at the sub-pixel level, the center of the search region with the highest correlation coefficient on the right image is selected as the matching point, thus obtaining the sub-pixel matching coordinates (x, y). Rij ,y Rij ).

[0148] S54. Perform seed point diffusion. Based on the coordinates of the corresponding point on the right image relative to the coordinates of the seed point on the left image, as determined in step S53, solve for the seed point (x...). Lij ,y Lij The pixel coordinate increment (Δu) on the left and right images ij ,Δv ij The specific formula is as follows:

[0149] Δu ij =u+u x dx+u y dy

[0150] Δv ij =v+v x dx+v y dy (5)

[0151] Centered on the seed point, expand outwards in four directions (up, down, left, right) with a fixed step size to obtain four image points. Based on the continuity of the image, the (Δu) of the seed point is... ij ,Δv ij The coarse matching coordinate increments of the four image points are assigned as follows: (x) Lij ,y Lij Taking one of the four image points (u1, v1) around a given point as an example, the coarsely matched value of (u1, v1) on the right image is (u1 + Δu). ij ,v1+Δv ij );

[0152] Then (u1+Δu) ij ,v1+Δv ij Using (u1, v1) as the initial position for fine matching, the fine matching result of (u1, v1) on the right image is obtained through iterative calculation. Using the same approach, the matching of all data points in the speckle division area in step S54 is completed.

[0153] S55. Three-dimensional reconstruction of data points within the speckle region: All matching point pairs obtained in step S54 are reconstructed on the GPU. The matching point pair data is copied to the GPU's global memory. The relative poses (R, T) of the first and second cameras, as well as the distortion model coefficients, are copied to the GPU's constant memory. A kernel function for GPU distortion correction and reconstruction calculations is written to complete the three-dimensional reconstruction of the data points on the GPU. Finally, the three-dimensional point data is copied back to the CPU.

[0154] S6. Acquire human speckle images over time and perform 3D reconstruction of data points at each state in the time series by matching image data points over time.

[0155] The process of image matching on time series and verification of matching results is as follows: Figure 9 As shown. t i The positions of all computed nodes on the time-mapping image are taken as t. i+1 The initial value for fine matching of the corresponding node on the time-time image, and at the same time, t i The deformation parameters of all data nodes on the time-mapping image are used as t i+1 The initial values ​​for the deformation parameters of the corresponding nodes on the image at time t are used for iteration. The image matching process on the time series involves using the coordinates of the image points at time t as the initial values ​​for matching at time t+1, and then performing a fine matching process. The fine matching process can be found in s53.

[0156] At time t+1, after the GPU completes the matching, the verification mainly includes: a) whether the equations in the iteration have no solution and whether the iteration has converged; b) the grayscale residual of the image sub-region; c) the reprojection error of the reconstructed 3D points; d) the depth of the reconstructed 3D point coordinates. A flag indicating a verification failure is assigned a value of -1, and the image point coordinates at time t+1 are copied back to the CPU. Based on the verification flag, for points that fail to match, the matching process expands to those points based on the surrounding successfully matched points, completing the matching on the CPU. A matching diagram is shown below. Figure 10 As shown.

[0157] S7. Copy the initial values ​​for fine matching and iteration to the GPU's global memory. Copy the left and right image data to the GPU's texture memory. Copy the relative poses (R, T) of the first and second cameras, as well as the distortion model coefficients, to the GPU's constant memory. Execute the fine matching kernel function to complete the fine matching process of the data points. After the GPU completes the matching, verify the following four verification items:

[0158] a) The iteration fails to converge due to the equation having no solution or the number of iterations being greater than 100.

[0159] b. The grayscale residual of the corresponding sub-region of the image is greater than 10.0;

[0160] c. The reprojection error of the reconstructed 3D points is greater than 0.8 pixels;

[0161] d. The depth of the reconstructed 3D point coordinates is greater than the object distance ± 0.5 times the object distance;

[0162] The system assigns a value of 1 to the flag indicating a correct data point match and a value of -1 to the flag indicating a failed data point match. It then outputs the correctly matched data points and the reconstructed 3D data points, completing the 3D reconstruction process.

[0163] The finely matched data point pairs and their transformation parameters are copied back to the CPU memory. The CPU uses the matching failure flags copied back from the GPU to identify failed matching points and repairs some of the finely matched node data. This process is then propagated towards the failed point using successfully matched data points surrounding it. A schematic diagram of the repair process for failed matching points is shown below. Figure 10 As shown in the diagram, solid dots represent data points that were successfully matched by the GPU, while hollow dots represent points that failed to match. The arrows in the diagram indicate the diffusion direction of surrounding data points when repairing the missing point on the CPU. After the failed matching points are repaired, the 3D reconstruction of that point is completed on the CPU.

[0164] S8. Output the reconstructed 3D point cloud data of the human body, including point cloud data in three states as follows: Figure 12 As shown, 6 is the human body 3D point cloud data in the reference state, 7 is the human body 3D point cloud data in state t1, and 8 is the human body 3D point cloud data in state t2.

[0165] Experimental Example 1:

[0166] In the experimental example, for a human image with the same number of data points, the number of data points for real-time calculation of the speckle region was 3200, achieved using both pure CPU computation and the GPU+CPU collaborative computation proposed in this embodiment of the invention. The computational efficiency of the two methods is shown in the graph, which compares the camera's shooting frame rate, the GPU+CPU collaborative computation frame rate, and the average frame rate of CPU multi-threaded computation (offline computation with the same amount of data). Figure 11 As shown.

[0167] Point cloud reconstruction based on digital image correlation methods currently uses CPU for offline measurement. That is, images acquired during human movement are first recorded, and then the point cloud reconstruction of the human body is completed through coarse matching, fine matching, diffusion, and three-dimensional reconstruction of seed points.

[0168] For calculations involving the same number of data points, such as Figure 11 In the previous method, the average computational efficiency of CPU multi-threading was 2.3 frames per second, which is far from meeting the camera's acquisition rate (20 frames per second). However, by adopting the GPU+CPU collaborative computing approach proposed in this patent, the computational efficiency of human point cloud data can reach 20 frames per second. It can be concluded that, provided that the data points are fully repaired, the computational efficiency is greatly improved. The fluctuations in the computational frame rate are caused by the different amounts of data repaired by the CPU.

[0169] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, several equivalent substitutions or obvious modifications can be made without departing from the concept of the present invention, and all such modifications, achieving the same performance or application, should be considered within the scope of protection of the present invention.

Claims

1. A method of real-time reconstruction of a dynamic human body, characterized in that, The steps of the method include: S1. Connect to the human body measurement device, which includes a first camera and a second camera; S2, calibrate the first camera and the second camera, solve the camera intrinsic parameters and the relative pose between the two cameras ; wherein and is a rotation matrix and a translation vector of the camera coordinate system of the second camera to the camera coordinate system of the first camera; S3. Reference State Image Acquisition: The first image acquired by the camera and used to divide the calculation area is considered as the reference state image; the speckle projector is turned on to project speckles onto the human body surface, while simultaneously acquiring human body speckle images; S4. Speckle region division: Based on the speckle distribution area and human body distribution area on the left image, the region to be calculated is divided to determine the range of left and right image matching and human body reconstruction. S5, three-dimensional reconstruction of reference state: first select a seed point on the left image, then search for the matching point of the seed point within the range of the strip set on the lower polar line, then complete the matching of all data points in the sub-area divided in S4, and finally reconstruct the three-dimensional point cloud of the human body in the reference state by using the calibration in S2. , S6. Image acquisition over time series: Simultaneously acquire image streams of human speckle patterns; S7. 3D Reconstruction of Human Speckle Data over Time Series: Based on the image stream acquired in step S6, and considering the continuity of human dynamic motion, the deformation parameters of the speckle data points from the previous moment relative to the reference state are directly used as the initial values ​​of the corresponding data points at the current moment. The fine matching process for each data point is completed on the GPU, and the matching results are verified on the CPU to correct any points that failed the fine matching. Finally, the 3D point cloud of the human body in the current state is reconstructed using the reconstruction method in step S5. S8. Real-time output of the reconstructed 3D point cloud of the human body over time. In step S7, the 3D reconstruction over time series is detailed as follows: S71. The time series of images acquired by the camera is represented by the sequence t0, t1, t2, t3 … tn; S72. Complete the data point matching and deformation parameter calculation for the image from time t0 to time t1; S73. Initial value data acquisition of image data points at time ti+1: The positions of all computation nodes on the image at time ti are used as the initial values ​​of fine matching of the corresponding nodes on the image at time ti+1. At the same time, the deformation parameters of all data nodes on the image at time ti are used as the initial values ​​of the iteration of the deformation parameters of the corresponding nodes on the image at time ti+1. S74, data copying, copying the initial value data in step S72 to the global memory of the GPU, copying the left and right image data to the texture memory of the GPU, copying the relative poses of the first camera and the second camera and the distortion model coefficients to the Constant memory of the GPU; S75. Execute the kernel function. The fine matching process of data points is completed in the GPU kernel function. The corresponding points after fine matching are reconstructed. After the GPU completes the matching, the data points that are correctly matched and those that are incorrectly matched are distinguished by verification. The correct data points after fine matching and the reconstructed 3D data points are output. S76. Data copy: Copy the data point pairs, deformation parameters of the point pairs, and three-dimensional point coordinates obtained in step S74 back to the CPU memory. S77. For the data point pairs copied back to the CPU in step S75, the CPU determines the matching failure points based on the matching failure flag bit copied back from the GPU and repairs some node data that failed to match precisely, and reconstructs the coordinates of the repair points on the CPU. S78. Assign the repaired point-pair data from step S77 to the initial values ​​of the data points in the image at time ti+2, and repeat steps S73 to S77 until the user stops the program.

2. The method of real-time reconstruction of a dynamic human body of claim 1, wherein, In step S1, the human body measuring device also includes a workstation, a speckle projector, a camera, and a speckle projector control box; the LED light source in the speckle projector is controlled by a command pulse to work, the LED light source generates short-wavelength blue light, and the speckle projector causes the light source to be projected onto the human body surface through the speckle sheet according to the optical path design requirements to produce a speckle pattern.

3. The method of real-time reconstruction of a dynamic human body of claim 1, wherein, In step S2, the calibration steps for the first camera and the second camera are as follows: S21. For the calibration plates placed in different positions within the camera's depth of field, the first camera and the second camera simultaneously and sequentially acquire images of the calibration plates at these multiple positions, generating multiple images in total. S22. Perform image processing on the acquired images, calculate the pixel coordinates of coded and non-coded points on the calibration board, and use the coordinates on each image and the three-dimensional coordinates on the calibration board to solve the camera's intrinsic parameters and the relative attitude between cameras according to the bundle adjustment algorithm.

4. The method of real-time reconstruction of a dynamic human body of claim 1, wherein, In step S4, the speckle region division step is detailed as follows: Using three basic shape tools—rectangle, ellipse, and polygon—as well as Boolean operations for intersection and union between these three shapes, complex speckle regions to be calculated are selected on the left camera image in the reference state, serving as the calculation range for subsequent image matching.

5. The method of real-time reconstruction of a dynamic human body of claim 1, wherein, In step S5, the steps for three-dimensional reconstruction of the reference state are as follows: S51, seed point selection, automatically selecting a seed point array on the image collected by the first camera according to the interval step parameter of the distribution of the seed points, and the seed point array is represented as wherein ; S52, coarse matching of seed points, according to the relative pose of the first camera and the second camera in S2 , using epipolar constraint in binocular geometry, the seed points are solved the estimated position on the image captured by the second camera (right image) , the position corresponds to a line on the right image; The correlation strategy of left and right image sub-regions, such as a normalized cross correlation function (NCC), is adopted to perform integer pixel search in the strip range of the epipolar line, and the point with the maximum correlation coefficient is selected as the matching point to obtain the integer pixel coordinates of the seed point matching ; S53, fine matching of the seed point, on the basis of step S52, to obtain The initial value is used for fine matching. First, a sub-pixel interpolation algorithm is used to perform gray value interpolation on the search domain to obtain a sub-pixel level coordinate and a corresponding gray value. After the sub-pixel interpolation is completed, a fine matching correlation coefficient calculation function is used to perform sub-pixel matching, and through the search of the sub-pixel, a search area center with the largest correlation coefficient on the right image is selected as the matching point, i.e. the sub-pixel matching coordinates are obtained ; S54, diffusion of the seed points, according to the coordinates of the corresponding points on the right image relative to the coordinates of the seed points on the left image, solve the coordinate increments of the corresponding points of the seed point array on the right image ; Centered on the seed point, expand outwards in four directions (up, down, left, right) with a certain step length to obtain four image points. Then, use the seed point... The coarse matching coordinate increments of these four image points are assigned, and then the fine matching process in step S53 is directly performed to obtain the fine matching coordinates and coordinate increments of these four points on the right image. Then, using these four points as the center, the same approach is adopted to complete the matching of all data points in the speckle area in step S54. S55. Three-dimensional reconstruction of data points within the speckle area: Complete the three-dimensional reconstruction of all matching point pairs obtained in step S54 on the GPU.

6. The method of real-time reconstruction of a dynamic human body of claim 1, wherein, In step S75, the verification distinguishes between data points with correct and incorrect matches, specifically including: verifying the verification items and assigning a value of 1 to the flag bit of a correctly matched data point and a value of -1 to the flag bit of a failed match; the verification items include: a) The iteration fails to converge due to the equation having no solution or too many iterations. b. The grayscale residual of the corresponding sub-region of the image is too large; c. The reprojection error of the reconstructed 3D points is greater than the threshold. d. The depth of the reconstructed 3D point coordinates is greater than the depth of field.

7. The method of real-time reconstruction of a dynamic human body of claim 1, wherein, In step S8, the three-dimensional point cloud data of the human body is output in real time.

8. The method of real-time reconstruction of a dynamic human body of claim 1, wherein, The intrinsic parameters mentioned in step S2 include focal length, principal point deviation, and distortion model coefficients. The calibration of the first and second cameras adopts a multi-step method and a bundle adjustment method. The multi-step method specifically involves placing the calibration plate at multiple different positions in the common field of view of the first and second cameras, and the two cameras simultaneously acquiring multiple pairs of images.

9. The method of real-time reconstruction of a dynamic human body of claim 1, wherein, In step S5, the set strip range is a range of 3 pixels above and below the epipolar line.

10. The real-time reconstruction method for dynamic human body as described in claim 3, characterized in that, The calibration plate mentioned in step S21 is a planar plate with a regular distribution of ring-shaped coded markers and circular non-coded markers. The size of the calibration plate used in calibration is consistent with the measurement area of ​​the camera, that is, within the depth of field of the camera, the image of the calibration plate occupies nearly 80% of the image area in the camera.

11. The method of real-time reconstruction of a dynamic human body of claim 5, wherein, In step S53, the zero-mean normalized minimum squared distance coefficient (ZNSSD) is used for sub-pixel search and matching of sub-regions; in step S55, the matched point pair data is copied to the GPU global memory, and the relative poses of the first camera and the second camera are set. The distortion model coefficients are copied to the constant memory of the GPU. The kernel functions used by the GPU for distortion removal and reconstruction calculations are used to complete the 3D reconstruction of the data points on the GPU. Finally, the 3D point data is copied back to the CPU.

Images