Single-frame three-dimensional reconstruction method for capturing moving object based on binocular camera
Through the combination of the non-maximum suppression algorithm and the binocular camera position matrix, the problem of difficulty in subject recognition and frame selection in three-dimensional reconstruction of binocular cameras is solved, the recognition accuracy and calculation speed of moving objects are improved, and the calculation cost is reduced .
Patent Information
- Application Number
- CN202411091186.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-09
- Publication Date
- 2024-11-22
AI Technical Summary
During the three-dimensional reconstruction of binocular cameras, due to parallax problems, it is difficult to identify subjects and frame selection, and the shadow interference of moving objects reduces the accuracy of reconstruction.
The non-maximum suppression algorithm is used to suppress noise. Combined with the grayscale value of the binocular camera, the motion amplitude and direction of the object are calculated. The motion information of the object is restored through the position matrix of the binocular camera, reducing the interference of the shadows and improving recognition. Accuracy and calculation speed.
It effectively solves the difficulty of subject recognition and frame selection in the three-dimensional reconstruction of moving objects by binocular cameras, improves recognition accuracy and calculation speed, and reduces calculation costs.
Smart Images

Figure CN119006713A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of computer vision, and in particular relates to a single-frame three-dimensional reconstruction method based on capturing a moving object with a binocular camera. Background Art
[0002] In recent years, the field of 3D reconstruction has developed rapidly and has been widely used in the fields of autonomous driving, positioning and navigation, and other intelligent vehicles. At the same time, in the research and development of virtual reality technology, 3D reconstruction technology has gradually highlighted its importance in intuitive object display.
[0003] For 3D reconstruction, a pair of "eyes" is needed to obtain information about the object. From the initial single camera to the current single-beam laser and then to LiDAR, they have different characteristics and specialties. Structured light 3D reconstruction based on a single camera is difficult to handle complex object structures, and the reconstructed object surface is uneven. There is almost no way to shoot objects in motion to obtain accurate information; 3D reconstruction based on a single-beam laser has high performance requirements for the algorithm and large data volume. When detecting moving objects, the movement of the object will cause the camera's photosensitivity error, which increases the difficulty of reconstruction; 3D reconstruction based on LiDAR has weak color and pattern recognition capabilities, and the 3D reconstruction algorithm for moving objects is complex and the model establishment is difficult. The binocular camera can obtain the depth information of the object through parallax, and is less affected by ambient light, has a flexible structure, and a wide range of applications. Compared with LiDAR equipment, it has lower cost and less algorithm difficulty. However, for binocular cameras, due to the parallax problem of binocular cameras, it is difficult to identify the subject and select the subject during 3D reconstruction, which in turn leads to difficulties in reconstructing the object. Moreover, for moving objects, identification and selection are more difficult and ambiguous, and the interference generated when the object moves will also hinder the reconstruction process of the object and reduce the accuracy of reconstruction. The present invention uses the high suppression of noise by non-maximum suppression to solve the problems of difficult subject recognition, selection and reconstruction when the binocular camera captures moving objects, improves the interference caused by the subject's smear under movement, uses the generated smear to calculate the gray value of the image, restores the object's motion amplitude and direction, and improves the accuracy and calculation speed of binocular camera recognition of moving objects. Summary of the invention
[0004] Purpose of the invention: In order to solve the problems existing in the above-mentioned prior art, the present invention provides a single-frame three-dimensional reconstruction method based on capturing moving objects with a binocular camera.
[0005] Technical solution: The present invention provides a single-frame 3D reconstruction method based on capturing a moving object with a binocular camera, comprising the following steps:
[0006] Step 1: Use the binocular camera to take a picture of the calibration plate for camera calibration, calculate the rotation matrix E and translation matrix H of the binocular camera, and then calculate the position matrix [E|H] of the binocular camera;
[0007] Step 2: Perform Gaussian function calculation on the current frame to generate the gray value matrix S i , where i = 1, 2; 1 represents the image corresponding to the left monocular camera, and 2 represents the image corresponding to the right monocular camera;
[0008] Step 3: Use the Laplacian operator to detect the edge of the gray value matrix to generate the edge point matrix M i ;
[0009] Step 4: Use the Sobel operator on the gray value matrix to calculate the gradient amplitude F of the pixel with coordinates (x, y) in the i-th image i (x,y) and the gradient direction θ i (x, y), and combine the gradient magnitudes of all pixels to form the gradient magnitude matrix F i , the gradient directions of all pixels are combined to form the gradient direction matrix θ i ;
[0010] Step 5: Based on the gradient magnitude and gradient direction of each element in step 4, the edge point matrix is updated using the non-maximum suppression algorithm to obtain the latest edge point matrix;
[0011] Step 6: Determine whether the number of edge points in the latest edge point matrix reaches the output threshold T. If so, go to step 8. Otherwise, take the next frame as the current frame, execute steps 2 and 3, and then go to step 7.
[0012] Step 7: Couple the edge point matrix of the current frame with the latest edge point matrix corresponding to the previous frame to obtain the latest edge point matrix of the current frame, and go to step 6;
[0013] Step 8: Fusion the latest edge point matrix with the binocular camera position matrix [E|H] to generate a spatial reconstruction matrix, and restore the elements in the spatial reconstruction matrix to the world coordinate system. Generate a spatial direction amplitude reconstruction matrix based on the gradient amplitude moment and the gradient direction matrix, and restore the spatial direction amplitude reconstruction matrix to the world coordinate system to obtain the subject’s motion amplitude and motion direction, and then use the iterative closest point algorithm to suppress noise.
[0014] Furthermore, the step 1 is specifically as follows: for the image of the calibration plate taken by the binocular camera, the Shi-Tomasi corner detection algorithm is applied to detect the corner point information of the calibration plate, and the rotation matrix E and translation matrix H of the binocular camera are calculated through the spatial position and the corresponding position of the corner point in the image, so as to calculate the position matrix [E|H] of the binocular camera according to the following formula:
[0015]
[0016] E represents the rotation matrix, H represents the translation matrix, e 11 Indicates the rotation relationship between the x-axis of the world coordinate system and the x-axis of the camera coordinate system, e 12 Indicates the rotation relationship between the x-axis of the world coordinate system and the y-axis of the camera coordinate system, e 13 is the rotation relationship between the x-axis of the world coordinate system and the z-axis of the camera coordinate system, e 21 Indicates the rotation relationship between the y-axis of the world coordinate system and the x-axis of the camera coordinate system, e 22 Indicates the rotation relationship between the y-axis of the world coordinate system and the y-axis of the camera coordinate system, e 23 Indicates the rotation relationship between the y-axis of the world coordinate system and the z-axis of the camera coordinate system, e 31 Indicates the rotation relationship between the z-axis of the world coordinate system and the x-axis of the camera coordinate system, e 32 Indicates the rotation relationship between the z-axis of the world coordinate system and the y-axis of the camera coordinate system, e 33 Indicates the rotation relationship between the z-axis of the world coordinate system and the z-axis of the camera coordinate system, h x Indicates the position of the origin of the camera coordinate system relative to the origin of the world coordinate system in the x-axis direction, h y Indicates the position of the origin of the camera coordinate system relative to the origin of the world coordinate system in the y-axis direction, h z Indicates the position of the origin of the camera coordinate system relative to the origin of the world coordinate system in the z-axis direction.
[0017] Furthermore, the step 2 is specifically as follows:
[0018] Step 2.1: Convert the RGB color space in the image to grayscale color space, and fit the grayscale value P of all pixels in the i-th image according to the following formula: i (x,y):
[0019] P i (x,y)=0.299R+0.587G+0.114B
[0020] Among them, R represents the red channel in a single pixel, G represents the green channel in a single pixel, and B represents the blue channel in a single pixel. The grayscale value P of all pixels i(x, y) are combined to form the grayscale space matrix P i ;
[0021] Step 2.2: Perform the following Gaussian function operation on all pixels in the i-th image to obtain Q i (x,y):
[0022]
[0023] Among them, σ x Represents the standard deviation of the coordinate values of the image pixels in the x direction, σ y Represents the standard deviation of the coordinate value of the image pixel in the y direction, μ x Represents the average value of the x coordinates of all pixels, μ y Represents the average value of the y coordinates of all pixels, σ x and σ y The expression is as follows:
[0024]
[0025] Where N represents the total number of pixels; x t Indicates the x-axis coordinate of the t-th pixel, y t represents the y-axis coordinate of the t-th pixel; Q corresponding to all pixels in the i-th image i (x, y) combines to form the Gaussian function gray value matrix Q i :
[0026] Step 2.3: Convolve the elements in the grayscale space matrix P1 of the image corresponding to the left monocular camera with the elements in the matrix Q1 obtained by Gaussian function operation of the image corresponding to the left monocular camera to obtain the grayscale value matrix S1 of the image corresponding to the left monocular camera, and convolve the elements in the grayscale space matrix P2 of the image corresponding to the right monocular camera with the elements in the matrix Q2 obtained by Gaussian function operation of the image corresponding to the right monocular camera to obtain the grayscale value matrix S2 of the image corresponding to the right monocular camera. The specific convolution operation is as follows:
[0027]
[0028] Among them, S i (x, y) represents the grayscale value of the pixel at position (x, y) in the i-th image, P i (i', j) represents the gray value of the pixel with coordinates (i', j) in the i-th image, Q i(x+i', y+j) represents the value obtained by performing Gaussian function operation on the pixel with coordinates (x+i', y+j) in the i-th image; k1 represents the value of half the radius of the convolution kernel in the horizontal direction, k2 represents the value of half the radius of the convolution kernel in the vertical direction, and N1 represents the matrix P i N2 represents the total number of rows in the matrix P i The total number of columns, Indicates rounding down; S i All elements in are combined to form the gray value matrix S i :
[0029] Furthermore, the step 3 is specifically as follows: using the Laplace operator to detect the gray value matrix S i The Edge:
[0030]
[0031] Among them, L i (x,y) represents the pair S i The value after Laplace operator is applied to (x,y). represents the partial derivative operation symbol, Represents the symbol of the quadratic partial derivative operation, S i (x, y) represents the pixel with coordinates (x, y) in the i-th image in the gray value matrix S i The value in L i (x,y) are combined to form the quadratic partial derivative matrix L i , and for L i Perform zero crossing point detection to obtain the edge point matrix M i .
[0032] Furthermore, the step 4 is specifically as follows: using the Sobel operator to i (x, y) performs convolution operation and obtains S i (x,y) horizontal gradient magnitude F ix (x, y) and the vertical gradient magnitude F iy (x, y), and use these two gradient magnitudes to calculate the gradient magnitude F of the pixel with coordinates (x, y) in the i-th image i (x,y), the formula is as follows, and F i (x,y) are combined to form the gradient magnitude matrix F i :
[0033]
[0034] According to F ix (x,y) and F iy(x, y) calculates the gradient direction θ of the pixel with coordinates (x, y) in the i-th image i (x,y), the formula is as follows, and θ i (x,y) is combined to form the gradient direction matrix θ i :
[0035]
[0036] Furthermore, the step 5 is: the edge point matrix M i For each pixel point on a single side, if the gradient amplitude of the pixel point is the local maximum along the gradient direction, the point is retained as an edge point in the edge point matrix, otherwise the point is regarded as a non-edge point.
[0037] Furthermore, in step 6, the output threshold T is calculated using the following formula:
[0038]
[0039] Among them, ∑ x,y F i (x,y) represents the sum of the gradient magnitudes in the gradient magnitude matrix, ∑ x,y Pi ( x,y) represents the matrix P i The sum of the gray values, W i Represents the number of edge points in the corresponding edge's latest edge point matrix; represents the weight of the gradient amplitude, ω represents the weight of the gray value, and the expression is:
[0040]
[0041] where σ F represents the standard deviation of the gradient amplitude, σ P Represents the standard deviation of the grayscale value, detects whether the number of edge points in the latest edge point matrix reaches the output threshold, if so, executes step 8, otherwise executes steps 2 to 3, and then executes step 7.
[0042] Furthermore, in step 7, the edge point matrix of the current frame is coupled with the latest edge point matrix corresponding to the previous frame as follows: if the pixel point with coordinates (x, y) in the i-th image of the current frame has a value of 1 in the edge point matrix of the current frame, and the pixel point with corresponding coordinates in the i-th image of the previous frame has a value of 1 in the latest edge point matrix corresponding to the i-th image of the previous frame, then the pixel point with coordinates (x, y) in the i-th image of the current frame has a value of 1 in the edge point matrix after coupling; otherwise, the pixel point with coordinates (x, y) in the i-th image of the current frame has a value of 0 in the edge point matrix after coupling, and step 6 is executed after coupling.
[0043] Furthermore, in step 8, the spatial direction amplitude reconstruction matrix is generated according to the gradient amplitude moment and the gradient direction matrix as follows:
[0044] Step 8.1: Use the weighted average method to fit the gradient amplitudes of the two images to obtain the weighted average gradient amplitude F3(x,y):
[0045] F3(x,y)=α·F1(x,y)+(1-α)·F2(x,y)
[0046] Among them, α is the amplitude weight, F1(x, y) is the gradient amplitude of the gradient amplitude matrix of the image corresponding to the left monocular camera at the position (x, y), F2(x, y) is the gradient amplitude of the gradient amplitude matrix of the image corresponding to the right monocular camera at the position (x, y), and F3(x, y) is combined to form the gradient amplitude matrix F3;
[0047] Step 8.2: Convert the values of θ1(x,y) and θ2(x,y) to v1 and v2 using the following formula:
[0048]
[0049] Wherein, v1 is the direction vector element of the element θ1(x, y) in the gradient direction matrix θ1 at the corresponding position in the matrix, and v2 is the direction vector element of the element θ2(x, y) in the gradient direction matrix θ2 at the corresponding position in the matrix;
[0050] Step 8.3: Apply the weighted average method to the two direction vectors using the following formula to calculate the weighted average direction vector v3:
[0051] v3=βv1+(1-β)v2
[0052] Among them, β is the direction weight,
[0053] Step 8.4: Convert v3 to an angle value according to the following formula:
[0054] θ3(x,y)=atan2(v 3a ,v 3b )
[0055] Among them, atan2 is a function that returns the degree of the angle between the vector from the origin to the point (a, b) and the positive direction of the a-axis, v 3a represents the horizontal coordinate of vector v3 in the rectangular coordinate system of the aOb plane, v 3b Represents the ordinate of vector v3 in the rectangular coordinate system of the aOb plane;
[0056] Step 8.5: Construct a new gradient direction matrix θ3, where θ3(x, y) is the angle value, and combine θ3(x, y) to form the gradient direction matrix θ3;
[0057] Step 8.6: Fuse the new gradient direction matrix θ3 and gradient amplitude matrix F3 with the position matrix [E|H of the binocular camera to generate a spatial direction amplitude reconstruction matrix, and then restore this reconstruction matrix to the world coordinate system to obtain the motion amplitude and motion direction of the subject.
[0058] Beneficial effects:
[0059] 1. In order to solve the problem of much interference and poor recognition accuracy when acquiring information for three-dimensional reconstruction of moving objects, this solution obtains edge point data of a single camera separately to reduce the impact of smear, and then uses the characteristic of binocular camera that can calculate depth and the main edge points obtained by a single camera to restore the main point cloud of the object in the world coordinate system. This point cloud is used to suppress the blur and smear problems caused by motion, so as to achieve the effect of identifying the subject and suppressing noise. The present invention can further reduce the computing cost while improving the anti-interference, accuracy and applicability.
[0060] 2. When identifying moving objects, the interference caused by the ghosting causes a lot of noise and the subject is blurred. The high suppression of blur noise by non-maximum value is used to effectively eliminate the interference caused by the ghosting phenomenon of the moving subject, which greatly improves the problems of difficult subject recognition, selection and reconstruction for binocular cameras under high-speed motion.
[0061] 3. In order to solve the problems of high information redundancy and high computational complexity caused by the ghosting of moving objects, the ghosting effect of the image is reused. By convolving the grayscale value of the image, the motion amplitude and direction of the object are obtained. The point cloud generation mechanism after binocular camera calibration can be used to restore the motion amplitude and direction of the object in three-dimensional space, effectively improving the accuracy and calculation speed of binocular camera in motion trajectory recognition. BRIEF DESCRIPTION OF THE DRAWINGS
[0062] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION
[0063] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.
[0064] like Figure 1 As shown, the present invention provides a single-frame 3D reconstruction method based on capturing a moving object with a binocular camera, which specifically includes the following steps:
[0065] (1) Use a binocular camera to take pictures of the calibration plate for camera calibration;
[0066] (2) Filter the current frame to generate a gray value matrix;
[0067] (3) Applying the Laplacian operator to the gray value matrix to detect edges and generate an edge point matrix;
[0068] (4) Applying the Sobel operator to the two gray value matrices to calculate the gradient magnitude matrix and the gradient direction matrix respectively;
[0069] (5) Applying the non-maximum suppression algorithm to the edge point matrix to generate the latest edge point matrix;
[0070] (6) Detect whether the number of edge points in the latest edge point matrix reaches the output threshold. If yes, execute step (8); otherwise, take the next frame as the current frame, execute steps (2) to (3), and then execute step (7);
[0071] (7) coupling the edge point matrix of the current frame with the latest edge point matrix corresponding to the previous frame to obtain the latest edge point matrix of the current frame, and executing step (6) after coupling;
[0072] (8) The generated reconstruction matrix is then added to the world coordinate system to construct a three-dimensional point cloud and suppress noise outside the edge.
[0073] Step (1) Use a binocular camera to take a picture of the calibration plate for camera calibration. The specific process is as follows:
[0074] Use a binocular camera to take a picture of the calibration plate, apply the Shi-Tomasi corner detection algorithm to detect the corner information of the calibration plate, and calculate the rotation matrix E and translation matrix H of the binocular camera through the spatial position of the corner points and the corresponding positions of the corner points in the image, so as to calculate the position matrix [E|H] of the binocular camera according to the following formula:
[0075]
[0076] R represents the rotation matrix, H represents the translation matrix, e 11 Indicates the rotation relationship between the x-axis of the world coordinate system and the x-axis of the camera coordinate system, e 12 Indicates the rotation relationship between the x-axis of the world coordinate system and the y-axis of the camera coordinate system, e 13 Indicates the rotation relationship between the x-axis of the world coordinate system and the z-axis of the camera coordinate system, e 21 Indicates the rotation relationship between the y-axis of the world coordinate system and the x-axis of the camera coordinate system, e 22 Indicates the rotation relationship between the y-axis of the world coordinate system and the y-axis of the camera coordinate system, e 23 Indicates the rotation relationship between the world coordinate system u-axis and the camera coordinate system z-axis, e 31Indicates the rotation relationship between the z-axis of the world coordinate system and the x-axis of the camera coordinate system, e 32 Indicates the rotation relationship between the z-axis of the world coordinate system and the y-axis of the camera coordinate system, e 33 Indicates the rotation relationship between the z-axis of the world coordinate system and the z-axis of the camera coordinate system, h x Indicates the position of the origin of the camera coordinate system relative to the origin of the world coordinate system in the x-axis direction, h y Indicates the position of the origin of the camera coordinate system relative to the origin of the world coordinate system in the y-axis direction, h z Indicates the position of the origin of the camera coordinate system relative to the origin of the world coordinate system in the z-axis direction.
[0077] Step (2) filters the current frame to generate a gray value matrix. The specific process is as follows:
[0078] For the single-frame images captured by the binocular camera, their respective grayscale space matrices are calculated respectively, and the RGB color space in the image is converted into the grayscale color space, so as to fit the grayscale value matrix of the image. The formula is as follows:
[0079] P(x,y)=0.299R+0.587G+0.114B
[0080] Among them, R represents the red channel in a single pixel, G represents the green channel in a single pixel, and B represents the blue channel in a single pixel. The grayscale value P of all pixels i (x, y) are combined to form the grayscale space matrix P i .
[0081] In the formula, because the human eye is more sensitive to green, the weighting of the green channel is higher than that of red and blue. Based on this, the grayscale space matrix converted by the left camera is assumed to be P1, and the grayscale space matrix converted by the right camera is assumed to be P2, which expresses each element in the grayscale value matrix within the image range, not the grayscale value of a single pixel.
[0082] Then, the Gaussian function operation on the grayscale values of the left and right images is performed respectively, and the related Gaussian function grayscale value matrix is generated. The formula is as follows:
[0083]
[0084] Among them, Q(x,y) represents the value obtained by performing Gaussian function operation on the pixel with coordinates (x,y) in the i-th image, σ x Represents the standard deviation of the coordinate values of the image pixels in the x direction, σ y Represents the standard deviation of the coordinate value of the image pixel in the y direction, μ x Represents the average value of the x coordinates of all pixels, μy represents the average value of the y coordinates of all pixels, N represents the total number of pixels; x t Indicates the x-axis coordinate of the t-th pixel, y t represents the u-axis coordinate of the t-th pixel; the Q corresponding to all pixels in the i-th image i (x, y) combines to form the Gaussian function gray value matrix Q i .
[0085] Based on this, let the Gaussian function gray value matrix fitted by the single frame captured by the left camera be Q1, and the Gaussian function gray value matrix fitted by the single frame captured by the right camera be Q2.
[0086] Convolve the grayscale space matrix of the single-side image and its Gaussian function grayscale value matrix P1 and Q1, p2 and Q2 to generate a new grayscale value matrix. The convolution formula is:
[0087]
[0088] Among them, S i (x, y) represents the gray value of the pixel at position (x, y) in the i-th image, p i (i', j) represents the gray value of the pixel with coordinates (i', j) in the i-th image, Q i (x+i', y+j) represents the value obtained by performing Gaussian function operation on the pixel with coordinates (x+i', y+j) in the i-th image; k1 represents the value of half the radius of the convolution kernel in the horizontal direction, k2 represents the value of half the radius of the convolution kernel in the vertical direction, and N1 represents the matrix P i N2 represents the total number of rows in the matrix P i The total number of columns, Indicates rounding down; S i The elements in are combined to form the gray value matrix S i .
[0089] Let the left gray value matrix resulting from the convolution be S1, and the right gray value matrix be S2.
[0090] Step (3) applies the Laplacian operator to the gray value matrix to detect edges and generate an edge point matrix. The specific process is:
[0091] For the two gray value matrices S1 and S2 generated after convolution, the Laplacian operator is applied to detect edges. The specific calculation formula is as follows:
[0092]
[0093] Where L(x,y) is the value after the Laplace operator is applied to S(x,y).i (x,y) are combined to form the quadratic partial derivative matrix L i , and for L i Perform zero crossing point detection to obtain edge point matrix M i , calculate the quadratic partial derivative matrix L i The formula is as follows.
[0094] Among them, the edge point matrix generated by S1 is M1, and the edge point matrix generated by S2 is M2.
[0095] The specific process of applying the Sobel operator to the two gray value matrices to calculate the gradient amplitude matrix and the gradient direction matrix is as follows:
[0096] For S1 and S2, the Sobel operator is applied to perform convolution operations respectively to obtain the gradient amplitude F in the horizontal direction of S(x,y) i (x, y) and the vertical gradient magnitude F i (x, y), and use these two gradient magnitudes to calculate the gradient magnitude F of the pixel with coordinates (x, y) in the i-th image i (x,y), the formula is as follows, and F i (x,y) are combined to form the gradient magnitude matrix F i :
[0097]
[0098] Therefore, F(x, y) is combined to form a gradient magnitude matrix F, then the gradient magnitude matrix generated by S1 is F1, and the gradient magnitude matrix generated by S2 is F2.
[0099] Calculate the gradient direction angle θ(x,y) of the pixel point with coordinates (x,y) in the i-th image. The formula is as follows, and combine θ(x,y) to form the gradient direction matrix θ:
[0100]
[0101] Therefore, the gradient direction matrix generated by S1 is θ1, and the gradient direction matrix generated by S2 is θ2.
[0102] Step (5) applies the non-maximum suppression algorithm to the edge point matrix to generate the latest edge point matrix. The specific process is:
[0103] For M1 and M2, non-maximum suppression is performed according to the gradient magnitude matrices F1 and F2 of all pixels and the gradient direction matrices θ1 and θ2 of all pixels. The following operations are performed:
[0104] Among them, for each pixel point (x, y) on a single side, if F(x, y) is a local maximum along the direction θ(x, y), then M is retained. i (x,y) as the edge point, otherwise, suppress M i (z,y) is set to a non-edge point.
[0105] Therefore, the latest edge point matrix is assumed to be M new1 and M new2 .
[0106] Step (6) detects whether the number of edge points in the matrix reaches the output threshold. If yes, execute step (8). Otherwise, execute steps (2) to (3) and then execute step (7). The specific process is as follows:
[0107] For the calculation of the number of edge points in the edge point matrix, the edge point matrix M i Traversal, there is the following formula:
[0108]
[0109] Where W is the total number of edge points, M i (x,y)=1 indicates an edge point, M i (x,y)=0 indicates a non-edge point,
[0110] Assume that the output threshold of the edge point value is T, and the following formula is used to calculate the output threshold T of the edge point value:
[0111]
[0112] Among them, σ F and σ P are the standard deviation of the gradient amplitude and grayscale value, respectively, so that and ω can reflect the variability of F(x,y) and P(x,y). The more variable the feature is, the smaller the weight is when calculating the edge point numerical threshold T. represents the weight of the gradient amplitude, and ω represents the weight of the gray value. They determine the importance of the gradient amplitude matrix F and the gray value matrix P in calculating the threshold. The size of the weight needs to be adjusted according to the actual situation. and the parameter values of ω.
[0113] Compare the total number of edge points W and the edge point value threshold T. If W ≥ T, execute step (8); otherwise, execute steps (2) to (3), and then execute step (7).
[0114] Step (7) couples the edge point matrices generated in the two frames. The specific process is as follows:
[0115] The SIFT algorithm is used to extract and match the key points in the edge point matrices M1 and M2, and the coincidence of the edge points in the matrix is determined by the following formula:
[0116]
[0117] Among them, m1 is an element in the edge point matrix of the current frame, m2 is an element in the latest edge point matrix corresponding to the previous frame, and m3 is an element in the coupled edge point matrix. Moreover, m1, m2, and m3 have the same position after alignment. Through the above formula, a new edge point matrix M3 is coupled, and then step (6) is executed.
[0118] The specific process of adding the generated reconstruction matrix to the world coordinate system in step (8) to construct a three-dimensional point cloud and suppress the noise outside the edge is as follows:
[0119] The spatial reconstruction matrix is generated by fusing the latest edge point matrix M with the position matrix [E|H] of the binocular camera to restore the elements in the spatial reconstruction matrix to the world coordinate system.
[0120] Through the spatial reconstruction matrix in the world coordinate system, the specific edges of the reconstructed subject can be clearly identified, and noise can be suppressed in areas that are unexpected to the edge points of the subject, mainly by using the Iterative Closest Point (ICP) algorithm, which is a method for aligning two point clouds. Through ICP, you can align the point cloud obtained by the binocular camera with the point cloud of the spatial reconstruction matrix, thereby identifying and suppressing points that are not at the edge of the subject.
[0121] For the gradient magnitude matrices F1 and F2, the weighted average method is applied to fit them to F3, and the formula is as follows:
[0122] F3(x,y)=α·f1(x,y)+(1-α)·F1(x,y)
[0123] Among them, α is the amplitude weight, the weight is adjusted according to the actual situation, and F3 is the new gradient amplitude matrix after fitting.
[0124] For the gradient direction matrices θ1 and θ2, since the elements in the gradient direction matrices are all angle numbers, we first need to convert the elements in the matrices into direction vectors. Taking one element in the gradient direction matrix as an example, the remaining elements are converted according to this process. The element in the original matrix θ1 is θ1(x, y), and the corresponding element in θ2 is θ2(x, y), then there is the following formula:
[0125]
[0126] Among them, v1 is the direction vector element of the element θ1(x, y) in the gradient direction matrix θ1 at the corresponding position in the matrix, and v2 is the direction vector element of the element θ2(x, y) in the gradient direction matrix θ2 at the corresponding position in the matrix. The weighted average method is applied to the two direction vector elements to calculate the weighted average direction vector v3. The formula is as follows:
[0127] v3=βv1+(1-β)v2
[0128] Among them, β is the direction weight, and the choice of weight is adjusted according to the actual situation. Then the vector element v3 after weighted average is converted back to the angle θ3 (x, y). The application formula is as follows:
[0129] θ3(x,y)=atan2(v 3a ,v 3b )
[0130] Among them, atan2 is a function that returns the degree of the angle between the vector from the origin to the point (a, b) and the positive direction of the a-axis, v 3a represents the horizontal coordinate of vector v3 in the rectangular coordinate system of the aOb plane, v 3b Represents the ordinate of vector v3 in the rectangular coordinate system of the aOb plane.
[0131] Place θ3(x,y) into the corresponding position of θ3 according to the coordinates, and place the remaining elements in the same way to construct a new gradient direction matrix θ3.
[0132] The gradient direction matrices θ1 and θ2 are fitted into a new gradient direction matrix θ3 through the above process.
[0133] By fusing the generated gradient direction matrix θ3 and the gradient amplitude matrix f3 with the position matrix [E|H] of the binocular camera, a spatial direction amplitude reconstruction matrix is generated, and this reconstruction matrix is restored to the world coordinate system to obtain the motion amplitude and motion direction of the subject.
Claims
1. A single-frame 3D reconstruction method based on capturing moving objects with a binocular camera, characterized in that: The steps include: Step 1: Use the binocular camera to take a picture of the calibration plate for camera calibration, calculate the rotation matrix E and translation matrix H of the binocular camera, and then calculate the position matrix [E|H] of the binocular camera; Step 2: Perform Gaussian function calculation on the current frame to generate the gray value matrix S i , where i = 1, 2; 1 represents the image corresponding to the left monocular camera, and 2 represents the image corresponding to the right monocular camera; Step 3: Use the Laplacian operator to detect the edge of the gray value matrix to generate the edge point matrix M i ; Step 4: Use the Sobel operator on the gray value matrix to calculate the gradient amplitude F of the pixel with coordinates (x, y) in the i-th image i (x,y) and the gradient direction θ i (x, y), and combine the gradient magnitudes of all pixels to form the gradient magnitude matrix F i , the gradient directions of all pixels are combined to form the gradient direction matrix θ i ; Step 5: Based on the gradient magnitude and gradient direction of each element in step 4, the edge point matrix is updated using the non-maximum suppression algorithm to obtain the latest edge point matrix; Step 6: Determine whether the number of edge points in the latest edge point matrix reaches the output threshold T. If so, go to step 8; otherwise, take the next frame as the current frame, execute steps 2 and 3, and then go to step 7; Step 7: Couple the edge point matrix of the current frame with the latest edge point matrix corresponding to the previous frame to obtain the latest edge point matrix of the current frame, and go to step 6; Step 8: Fusion the latest edge point matrix with the binocular camera position matrix [E|H] to generate a spatial reconstruction matrix, and restore the elements in the spatial reconstruction matrix to the world coordinate system. Generate a spatial direction amplitude reconstruction matrix based on the gradient amplitude moment and the gradient direction matrix, and restore the spatial direction amplitude reconstruction matrix to the world coordinate system to obtain the subject’s motion amplitude and motion direction, and then use the iterative closest point algorithm to suppress noise.
2. The single-frame 3D reconstruction method based on binocular camera capturing moving objects according to claim 1, characterized in that: The step 1 is specifically as follows: for the image of the calibration plate taken by the binocular camera, the Shi-Tomasi corner detection algorithm is used to detect the corner point information of the calibration plate, and the position matrix [E|H] of the binocular camera is calculated according to the following formula through the spatial position of the corner point and the corresponding position of the corner point in the image: E represents the rotation matrix, H represents the translation matrix, e 11 Indicates the rotation relationship between the x-axis of the world coordinate system and the x-axis of the camera coordinate system, e 12 Indicates the rotation relationship between the x-axis of the world coordinate system and the y-axis of the camera coordinate system, e 13 Indicates the rotation relationship between the x-axis of the world coordinate system and the z-axis of the camera coordinate system, e 21 Indicates the rotation relationship between the y-axis of the world coordinate system and the x-axis of the camera coordinate system, e 22 Indicates the rotation relationship between the y-axis of the world coordinate system and the y-axis of the camera coordinate system, e 23 Indicates the rotation relationship between the y-axis of the world coordinate system and the z-axis of the camera coordinate system, e 31 Indicates the rotation relationship between the z-axis of the world coordinate system and the x-axis of the camera coordinate system, e 32 Indicates the rotation relationship between the z-axis of the world coordinate system and the y-axis of the camera coordinate system, e 33 Indicates the rotation relationship between the z-axis of the world coordinate system and the z-axis of the camera coordinate system, h x Indicates the position of the origin of the camera coordinate system relative to the origin of the world coordinate system in the x-axis direction, h y Indicates the position of the origin of the camera coordinate system relative to the origin of the world coordinate system in the y-axis direction, h z Indicates the position of the origin of the camera coordinate system relative to the origin of the world coordinate system in the z-axis direction.
3. The single-frame 3D reconstruction method based on binocular camera capturing moving objects according to claim 1, characterized in that: The step 2 is specifically as follows: Step 2.1: Convert the RGB color space in the image to grayscale color space, and fit the grayscale value P of all pixels in the i-th image according to the following formula: i (x,y): P i (x,y)=0.299R+0.587G+0.114B Among them, R represents the red channel in a single pixel, G represents the green channel in a single pixel, and B represents the blue channel in a single pixel. The grayscale values of all pixels P i (x, y) are combined to form the grayscale space matrix P i ; Step 2.2: Perform the following Gaussian function operation on all pixels in the i-th image to obtain Q i (x,y): Among them, σ x Represents the standard deviation of the coordinate values of the image pixels in the x direction, σ y Represents the standard deviation of the coordinate value of the image pixel in the y direction, μ x Represents the average value of the x coordinates of all pixels, μ y represents the average value of the y coordinates of all pixels, σ x and σ y The expression is as follows: Where N represents the total number of pixels; x t Indicates the x-axis coordinate of the t-th pixel, y t represents the y-axis coordinate of the t-th pixel; Q corresponding to all pixels in the i-th image i (x, y) is combined to form the Gaussian function gray value matrix Q i ; Step 2.3: Convolve the elements in the grayscale space matrix P1 of the image corresponding to the left monocular camera with the elements in the matrix Q1 obtained by Gaussian function operation of the image corresponding to the left monocular camera to obtain the grayscale value matrix S1 of the image corresponding to the left monocular camera, and convolve the elements in the grayscale space matrix P2 of the image corresponding to the right monocular camera with the elements in the matrix Q2 obtained by Gaussian function operation of the image corresponding to the right monocular camera to obtain the grayscale value matrix S2 of the image corresponding to the right monocular camera. The specific convolution operation is as follows: Among them, S i (x, y) represents the gray value of the pixel with coordinates (x, y) in the i-th image, P i (i', j) represents the gray value of the pixel with coordinates (i', j) in the i-th image, Q i (x+i', y+j) represents the value obtained by performing Gaussian function operation on the pixel with coordinates (x+i', y+j) in the i-th image; k1 represents the value of half the radius of the convolution kernel in the horizontal direction, k2 represents the value of half the radius of the convolution kernel in the vertical direction, and N1 represents the matrix P i N2 represents the total number of rows in the matrix P i The total number of columns, Indicates rounding down; S i (x,y) are combined to form the gray value matrix S i .
4. The single-frame 3D reconstruction method based on binocular camera capturing moving objects according to claim 1, characterized in that: The step 3 is specifically as follows: using the Laplace operator to detect the gray value matrix S i The Edge: Among them, L i (x,y) represents the pair S i The value after Laplace operator is applied to (x,y). represents the partial derivative operation symbol, Represents the symbol of the quadratic partial derivative operation, S i (x, y) represents the pixel with coordinates (x, y) in the i-th image in the gray value matrix S i The value in L i (x,y) are combined to form the quadratic partial derivative matrix L i , and for L i Perform zero crossing point detection to obtain the edge point matrix M i .
5. The single-frame 3D reconstruction method based on binocular camera capturing moving objects according to claim 1, characterized in that: The step 4 is specifically as follows: using the Sobel operator to calculate S i (x, y) performs convolution operation and obtains S i (x,y) horizontal gradient magnitude F ix (x, y) and the vertical gradient magnitude F iy (x, y), and use these two gradient magnitudes to calculate the gradient magnitude F of the pixel with coordinates (x, y) in the i-th image i (x,y), the formula is as follows, and F i (x,y) are combined to form the gradient magnitude matrix F i : According to F ix (x,y) and F iy (x,y) calculates the gradient direction θ of the pixel with coordinates (x,y) in the i-th image i (x,y), the formula is as follows, and θ i (x,y) is combined to form the gradient direction matrix θ i :
6. The single-frame 3D reconstruction method based on binocular camera capturing moving objects according to claim 1, characterized in that: The step 5 is: edge point matrix M i For each pixel point on a single side, if the gradient amplitude of the pixel point is the local maximum along the gradient direction, the point is retained as an edge point in the edge point matrix, otherwise the point is regarded as a non-edge point.
7. The single-frame 3D reconstruction method based on binocular camera capturing moving objects according to claim 3, characterized in that: In step 6, the output threshold T is calculated using the following formula: Among them, ∑ x,y F i (x,y) represents the sum of the gradient magnitudes in the gradient magnitude matrix, ∑ x,y P i (x,y) represents the grayscale space matrix P i The sum of the gray values, W i Indicates the number of edge points in the corresponding latest edge point matrix; represents the weight of the gradient amplitude, ω represents the weight of the gray value, and the expression is: where σ F represents the standard deviation of the gradient amplitude, σ P Represents the standard deviation of grayscale values.
8. The single-frame 3D reconstruction method based on binocular camera capturing moving objects according to claim 1, characterized in that: In step 7, the edge point matrix of the current frame is coupled with the latest edge point matrix corresponding to the previous frame as follows: if the pixel point with coordinates (x, y) in the i-th image of the current frame has a value of 1 in the edge point matrix of the current frame, and the pixel point with corresponding coordinates in the i-th image of the previous frame has a value of 1 in the latest edge point matrix corresponding to the i-th image of the previous frame, then the pixel point with coordinates (x, y) in the i-th image of the current frame has a value of 1 in the edge point matrix after coupling, otherwise the pixel point with coordinates (x, y) in the i-th image of the current frame has a value of 0 in the edge point matrix after coupling.
9. The single-frame 3D reconstruction method based on binocular camera capturing moving objects according to claim 1, characterized in that: In step 8, the spatial direction amplitude reconstruction matrix is generated according to the gradient amplitude matrix and the gradient direction matrix as follows: Step 8.1: Use the weighted average method to fit the gradient amplitudes of the two images to obtain the weighted average gradient amplitude F3(x,y): F3(x,y)=α·F1(x,y)+(1-α)·F2(x,y) Among them, α is the amplitude weight, F1(x, y) is the gradient amplitude of the gradient amplitude matrix of the image corresponding to the left monocular camera at the position (x, y), F2(x, y) is the gradient amplitude of the gradient amplitude matrix of the image corresponding to the right monocular camera at the position (x, y), and F3(x, y) is combined to form the gradient amplitude matrix F3; Step 8.2: Convert the values of θ1(x,y) and θ2(x,y) to v1 and v2 using the following formula: Wherein, v1 is the direction vector element of the element θ1(x, y) in the gradient direction matrix θ1 at the corresponding position in the matrix, and v2 is the direction vector element of the element θ2(x, y) in the gradient direction matrix θ2 at the corresponding position in the matrix; Step 8.3: Apply the weighted average method to the two direction vectors using the following formula to calculate the weighted average direction vector v3: v3=βv1+(1-β)v2 Among them, β is the direction weight, Step 8.4: Convert v3 to an angle value according to the following formula: θ3(x,y)=atan2(v 3a ,v 3b ) Among them, atan2 is a function that returns the angle between the vector from the origin to the point (a, b) and the positive direction of the a-axis, v 3a represents the horizontal coordinate of vector v3 in the rectangular coordinate system of the aOb plane, v 3b Represents the ordinate of vector v3 in the rectangular coordinate system of the aOb plane; Step 8.5: Construct a new gradient direction matrix θ3, where θ3(x, y) is the angle value of the corresponding position of the matrix, and combine θ3(x, y) to form θ3; Step 8.6: Fusion the new gradient direction matrix θ3 and gradient amplitude matrix F3 with the binocular camera position matrix [E|H] to generate a spatial direction amplitude reconstruction matrix, and then restore this reconstruction matrix to the world coordinate system to obtain the subject's motion amplitude and direction.
