A large scene high-speed depth calculation method

Abstract

The application discloses a large-scene high-speed depth calculation method, comprising the following steps: firstly, calibrating binocular cameras and a depth camera respectively, and determining the internal parameters and external parameters of the binocular cameras and the depth camera; pre-processing image data collected by the binocular cameras; using a fast positioning algorithm to quickly scan and analyze a large scene in the image data, so as to position a target region; mapping the positioned target region to a pixel coordinate system of the depth camera; and using the depth camera to calculate the depth of points in the target region mapped to the pixel coordinate system of the depth camera. The method realizes fast acquisition of depth information of a target region in a large scene by using a phased strategy, combining fast positioning of a target region and depth calculation technology.

Description

Technical Field

[0001] This invention relates to the field of computer vision and 3D reconstruction technology, specifically to a high-speed depth calculation method for large scenes. Background Technology

[0002] In the fields of computer vision and image processing, full 3D reconstruction of large scenes is a challenging task. Traditional 3D reconstruction methods typically require global optimization of the entire scene, and the computational complexity increases exponentially with the scene size. This results in slow processing speeds for large scenes, limiting real-time performance and efficiency, especially for large-scale scenes. Furthermore, traditional methods can be affected by environmental changes and occlusion, leading to low reconstruction accuracy or failure.

[0003] To address these issues, several improved methods for large-scale 3D reconstruction have been proposed in recent years, such as parallel computing-based algorithms and GPU-accelerated methods. However, these methods still have certain limitations, such as high hardware requirements and high implementation complexity.

[0004] Therefore, it is necessary to develop a faster and more efficient method to solve the problem of 3D reconstruction of large scenes. Summary of the Invention

[0005] This invention addresses the shortcomings of existing technologies by proposing a high-speed depth calculation method for large scenes, aiming to solve the problem of slow speed in processing large scenes using traditional methods. This method, through a phased strategy, combines rapid target region localization and depth calculation techniques to quickly acquire depth information of target regions in large scenes.

[0006] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:

[0007] A high-speed depth calculation method for large-scale scenes includes the following steps:

[0008] S100. Calibrate the stereo camera and depth camera respectively, and determine the intrinsic and extrinsic parameter matrices of the stereo camera and depth camera.

[0009] S200: Preprocess the image data acquired by the binocular camera;

[0010] S300: Use a fast positioning algorithm to quickly scan and analyze large scenes in image data, thereby locating the target area;

[0011] S400: Map the located target area to the pixel coordinate system of the depth camera;

[0012] S500: Calculates the depth of points within the target region mapped to the pixel coordinate system of the depth camera using a depth camera.

[0013] Preferably, the method for determining the intrinsic parameter matrix and the extrinsic parameter matrix is ​​as follows:

[0014] S101. Select a set of fixed calibration points, which can be observed simultaneously within the field of view of the binocular camera and the depth camera;

[0015] S102. By capturing images of spatial calibration points, the intrinsic parameter matrices K1 and K2 of the binocular camera and the intrinsic parameter matrix K3 of the depth camera are determined.

[0016] S103. By obtaining the mapping relationship between each camera pixel and the corresponding real space coordinates, and combining the intrinsic parameter matrices K1, K2, and K3, calculate the extrinsic parameter matrices [R1|t1] and [R2|t2] of the stereo camera and the extrinsic parameter matrix [R3|t3] of the depth camera.

[0017] Preferably, in step S103, the perspective model and the known coordinates of the calibration points are used to solve the extrinsic parameter matrix.

[0018] Preferably, the method for solving the extrinsic parameter matrix is ​​as follows:

[0019] Suppose we have the coordinates of four points in three-dimensional space that are not on the same plane, and their corresponding pixel points:

[0020] P word1 =(X1,Y1,Z1),p pixel1 =(x1,y1,1)

[0021] P word2 =(X2,Y2,Z2),p pixel2 =(x2,y2,1)

[0022] P word3 =(X3,Y3,Z3),p pixel3 =(x3,y3,1)

[0023] P word4 =(X4,Y4,Z4),p pixel4 =(x4,y4,1)

[0024] Based on the perspective model, the following equations can be established:

[0025]

[0026] Where s is the scaling factor, K is the intrinsic parameter matrix, and R and t are the camera's rotation matrix and translation vector, respectively, the extrinsic parameter matrix [R|t] can be calculated.

[0027] Preferably, the image data preprocessing method includes noise reduction and grayscale processing.

[0028] Preferably, in step 300, the method for locating the target area is as follows:

[0029] S301. Extract feature points from the image data acquired by the binocular camera using feature point descriptors;

[0030] S302. Combine the intrinsic and extrinsic parameter matrices of the binocular camera to calculate the world coordinates that match the feature points;

[0031] S303. Calculate the depth information of the missing region using bilinear interpolation.

[0032] S304. Combine depth information and image to generate point cloud, aggregate the point cloud into target region according to user-defined conditions, and output.

[0033] Preferably, step S301 further includes feature point matching between the binocular cameras. The feature point matching method is to apply geometric verification to identify and remove erroneous matches. By calculating the fundamental matrix or essential matrix, the method checks whether each pair of matching points follows the corresponding geometric constraints and eliminates inconsistent matches.

[0034] Preferably, in step S302, the method for calculating the world coordinates of feature point matching is as follows:

[0035] First, let:

[0036]

[0037] The problem of 3D depth calculation is transformed into solving equation M. 4*3 X 3*1 =N 4*1 The problem involves the X vector, which contains the world coordinates of the feature point to be measured, denoted as (x, y, z), and its matrix expression is as follows:

[0038]

[0039]

[0040] X represents the world coordinates of the measured point. The world coordinates of the feature point in the world coordinate system are obtained by solving the least squares method.

[0041] Preferably, in step S301, the selected feature descriptor has scale rotation invariance.

[0042] Preferably, in step S400, the mapping method for the target region is as follows:

[0043] S401. Use the binocular camera intrinsic parameter matrix to convert pixel coordinates into points in the binocular camera coordinate system;

[0044] S402. Use the stereo camera extrinsic matrix to convert the stereo camera coordinates to world coordinates;

[0045] S403. Convert world coordinates to points in the depth camera coordinate system using the binocular camera extrinsic parameter matrix;

[0046] S404. Use the stereo camera intrinsic parameter matrix to convert the stereo camera coordinates into depth camera pixel coordinates.

[0047] Preferably, in step S500, depth calculation is performed using the principle of structured light.

[0048] This invention has the following characteristics and beneficial effects:

[0049] By employing the above technical solution, the method of this invention not only provides an effective way to quickly acquire depth information of target areas in large scenes, but also achieves significant improvements in processing efficiency and accuracy. Compared with traditional methods, the method proposed in this invention does not require a large amount of time and computing resources to perform global optimization of the entire scene. Instead, it achieves efficient depth information acquisition by quickly locating the target area and using a depth camera for depth calculation. Therefore, this invention is of particular importance for application scenarios with high real-time requirements, such as real-time obstacle detection in autonomous driving systems and rapid target tracking in intelligent monitoring systems. In addition, this method can also provide a more suitable solution for faster and more accurate object localization and 3D reconstruction in the field of industrial automation. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 A complete flowchart is provided for embodiments of this application.

[0052] Figure 2 A scene diagram provided for an embodiment of this application.

[0053] Figure 3 This is a flowchart illustrating the camera calibration process provided in an embodiment of this application.

[0054] Figure 4 A flowchart illustrating the large-scene area positioning process provided in this application embodiment.

[0055] Figure 5 A flowchart illustrating the target region mapping process provided in this application embodiment. Detailed Implementation

[0056] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0057] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0058] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art will understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0059] The invention can be applied in various fields such as autonomous driving, intelligent monitoring, industrial automation, virtual reality and augmented reality.

[0060] The overall process of implementing the method of this invention is as follows: Figure 1 As shown in the accompanying drawings. The following will be illustrated in conjunction with the attached figures. Figure 2 The paint loading system illustrated herein provides a complete and clear description of the implementation method, technical means, and operation process in this embodiment of the invention. In this embodiment, a short-focal-length binocular camera and a long-focal-length depth camera are used in combination. The entire loading scene is large and contains multiple smaller scenes. Simultaneously, the system requires a certain loading speed, therefore it is essential to quickly and accurately determine the stacking area range and rapidly reconstruct the depth information of the objects in the stacking area before transmitting it to the motion system for handling. See also... Figure 1 The figure shows a high-speed depth calculation method for large scenes according to the present invention, which includes at least the following steps:

[0061] Step S100: Camera Calibration:

[0062] Calculate the intrinsic parameter matrices K1, K2 and extrinsic parameter matrices [R1|t1], [R2|t2] of the stereo camera, and the intrinsic parameter matrix K3 and extrinsic parameter matrix [R3|t3] of the depth camera, and determine the transformation matrix between the image coordinate systems of each camera;

[0063] Step S200: Image preprocessing:

[0064] Preprocessing is performed on the image data acquired by the binocular camera to improve the accuracy and efficiency of subsequent processing;

[0065] Step S300: Large scene area positioning:

[0066] A fast localization algorithm is used to quickly scan and analyze a large scene to identify areas that may contain the target region. This step aims to narrow down the search area in the shortest possible time, thereby improving the efficiency of subsequent depth calculations.

[0067] Step S400: Target Region Mapping:

[0068] The target area determined by the binocular camera system will be mapped to the pixel coordinate system of the depth camera.

[0069] Step S500: Small-area depth detection:

[0070] The depth of points within the target area mapped in step S400 is calculated using a depth camera.

[0071] In this embodiment, due to the small focal length, the depth accuracy of the points calculated by region localization is relatively low. Therefore, after initially screening the point depths in the large scene, all points close to the minimum depth value are retained and merged. The merged points are represented by a set of region coordinates including these points. Next, these region coordinates are transformed into the depth camera coordinate system through target region mapping. To perform accurate depth calculation, the depth camera is fixed to the end of a robotic arm, and the robotic arm is controlled to move the depth camera to the corresponding target region small scene for depth calculation. In this way, efficient and accurate target region depth information can be obtained in large scenes. To verify the effectiveness of this method, depth information is calculated at 10m... 2 50m 2 100m 2 In the scenario, experiments were conducted and data were recorded using a pure structured light scheme, a pure binocular vision scheme, and the proposed scheme, as shown in the table below, which fully demonstrates the feasibility of the proposed method.

[0072]

[0073] like Figure 3As shown, step S200 includes the following sub-steps:

[0074] Step S101: Determine the spatial calibration point:

[0075] Determine spatial calibration points for camera calibration. These calibration points are fixed in position in the scene, can be observed simultaneously by two monocular cameras and a depth camera, and are not coplanar.

[0076] Step S102: Calculate the camera intrinsic parameter matrix:

[0077] By capturing images of spatial calibration points, the intrinsic parameter matrices K1 and K2 of the stereo camera and the intrinsic parameter matrix K3 of the depth camera can be determined using a series of methods. In this embodiment, the intrinsic parameter matrices K1, K2, and K3 are obtained using the Zhang Zhengyou calibration method.

[0078] Step S103: Calculate the camera extrinsic parameter matrix:

[0079] Given the extrinsic parameters [R1|t1] and [R2|t2] of the stereo camera and the extrinsic parameter matrix [R3|t3] of the depth camera, the calculation process for the extrinsic parameter matrix of a single camera is as follows:

[0080] Suppose we have the coordinates of four points in three-dimensional space that are not on the same plane, and their corresponding pixel points:

[0081] P word1 =(X1,Y1,Z1),p pixel1 =(x1,y1,1)

[0082] P word2 =(X2,Y2,Z2),p pixel2 =(x2,y2,1)

[0083] P word3 =(X3,Y3,Z3),p pixel3 =(x3,y3,1)

[0084] P word4 =(X4,Y4,Z4),p pixel4 =(x4,y4,1)

[0085] Based on the perspective model, the following equations can be established:

[0086]

[0087] Where s is the scaling factor, K is the intrinsic parameter matrix, and R and t are the camera's rotation matrix and translation vector, respectively, the extrinsic parameter matrix [R|t] can be calculated.

[0088] Furthermore, such as Figure 4As shown, step S300 includes the following sub-steps:

[0089] Step S301: Feature point extraction from the stereo camera:

[0090] This step extracts feature points from the two images of the binocular camera system. A suitable feature point descriptor is selected to enable the computer to identify and describe key regions in the images that remain unchanged across multiple viewpoints, making them suitable for subsequent matching. In this embodiment, no epipolar correction is performed between the binocular cameras, so the ORB descriptor is used.

[0091] Step S302: Feature point matching between binocular cameras:

[0092] By comparing feature descriptors, matching feature point pairs are found between two images. The coordinates of the same object point in the two camera spaces are stored one-to-one, denoted as Point. L (u L ,v L Point R (u R ,v R This embodiment uses K-Nearest Neighbors for feature point matching;

[0093] Step S303: Geometric verification between feature points:

[0094] To improve matching accuracy, geometric verification is applied to identify and remove erroneous matches. By calculating the fundamental or essential matrices and using them to check whether each pair of matching points follows the corresponding geometric constraints, inconsistent matches can be eliminated.

[0095] Step S304: Depth estimation of feature points:

[0096] make

[0097]

[0098] The problem of 3D depth calculation is transformed into solving equation M. 4*3 X 3*1 =N 4*1 The problem involves a vector X containing the world coordinates of the feature point to be measured, denoted as (x, y, z), where the matrix...

[0099]

[0100]

[0101] X represents the world coordinates of the measured point. The world coordinates of the feature point in the world coordinate system are solved by the least squares method, which gives the depth information of the point. The comprehensible depth information can complete the information of the Z axis.

[0102] Step S305: Point cloud interpolation:

[0103] After obtaining the initial depth information and the corresponding 3D points (point cloud), there may be holes or missing regions in the point cloud, especially in areas where matching points are sparse or disparity calculation is difficult. This embodiment uses bilinear interpolation to calculate the depth of unknown points;

[0104] Step S306: Area positioning:

[0105] Based on the actual situation, a suitable algorithm is selected to identify the required point cloud region, and the region is output for subsequent calculations. In this embodiment, the top of the palletized items needs to be captured; therefore, the region of points with depths near the minimum point and depth differences not exceeding 5mm is selected as the area for small-range depth detection.

[0106] like Figure 5 As shown, step S400 includes the following sub-steps:

[0107] Step S401: Convert pixel coordinates to points in the camera coordinate system:

[0108] For each point in camera 1, use the intrinsic parameter matrix K1 to transform them from pixel coordinates to points in the camera coordinate system of camera 1. If the points are represented in homogeneous coordinates, the transformation formula is: P camera1 =K1 -1 *p pixel1 ;

[0109] Step S402: Convert camera coordinates to points in the world coordinate system:

[0110] Using the extrinsic parameter matrix of camera 1, each point is transformed from the camera coordinate system to the world coordinate system. The transformation formula is: Pworld = R1 T *(pcamera1-t1);

[0111] Step S403: Transform the world coordinate system into points in the second camera coordinate system:

[0112] Using the extrinsic parameter matrix of camera 2, each point is transformed from the world coordinate system to the camera coordinate system of camera 2. The transformation formula is: P camera2 =R2*P world +t2;

[0113] Step S404: Project the points in the second camera coordinate system onto the pixel coordinate system of camera 2:

[0114] Using the intrinsic parameter matrix of camera 2, each point is transformed from the camera coordinate system of camera 2 to the image coordinate system of camera 2. The transformation formula is: P pixel1 =K2*P camera2 .

[0115] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions, and variations can be made to these embodiments, including components, without departing from the principles and spirit of the present invention, and these variations still fall within the protection scope of the present invention.

Claims

1. A high-speed depth calculation method for large-scale scenes, characterized in that, Includes the following steps: S100. Calibrate the stereo camera and depth camera respectively, and determine the intrinsic and extrinsic parameter matrices of the stereo camera and depth camera. S200: Preprocess the image data acquired by the binocular camera; S300: Use a fast positioning algorithm to quickly scan and analyze large scenes in image data, thereby locating the target area; S301. Extract feature points from the left and right images captured by the stereo camera using feature point descriptors; S302. Perform feature point matching between the binocular cameras and apply geometric verification to remove incorrect matches; S303. Combine the intrinsic and extrinsic parameter matrices of the binocular camera to calculate the world coordinates of the matching feature points; S304. Complete the depth information of the missing region using bilinear interpolation. S305. Generate point clouds by combining depth information and images; S306. Aggregate the point cloud into a target region according to the conditions set by the user and output it; S400: Map the located target area to the pixel coordinate system of the depth camera; Step S400 includes the following sub-steps: Step S401: Convert pixel coordinates to points in the camera coordinate system: For each point in camera 1, use the intrinsic parameter matrix Convert them from pixel coordinates to points in the camera coordinate system of camera 1, where the points are represented in homogeneous coordinates. The conversion formula is: ; Step S402: Convert camera coordinates to points in the world coordinate system: Using the extrinsic parameter matrix of camera 1, each point is transformed from the camera coordinate system to the world coordinate system. The transformation formula is as follows: ; Step S403: Transform the world coordinate system into points in the second camera coordinate system: Using the extrinsic parameter matrix of camera 2, each point is transformed from the world coordinate system to the camera coordinate system of camera 2. The transformation formula is as follows: ; Step S404: Project the points in the second camera coordinate system onto the pixel coordinate system of camera 2: Using the intrinsic parameter matrix of camera 2, each point is transformed from the camera coordinate system of camera 2 to the image coordinate system of camera 2. The transformation formula is as follows: ; S500: Calculates the depth of points within the target region mapped to the pixel coordinate system of the depth camera using a depth camera.

2. The method for high-speed depth calculation in large scenes according to claim 1, characterized in that, The method for determining the intrinsic parameter matrix and the extrinsic parameter matrix is ​​as follows: S101. Select a set of fixed calibration points, which can be observed simultaneously within the field of view of the binocular camera and the depth camera; S102. By capturing images of spatial calibration points, the intrinsic parameter matrix of the stereo camera is determined. , Intrinsic parameter matrix of depth camera ; S103. By obtaining the mapping relationship between each camera pixel and its corresponding real-world coordinates, combined with the intrinsic parameter matrix... , , Solve the extrinsic parameter matrix of the stereo camera. , extrinsic matrix of depth camera .

3. The method for high-speed depth calculation in large scenes according to claim 2, characterized in that, In step S103, the perspective model and the known coordinates of the calibration points are used to solve the extrinsic parameter matrix.

4. The method for high-speed depth calculation in large scenes according to claim 3, characterized in that, The method for solving the extrinsic parameter matrix is ​​as follows: Suppose we have the coordinates of four points in three-dimensional space that are not on the same plane, and their corresponding pixel points: ； Based on the perspective model, the following equations can be established: ； in, It is a scaling factor. It is an intrinsic parameter matrix. and The extrinsic parameter matrix can be calculated from the camera's rotation matrix and translation vector. .

5. The method for high-speed depth calculation in large scenes according to claim 1, characterized in that, The image data preprocessing methods include noise reduction and grayscale processing.

6. The method for high-speed depth calculation in large scenes according to claim 5, characterized in that, Step S302 also includes feature point matching between the binocular cameras. The feature point matching method is to apply geometric verification to identify and remove erroneous matches. By calculating the fundamental matrix or essential matrix, it is checked whether each pair of matching points follows the corresponding geometric constraints and inconsistent matches are eliminated.

7. The method for high-speed depth calculation in large scenes according to claim 5, characterized in that, In step S303, the method for calculating the world coordinates of feature point matching is as follows: First, let: ； Transform the 3D depth calculation problem into solving equations The problem is that the vector contains the world coordinates of the feature point to be measured, denoted as , and the matrix expression is as follows: ； ， ； Given the world coordinates of the measured point, the world coordinates of the feature point in the world coordinate system are solved using the least squares method.

8. A high-speed depth calculation method for large scenes according to claim 5 or 6, characterized in that, In step S301, the selected feature descriptor has scale rotation invariance.

9. The method for high-speed depth calculation in large scenes according to claim 1, characterized in that, In step S500, depth calculation is performed using the principle of structured light.

Images