Large-scene high-speed depth calculation method

Abstract

The invention discloses a large-scene high-speed depth calculation method, which comprises the following steps of: firstly, respectively calibrating a binocular camera and a depth camera, and determining internal parameters and external parameters of the binocular camera and the depth camera; preprocessing the image data acquired by the binocular camera; rapidly scanning and analyzing a large scene in the image data by using a rapid positioning algorithm so as to position a target area; mapping the positioned target area to a pixel coordinate system of a depth camera; depth calculation is performed using the depth camera on points within the target area mapped to the pixel coordinate system of the depth camera. According to the method, the depth information of the target area is rapidly acquired in a large scene by combining the rapid positioning of the target area and the depth calculation technology through a staged strategy.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and three-dimensional reconstruction, and in particular to a high-speed depth calculation method for a large scene. Background Art

[0002] In the field of computer vision and image processing, full 3D reconstruction of large scenes is a challenging task. Traditional 3D reconstruction methods usually require global optimization of the entire scene, and the computational complexity increases exponentially with the increase in the size of the scene, which leads to the problem of slow processing of large scenes, limiting real-time performance and efficiency, especially for large scenes. In addition, traditional methods may also be affected by environmental changes and occlusions, resulting in low reconstruction accuracy or failure.

[0003] In order to solve these problems, some improved methods for large-scale 3D reconstruction have been proposed in recent years, such as algorithms based on parallel computing and methods based on GPU acceleration. However, these methods still have certain limitations, such as high requirements for hardware equipment and high implementation complexity.

[0004] Therefore, it is necessary to develop a faster and more efficient method to solve the 3D reconstruction problem of large scenes. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention proposes a high-speed depth calculation method for large scenes, aiming to solve the problem that the traditional method is slow when processing large scenes. The method uses a phased strategy, combines rapid positioning of the target area and depth calculation technology, and realizes the rapid acquisition of depth information of the target area in a large scene.

[0006] In order to solve the above technical problems, the technical solution of the present invention is:

[0007] A high-speed depth calculation method for a large scene includes the following steps:

[0008] S100, calibrating the binocular camera and the depth camera respectively, and determining the intrinsic parameter matrix and the extrinsic parameter matrix of the binocular camera and the depth camera;

[0009] S200, preprocessing the image data collected by the binocular camera;

[0010] S300, using a fast positioning algorithm to quickly scan and analyze a large scene in the image data, thereby locating a target area;

[0011] S400, mapping the located target area to the pixel coordinate system of the depth camera;

[0012] S500: Utilize a depth camera to perform depth calculation on points within a target area mapped to a pixel coordinate system of the depth camera.

[0013] Preferably, the method for determining the internal parameter matrix and the external parameter matrix is:

[0014] S101, selecting a set of calibration points at fixed positions, wherein the calibration points can be simultaneously observed within the field of view of the binocular camera and the depth camera;

[0015] S102, by taking images of the spatial calibration points, thereby determining the intrinsic parameter matrices K1 and K2 of the binocular camera and the intrinsic parameter matrix K3 of the depth camera;

[0016] S103. By obtaining the mapping relationship between each camera pixel and the corresponding real space coordinates, combined with the intrinsic parameter matrices K1, K2, and K3, the extrinsic parameter matrices [R1|t1], [R2|t2] of the binocular camera and the extrinsic parameter matrix [R3|t3] of the depth camera are solved.

[0017] Preferably, in step S103, the perspective model and known coordinates of calibration points are used to solve the extrinsic parameter matrix.

[0018] Preferably, the method for solving the external parameter matrix is:

[0019] Assume that in three-dimensional space, there are coordinates of four points that are not on the same plane, and their corresponding pixels:

[0020] P word1 =(X1,Y1,Z1),p pixel1 =(x1,y1,1)

[0021] P word2 =(X2,Y2,Z2),p pixel2 =(x2,y2,1)

[0022] P word3 =(X3,Y3,Z3),p pixel3 =(x3,y3,1)

[0023] P word4 =(X4,Y4,Z4),p pixel4 =(x4,y4,1)

[0024] According to the perspective model, the following equation can be established:

[0025]

[0026] Among them, s is the scale factor, K is the intrinsic parameter matrix, R and t are the rotation matrix and translation vector of the camera respectively, and the extrinsic parameter matrix [R|t] can be calculated.

[0027] Preferably, the image data preprocessing method includes denoising and grayscale processing.

[0028] Preferably, in step 300, the method for locating the target area is:

[0029] S301, extracting feature points in image data collected by a binocular camera through feature point descriptors;

[0030] S302, combining the internal parameter matrix and the external parameter matrix of the binocular camera to calculate the world coordinates matching the feature points;

[0032] S303, calculating the depth information of the missing area by a bilinear interpolation method;

[0033] S304: Generate a point cloud by combining the depth information and the image, aggregate the point cloud into a target area according to the conditions set by the user, and output it.

[0034] Preferably, step S301 also includes feature point matching between binocular cameras, and the feature point matching method is to apply geometric verification, identify and remove erroneous matches, calculate the basic matrix or the essential matrix, check whether each pair of matching points complies with the corresponding geometric constraints, and eliminate inconsistent matches.

[0035] Preferably, in step S302, the method for calculating the world coordinates of the feature point matching is:

[0036] First, suppose:

[0037]

[0038] Convert the three-dimensional depth calculation problem into solving equation M 4*3 X 3*1 =N 4*1 The content of the X vector is the world coordinates of the feature point that needs to be measured, recorded as (x, y, z), and the matrix expression is as follows:

[0039]

[0040]

[0041] X is the world coordinate of the measured point, and the world coordinate of the feature point in the world coordinate system is solved by the least squares method.

[0042] Preferably, in step S301, the selected feature descriptor has scale rotation invariance.

[0043] Preferably, in step S400, the target area is mapped by:

[0044] S401, using the binocular camera intrinsic parameter matrix to convert the pixel coordinates into points in the binocular camera coordinate system;

[0045] S402, using the binocular camera extrinsic matrix to convert the binocular camera coordinates into world coordinates;

[0046] S403, converting the world coordinates into points in the depth camera coordinate system through the binocular camera extrinsic matrix;

[0047] S404: Use the stereo camera intrinsic parameter matrix to convert the stereo camera coordinates into depth camera pixel coordinates.

[0048] Preferably, in step S500, depth calculation is performed using the principle of structured light.

[0049] The present invention has the following characteristics and beneficial effects:

[0050] By adopting the above technical solution, the method of the present invention not only provides an effective way to quickly obtain the depth information of the target area in a large scene, but also achieves significant improvements in processing efficiency and accuracy. Compared with the traditional method, the method proposed in the present invention does not need to spend a lot of time and computing resources to globally optimize the entire scene, but achieves efficient depth information acquisition by quickly locating the target area and using a depth camera to perform depth calculation. Therefore, the present invention is of particular importance for application scenarios with high real-time requirements, such as real-time obstacle detection in autonomous driving systems, fast target tracking in intelligent monitoring systems, etc. In addition, the method can also provide a more suitable solution for providing faster and more accurate object positioning and three-dimensional reconstruction functions in the field of industrial automation. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0052] Figure 1 A complete flow chart is provided for the embodiments of the present application.

[0053] Figure 2 A scene diagram provided for an embodiment of the present application.

[0054] Figure 3 A camera calibration flowchart provided in an embodiment of the present application.

[0055] Figure 4 A flowchart for large scene area positioning provided in an embodiment of the present application.

[0056] Figure 5A target area mapping flow chart provided for an embodiment of the present application. DETAILED DESCRIPTION

[0057] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0058] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and the like are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first", "second", and the like may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0059] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood by specific circumstances.

[0060] The application scenarios of the present invention can be in multiple fields such as autonomous driving, intelligent monitoring, industrial automation, virtual reality and augmented reality.

[0061] The overall process of the method of the present invention is as follows Figure 1 The following will be combined with the accompanying drawings to Figure 2 The paint feeding system shown is used as an object to provide a complete and clear description of the implementation method, technical means, and operation process in the embodiment of the present invention. In this embodiment, a short-focal-length binocular camera and a long-focal-length depth camera are used in combination. The entire feeding scene is large and has multiple small scenes. At the same time, the system has certain requirements for the feeding speed, so the stacking area must be determined quickly and accurately, and the depth information of the objects in the stacking area must be restored at high speed and then transmitted to the motion system for transportation. See Figure 1 , which is a large scene high-speed depth calculation method of the present invention, at least comprising the following steps:

[0062] Step S100: Camera calibration:

[0063] Calculate the intrinsic parameter matrix K1, K2 and extrinsic parameter matrix [R1|t1], [R2|t2] of the binocular camera, as well as the intrinsic parameter matrix K3 and extrinsic parameter matrix [R3|t3] of the depth camera, and determine the transformation matrix between the image coordinate systems of each camera;

[0064] Step S200: Image preprocessing:

[0065] Pre-process the image data collected by the binocular camera to improve the accuracy and efficiency of subsequent processing;

[0066] Step S300: Large scene area positioning:

[0067] Use fast positioning algorithms to quickly scan and analyze large scenes to determine areas that may contain target areas. This step aims to narrow the search range in the shortest possible time, thereby improving the efficiency of subsequent depth calculations;

[0068] Step S400: Target area mapping:

[0069] Map the target area determined by the binocular camera system to the pixel coordinate system of the depth camera;

[0070] Step S500: Small range depth detection:

[0071] The depth camera is used to calculate the depth of points within the target area mapped in step S400.

[0072] In this embodiment, due to the small focal length, the depth accuracy of the points calculated by the regional positioning is relatively low. Therefore, after the preliminary screening of the point depths in the large scene, all points close to the minimum depth value are retained and merged. The merged points will be represented by a set of regional coordinates including these points. Next, these regional coordinates are converted to the depth camera coordinate system through target area mapping. In order to perform accurate depth calculations, the depth camera is fixed at the end of the robotic arm, and the robotic arm is controlled to move the depth camera to the corresponding target area small scene for depth calculation. In this way, efficient and accurate acquisition of target area depth information can be achieved in large scenes. To verify the effectiveness of this method, the depth camera was measured at 10m 2 、50m 2 、100m 2 In the scenario, experiments were conducted using the pure structured light solution, the pure binocular vision solution, and this solution, and the data were recorded, as shown in the following table, which fully proves the feasibility of this method.

[0073]

[0074] like Figure 3 As shown, the step S200 includes the following sub-steps:

[0075] Step S101: Determine the spatial calibration point:

[0076] Determine the spatial calibration points used for camera calibration. These calibration points are fixed in the scene, can be observed by two monocular cameras and the depth camera at the same time, and are not coplanar;

[0077] Step S102: Calculate the camera intrinsic parameter matrix:

[0078] By taking images of the spatial calibration points, the intrinsic parameter matrices K1 and K2 of the binocular camera and the intrinsic parameter matrix K3 of the depth camera can be determined through a series of methods. In this embodiment, the intrinsic parameter matrices K1, K2, and K3 are obtained by the Zhang Zhengyou calibration method;

[0079] Step S103: Calculate the camera extrinsic matrix:

[0080] Determine the extrinsic matrix [R1|t1], [R2|t2] of the binocular camera and the extrinsic matrix [R3|t3] of the depth camera. The calculation process of the extrinsic matrix of a single camera is as follows:

[0081] Assume that in three-dimensional space, there are coordinates of four points that are not on the same plane, and their corresponding pixels:

[0082] P word1 =(X1,Y1,Z1),p pixel1 =(x1,y1,1)

[0083] P word2 =(X2,Y2,Z2),p pixel2 =(x2,y2,1)

[0084] P word3 =(X3,Y3,Z3),p pixel3 =(x3,y3,1)

[0085] P word4 =(X4,Y4,Z4),p pixel4 =(x4,y4,1)

[0086] According to the perspective model, the following equation can be established:

[0087]

[0088] Among them, s is the scale factor, K is the intrinsic parameter matrix, R and t are the rotation matrix and translation vector of the camera respectively, and the extrinsic parameter matrix [R|t] can be calculated.

[0089] Further, such as Figure 4As shown, the step S300 includes the following sub-steps:

[0090] Step S301: Feature point extraction of binocular camera:

[0091] In this step, feature points are extracted from the two images of the binocular camera system. Selecting appropriate feature point descriptors enables the computer to identify and describe key areas in the image, which remain unchanged under multiple viewing angles and are suitable for the subsequent matching process. In this embodiment, no epipolar correction is performed between the binocular cameras, so the ORB descriptor is used;

[0092] Step S302: Feature point matching between binocular cameras:

[0093] By comparing the feature descriptors, matching feature point pairs are found between the two images, and the coordinates of the two camera spaces corresponding to the same object point are stored one by one, recorded as Point L (u L ,v L ),Point R (u R ,v R ). This embodiment uses K-Nearest Neighbors to perform feature point matching;

[0094] Step S303: Geometric verification between feature points:

[0095] In order to improve the accuracy of matching, geometric checks are applied to identify and remove incorrect matches. Inconsistent matches can be excluded by calculating the fundamental matrix or the essential matrix and using them to check whether each pair of matching points follows the corresponding geometric constraints;

[0096] Step S304: Depth estimation of feature points:

[0097] make

[0098]

[0099] Convert the three-dimensional depth calculation problem into solving equation M 4*3 X 3*1 =N 4*1 The problem is that the content of the X vector is the world coordinates of the feature point that needs to be measured, recorded as (x, y, z), where the matrix

[0100]

[0101]

[0102] X is the world coordinate of the measured point. The world coordinate of the feature point in the world coordinate system is solved by the least square method, that is, the depth information of the point is obtained. The understandable depth information can complement the information of the Z axis;

[0103] Step S305: Point cloud interpolation:

[0104] After obtaining the initial depth information and the corresponding 3D points (point cloud), there may be holes or missing areas in the point cloud, especially in areas where the matching points are sparse or the parallax calculation is difficult. This embodiment calculates the depth of the unknown point by bilinear interpolation;

[0105] Step S306: Area positioning:

[0106] According to the actual situation, select the appropriate algorithm to select the required point cloud area, and output the area for subsequent calculation. In this embodiment, the top of the stacked items needs to be captured, so the area of ​​the point set with a depth near the minimum point and a depth difference of no more than 5mm is used as the area for small-range depth detection.

[0107] like Figure 5 As shown, the step S400 includes the following sub-steps:

[0108] Step S401: Convert pixel coordinates to points in the camera coordinate system:

[0109] For each point in camera 1, use the intrinsic parameter matrix K1 to transform them from pixel coordinates to points in the camera coordinate system of camera 1. If the points are expressed in homogeneous coordinates, the conversion formula is: camera1 =K1 -1 *p pixel1 ;

[0110] Step S402: Convert the camera coordinates to points in the world coordinate system:

[0111] Use the external parameter matrix of camera 1 to transform each point from the camera coordinate system to the world coordinate system. The conversion formula is: Pworld = R1 T *(pcamera1-t1);

[0112] Step S403: Convert the world coordinate system to a point in the second camera coordinate system:

[0113] Use the external parameter matrix of camera 2 to transform each point from the world coordinate system to the camera coordinate system of camera 2. The conversion formula is: camera2 =R2*P world +t2;

[0114] Step S404: Project the points in the second camera coordinate system to the pixel coordinate system of camera 2:

[0115] Using the intrinsic parameter matrix of camera 2, each point is transformed from the camera coordinate system of camera 2 to the image coordinate system of camera 2. The conversion formula is: pixel1 =K2*P camera2 .

[0116] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions and variations of these embodiments including components are made without departing from the principles and spirit of the present invention, and still fall within the scope of protection of the present invention.

Claims

1. A high-speed depth calculation method for a large scene, characterized in that: The steps include: S100, calibrating the binocular camera and the depth camera respectively, and determining the intrinsic parameter matrix and the extrinsic parameter matrix of the binocular camera and the depth camera; S200, preprocessing the image data collected by the binocular camera; S300, using a fast positioning algorithm to quickly scan and analyze a large scene in the image data, thereby locating a target area; S400, mapping the located target area to the pixel coordinate system of the depth camera; S500: Utilize a depth camera to perform depth calculation on points within a target area mapped to a pixel coordinate system of the depth camera.

2. A large scene high-speed depth calculation method according to claim 1, characterized in that: The method for determining the internal parameter matrix and the external parameter matrix is: S101, selecting a set of calibration points at fixed positions, wherein the calibration points can be simultaneously observed within the field of view of the binocular camera and the depth camera; S102, by taking images of the spatial calibration points, thereby determining the intrinsic parameter matrices K1 and K2 of the binocular camera and the intrinsic parameter matrix K3 of the depth camera; S103. By obtaining the mapping relationship between each camera pixel and the corresponding real space coordinates, combined with the intrinsic parameter matrices K1, K2, and K3, the extrinsic parameter matrices [R1|t1], [R2|t2] of the binocular camera and the extrinsic parameter matrix [R3|t3] of the depth camera are solved.

3. A large scene high-speed depth calculation method according to claim 1, characterized in that: In step S103, the perspective model and the known coordinates of the calibration points are used to solve the extrinsic parameter matrix.

4. A large scene high-speed depth calculation method according to claim 3, characterized in that: The solution method of the external parameter matrix is: Assume that in three-dimensional space, there are coordinates of four points that are not on the same plane, and their corresponding pixels: P word1 =(X1,Y1,Z1),p pixel1 =(x1,y1,1) P word2 =(X2,Y2,Z2),p pixel2 =(x2,y2,1) P word3 =(X3,Y3,Z3),p pixel3 =(x3,y3,1) P word4 =(X4,Y4,Z4),p pixel4 =(x4,y4,1) According to the perspective model, the following equation can be established: Among them, s is the scale factor, K is the intrinsic parameter matrix, R and t are the rotation matrix and translation vector of the camera respectively, and the extrinsic parameter matrix [R|t] can be calculated.

5. A large scene high-speed depth calculation method according to claim 1, characterized in that: The image data preprocessing method includes denoising and grayscale processing.

6. A large scene high-speed depth calculation method according to claim 1, characterized in that: In step 300, the method for locating the target area is: S301, extracting feature points in image data collected by a binocular camera through feature point descriptors; S302, combining the internal parameter matrix and the external parameter matrix of the binocular camera to calculate the world coordinates matching the feature points; S303, calculating the depth information of the missing area by a bilinear interpolation method; S304: Generate a point cloud by combining the depth information and the image, aggregate the point cloud into a target area according to the conditions set by the user, and output it.

7. A large scene high-speed depth calculation method according to claim 6, characterized in that: The step S301 also includes feature point matching between binocular cameras. The feature point matching method is to apply geometric verification, identify and remove erroneous matches, calculate the basic matrix or the essential matrix, check whether each pair of matching points complies with the corresponding geometric constraints, and eliminate inconsistent matches.

8. A large scene high-speed depth calculation method according to claim 6, characterized in that: In step S302, the method for calculating the world coordinates of the feature point matching is: First, suppose: Convert the three-dimensional depth calculation problem into solving equation M 4*3 X 3*1 =N 4*1 The content of the X vector is the world coordinates of the feature point that needs to be measured, recorded as (x, y, z), and the matrix expression is as follows: X is the world coordinate of the measured point, and the world coordinate of the feature point in the world coordinate system is solved by the least squares method.

9. A large scene high-speed depth calculation method according to claim 6, characterized in that: In step S301, the selected feature descriptor has scale and rotation invariance.

10. A large scene high-speed depth calculation method according to claim 1, characterized in that: In step S500, depth calculation is performed using the structured light principle.

Images