An unmanned target detection method based on camera fusion solid-state laser radar

Abstract

The application discloses an unmanned target detection method based on camera fusion solid-state laser radar, one camera and one solid-state laser radar are arranged at four positions of front, rear, left and right of a vehicle; the four-position camera and solid-state laser radar are jointly calibrated respectively, and image data and point cloud data are time-space synchronized; the image data and point cloud data are respectively enhanced; neural networks are used to extract image features and point cloud features respectively, then the image features and point cloud features of a single position are fused respectively, and then the fusion information of the four positions is subjected to secondary feature fusion; the extracted fusion features are input into a prediction network to identify the obstacle category, position and pose information under a mine. The application fuses single-azimuth camera and solid-state laser radar information, then fuses multi-azimuth information, and adopts information of a whole four-around view, so that high-precision three-dimensional target detection can be realized.

Description

Technical Field

[0001] This invention relates to a target detection method, specifically to an unmanned vehicle target detection method based on camera fusion solid-state lidar. Background Technology

[0002] With the rapid development of intelligent technology, autonomous driving technology has emerged. However, its safety and reliability have also received increasing attention. It is inseparable from the perception of the environment around the vehicle. Accurate identification of targets and obstacles can help autonomous vehicles to plan paths in advance and take emergency control measures.

[0003] In harsh environments such as underground coal mines and petrochemical plants, with significant dust and water mist and unstable light levels, traditional single cameras or LiDAR are insufficient to meet the requirements. There is an urgent need for camera-LiDAR fusion technology to detect targets. Currently, the most commonly used method is mechanical LiDAR combined with a camera for detection. However, considering cleaning measures, mechanical LiDAR suffers a significant decrease in accuracy due to the cleaning equipment, and its scanning range is limited, making high-precision target detection impossible. Furthermore, the field of view angles of the images and LiDAR acquisition differ considerably, resulting in poor consistency and making it difficult to achieve absolute uniformity. Summary of the Invention

[0004] To address the problems existing in the prior art, this invention provides an unmanned target detection method based on camera fusion solid-state LiDAR, which can unify the field of view of the camera and solid-state LiDAR to achieve high-precision three-dimensional target detection.

[0005] To achieve the above objectives, the technical solution adopted by this invention is: an unmanned target detection method based on camera fusion solid-state lidar, comprising the following steps:

[0006] S1. Install a camera and a solid-state lidar at the front, rear, left, and right of the vehicle, respectively;

[0007] S2. Perform joint calibration on the four position cameras and solid-state LiDAR respectively, and synchronize the image data and point cloud data in time;

[0008] S3. Perform enhancement processing on the image data and point cloud data respectively;

[0009] S4. Use neural networks to extract image features and point cloud features respectively, then fuse the image features and point cloud features at a single location respectively, and then perform secondary feature fusion on the fused information at four locations.

[0010] S5. Input the extracted fusion features into the prediction network to identify the type, location, and pose information of obstacles in the mine.

[0011] Furthermore, in step S1, 32-line solid-state LiDARs are used at the front and rear of the vehicle, and 16-line solid-state LiDARs are used at the left and right of the vehicle. The vertical distance between the camera and the solid-state LiDAR at the same position is 5cm to 15cm, and the cameras and solid-state LiDARs at the four positions are set on the same horizontal plane.

[0012] Furthermore, the method for joint calibration of the four position cameras and solid-state LiDAR in step S2 is as follows: Through manual calibration and alignment, the position transformation matrix of each individual position camera and solid-state LiDAR is obtained; a world coordinate system is established; and the extrinsic parameters of each of the four cameras and solid-state LiDAR are obtained for calibration. The rotation and translation matrices of the world coordinate system and the camera are set to R0 and R1 respectively. c T c The non-homogeneous linear coordinate in the camera coordinate system is X. w X c Then: X c =R c *X w +T c Set the rotation and translation matrices of the solid-state lidar and camera to R, respectively. l T l A point P in a solid-state lidar l The non-homogeneous linear coordinates in the relative coordinate system and the camera coordinate system are X l X c Then: X c =R l *X l +T l .

[0013] Furthermore, the method for time synchronization of image data and point cloud data in step S2 is as follows: The sampling periods for the camera and solid-state LiDAR are set to T... C T L Determine 1 / 2T C >T L 1 / 2T C With T L Merge timestamp data; determine 1 / 2T C <T L , will T C With T L The timestamp data is fused to achieve spatiotemporal synchronization between image data and point cloud data.

[0014] Furthermore, in step S3, the image data enhancement process utilizes the self-attention mechanism of the reconstruction function to preserve the image data structure and reconstruct the image data; the point cloud data enhancement process uses Gaussian filtering to perform Gaussian weighted averaging on the points in the neighborhood of each point, calculates the weights within the neighborhood determined by its covariance matrix, and then averages them with the center point value.

[0015] Furthermore, in step S4, the method for fusing image features and point cloud features at a single location involves setting the feature layer extracted from the image to Nc. i The feature layer extracted from the point cloud is set to Nl. i In the feature extraction layer stage, image features Nc are extracted. i Point cloud features Nl i+2 Perform cross-modal fusion.

[0016] Furthermore, in step S4, the fusion information of the image features and point cloud features at the four single locations are R1, R2, R3, and R4, respectively. The fusion features at the four locations are then fused a second time, R... a =W1*R1+W2*R2+W3*R3+W4*R4, where W1, W2, W3, and W4 are the weight values ​​of the fused features at the four positions in front, behind, left, and right of the camera, respectively.

[0017] Furthermore, in step S5, the prediction head position of the prediction network is set to four dimensions, namely S1(2... i *2 i S2(2) i+1 *2 i+1 S3(2) i+2 *2 i+2 S4(2) i+3 *2 i+3 ), i max (2 i+3 The feature width (≤ output feature width value) is finally mapped to a linear matrix (P1, P2, ..., x, y, z, R), where P represents the category information, x, y, z represent the three-dimensional spatial information, and R represents the rotation angle information.

[0018] Compared with existing technologies, this invention utilizes the fusion of information from a single-directional camera and a solid-state LiDAR, and then fuses information from multiple directions, taking information from all four sides, to achieve high-precision 3D target detection in special environments. Joint calibration of the camera and solid-state LiDAR enables high-precision synchronization of image data and point cloud data. Enhancement algorithms for image and point cloud data remove noise, improving the quality of image and point cloud data. Furthermore, secondary fusion of feature information from multiple perspectives expands the detection range, solving the problem of limited scanning range and improving the ability to express the features of the detected target, thus achieving the goal of high-precision 3D target detection. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the process of the present invention;

[0020] Figure 2 This is a schematic diagram of the arrangement of the camera and solid-state lidar in this invention;

[0021] Figure 3 This is a feature fusion diagram of image information and point cloud information of the present invention. Detailed Implementation

[0022] The invention will now be further described with reference to the accompanying drawings.

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] like Figures 1 to 3 As shown, this invention provides a target detection method for unmanned vehicles based on camera fusion solid-state LiDAR. This embodiment takes a trackless rubber-wheeled vehicle as an example, and the specific detection method is as follows:

[0025] S1: As Figure 2As shown, a camera and a solid-state lidar are installed at four positions on the trackless rubber-tired vehicle: front, rear, left, and right. The front and rear lidars are both 32-line lidars, placed in the middle of the vehicle. The left and right lidars are both 16-line lidars, placed at approximately two-thirds of the distance from the front of the vehicle. The vertical distance between the cameras and lidars at the same location is set at 5–15 cm. All four cameras and lidars are placed on the same horizontal plane, and are mounted on pillars between 1.5L and 2L of the vehicle's height L. The cameras are used to collect image data from the four locations on the trackless rubber-tired vehicle, and the lidars are used to collect point cloud data from the four locations.

[0026] S2: Perform joint calibration of the cameras and solid-state LiDAR at the four locations: Obtain the position transformation matrices of the cameras and solid-state LiDAR at individual locations; establish a world coordinate system through manual calibration and alignment; obtain the extrinsic parameters of each of the four cameras and solid-state LiDAR; and set the rotation and translation matrices between the world coordinate system and the cameras to R. c T c The non-homogeneous linear coordinate in the camera coordinate system is X. w X c Then: X c =R c *X w +T c ;

[0027] Set the rotation and translation matrices of the solid-state lidar and camera to R. l T l A point P in a solid-state lidar l The non-homogeneous linear coordinates in the relative coordinate system and the camera coordinate system are X l X c Then: X c =R l *X l +T l .

[0028] Time synchronization of image data and point cloud data: Set the sampling periods of the camera and solid-state LiDAR to T. C T L Determine 1 / 2T C >T L 1 / 2T C With T L The timestamp data is merged, and at this point, 1 / 2T is selected. C Image data at specific time points as pre-fusion data D C1 D L1 Point cloud data at specific times as pre-fused data D L Then 1 / 2TC The image data and point cloud data at each time point are set to D. c1 and D L1 1 / 2T C ~T C The image data and point cloud data at each time point are set to D. c2 and D L2 ; Determine 1 / 2T C <T L , will T C With T L The timestamp data is calibrated and unified to achieve time synchronization between image data and solid-state LiDAR point cloud data, and then the T... C and T L All data are set to T L Time point data.

[0029] S3: Enhance the image data and point cloud data separately to remove noise and obtain high-quality preprocessed data.

[0030] First, the image data is enhanced to remove noise. Then, the self-attention mechanism of the reconstruction function is used to preserve the image data structure and calculate the image pixels to achieve image data reconstruction.

[0031] Then, the point cloud data is enhanced to remove noise. Gaussian filtering is used to perform a Gaussian weighted average of the points in the neighborhood of each point. The weights within the neighborhood determined by the covariance matrix are calculated and then averaged with the center point value.

[0032] S4: A neural network is used to extract image features and point cloud features separately. Then, the image features and point cloud features at a single location are fused separately. Finally, the fused information from the four locations is subjected to secondary feature fusion. First, the first feature fusion is performed, setting the feature layer extracted from the image to Nc. i The feature layer extracted from the point cloud is set to Nl. i Set the fused feature layer to Nf k In the feature extraction layer stage, image feature Nc1 and point cloud feature Nl3 are fused across modally to form Rf1. Then, image feature Nc2 and point cloud feature Nl4 are fused across modally to form Rf2, and so on, achieving the fusion of image features and point cloud features at a single location. Finally, the fused features at the four locations are R1, R2, R3, and R4, respectively. These four fused features are then fused a second time, R... a =W1*R1+W2*R2+W3*R3+W4*R4, where W1, W2, W3, and W4 are the weight values ​​of the fused features at the four positions in front, behind, left, and right of the camera, respectively.

[0033] S5: Input the extracted fusion features Ra into the prediction network, and set the predicted head position to a four-dimensional size S1(2). i *2 i S2(2) i+1 *2 i+1 S3(2) i+2 *2 i+2 S4(2) i+3 *2 i+3 ), i max (2 i+3 The feature width (≤ output feature width value) is finally mapped to a linear matrix (P1, P2, ..., x, y, z, R), where P represents the category information, x, y, z represent the three-dimensional spatial information, and R represents the rotation angle information.

[0034] A ground tunnel was selected as the test site. The camera's sampling period was 30ms, and the solid-state LiDAR's sampling period was 20ms. It can be determined that the 20ms sampling period of the solid-state LiDAR is greater than half the camera's sampling period of 15ms. Therefore, the sampling timestamps of the camera and the solid-state LiDAR were calibrated and unified. In the secondary feature fusion stage, considering that the vehicle's front and rear position data are more important, W1 and W2 were set to 0.6 and 0.6, respectively, and W3 and W4 were set to 0.2 and 0.2, respectively, achieving high-precision fusion of the four position information. Then, the camera information and the solid-state LiDAR information were fused. Finally, the features were input into the network prediction head. The Person information output prediction head size was (4, 8, 16, 32). However, due to the small size of Person, non-maximum suppression was used, and a size of 8 was used as the predicted output size, outputting the detection data (Person, 120, 87, 23, 80). The output prediction head size for the Car information is (8, 16, 32, 64). Since the size of the car is relatively moderate, according to the non-maximum suppression operation, 64 is used as the output size for prediction, and the output detection data is (car, 386, 267, 35, 34).

[0035] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0036] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any minor modifications, equivalent substitutions, and improvements made to the above embodiments based on the technical essence of the present invention should be included within the protection scope of the present invention.

Claims

1. A target detection method for unmanned vehicles based on camera fusion solid-state lidar, characterized in that, Includes the following steps: S1. Install a camera and a solid-state lidar at the front, rear, left, and right of the vehicle, respectively; S2. Perform joint calibration on the four position cameras and solid-state LiDAR respectively, and synchronize the image data and point cloud data in time; The method for synchronizing image data and point cloud data in step S2 is as follows: Set the sampling periods of the camera and solid-state LiDAR to T respectively. C T L Determine 1 / 2T C > T L 1 / 2T C With T L Merge timestamp data; determine 1 / 2T C < T L , will T C With T L The timestamp data is fused to achieve spatiotemporal synchronization of image data and point cloud data; S3. Perform enhancement processing on the image data and point cloud data respectively; S4. Use neural networks to extract image features and point cloud features respectively, then fuse the image features and point cloud features at a single location respectively, and then perform secondary feature fusion on the fused information at four locations. The method for fusing image features and point cloud features at a single location in step S4 is to set the feature layer extracted from the image to Nc. i The feature layer extracted from the point cloud is set to Nl. i In the feature extraction layer stage, image features Nc are extracted. i Point cloud features Nl i+2 Perform cross-modal fusion; In step S4, the fusion information of image features and point cloud features at four single locations are R1, R2, R3, and R4, respectively. The fusion features at the four locations are then fused a second time. W1, W2, W3, and W4 are the weight values ​​of the fused features at the four positions in front, behind, left, and right of the camera, respectively. S5. Input the extracted fusion features into the prediction network to identify the type, location, and pose information of obstacles in the mine.

2. The unmanned target detection method based on camera fusion solid-state lidar according to claim 1, characterized in that, In step S1, 32-line solid-state LiDARs are used at the front and rear of the vehicle, and 16-line solid-state LiDARs are used at the left and right of the vehicle. The vertical distance between the camera and the solid-state LiDAR at the same position is 5cm to 15cm. The cameras and solid-state LiDARs at the four positions are set on the same horizontal plane.

3. The unmanned target detection method based on camera fusion solid-state lidar according to claim 1, characterized in that, The method for joint calibration of the four position cameras and solid-state lidar in step S2 is as follows: by manually calibrating and aligning, the position transformation matrix of a single position camera and solid-state lidar is obtained, a world coordinate system is established, and the external parameters of the four cameras and solid-state lidar are obtained for calibration. Set the world coordinate system and the camera's rotation and translation matrices as R, respectively. c T c The non-homogeneous linear coordinate in the camera coordinate system is X. w X c ,but: ; Set the rotation and translation matrices of the solid-state LiDAR and camera to R respectively. l T l A point P in a solid-state lidar l The non-homogeneous linear coordinates in the relative coordinate system and the camera coordinate system are X, respectively. l X c ,but: .

4. The unmanned target detection method based on camera fusion solid-state lidar according to claim 1, characterized in that, In step S3, the image data enhancement process utilizes the self-attention mechanism of the reconstruction function to preserve the image data structure and reconstruct the image data. The point cloud data enhancement process uses Gaussian filtering to perform Gaussian weighted averaging on the points in the neighborhood of each point, calculates the weights within the neighborhood determined by its covariance matrix, and then averages them with the center point value.

Images