A target detection method based on infrared camera and three-dimensional laser radar

Abstract

The application relates to a target anomaly detection method, in particular to a target detection method based on an infrared camera and a three-dimensional laser radar, and solves the technical problem that the detection result of a single detection method is poor in a specific scene. The target detection method based on the infrared camera and the three-dimensional laser radar comprises the following steps: 1) building an electronic fence system; 2) detecting the boundary of a calibration board and each point thereon respectively to obtain an infrared image and point cloud data; 3) performing data registration on the infrared image and the point cloud data to obtain a rigid transformation matrix; 4) simultaneously collecting same scene data to obtain target infrared data and target point cloud data; 5) obtaining an infrared detection result; 6) obtaining a point cloud detection result; 7) performing decision-level fusion on the infrared detection result and the point cloud detection result by using the rigid transformation matrix to obtain a final dynamic target fusion anomaly detection result, and completing all-day target anomaly detection.

Description

TECHNICAL FIELD

[0001] The application relates to a target anomaly detection method, in particular to a target detection method based on an infrared camera and a three-dimensional laser radar. BACKGROUND

[0002] At present, target anomaly detection methods based on various detection means are emerging in an endless stream, and the detection effect has reached a good precision, so that the improvement space is getting smaller and smaller. However, most single detection means are only applicable to some specific scenes and have certain limitations, and cannot maintain an all-day working state, so different detection means need to be used in combination to combine their respective advantages to adapt to diversified application scenes.

[0003] With the development of the fields of security and protection, automatic driving, unmanned aerial vehicle navigation and the like, the demand for a detection system capable of reliable operation at any time and under any weather conditions is increasing. All-day anomaly detection requires that the detection system not only works in the daytime, but also works at night and maintains a certain precision and accuracy, and is not affected even in severe weather conditions, but the existing single detection means are difficult to meet the above requirements in terms of target anomaly detection results in specific scenes.

[0004] Infrared imaging technology is a passive detection method that uses the infrared energy radiated by an object to form an image, and is a passive detection method that is not affected by visible light and can provide clear images at night or in severe weather conditions. The infrared sensor can capture the thermal characteristics of the target, which is particularly useful for detecting hidden targets. However, due to the low resolution of infrared imaging when facing a background with similar temperature, the ability to finely depict the target is poor, and there is a lack of depth information, thus limiting its application range. On the other hand, three-dimensional laser scanning (Lidar) technology constructs a three-dimensional model of an object by emitting laser pulses and measuring the reflection time, and is an active detection method that can provide accurate distance measurement and high-resolution spatial information. Although Lidar may not perform well in severe weather conditions, its effectiveness will be affected, but it has incomparable advantages in providing detailed spatial data.

[0005] In view of the limitations of single detection means, it is extremely important to combine infrared imaging and three-dimensional laser scanning technology. SUMMARY

[0006] The application aims to solve the technical problem of poor target anomaly detection results of the existing single detection means in specific scenes, and provides a target detection method based on an infrared camera and a three-dimensional laser radar.

[0007] In order to achieve the above-mentioned purpose, the application adopts the following technical scheme:

[0008] A target detection method based on infrared camera and three-dimensional laser radar, which is characterized by comprising the following steps:

[0009] 1) Building an electronic fence system with infrared camera and three-dimensional laser radar;

[0010] 2) Preparing a calibration board, using the infrared camera and three-dimensional laser radar in the electronic fence system to detect the boundaries of the calibration board and each point thereon, respectively, to obtain infrared images and point cloud data;

[0011] 3) Data registration of the infrared images and point cloud data to obtain a rigid transformation matrix;

[0012] 4) Simultaneous acquisition of data of the same scene by the infrared camera and three-dimensional laser radar in the electronic fence system to obtain target infrared data and target point cloud data;

[0013] 5) Sending the target infrared data to the infrared data processing module of the infrared camera for processing to obtain an infrared detection result;

[0014] 6) Sending the target point cloud data to the point cloud data processing module of the three-dimensional laser radar for processing to obtain a point cloud detection result;

[0015] 7) Decision-level fusion of the infrared detection result and the point cloud detection result using the rigid transformation matrix to obtain a final dynamic target fusion anomaly detection result, completing all-day target anomaly detection.

[0016] Further, step 1) is specifically:

[0017] The infrared camera is set on the three-dimensional laser radar, parallel calibration is performed thereon, and it is fixed to complete the construction of the electronic fence system.

[0018] Further, step 3) is specifically:

[0019] 3.1, calculating the 3D plane of the calibration board in the camera coordinate system into which the infrared image is back-projected and its boundaries wherein C represents the infrared camera; i represents the number, i = 1, 2, 3, … N, N represents the number of infrared images; j represents the number of the four boundaries of the calibration board, j = 1, 2, 3, 4;

[0020] 3.2, using RANSAC algorithm to detect the plane where the calibration board is located in the point cloud coordinate system and its boundaries L represents the three-dimensional laser radar;

[0021] 3.3, calculating the initial value of the rotation matrix using the minimization function

[0022]

[0023] wherein, is a direction vector of the boundary , is a direction vector of the boundary , is a normal vector of the plane , is a normal vector of the plane , is a rotation matrix from the point cloud coordinate system to the camera coordinate system, and argmin represents minimizing a function;

[0024] 3.4. Solving the initial value of the translation matrix using linear least squares method

[0025]

[0026] wherein, H is a coefficient matrix, A ij is a process calculation amount, I is a unit matrix, and b is a column matrix, is the centroid of the plane , is the centroid of the four boundaries of the plane , is the distance from the plane to the origin of the camera coordinate system, is the data point on the plane ;

[0027] 3.5. Correcting the rotation matrix and the translation matrix using the minimizing function according to the initial value of the rotation matrix and the translation matrix to obtain the rigid transformation matrix

[0028]

[0029] wherein, m is a loop variable; N i is the number of data points on the plane ; is the data point on the boundary , and k is a loop variable; K ij is the number of data points on the boundary .

[0030] Further, step 5) is specifically:

[0031] 5.1, send the target infrared data into the infrared data processing module of the infrared camera, and mark the nth frame image and the n-1th frame image as f n and f n-1 respectively, the gray value of the corresponding pixel points of the two frames of images is marked as f n (x, y) and f n-1 (x, y), and the difference image D n is calculated by the following formula:

[0032] D n (x, y) = |f n (x, y) - f n-1 (x, y)|

[0033] In the formula, x represents the coordinate in the horizontal direction of the image plane, and y represents the coordinate in the vertical direction of the image plane.

[0034] 5.2, morphological processing is performed on the difference image D n (x, y) to obtain a target image.

[0035] 5.3, set a first threshold value, if the pixel value of the target image is greater than or equal to the first threshold value, keep the pixel point; otherwise, remove the pixel point; and all the retained pixel points are taken as the infrared detection result.

[0036] Further, step 6) is specifically:

[0037] 6.1, send the target point cloud data into the point cloud data processing module of the three-dimensional laser radar, minimize the distance between the target point cloud data, and match the target point cloud data;

[0038] 6.2, search the matched target point cloud data to obtain the nearest neighbor point;

[0039] 6.3, set a second threshold value, if the nearest neighbor point is greater than or equal to the second threshold value, take it as a target point; otherwise, do not process;

[0040] 6.4, extract the coordinate position of the target point, perform clustering analysis, and obtain different types of targets;

[0041] 6.5, visualize the target to obtain a point cloud detection result.

[0042] Further, in step 6.1, the distance between the target point cloud data is minimized by using the ICP algorithm.

[0043] Further, in step 6.2, the search is a KD tree search.

[0044] Further, in step 6.4, the clustering is DBSCAN clustering analysis.

[0045] Advantages of the present application:

[0046] 1. The target detection method based on infrared cameras and three-dimensional laser radars of the present application uses the fusion of infrared cameras and three-dimensional laser radars for dynamic target all-weather anomaly detection, solves the problem of poor detection effect of single means on dynamic targets in specific scenarios, and realizes efficient identification and tracking of dynamic targets in various environmental conditions all day long.

[0047] 2. The target detection method based on infrared cameras and three-dimensional laser radars of the present application not only integrates the all-weather detection capability of infrared imaging and the high-precision three-dimensional spatial information acquisition capability of three-dimensional laser radars, but also realizes efficient fusion of the two through algorithms, significantly improving robustness and detection accuracy.

[0048] 3. The target detection method based on infrared cameras and three-dimensional laser radars of the present application can obtain complementary information through infrared cameras and three-dimensional laser radars. In the case of night or low visibility, infrared imaging can make up for the shortcomings of three-dimensional laser radars; while in the case of needing detailed spatial data, three-dimensional laser radars can make up for the shortcomings of infrared imaging. In addition, it can also enhance adaptability and flexibility, so that it can perform well in a wide range of environments.

[0049] 4. The target detection method based on infrared cameras and three-dimensional laser radars of the present application can provide stable and reliable dynamic target detection and tracking services in various weather conditions, and is particularly suitable for key areas that need continuous monitoring, such as airports, borders, and important facilities.

[0050] 5. The target detection method based on infrared cameras and three-dimensional laser radars of the present application not only improves the accuracy of detection, but also reduces the false alarm rate, ensuring effective monitoring all day long, and bringing new possibilities and development directions to the fields of safety protection and intelligent traffic management. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 is a flowchart of an embodiment of the target detection method based on infrared cameras and three-dimensional laser radars of the present application;

[0052] Figure 2 is a flowchart of the target infrared data processing module processing target infrared data in an embodiment of the target detection method based on infrared cameras and three-dimensional laser radars of the present application;

[0053] Figure 3 is a flowchart of the point cloud data processing module processing target point cloud data in an embodiment of the target detection method based on infrared cameras and three-dimensional laser radars of the present application. DETAILED DESCRIPTION

[0054] AsFigure 1 As shown, a target detection method based on an infrared camera and a 3D lidar includes the following steps:

[0055] 1) Build an electronic fence system using infrared cameras and 3D LiDAR;

[0056] The infrared camera is set on the 3D lidar, parallelized and fixed to complete the construction of the electronic fence system.

[0057] 2) The infrared camera and 3D lidar in the electronic fence system are used to detect the boundary of the calibration board and its points to obtain infrared images and point cloud data.

[0058] 3) Register the infrared image and point cloud data to obtain the rigid transformation matrix;

[0059] 3.1 Calculate the 3D plane of the infrared image back-projected onto the calibration plate in the camera coordinate system. and its boundaries Where C represents the infrared camera; i represents the number, i = 1, 2, 3, ... N, and N represents the number of infrared images; j represents the number of the four boundaries of the calibration plate, j = 1, 2, 3, 4;

[0060] 3.2. Use the RANSAC algorithm to detect the plane where the calibration plate is located in the point cloud coordinate system. and its boundaries L represents a 3D lidar;

[0061] 3.3 Calculate the initial value of the rotation matrix using a minimization function.

[0062]

[0063] In the formula, It is a boundary directional vector, It is a boundary directional vector, It is a plane The normal vector, It is a plane The normal vector, It is the rotation matrix from the point cloud coordinate system to the camera coordinate system, and argmin represents the minimization function;

[0064] 3.4 Obtain the initial value of the translation matrix based on the following two constraints.

[0065]

[0066] To avoid deviation, a plane is used. center of mass and the centroids of each of the four boundaries Calculate the translation matrix

[0067] make Then for each pose, we have:

[0068]

[0069] By summing the above two equations for all poses, a system of linear equations is formed:

[0070]

[0071] Use the linear least squares method to solve for the initial value of the translation matrix. That is, find Make Minimize. For this type of problem, a closed-form solution exists:

[0072]

[0073] In the formula, I is the identity matrix; It is a boundary The data points are on the loop, and k is the loop variable. It is a plane Data points on; It is a plane Data points on; It is a plane Distance to the origin of the camera coordinate system; is the translation matrix from the point cloud coordinate system to the camera coordinate system; T is the transpose matrix; H is the coefficient matrix; b is the column matrix; where,

[0074]

[0075] 3.5. Based on the initial value and initial value Correcting the rotation matrix using a minimization function Translation matrix Obtain the rigid transformation matrix

[0076]

[0077] In the formula, m is the loop variable, m = 1, 2, ..., N i N i It is a plane Number of data points on; K ij It is a boundary The number of data points on the data;

[0078] 4) Utilize the infrared camera and 3D lidar in the electronic fence system to simultaneously collect data from the same scene, thereby obtaining target infrared data and target point cloud data;

[0079] 5) such as Figure 2 As shown, the target infrared data is sent to the infrared data processing module of the infrared camera for processing to obtain the infrared detection result.

[0080] 5.1. Send the target infrared data into the infrared data processing module of the infrared camera, and denot the nth frame image and the (n-1)th frame image as f respectively. n and f n-1 The grayscale values ​​of corresponding pixels in two frames are denoted as f. n (x,y) and f n-1 The difference image D is obtained by calculating (x,y) using the following formula. n :

[0081] D n (x,y)=|f n (x,y)-f n-1 (x,y)|

[0082] In the formula, x represents the horizontal coordinate on the image plane, and y represents the vertical coordinate on the image plane;

[0083] 5.2. For the difference image D n Morphological processing is performed on (x,y) to obtain the target image;

[0084] 5.3. Perform thresholding on the target image;

[0085] Set a first threshold. If the pixel value of the target image is greater than or equal to the first threshold, the pixel is retained; otherwise, the pixel is removed. All retained pixels are used as the infrared detection result.

[0086] 6) such as Figure 3 As shown, the target point cloud data is sent to the point cloud data processing module of the 3D LiDAR for processing to obtain the point cloud detection results.

[0087] 6.1 The target point cloud data is sent to the point cloud data processing module of the 3D LiDAR, the ICP algorithm is used to minimize the distance between the target point cloud data, and the target point cloud data is matched.

[0088] 6.2. Use a KD (K-Demensional, KD) tree to search the matched target point cloud data to obtain the nearest neighbor points;

[0089] 6.3 Set a second threshold. If the nearest neighbor is greater than or equal to the second threshold, then use it as the target point; otherwise, do nothing.

[0090] 6.4, Extract the coordinate position of the target point, and perform DBSCAN (Density Based Spatial Clustering of Applications with Noise) clustering analysis to obtain different types of targets;

[0091] 6.5, Visualize the target to obtain a point cloud detection result;

[0092] 7) Use the rigid transformation matrix to perform decision-level fusion of the infrared detection result and the point cloud detection result to obtain a final dynamic target fusion anomaly detection result, and complete all-day target anomaly detection.

Claims

1. A target detection method based on an infrared camera and a three-dimensional lidar, characterized in that, Includes the following steps: 1) Build an electronic fence system using infrared cameras and 3D LiDAR; 2) Prepare a calibration board. Use the infrared camera and 3D LiDAR in the electronic fence system to detect the boundary of the calibration board and each point on it to obtain infrared images and point cloud data. 3) Register the infrared image and point cloud data to obtain the rigid transformation matrix: 3.1 Calculate the 3D plane of the infrared image back-projected onto the calibration plate in the camera coordinate system. and its boundaries Where C represents the infrared camera; i represents the number, i = 1, 2, 3, ... N, and N represents the number of infrared images; j represents the number of the four boundaries of the calibration plate, j = 1, 2, 3, 4; 3.

2. Use the RANSAC algorithm to detect the plane where the calibration plate is located in the point cloud coordinate system. and its boundaries L represents a 3D lidar; 3.3 Calculate the initial value of the rotation matrix using a minimization function. In the formula, It is a boundary directional vector, It is a boundary directional vector, It is a plane The normal vector, It is a plane The normal vector, It is the rotation matrix from the point cloud coordinate system to the camera coordinate system, and argmin represents the minimization function; 3.4 Solving for the initial value of the translation matrix using the linear least squares method In the formula, H is the coefficient matrix. A ij The computational complexity is as follows: I is the identity matrix, and b is the column matrix. For plane The center of mass, For plane The centroids of the four boundaries, For plane Distance to the origin of the camera coordinate system For plane Data points on; 3.

5. Based on the initial values ​​of the rotation matrix Translation matrix Correcting the rotation matrix using a minimization function Translation matrix Obtain the rigid transformation matrix In the formula, m is the loop variable; N i It is a plane The number of data points on the data; It is a boundary The data points are on the loop, where k is the loop variable; K ij It is a boundary The number of data points on the data; 4) Utilize the infrared camera and 3D lidar in the electronic fence system to simultaneously collect data from the same scene, thereby obtaining target infrared data and target point cloud data; 5) The target infrared data is sent to the infrared data processing module of the infrared camera for processing to obtain the infrared detection results; 6) The target point cloud data is sent to the point cloud data processing module of the 3D LiDAR for processing to obtain the point cloud detection results; 7) The infrared detection results and point cloud detection results are fused at the decision level using a rigid transformation matrix to obtain the final dynamic target fusion anomaly detection result, thus completing all-day target anomaly detection.

2. The target detection method based on an infrared camera and a three-dimensional lidar according to claim 1, characterized in that, Step 1) specifically involves: An infrared camera is mounted on a 3D lidar unit, calibrated in parallel, and fixed in place to complete the construction of the electronic fence system.

3. The target detection method based on an infrared camera and a three-dimensional lidar according to claim 2, characterized in that, Step 5) specifically refers to: 5.

1. Send the target infrared data into the infrared data processing module of the infrared camera, and denot the nth frame image and the (n-1)th frame image as f respectively. n and f n-1 The grayscale values ​​of corresponding pixels in two frames are denoted as f. n (x,y) and f n-1 The difference image D is obtained by calculating (x,y) using the following formula. n : D n (x,y)=|f n (x,y)-f n-1 (x,y)| In the formula, x represents the horizontal coordinate on the image plane, and y represents the vertical coordinate on the image plane; 5.

2. For the difference image D n Morphological processing is performed on (x,y) to obtain the target image; 5.3 Set a first threshold. If the pixel value of the target image is greater than or equal to the first threshold, retain the pixel; otherwise, remove the pixel; and use all retained pixels as the infrared detection result.

4. The target detection method based on an infrared camera and a three-dimensional lidar according to claim 3, characterized in that, Step 6) specifically refers to: 6.

1. Send the target point cloud data into the point cloud data processing module of the 3D LiDAR, minimize the distance between the target point cloud data, and match the target point cloud data; 6.

2. Search the matched target point cloud data to obtain the nearest neighbor points; 6.3 Set a second threshold. If the nearest neighbor is greater than or equal to the second threshold, then use it as the target point. Otherwise, no action will be taken; 6.4 Extract the coordinates of the target points and perform cluster analysis to obtain different types of targets; 6.

5. Visualize the target to obtain the point cloud detection results.

5. The target detection method based on an infrared camera and a three-dimensional lidar according to claim 4, characterized in that: In step 6.1, the distance between the target point cloud data is minimized using the ICP algorithm.

6. The target detection method based on an infrared camera and a three-dimensional lidar according to claim 5, characterized in that: In step 6.2, the search is performed using a KD-tree search.

7. The target detection method based on an infrared camera and a three-dimensional lidar according to claim 6, characterized in that: In step 6.4, the clustering is DBSCAN clustering analysis.

Images