Infrared image edge detection method based on three-dimensional laser point cloud data guidance

Abstract

The invention provides an infrared image edge detection method based on three-dimensional laser point cloud data guidance, and belongs to the technical field of mobile robot autonomous environment perception. According to the method, edge points extracted from three-dimensional point cloud data are projected to an infrared image, edge detection is performed on the infrared image based on a DoG edge detection operator, and region search is performed by taking a pixel point obtained by projection as a starting point, so that the number of edge point detection of the infrared image is increased. According to the method, three-dimensional laser edge point information detected by a laser radar is used as priori, selection of a starting point of a region search part in an infrared image edge detection process is guided, the influence of infrared image noise is avoided, and the infrared image edge detection effect is enhanced; meanwhile, the method has the advantages of being few in process and small in calculation amount, and the real-time detection effect can be achieved on the industrial personal computer.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous environmental perception of mobile robots, and relates to data fusion between an infrared camera and a three-dimensional laser radar, and in particular to a method for improving the number of infrared image edge point detections based on three-dimensional laser point cloud data guidance. Background Art

[0002] In recent years, mobile robots have been applied to various fields and have completed many tasks on behalf of humans, such as exploration, mapping, search and rescue in disaster environments, etc. Mobile robots need to fully perceive the surrounding environment when completing tasks, so they are usually equipped with sensors such as lidar and visible light cameras. However, in challenging environments such as darkness, rain, or smoke, the performance of visible light cameras will be affected. Infrared cameras have all-day perception capabilities, have better spectral penetration performance in the presence of media such as fog, dust, and smoke, and are not affected by light and can work normally in dark environments. Therefore, infrared cameras are also increasingly used in the field of autonomous environmental perception technology for mobile robots. Edge information in infrared images is an important feature, and infrared image edge detection technology is of great significance for using infrared cameras to perceive the environment.

[0003] At present, commonly used edge detection algorithms include LoG operator, Sobel operator and Canny operator. These algorithms have the advantages of being simple and fast, but due to the characteristics of high noise and low contrast of infrared images, these operators are not effective when applied to infrared images. Reference (Wang Huihua, Yang Xiaogang. Research on infrared image edge detection based on adaptive threshold method [J]. Laser Journal, 2022, 43(11): 124-128.) The maximum inter-class variance method is used to calculate the adaptive threshold of infrared image edge detection and complete the initial edge extraction of the infrared image. Then, the initial edge expected gradient field and energy function are constructed to filter and enhance the initial edge, and finally the target edge image is obtained. This method improves the edge detection effect of infrared images to a certain extent, but is limited by the singleness of the information obtained by the sensor, and the final effect is limited. The paper (He Qian, Liu Boyun. An infrared image edge detection algorithm based on DexiNed [J]. Infrared Technology, 2021, 43(09): 876-884.) proposed an infrared image edge detection algorithm based on deep learning. Based on DexiNed (XSoria, A.Sappa, P.Humanante, and A.Akbarinia. Dense extreme inception network for edge detection [J]. Pattern Recognit., vol. 139, 2023, Art. no. 109461.), the network size was reduced, image-level differences were introduced into the loss function, and the parameters of the loss function were carefully set to optimize the network performance. Due to the lack of infrared image edge detection datasets, the paper can only train the network by adjusting the natural image edge detection dataset to approximate the infrared image edge detection dataset. However, this method may affect the network's ability to extract edge information from infrared images. In addition, the infrared image edge detection algorithm based on deep learning has a large amount of computation, which makes it impossible to run in real time on lightweight devices such as small industrial computers. Summary of the invention

[0004] In order to solve the shortcomings of the prior art, the present invention provides an infrared image edge detection method based on the guidance of three-dimensional laser point cloud data. The present invention projects the edge points extracted from the three-dimensional point cloud data to the infrared image, performs edge detection on the infrared image based on the DoG edge detection operator, and performs regional search with the projected pixel points as the starting point, thereby increasing the number of infrared image edge point detections. The method of the present invention has the characteristics of fewer processes and less computational complexity, and can achieve real-time detection effects on industrial computers.

[0005] Technical solution of the present invention:

[0006] An infrared image edge detection method based on three-dimensional laser point cloud data guidance adopts laser prior to guide infrared image edge detection, comprising:

[0007] Extract 3D laser edge points based on the geometric curvature of 3D laser point cloud;

[0008] Projecting the three-dimensional laser edge points onto an infrared image;

[0009] Filter out noise and low-contrast areas in infrared images and extract significant edges;

[0010] The pixel point obtained by projecting the 3D laser edge point onto the infrared image is used as the starting point, and a regional search is performed around it to extract new edges or connect broken edges.

[0011] Furthermore, the three-dimensional laser edge point extraction based on the geometric curvature of the three-dimensional laser point cloud specifically includes:

[0012] Assume that any frame of 3D laser point cloud data is P, the laser radar coordinate system itself is L, and for point i (i∈P), its coordinate value is expressed as Then the curvature value c at this point is i The calculation is as follows:

[0013]

[0014] Among them, S is the set of continuous adjacent points before and after point i and located on the same scan line as point i. Generally, n points are selected before and after the scan line where point i is located to form S. For the geometric curvature c i If the value is larger, it means that the geometric change of the scanning point at that location is more "sharp", and it is called a geometric edge point. If the value is smaller, it means that the geometric change of the scanning point at that location is more "flat", and it is called a geometric plane point.

[0015] Set the threshold c th , if c i Greater than c th , then add the 3D laser point to the 3D laser edge point set.

[0016] Furthermore, the specific process of projecting the 3D laser edge point onto the infrared image is as follows:

[0017] Calibrate the infrared camera internal parameters to obtain the internal parameter matrix K;

[0018] Jointly calibrate the infrared camera and lidar to obtain the external parameter matrix T between the camera coordinate system and the lidar coordinate system LC ;

[0019] Let p Lis a point in the edge point set of the three-dimensional laser point cloud, located in the laser radar coordinate system; according to formula (2), p L Convert to the camera coordinate system and get p C :

[0020] p C =T LC p L (2)

[0021] p uv =Kp C (3)

[0022] According to formula (3), p C Projected onto the two-dimensional image, we get pixel point p uv .

[0023] Furthermore, the DoG edge detection operator is used to filter out noise and low contrast areas in the infrared image and extract significant edges, as shown in formula (4):

[0024]

[0025] in, represents the image obtained by Gaussian blurring the infrared image I with two Gaussian kernels of different sizes, σ and kσ, and performing difference analysis; x, y represent the coordinate values ​​of the infrared image pixel points, It represents the pixel value of the pixel (x, y) after Gaussian blurring the infrared image with a Gaussian kernel of size σ. This can not only improve the visibility of the edge, but also suppress the noise of the image.

[0026] Assume that image I1 is an image filtered with Gaussian kernel σ, that is, I1 = I*G(x, y, σ); calculate the gradient information based on image I1, calculate the gradient amplitude and the gradient dx, dy in the x and y directions respectively, and combine the gradient amplitude with The pixel value of the infrared image is used to set a first threshold to extract significant edges and remove false edges generated by noise and low contrast areas in the infrared image.

[0027] Furthermore, the first threshold is set between 0.3 and 0.4.

[0028] Furthermore, the pixel point obtained by projecting the three-dimensional laser edge point onto the infrared image is used as the starting point, and a regional search is performed around it, thereby improving the effect of infrared image edge detection; the specific process is:

[0029] First, the pixels are divided into two categories: horizontal and vertical according to the size of the gradient in the x and y directions; when dx≥dy, the current pixel is in the vertical direction; when dx<dy, the current pixel is in the horizontal direction;

[0030] The pixel point p obtained by projecting the edge point of the 3D laser point cloud onto the infrared image uv as a starting point, and perform area search around it to extract new edges or connect broken edges;

[0031] In the process of searching the edge point area, the current pixel point and the previous pixel point are considered at the same time. Specifically, when the directions of the two pixels before and after are the same, that is, the directions of the two pixels before and after are both vertical or horizontal, there are three candidate points for the next exploration. Assuming that the current exploration direction is horizontal to the right and the coordinate value of the current pixel point is (x, y), the coordinate values ​​of the three candidate points are (x+1, y), (x+1, y+1) and (x+1, y-1) respectively; when the directions of the two pixels before and after are different, the exploration is continued along the direction of the previous pixel point first, so there are two candidate points for the next exploration. Assuming that the exploration direction of the previous pixel point is horizontal to the right, the direction of the current pixel point is vertical and the coordinate value of the current pixel point is (x, y), the coordinate values ​​of the two candidate points are (x+1, y+1) and (x+1, y-1) respectively; the pixel point with the largest gradient amplitude is selected from all candidate points as the next edge point; in each exploration process, the pixel point whose direction changes will be marked and stacked. When the exploration along the original direction is completed, the top pixel of the stack will be popped out in turn and used as the starting point to continue the exploration.

[0032] Furthermore, the search stops when encountering the following two situations: 1) the next pixel point explored has been marked as an edge point, that is, it already belongs to another edge, which indicates that the edges starting from two different starting points are successfully connected; 2) the candidate points explored in the next step are all weak edge pixels, that is, the gradient amplitude of the candidate points is less than the second threshold.

[0033] Furthermore, the second threshold is set to be smaller than the first threshold, so that edge information missed by the DoG edge detection operator can be detected.

[0034] Furthermore, the second threshold is set to half of the first threshold.

[0035] Beneficial effects of the present invention: The present invention uses the three-dimensional laser edge point information detected by the laser radar as a priori to guide the selection of the starting point of the regional search part in the infrared image edge detection process, thereby avoiding the influence of infrared image noise and enhancing the effect of infrared image edge detection. At the same time, the method of the present invention has the characteristics of being lightweight and can be run in real time on an industrial computer. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a flow chart of the present invention.

[0037] Figure 2Schematic diagram of the area search method.

[0038] Figure 3 This is a diagram of the data acquisition device of the present invention.

[0039] Figure 4 Indoor corridor images captured by infrared cameras.

[0040] Figure 5 It is the edge image output after the first stage of laser prior-guided infrared image edge detection.

[0041] Figure 6 It is the edge image output after the second stage of laser prior-guided infrared image edge detection. DETAILED DESCRIPTION

[0042] The specific implementation of the present invention is described in detail below in conjunction with the technical scheme and the accompanying drawings.

[0043] The data acquisition platform selected by the present invention is as follows Figure 3 As shown in the figure, it includes an infrared camera (TE-EV1), a multi-line laser radar (VLP-16), an industrial computer (NUC), an inertial measurement unit (Xsens), and a power supply. Among them, the multi-line laser radar and infrared camera are responsible for real-time data collection, and the industrial computer is responsible for image processing, coordinate system conversion calculation, etc.

[0044] like Figure 1 As shown, this embodiment provides an infrared image edge detection method based on three-dimensional laser point cloud data guidance, the steps are as follows:

[0045] Step 1) 3D laser edge point extraction

[0046] During the operation of the device, for each frame of 3D laser point cloud received, the geometric curvature of each point in the point cloud is calculated, and edge feature points and plane feature points are divided based on the size of the geometric curvature. The specific process is as follows:

[0047] First, calculate the curvature of any point i in the 3D laser point cloud P. The formula is as follows:

[0048]

[0049] Then, traverse each scan line and divide all the points on this line into m continuous sets. Arrange the points in each set in descending order according to the geometric curvature calculated by the above method, and search for the points with curvature greater than the geometric edge threshold from the point with the maximum curvature. The points are added to the three-dimensional laser edge point set. In the present invention, m=6 is selected.

[0050] Step 2) Project 3D laser edge points onto infrared image

[0051] First, the infrared camera internal parameters and the external parameters between the lidar and the infrared camera are calibrated; the calibrated infrared camera internal parameter matrix K and the external parameter matrix T between the two sensors LC They are:

[0052]

[0053] When the device is running, the collection time of the laser radar data and the infrared camera data is not completely synchronized. Suppose the collection time of the laser radar data is t m , the time of infrared camera data collection is t n Running the lightweight laser SLAM algorithm can calculate the device at t m With t n The pose transformation matrix T between two moments mn Before projecting the 3D laser edge points onto the infrared image in step 2), it is necessary to first m The 3D laser edge point set collected at time t is converted to n At this moment, in the laser radar coordinate system, as shown in the formula:

[0054]

[0055] in, t m The three-dimensional laser edge point in the laser radar coordinate system at this moment, t n The three-dimensional laser edge point in the laser radar coordinate system at that moment.

[0056] Will Convert to camera coordinate system:

[0057]

[0058] Then according to formula (3) Projected onto the two-dimensional image to obtain pixel point p uv .

[0059] Step 3) Laser prior guided infrared image edge detection

[0060] So far, a two-dimensional pixel point set formed by the projection of three-dimensional laser edge points has been obtained. The laser prior-guided infrared image edge detection part is mainly divided into two stages. In the first stage, the DoG edge detection operator is used to detect the edge of the infrared image, and the first threshold is set to remove the image noise and the false edges caused by the low-contrast area. The first threshold th1=0.4 set in this embodiment. In the second stage, a suitable starting point is selected and a regional search is performed around it. For pictures taken by ordinary cameras, the local maximum point of the gradient is generally searched globally as the starting point of the regional search. However, there are many noise points in the infrared image, which will affect the accuracy of the selection of the local maximum point of the gradient. In order to solve this problem, this embodiment takes the point in the obtained two-dimensional pixel point set as the starting point, and performs a regional search around it, so as to extract new edges or connect broken edges. Figure 2 Schematic diagram of the area search method. In the figure, the pixels marked with V represent the vertical pixels, and the pixels marked with H represent the horizontal pixels. Figure 2 The first row shows the distribution of the next candidate point when the two pixels before and after are in the same direction; if the exploration direction is horizontal to the right and the current pixel coordinate value is (x, y), then the coordinate values ​​of the three candidate points are (x+1, y), (x+1, y+1) and (x+1, y-1), and the same is true when the exploration direction is left, downward or upward. Figure 2 The second row shows the distribution of the next candidate point when the directions of the two pixels before and after are different; assuming that the exploration direction of the previous pixel point is horizontal to the right, the direction of the current pixel point is vertical and the coordinate value of the current pixel point is (x, y), then the coordinate values ​​of the two candidate points are (x+1, y+1) and (x+1, y-1), respectively. The same is true when the exploration direction of the previous pixel point is left, downward or upward. From the candidate points whose gradient amplitude is greater than the second threshold, select the point with the largest gradient amplitude as the pixel point to be explored next. Set the second threshold to half of the first threshold, and the present invention is set to th2=0.2. If the gradient amplitudes of all candidate points are less than the second threshold or the pixel point to be explored next has been marked as an edge point, stop exploring in the current direction.

[0061] The handheld device walks in the corridor, and runs the method of the present invention in real time during the walking process, performs edge detection on the infrared image collected in real time, and captures the edge detection effect diagram at time t. Figure 4 is the original infrared image collected by the infrared camera at time t, Figure 5 for Figure 4 The edge image output after the first stage in step 3) is Figure 6 for Figure 4 The edge image output after the second stage in step 3). Figure 5It can be seen that using only the information of a single sensor of an infrared camera can easily miss some edge information in the infrared image, such as Figure 4 The edge of the notice board between the two door frames. Figure 6 It can be seen that by integrating the edge point prior information in the laser point cloud data, it can be successfully detected. Figure 5 This demonstrates the effectiveness of the method of the present invention.

[0062] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An infrared image edge detection method based on three-dimensional laser point cloud data guidance, characterized in that: include: Extract 3D laser edge points based on the geometric curvature of 3D laser point cloud; Projecting the three-dimensional laser edge points onto an infrared image; Filter out noise and low-contrast areas in infrared images and extract significant edges; The pixel point obtained by projecting the 3D laser edge point onto the infrared image is used as the starting point, and a regional search is performed around it to extract new edges or connect broken edges.

2. The infrared image edge detection method based on three-dimensional laser point cloud data guidance according to claim 1 is characterized in that: The specific process of extracting 3D laser edge points based on the geometric curvature of the 3D laser point cloud includes: Assume that any frame of 3D laser point cloud data is P, the laser radar coordinate system itself is L, and for point i∈P, its coordinate value is expressed as Then the curvature value c at this point is i The calculation is as follows: Among them, S is the set of continuous adjacent points before and after point i and located on the same scan line as point i, which is composed of n points before and after the scan line where point i is located; Set the threshold c th , if c i Greater than c th , then add the 3D laser point to the 3D laser edge point set.

3. The infrared image edge detection method based on three-dimensional laser point cloud data guidance according to claim 2 is characterized in that: The specific process of projecting the 3D laser edge points onto the infrared image is as follows: Calibrate the infrared camera internal parameters to obtain the internal parameter matrix K; Jointly calibrate the infrared camera and lidar to obtain the external parameter matrix T between the camera coordinate system and the lidar coordinate system LC ; Let p L is a point in the edge point set of the three-dimensional laser point cloud, located in the laser radar coordinate system; according to formula (2), p L Convert to the camera coordinate system and get p C : p C =T LC p L (2) p uv =Kp C (3) According to formula (3), p C Projected onto the two-dimensional image, we get pixel point p uv .

4. The infrared image edge detection method based on three-dimensional laser point cloud data guidance according to claim 3 is characterized in that: The DoG edge detection operator is used to filter out noise and low contrast areas in infrared images and extract significant edges, as shown in formula (4): in, represents the image obtained by Gaussian blurring the infrared image I with two Gaussian kernels of different sizes, σ and kσ, and performing difference analysis; x, y represent the coordinate values ​​of the infrared image pixel points, It represents the pixel value of the pixel point (x, y) after Gaussian blurring the infrared image with a Gaussian kernel of size σ; Assume that image I1 is an image filtered with Gaussian kernel σ, that is, I1 = I*G(x, y, σ); calculate the gradient information based on image I1, calculate the gradient amplitude and the gradient dx, dy in the x and y directions respectively, and combine the gradient amplitude with The pixel value of the infrared image is used to set a first threshold to extract significant edges and remove false edges generated by noise and low contrast areas in the infrared image.

5. The infrared image edge detection method based on three-dimensional laser point cloud data guidance according to claim 4 is characterized in that: The first threshold is set between 0.3 and 0.

4.

6. The infrared image edge detection method based on three-dimensional laser point cloud data guidance according to claim 4 or 5, characterized in that: The pixel point obtained by projecting the three-dimensional laser edge point onto the infrared image is used as the starting point, and an area search is performed around it. The specific process is as follows: According to the size of the gradient in the x and y directions, the pixels are divided into two categories: horizontal and vertical directions; when dx≥dy, the current pixel is in the vertical direction; when dx<dy, the current pixel is in the horizontal direction; The pixel point p obtained by projecting the edge point of the 3D laser point cloud onto the infrared image uv as a starting point, and perform area search around it to extract new edges or connect broken edges; In the process of searching the edge point area, when the directions of the two pixels are the same, that is, the directions of the two pixels are both vertical or horizontal, there are three candidate points for the next exploration; when the directions of the two pixels are different, it is preferred to continue exploring along the direction of the previous pixel, so there are two candidate points for the next exploration; the pixel with the largest gradient amplitude is selected as the next edge point among all the candidate points; in each exploration process, the pixel points whose direction has changed will be marked and stacked, and when the exploration is completed along the original direction, the top pixels of the stack will be popped out one by one and used as the starting point to continue the exploration.

7. The infrared image edge detection method based on three-dimensional laser point cloud data guidance according to claim 6 is characterized in that: The search stops when encountering the following two situations: 1) the next pixel explored has been marked as an edge point, that is, it already belongs to another edge, which indicates that the edges starting from two different starting points are successfully connected; 2) the candidate points explored in the next step are all weak edge pixels, that is, the gradient amplitude of the candidate points is less than the second threshold.

8. The infrared image edge detection method based on three-dimensional laser point cloud data guidance according to claim 7 is characterized in that: The set second threshold is smaller than the first threshold.

9. The infrared image edge detection method based on three-dimensional laser point cloud data guidance according to claim 8, characterized in that: The second threshold is set to half of the first threshold.

Images