An ultra-wide-angle infrared and visible dual-band stereo vision perception device and method

Abstract

This specification provides an ultra-wide-angle infrared and visible light dual-band stereo vision perception device and method, characterized by comprising: simultaneously acquiring infrared images and visible light images using the ultra-wide-angle infrared binocular stereo camera and the ultra-wide-angle visible light binocular stereo camera; performing binocular stereo matching on the infrared and visible light image data respectively using the SAD algorithm; performing fractional disparity calculation and smoothing processing on the matched infrared and visible light images to output corresponding infrared 3D point clouds and visible light 3D point clouds; and performing point cloud fusion and image fusion on the infrared 3D point clouds and the visible light 3D point clouds to obtain fused 3D point cloud data and fused image data, thereby improving the accuracy, density, and confidence of the 3D stereo data.

Description

Technical Field

[0001] This invention relates to the field of stereoscopic vision recognition technology, and in particular to an ultra-wide-angle infrared and visible light dual-band stereoscopic vision sensing device and method. Background Technology

[0002] Stereo vision recognition technology has become an important technology for precision inspection in various fields. It has significant application value in many areas such as safe driving, defect detection of objects, and 3D model construction. Achieving all-weather 3D stereo vision perception through passive means is an important research direction in computer stereo vision.

[0003] In existing technologies, stereo vision recognition mainly employs two methods. One method uses active light sources such as 3D LiDAR, ToF, and structured light, but these methods suffer from short light source lifespans and weak counter-surveillance capabilities. The other method uses passive light sources based on cameras, which suffers from issues such as unclear target features in low-light conditions like rain, fog, or nighttime, requiring additional supplementary lighting. Furthermore, stereo matching often utilizes the BM or SGBM algorithms, both of which have high point cloud density, resulting in high computational complexity and low accuracy. The SAD algorithm offers relatively high accuracy, but its point cloud is sparse, consisting mostly of marginal, effective points.

[0004] Based on the above, it is necessary to propose a stereo vision perception method with dual-band light source, high precision, high density and high confidence. Summary of the Invention

[0005] To overcome the problems existing in related technologies, the present invention provides an ultra-wide-angle infrared and visible light dual-band stereo vision perception device and method to solve the technical problems of low point cloud accuracy, sparse number and mostly marginal effective points in the prior art.

[0006] This specification provides one or more embodiments of an ultra-wide-angle infrared and visible light dual-band stereo vision perception method, including:

[0007] Infrared and visible light images are acquired simultaneously using an ultra-wide-angle infrared binocular stereo camera and an ultra-wide-angle visible light binocular stereo camera.

[0008] The infrared image and the visible light image are respectively subjected to binocular stereo matching using the SAD algorithm. The matched infrared image and visible light image data are then subjected to fractional disparity calculation and smoothing processing to output the corresponding infrared 3D point cloud and visible light 3D point cloud.

[0009] The infrared 3D point cloud and the visible light 3D point cloud are fused together to obtain fused 3D point cloud data and fused image data.

[0010] Furthermore, the stereo matching using the SAD algorithm includes:

[0011] The SAD algorithm is used for stereo matching with binoculars. Point cloud information is obtained based on the set out-of-limit matching threshold and successful matching threshold, and the calculation is as follows:

[0012]

[0013] Where disp is the parallax, P L (i, j) represents the grayscale value of the pixel to be matched in the reference image, P R (i, j) represents the grayscale value of the matching point in the registration diagram, and (h, k) represents the size of the moving window.

[0014] Furthermore, the calculation of fractional parallax for the matched infrared and visible light images specifically involves:

[0015] For the matched infrared and visible light images, any pair of pixels is randomly selected for matching. After the current pixel is successfully matched, the next adjacent pixel is matched. The fractional parallax is calculated using the matching value of the current pixel and the matching values ​​of the previous and next pixels to obtain the infrared and visible light images with fractional parallax.

[0016] The calculation of decimal disparity is specifically as follows:

[0017] △d=0.5*(ya-yc) / (yc-yb)ya <yc;

[0018] △d=0.5*(ya-yc) / (ya-yb)ya>yc;

[0019] Where ya, yb, and yc represent the SAD values ​​of three adjacent pixels in the data that were successfully matched using SAD, and △d represents the fractional disparity. The disparity value after calculating the fractional disparity is d+△d.

[0020] Furthermore, the smoothing process performed on the matched infrared and visible light images includes:

[0021] Calculate the minimum Euclidean distance between adjacent pixel point clouds. If the minimum Euclidean distance between adjacent pixel point clouds is less than a preset threshold, take a single pixel as the center and the 8 adjacent pixels as the value range to take the mode of the pixel value. Replace the pixel value of the pixel whose minimum Euclidean distance is less than the preset threshold with the mode of the pixel value.

[0022] Furthermore, the smoothing process for the matched infrared and visible light images also includes filling in missing points in the point cloud data of the infrared and visible light images, specifically:

[0023] Using any pixel with a value of 0 as the center, search for non-zero pixels in eight directions around it until a non-zero pixel is encountered. Then, select the non-zero pixel that appears most frequently (more than 4 times) to fill the non-zero pixel in the stereo matching.

[0024] Furthermore, before the step of fusing the infrared three-dimensional point cloud with the visible light three-dimensional point cloud, the infrared image and the visible light image after fractional parallax calculation and smoothing are respectively subjected to image fusion algorithm to highlight the significant targets in the image.

[0025] On the other hand, an ultra-wide-angle infrared and visible light dual-band stereoscopic vision sensing device is provided, including:

[0026] The synchronization control module is used to control the ultra-wide-angle infrared binocular stereo camera and the ultra-wide-angle visible light binocular stereo camera to synchronously acquire infrared images and visible light images;

[0027] The preprocessing module is used to perform stereo matching of the infrared image and the visible light image data using the SAD algorithm, and to perform fractional disparity calculation and smoothing processing on the matched infrared image and visible light image to obtain the corresponding infrared three-dimensional point cloud and visible light three-dimensional point cloud.

[0028] The fusion module performs point cloud fusion and image fusion on the infrared 3D point cloud and the visible light 3D point cloud to obtain fused 3D point cloud data and fused image data.

[0029] Furthermore, the preprocessing module is used to perform binocular stereo matching on the infrared image and the visible light image data respectively using the SAD algorithm, specifically including:

[0030] The SAD algorithm is used for stereo matching with binoculars. Point cloud information is obtained based on the set out-of-limit matching threshold and successful matching threshold, and the calculation is as follows:

[0031]

[0032] Where disp is the parallax, P L (i, j) represents the grayscale value of the pixel to be matched in the reference image, P R (i, j) represents the grayscale value of the matching point in the registration diagram, and (h, k) represents the size of the moving window.

[0033] Furthermore, the preprocessing module performs fractional parallax calculation on the matched infrared and visible light images, specifically for:

[0034] For the matched infrared and visible light images, any pair of pixels is randomly selected for matching. After the current pixel is successfully matched, the next adjacent pixel is matched. The fractional disparity is calculated and output using the matching value of the current pixel and the matching values ​​of the pixels before and after it.

[0035] This invention provides an ultra-wide-angle infrared and visible light dual-band stereoscopic vision sensing device and method, which has the following advantages compared with the prior art:

[0036] By using ultra-wide-angle infrared binocular stereo cameras and ultra-wide-angle visible light binocular stereo cameras, a wide-band passive light source can be achieved. The matched images are then subjected to fractional parallax calculation and smoothing processing to obtain high-precision, high-density, and high-confidence 3D point cloud data and fused images. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in one or more embodiments of this specification or in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 A flowchart of an ultra-wide-angle infrared and visible light dual-band stereo vision perception method provided for one or more embodiments of this specification;

[0039] Figure 2 A schematic diagram illustrating the calculation of fractional parallax in an ultra-wide-angle infrared and visible light dual-band stereo vision perception method provided in one or more embodiments of this specification;

[0040] Figure 3 A schematic diagram illustrating normal and incorrect matching of an ultra-wide-angle infrared and visible light dual-band stereo vision perception method provided in one or more embodiments of this specification;

[0041] Figure 4 The images captured by this specification and the effect diagram of binocular stereo matching of 0 for one or more embodiments of this specification;

[0042] Figure 5 The diagram shows the effect of spreading the original data in eight directions, as provided in one or more embodiments of this specification.

[0043] Figure 6 Different effect diagrams of infrared and visible light vehicle detection provided in one or more embodiments of this specification;

[0044] Figure 7A graph of the infrared red line provided for one or more embodiments of this specification;

[0045] Figure 8 A schematic diagram of fused 3D point cloud data provided for one or more embodiments of this specification;

[0046] Figure 9 A block diagram of an ultra-wide-angle infrared and visible light dual-band stereo vision sensing device provided for one or more embodiments of this specification;

[0047] Figure 10 A schematic diagram of the internal structure of an ultra-wide-angle infrared and visible light dual-band stereo vision sensing device provided for one or more embodiments of this specification;

[0048] Figure 11 This is a schematic diagram of the outer shell structure of an ultra-wide-angle infrared and visible light dual-band stereo vision sensing device provided for one or more embodiments of this specification. Detailed Implementation

[0049] To enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of this invention.

[0050] The present invention will now be described in detail with reference to specific embodiments and accompanying drawings.

[0051] Method Implementation Examples

[0052] Figure 1 This flowchart illustrates an ultra-wide-angle infrared and visible light dual-band stereo vision perception method provided in one or more embodiments of this specification. According to embodiments of the present invention, an ultra-wide-angle infrared and visible light dual-band stereo vision perception method is provided, comprising:

[0053] Step 101: Simultaneously acquire infrared and visible light images using an ultra-wide-angle infrared binocular stereo camera and an ultra-wide-angle visible light binocular stereo camera;

[0054] Step 102: Perform stereo matching of the infrared image and the visible light image data using the SAD algorithm, calculate the fractional disparity and smooth the matching infrared image and visible light image, and output the corresponding infrared 3D point cloud and visible light 3D point cloud.

[0055] Step 103: Perform point cloud fusion and image fusion on the infrared 3D point cloud and the visible light 3D point cloud to obtain fused 3D point cloud data and fused image data.

[0056] The embodiments of the present invention can realize a wide-band passive light source by using an ultra-wide-angle infrared binocular stereo camera and a visible light binocular stereo camera. The matched images are subjected to fractional parallax calculation and smoothing processing to obtain high-precision, high-density, and high-confidence three-dimensional point cloud data and fused images.

[0057] In this embodiment, step 102 involves performing stereo matching using the SAD algorithm, including:

[0058] The SAD algorithm in a 4x4 pattern is used for stereo matching with binoculars. Point cloud information is obtained based on the set out-of-limit matching threshold and successful matching threshold, and the calculation is as follows:

[0059]

[0060] Where disp is the parallax, P L (i, j) represents the grayscale value of the pixel to be matched in the reference image, P R (i, j) represents the grayscale value of the matching point in the registration diagram, and (h, k) represents the size of the moving window.

[0061] In this embodiment, step 102 involves calculating the fractional parallax of the matched infrared and visible light images, specifically as follows:

[0062] For the matched infrared and visible light images, any pair of pixels is randomly selected for matching. After the current pixel is successfully matched, the next adjacent pixel is matched. The fractional parallax is calculated using the matching value of the current pixel and the matching values ​​of the previous and next pixels to obtain the infrared and visible light images with fractional parallax.

[0063] To accurately calculate the value of Δd, this invention uses the following formula:

[0064] △d=0.5*(ya-yc) / (yc-yb)ya <yc;

[0065] △d=0.5*(ya-yc) / (ya-yb)ya>yc;

[0066] Where ya, yb, and yc represent the SAD values ​​of three adjacent pixels in the data that were successfully matched using SAD, and △d represents the fractional disparity. The disparity value after calculating the fractional disparity is d+△d. Figure 2 A schematic diagram illustrating the fractional parallax calculation of an ultra-wide-angle infrared and visible light dual-band stereo vision perception method provided in one or more embodiments of this specification, with reference to... Figure 2The change of SAD is represented by two straight lines. Due to integer matching, the result B is the output value, while the position of the intersection point D is the accurate value after stereo matching.

[0067] In the diagram, A, B, C, and D represent the calculated SAD value and the calculated pixel offset, respectively. For example, in the left image, the SAD value for matching the 5th pixel in the binocular left and right images is 25, i.e., A(5, 25); for matching the 6th pixel, it is B(6, 20); and for matching the 7th pixel offset, it is C(7, 30). However, the actual true value is D(5.75, 15). Since the image is based on integers, only point B can be taken as the result data. To increase the matching accuracy, this algorithm implements decimal calculation. The right image is similar. ya, yb, and yc represent the matching SAD values, and yd represents the actual SAD value, i.e., yd = d + Δd. After calculation, the algorithm uses a binary search to obtain 1 / 4 decimals of Δd, i.e., 0.25, 0.5, and 0.75, which can improve the final detection accuracy to a certain extent.

[0068] In reality, SAD is calculated based on the absolute value of the difference d between small regions of the left and right images, and is therefore affected by ambient light and calibration accuracy. As a result, the actual accuracy becomes about 1 / 4 pixel, improving the accuracy of 3D data by 1 / 4 decimal place. By increasing the density of 3D data through algorithms, the accuracy of data at long distances is further improved.

[0069] As one embodiment, the smoothing process performed on the matched infrared and visible light images in step 102 includes:

[0070] Calculate the minimum Euclidean distance between adjacent pixel point clouds. If the minimum Euclidean distance between adjacent pixel point clouds is less than a preset threshold, take a single pixel as the center and the 8 adjacent pixels as the value range to take the mode of the pixel value. Replace the pixel value of the pixel whose minimum Euclidean distance is less than the preset threshold with the mode of the pixel value.

[0071] Because the SAD algorithm calculates small regions and mechanically segments the image, occasional identical patterns may exist within the search range, causing interference in the image and preventing the acquisition of correct values. Random mismatches may occur, but relative to the majority of data, they constitute a small proportion and can be corrected by taking the mode of a 3x3 grid. Figure 3 As shown: In normal matching, Figure 3 (a) has a center value of 17, and the mode of the 8 adjacent pixels in the 3x3 grid is 18. Since the center value is within the maximum frequency range of 18±1, the center value of 17 is not replaced. The result is as follows. Figure 3 As shown in (b). In incorrect matches, Figure 3(c) has a center value of 40. The mode of the 8 adjacent pixels in the 3x3 grid is 18. The center value is not within the maximum frequency range of 18±1, indicating a significant offset. Replacing the center value of 40 with the mode of 18 yields the following result: Figure 3 As shown in (d), data smoothing is achieved for the stereo matching results by replacing pixels with larger offsets.

[0072] As another embodiment, the smoothing process for the matched infrared and visible light images in step 102 further includes filling in missing point cloud data in the infrared and visible light images, specifically:

[0073] Using any pixel with a value of 0 as the center, search for non-zero pixels in eight directions around it until a non-zero pixel is encountered. Then, select the non-zero pixel that appears most frequently (more than 4 times) to fill the non-zero pixel in the stereo matching.

[0074] Figure 4 The images shown are captured using one or more embodiments of this specification, and the resulting images are stereoscopically matched to a value of 0. Figure 4 (a) is a photograph. Figure 4 (b) shows the result of a stereo matching error of 0. Initially, the SAD algorithm only matches valid data at locations with edge points. At other locations, since there are no obvious edge contours, a successful match occurs when the offset is 0. Figure 4 As shown, it can be seen that there is valid data only in the locations where the object has obvious edges, while other locations are empty.

[0075] Figure 5 This invention provides an example of the effect of applying eight-directional diffusion to the original data, as shown in one or more embodiments of this specification. The invention employs a dual-judgment method of using the mode and frequency of the eight-directional filling approach for hole filling. For data with a stereo matching error of 0, the data is diffused pixelally along eight directions: top, bottom, left, right, upper left, upper right, lower left, and lower right, until a non-zero data point is encountered. The non-zero data point with the highest frequency (more than four times) is then used to fill the data with a stereo matching error of 0, resulting in eight diffused images, as shown below. Figure 5 As shown, Figure 5 (a) Figure 5 (b) Figure 5 (c) Figure 5 (d) These are the spread maps obtained by spreading along the top, bottom, left, and right directions of the image, respectively. Figure 5 (e) Figure 5 (f) Figure 5 (g) Figure 5(h) are the spread maps obtained by spreading along the upper left, upper right, lower left, and lower right directions of the image, respectively. In this embodiment, data spread increases the order of magnitude of the 3D point cloud data, making subsequent applications more accurate.

[0076] As one example, Figure 6 Different effect diagrams of infrared and visible light image detection and highlighting of target objects provided in one or more embodiments of this specification, wherein, Figure 6 (a) is a schematic diagram of highlighting special objects in a visible light image using an image fusion algorithm; Figure 6 (b) is a schematic diagram illustrating the highlighting of special objects in an infrared image using an image fusion algorithm. Before the point cloud fusion step in step 103, which involves fusing the infrared 3D point cloud with the visible light 3D point cloud, this invention employs threshold filtering and artificial intelligence detection to identify special objects in the two images in order to prominently display significant targets in the image. Pixel fusion is then used to perform fractional disparity calculation and smoothing processing on the processed image, and the image fusion algorithm is used to highlight these special objects. For example, see reference... Figure 6 Visible light is used to detect vehicles in the image using artificial intelligence deep learning, while infrared light is used to extract vehicles from the image based on temperature differences. The two methods are then pixel-wise fused to create unique vehicle markers, which are then referenced... Figure 7 , Figure 7 The curve of the red line in the infrared image provided in one or more embodiments of this specification will Figure 6 The curve of the red line in the mid-infrared image is shown in Figure 7 middle.

[0077] As one example, Figure 8 The fused 3D point cloud data provided for one or more embodiments of this specification.

[0078] In step 103, point cloud fusion involves displaying the point cloud data obtained from the ultra-wide-angle infrared stereo camera and the point cloud data obtained from the visible light stereo camera in the same three-dimensional coordinate system, resulting in denser point cloud data. Simultaneously, by combining data from the same point location, such as... Figure 8 As shown.

[0079] This invention provides an algorithm for image fusion and 3D point cloud fusion based on infrared and visible light dual-band, which can highlight the highlights of things and present special things in a focused manner.

[0080] To achieve the above embodiments, the present invention provides a hardware layout between an infrared and visible light dual-band stereo camera and a controller, enabling high-speed acquisition of data from the infrared and visible light dual-band stereo camera on the same plane, synchronously, and within the same frame. Figure 9A block diagram illustrating an ultra-wide-angle infrared and visible light dual-band stereoscopic vision perception device provided in one or more embodiments of this specification, the device comprising:

[0081] The synchronization control module is used to control the ultra-wide-angle infrared binocular stereo camera and the ultra-wide-angle visible light binocular stereo camera to synchronously acquire infrared images and visible light images;

[0082] The preprocessing module is used to perform stereo matching of the infrared image and the visible light image data using the SAD algorithm, and to perform fractional disparity calculation and smoothing processing on the matched infrared image and visible light image to obtain the corresponding infrared three-dimensional point cloud and visible light three-dimensional point cloud.

[0083] The fusion module performs point cloud fusion and image fusion on the infrared 3D point cloud and the visible light 3D point cloud to obtain fused 3D point cloud data and fused image data.

[0084] The synchronous control module can acquire image data of a wide-band passive light source. The preprocessing module performs fractional parallax calculation and smoothing on the matched image to obtain high-precision, high-density, and high-confidence 3D point cloud data. The fusion module obtains a high-density, high-confidence 3D point cloud fusion image that highlights the target object through point cloud fusion and image fusion.

[0085] In one specific embodiment, a schematic diagram of a device is provided that enables the simultaneous setting of the ultra-wide-angle infrared binocular stereo camera and the visible light binocular stereo camera, and the synchronous acquisition of infrared and visible light images; (Refer to...) Figure 10 and Figure 11 , Figure 10 This is a schematic diagram of the structure of an ultra-wide-angle infrared and visible light dual-band stereo vision sensing device provided in one or more embodiments of this specification. Figure 11 This specification provides a schematic diagram of the housing structure of an ultra-wide-angle infrared and visible light dual-band stereo vision sensing device according to one or more embodiments. In one embodiment, the device includes an ultra-wide-angle infrared binocular stereo camera 1, an ultra-wide-angle visible light binocular stereo camera 2, an FPGA controller 3, an ARM high-speed processor 4, a data switch 5, and a housing 6. The synchronization control module and the preprocessing module are implemented by the FPGA controller 3, and the fusion module is implemented by the ARM high-speed processor 4.

[0086] The housing 6 in this invention encapsulates all devices, preventing camera position changes due to external forces and protecting the ARM high-speed processor from external interference. The ultra-wide-angle infrared binocular stereo camera 1 is a far-infrared thermal imaging camera with a resolution of 1280*720p, and the ultra-wide-angle visible light binocular stereo camera 2 is a visible light camera with a resolution of 1280*720p. The ultra-wide-angle infrared binocular stereo camera 1 and the ultra-wide-angle visible light binocular stereo camera 2 are horizontally parallel to each other on the housing 6, and the imaging base planes of both cameras are on the same vertical plane. The baseline of the center point of the imaging base plane of the cameras should be consistent, and they are mainly responsible for sensing objects in front of the lens. The ultra-wide-angle infrared binocular stereo camera 1 and the visible ultra-wide-angle light binocular stereo camera 2 are connected to the FPGA controller 3 via an LVDS harness. The FPGA controller 3 is connected to the ARM processor via the data switch 5. A high-speed processor 4 is connected; among them, the FPGA camera controller 3 has camera synchronization control capability, serves as a synchronization control module, and also has the function of synchronous communication between FPGAs to ensure that the four cameras run synchronously in the same frame, while calculating point cloud information; the ARM high-speed processor 4 is used for data processing tasks such as image fusion, hole filling, data correction, and 3D point cloud fusion; the data transmission switch 5 is used to assist the ARM processor 4 and the FPGA controller 3 in data communication and to provide a data transmission interface for subsequent applications. The independent device data interface between the FPGA controller 3 and the ARM high-speed processor 4 is a gigabit network port, which is connected through the data transmission switch 5, and the data is transmitted using the UDP protocol.

[0087] In this embodiment, the FPGA controller 3 achieves synchronous control of the ultra-wide-angle infrared binocular stereo camera 1 and the ultra-wide-angle visible light binocular stereo camera 2 by setting a synchronization signal, and acquires infrared image data and visible light image data of the same frame. Each group of cameras should also maintain synchronous exposure and wide dynamic range. A synchronization signal is set between the two FPGAs. After receiving the signal, the camera image data is read simultaneously to maximize the data synchronization of the four cameras. This can avoid the occurrence of ghost images, image trailing, and calibration deviation caused by image asynchrony during movement.

[0088] In one embodiment, the preprocessing module performs stereo matching using the SAD algorithm, and the matching includes:

[0089] The SAD algorithm is used for stereo matching with binoculars. Point cloud information is obtained based on the set out-of-limit matching threshold and successful matching threshold, and the calculation is as follows:

[0090]

[0091] Where disp is the parallax, P L(i, j) represents the grayscale value of the pixel to be matched in the reference image, P R (i, j) represents the grayscale value of the matching point in the registration diagram, and (h, k) represents the size of the moving window.

[0092] In one embodiment, the preprocessing module performs fractional parallax calculation on the matched infrared and visible light images, specifically as follows:

[0093] For the matched infrared and visible light images, any pair of pixels is randomly selected for matching. After the current pixel is successfully matched, the next adjacent pixel is matched. The fractional disparity is calculated and output using the matching value of the current pixel and the matching values ​​of the pixels before and after it.

[0094] In this embodiment, due to slight differences in the intrinsic and extrinsic parameters of individual stereo cameras, there may be vertical, horizontal, or even image distortion during imaging. The camera acquires a checkerboard image, and the image is corrected using a software correction method—the existing stereo camera calibration method in MATLAB—to compensate for the individual differences in the hardware.

[0095] In this embodiment, when performing binocular stereo matching, the positions of the infrared image data and visible light image data are first calibrated and corrected. The pixel deviations caused by the intrinsic and extrinsic parameters of each camera are adjusted to ensure the consistency of the pixels in the horizontal direction and to ensure the accuracy of the matching process.

[0096] Specifically, parameter calibration is performed on the ultra-wide-angle infrared binocular camera 1 and the ultra-wide-angle visible light binocular camera 2. Both ultra-wide-angle infrared and visible light cameras use the same calibration method, namely checkerboard calibration. The ultra-wide-angle infrared binocular stereo camera 1 is calibrated by adding a heating and cooling module to the back of the black and white checkerboard, adjusting its temperature to change the infrared imaging effect. The black and white checkerboard calibration board uses a fixed background scene, and mechanical control allows the calibration board to move omnidirectionally, ensuring that each point of the ultra-wide-angle visible light binocular stereo camera 2 and the ultra-wide-angle infrared binocular stereo camera 1 participates in the calibration calculation. The intrinsic and extrinsic parameters of the ultra-wide-angle infrared binocular camera 1 and the ultra-wide-angle visible light binocular camera 2 are used to correct the infrared and visible light images, obtaining corrected infrared and visible light image data, improving the consistency of the acquired images, and simultaneously achieving accurate and rapid stereo matching of the infrared and visible light image data.

[0097] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for apparatus or system embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The apparatus and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0098] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and the contents not described in detail in the specification of the present invention are well known to those skilled in the art.

Claims

1. An ultra-wide-angle infrared, visible dual-band stereo vision perception method, characterized in that, include: Infrared and visible light images are acquired simultaneously using an ultra-wide-angle infrared binocular stereo camera and an ultra-wide-angle visible light binocular stereo camera. The infrared and visible light image data are respectively subjected to binocular stereo matching using the SAD algorithm. Decimal disparity calculation and smoothing processing are then performed on the matched infrared and visible light images to output the corresponding infrared and visible light 3D point clouds. Specifically, the decimal disparity calculation for the matched infrared and visible light images involves: For the matched infrared and visible light images, any pair of pixels is randomly selected for matching. After the current pixel is successfully matched, the next adjacent pixel is matched. The fractional disparity is calculated using the matching value of the current pixel and the matching values ​​of the pixels before and after it, and the infrared and visible light images with fractional disparity are obtained. The decimal disparity calculation is specifically as follows: When ya <yc,△d=0.5*(ya-yc) / (yc-yb); When ya>yc, △d=0.5*(ya-yc) / (ya-yb); Where ya, yb, and yc represent the SAD values ​​of three adjacent pixels in the data that were successfully matched using SAD, and △d represents the fractional disparity. The disparity value after calculating the fractional disparity is d+△d. The infrared 3D point cloud and the visible light 3D point cloud are fused together to obtain fused 3D point cloud data and fused image data.

2. The ultra-wide-angle infrared and visible dual-band stereovision perception method of claim 1, wherein, The stereo matching using the SAD algorithm is specifically as follows: The SAD algorithm is used for stereo matching with binoculars. Point cloud information is obtained based on the set out-of-limit matching threshold and successful matching threshold, and the calculation is as follows: ； in, disp For parallax, P L ( i, j () represents the grayscale value of the pixel to be matched in the reference image. P R ( i, j ) represents the grayscale value of the matching point in the registration diagram, ( h, k () represents the size of the movable window.

3. The ultra-wideband infrared, visible dual-band stereovision perception method of claim 1, wherein, The smoothing process for the matched infrared and visible light images includes: Calculate the minimum Euclidean distance between adjacent pixel point clouds. If the minimum Euclidean distance between adjacent pixel point clouds is less than a preset threshold, take a single pixel as the center and the 8 adjacent pixels as the value range to take the mode of the pixel value. Replace the pixel value of the pixel whose minimum Euclidean distance is less than the preset threshold with the mode of the pixel value.

4. The ultra-wideband infrared, visible dual-band stereovision perception method of claim 1 or 2, wherein, The smoothing process for the matched infrared and visible light images also includes filling in missing points in the point cloud data of the infrared and visible light images, specifically: Using any pixel with a value of 0 as the center, search for non-zero pixels in eight directions around it until a non-zero pixel is encountered. Then, select the non-zero pixel that appears most frequently (more than 4 times) to fill the non-zero pixel in the stereo matching. 5.The method of claim 1, wherein, Before the step of fusing the infrared 3D point cloud with the visible light 3D point cloud, the method further includes: After fractional disparity calculation and smoothing, infrared and visible light images are used to highlight salient targets in the images through image fusion algorithms.

6. An ultra-wide-angle infrared, visible dual-band stereo-vision perception device, characterized in that, include The synchronization control module is used to control the ultra-wide-angle infrared binocular stereo camera and the ultra-wide-angle visible light binocular stereo camera to simultaneously acquire infrared and visible light images; The preprocessing module is used to perform stereo matching of the infrared image and the visible light image data using the SAD algorithm, and to perform fractional disparity calculation and smoothing processing on the matched infrared and visible light images to obtain the corresponding infrared 3D point cloud and visible light 3D point cloud; the preprocessing module is specifically used for: For the matched infrared and visible light images, randomly select any pair of pixels for matching. After the current pixel is successfully matched, perform matching on the next adjacent pixel. Calculate the fractional disparity output using the matching value of the current pixel and the matching values ​​of the pixels before and after it. The decimal disparity calculation is specifically as follows: When ya <yc,△d=0.5*(ya-yc) / (yc-yb); When ya>yc, △d=0.5*(ya-yc) / (ya-yb); Where ya, yb, and yc represent the SAD values ​​of three adjacent pixels in the data that were successfully matched using SAD, and △d represents the fractional disparity. The disparity value after calculating the fractional disparity is d+△d. The fusion module is used to perform point cloud fusion and image fusion between the infrared three-dimensional point cloud and the visible light three-dimensional point cloud to obtain fused three-dimensional point cloud data and fused image data.

7. The ultra-wideband infrared, visible dual-band stereovision perception apparatus of claim 6, wherein, The preprocessing module is used to perform binocular stereo matching on the infrared image and the visible light image data respectively using the SAD algorithm, specifically including: The SAD algorithm is used for stereo matching with binoculars. Point cloud information is obtained based on the set out-of-limit matching threshold and successful matching threshold, and the calculation is as follows: ； in, disp For parallax, P L ( i, j () represents the grayscale value of the pixel to be matched in the reference image. P R ( i, j ) represents the grayscale value of the matching point in the registration diagram, ( h, k () represents the size of the movable window.

Images