Depth learning based binocular stereo vision ranging method and system for heterogeneous images

Abstract

The present disclosure relates to a deep learning based heterogenous image binocular stereo vision ranging method and system. The method comprises: obtaining a first type image and a second type image, the first and second type images being captured at the same time for the same scene via first and second cameras at preset relative positions and a field of view of the first type image being greater than that of the second type image; processing at least one of the first and second type images; performing coplanar line alignment on the processed first and second type images; predicting a disparity map associated with the first and second type images; and obtaining distance information of a target to be measured in the scene based on the disparity map. In this way, an accurate disparity map can be generated using heterogenous images without increasing the structure of the system, and the distance information of the target can be accurately obtained, while the target can be detected and tracked in different wavebands.

Description

Technical Field

[0001] This disclosure generally relates to the field of computer science, and particularly to a method and system for heterogeneous image binocular stereo vision ranging based on deep learning. Background Technology

[0002] Computer vision is a comprehensive technology that utilizes sensors such as cameras to observe the external world and uses image processing systems to make judgments. However, images acquired by CCD or CMOS cameras typically only retain two-dimensional information, losing the depth information of the target. Therefore, acquiring the three-dimensional information of the target helps users make more accurate and objective judgments. Currently, non-contact optical ranging technologies for objects are mainly divided into three types: structured light method, time-of-flight (TOF) method, and binocular stereo vision method. Among them, structured light method and TOF method are active ranging technologies, while using binocular stereo vision method to obtain the depth information of the target is a passive ranging technology.

[0003] In practical applications, structured light ranging technology is easily affected by ambient lighting, resulting in larger errors when measuring long distances. It is also susceptible to reflections from smooth surfaces such as mirrors. In contrast, Time-of-Flight (TOF) depth cameras utilize the TOF method for ranging, offering higher accuracy over medium to long distances, but exhibiting poorer accuracy for close-up targets.

[0004] Binocular stereo vision works by simulating the human eye's distance perception mechanism. It uses two cameras to capture left and right views of the same scene from different locations at the same time, generating a disparity map. Then, based on triangulation, a depth map is calculated, ultimately revealing the target's distance. Unlike the other two technologies, binocular stereo vision does not require an additional light source for imaging, thus avoiding the need to emit detection signals for distance measurement and revealing the observer's position. This offers advantages such as non-contact operation and strong concealment.

[0005] Compared to two-dimensional images, acquiring three-dimensional information about a target helps users perceive their environment more accurately and objectively. Currently, infrared-visible light imaging is widely used in autonomous driving assistance systems, disaster relief robots, and small drones to achieve visual tasks such as target recognition, tracking, and scene enhancement. However, these visual tasks only use the two-dimensional information of the image, losing the depth information of the target within the image. Researching stereo vision technology using heterogeneous images can not only utilize the complementarity of two different image bands to achieve target recognition and tracking in complex environments, but also simultaneously acquire relative position, depth, and size information of external targets. With the rapid development of multi-sensor data matching and fusion technologies, achieving complementary advantages from multiple sensors and utilizing image information from different bands to realize binocular stereo vision is becoming increasingly feasible.

[0006] Therefore, a stereo vision ranging method and system are needed to effectively improve the efficiency of heterogeneous image acquisition systems, while acquiring relative position information, depth information, and size information of external targets, so as to achieve accurate identification and tracking of targets in complex environments. Summary of the Invention

[0007] According to an example embodiment of this disclosure, a scheme for heterogeneous image binocular stereo vision ranging based on deep learning is provided.

[0008] In a first aspect of this disclosure, a deep learning-based binocular stereo vision ranging method for heterogeneous images is provided. The method includes: acquiring a first type image and a second type image, wherein the first type image and the second type image are captured simultaneously for the same scene by a first camera and a second camera located at preset relative positions, and the field of view of the first type image is larger than that of the second type image; processing at least one of the first type image and the second type image such that the resolution of the first type image is equal to the resolution of the second type image; performing coplanar alignment on the processed first type image and the second type image; inputting the coplanar aligned first type image and the second type image into a deep learning-based stereo matching network model for stereo matching to predict a disparity map associated with the first type image and the second type image; and obtaining distance information of a target to be measured in the scene based on the disparity map.

[0009] In some embodiments, processing at least one of the first type image and the second type image may further include: converting the second type image into a pseudo first type image; and processing the first type image and the pseudo first type image such that the resolution of the first type image is equal to the resolution of the pseudo first type image.

[0010] In some embodiments, the first type of image may include a visible light image; the second type of image may include a long-wave infrared image; the first camera may include a visible light camera; and the second camera may include a long-wave infrared camera.

[0011] In some embodiments, acquiring a first type of image and a second type of image may include: performing single-target calibration on the first camera and the second camera respectively to obtain at least the following parameters of the first camera and the second camera: intrinsic parameter matrix, extrinsic parameter matrix and distortion vector; and capturing the first type of image and the second type of image using the first camera and the second camera.

[0012] In some embodiments, coplanar alignment of the processed first-type image and second-type image may include: stereo calibration of the first camera and the second camera to obtain the rotation and translation relationship between the camera coordinate systems corresponding to the first camera and the second camera respectively; and coplanar alignment of the first-type image and the second-type image based on parameter information associated with the monocular calibration and the stereo calibration.

[0013] In some embodiments, processing at least one of the first type image and the second type image such that the resolution of the first type image is equal to the resolution of the second type image may include: cropping and scaling the first type image so that the resolution of the first type image is equal to the resolution of the second type image.

[0014] In some embodiments, performing coplanar row alignment on the processed first type image and second type image may further include: using the Bouguet algorithm to perform stereo correction on the cropped and scaled first type image and second type image to obtain coplanar row aligned first type image and second type image.

[0015] In some embodiments, obtaining distance information of the target to be measured in the scene based on the disparity map may include: solving for the depth information of the scene using the principle of triangulation based on parameter information corresponding to the first type image and the second type image after coplanar alignment.

[0016] In some embodiments, single-target calibration of the first camera and the second camera may include: in response to determining that the second type of image is a long-wave infrared image, calibrating the second type of image using a calibration plate with a heating plate, wherein the calibration plate has thermal conductivity corresponding to a color grid.

[0017] In a second aspect of this disclosure, a deep learning-based heterogeneous image binocular stereo vision ranging system is provided for performing the method described in the first aspect of this disclosure.

[0018] It should be understood that the description in the Summary of the Invention is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0019] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements. The drawings are provided for a better understanding of the present invention and are not intended to limit the scope of this disclosure, wherein:

[0020] Figure 1 A schematic flowchart of a binocular stereo vision ranging method for heterogeneous images based on deep learning is shown.

[0021] Figure 2 A schematic diagram of the lateral resolution calculation of a visible light image according to some embodiments of the present disclosure is shown;

[0022] Figure 3 A schematic diagram of the epipolar geometry is shown when an object point P is imaged in the left and right cameras of a binocular stereo vision system according to some embodiments of the present disclosure.

[0023] Figure 4 A schematic diagram showing the coplanar alignment of the same spatial object point on the image planes of two cameras, according to some embodiments of the present disclosure;

[0024] Figure 5 A schematic diagram showing the effect of left and right views after stereoscopic correction according to some embodiments of the present disclosure is provided;

[0025] Figure 6 A schematic diagram illustrating a specific implementation scheme of a heterogeneous image binocular stereo vision ranging method according to some embodiments of the present disclosure is shown.

[0026] Figure 7 It shows according to Figure 6 The diagram shows the prediction effect of the scheme, where (a) is a long-wave infrared image, (b) is a visible light image, and (c) is a parallax map.

[0027] Figure 8 A schematic diagram of a specific implementation scheme of a heterogeneous image binocular stereo vision ranging method according to another embodiment of the present disclosure is shown;

[0028] Figure 9 It shows according to Figure 8 The diagram shows the prediction effect of the scheme, where (a) is a long-wave infrared image, (b) is a pseudo-visible light image, (c) is a visible light image, and (d) is a parallax map.

[0029] Figure 10 An overall overview of a network model according to some embodiments of this disclosure is shown;

[0030] Figure 11 An overall overview of a network model according to another embodiment of this disclosure is shown;

[0031] Figure 12 A block diagram of a computing device capable of implementing several embodiments of the present disclosure is shown. Detailed Implementation

[0032] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0033] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0034] As mentioned earlier, current autonomous driving assistance systems, disaster relief robots, and small drones widely use infrared-visible light imaging to achieve visual tasks such as target recognition, tracking, and scene enhancement. These visual tasks only use the two-dimensional information of the image, losing the depth information of the target in the image. However, using heterogeneous binocular stereo vision technology can also obtain the target's relative position information, depth information, and size information. At the same time, in practical applications, the vision system hardware structure also requires high size and stability.

[0035] To address the above issues, embodiments of this disclosure employ a single first camera (e.g., a visible light camera) and a single second camera (e.g., a long-wave infrared camera) to acquire heterogeneous images of the same scene. These images are then input as left and right views into a specially designed deep learning-based stereo matching network to obtain a disparity map. The distance between objects in the scene and the camera array is then calculated using a classic binocular stereo vision pinhole camera model. In this way, a data acquisition system consisting of a single first camera (e.g., a visible light camera) and a single second camera (e.g., a long-wave infrared camera) is constructed. Without increasing the system structure, an accurate disparity map is generated using two heterogeneous images. Furthermore, the distance information of the target, including relative position, depth, and size information, is calculated, for example, based on the principle of triangulation. Simultaneously, due to the imaging advantages of the heterogeneous images, target detection and tracking at different wavelengths can be achieved while simultaneously performing distance measurement. Moreover, the system, with its lightweight hardware structure, can handle more visual processing tasks, making it easy to integrate into drones, automobiles, and intelligent robots to provide visual navigation.

[0036] The following will combine Figures 1 to 12 A detailed description of exemplary embodiments according to this disclosure is provided.

[0037] Figure 1 A schematic flowchart of a binocular stereo vision ranging method for heterogeneous images based on deep learning is shown.

[0038] Reference Figure 1 In box 101, a first type image and a second type image are acquired. The first type image and the second type image are captured simultaneously on the same scene by a first camera and a second camera located at preset relative positions, respectively, and the field of view of the first type image is larger than that of the second type image. In box 103, at least one of the first type image and the second type image is processed so that the resolution of the first type image is equal to the resolution of the second type image. In box 105, the processed first type image and the second type image are aligned in a coplanar row. In box 107, the first type image and the second type image after coplanar row alignment are input into a deep learning-based stereo matching network model for stereo matching to predict the disparity map associated with the first type image and the second type image. In box 109, based on the disparity map, the distance information of the target to be tested in the scene is obtained.

[0039] In one embodiment, the first type of image may be, for example, a visible light image, and the second type of image may be, for example, a long-wave infrared image. Correspondingly, the first camera may be, for example, a visible light camera, and the second camera may be, for example, a long-wave infrared camera. It should be understood that the first type of image, the second type of image, the first camera, and the second camera may also be any other suitable image or camera, and this disclosure does not limit them.

[0040] Visible light images are rich in color, texture, and edge detail, and have high contrast, making them suitable for human observation and target identification. Long-wave infrared images reflect the thermal radiation information of the measured environment, especially since the 8–14 μm long-wave infrared wavelength has strong atmospheric transmission capabilities. Therefore, using long-wave infrared and visible light cameras to create images and leveraging their complementary advantages to establish a binocular system is an effective way to realize multi-domain applications of binocular stereo vision. The following will describe in detail exemplary embodiments of this disclosure, using a first type of image (visible light image), a second type of image (long-wave infrared image), a first camera (visible light camera), and a second camera (long-wave infrared camera).

[0041] In one embodiment, before acquiring visible light and long-wave infrared images, single-camera calibration of the first and second cameras is required to obtain parameters such as the intrinsic parameter matrix, extrinsic parameter matrix, and distortion vector of each camera. Camera calibration is a fundamental preparatory step for achieving binocular stereo vision. In one embodiment, the Zhang Zhengyou checkerboard calibration method can be selected for camera calibration. This method utilizes a linear model camera calibration method to solve for camera parameters and sets these parameters as initial values, while also considering the nonlinear distortion of the lens. Finally, it combines an optimization algorithm to achieve calibration. This method not only has high calibration accuracy but also strong versatility, applicable not only to the calibration of visible light cameras but also to the calibration of long-wave infrared cameras.

[0042] In one embodiment, since long-wave infrared cameras image based on the thermal radiation information of external objects, they cannot perceive the color and texture information of objects. Calibrating the camera using the Zhang Zhengyou checkerboard calibration method requires obtaining accurate black-and-white grid intersections from the image captured by the camera. Ordinary optical calibration plates are not suitable for infrared cameras. To address this problem, in one embodiment, a calibration plate with a heating plate can be used. This calibration plate has different thermal conductivity at different color grids, thus presenting black-and-white squares representing different thermal radiation information on the infrared image. Simultaneously, the surface of the calibration plate also has a checkerboard pattern of black and white blocks, thereby achieving the calibration of a visible light camera. The calibration plate with the heating plate has thermal conductivity corresponding to the color grid.

[0043] In one embodiment, specifically, the Zhang Zhengyou calibration method described above can be used to calculate the intrinsic parameter matrix, extrinsic parameter matrix, and distortion vector of the long-wave infrared camera and the visible light camera, respectively. The intrinsic parameter matrix K is:

[0044]

[0045] Where f is the lens focal length, dx and dy are the pixel sizes of the sensor in the horizontal and vertical directions, respectively, and u0 and v0 are the horizontal and vertical coordinates of the intersection point of the camera optical axis and the imaging plane in the image pixel coordinate system.

[0046] In one embodiment, an infrared camera and a visible light camera simultaneously acquire left and right views. Because the focal lengths of the lenses and the target surface sizes of the two cameras differ, their field of view sizes differ, and the aspect ratios of the original images acquired by the two cameras are also inconsistent. Therefore, the field of view of the other camera must first be adjusted according to the smaller field of view of the camera to ensure that the aspect ratios of the images acquired by the two cameras are the same. Furthermore, for subsequent stereo correction of the two images, their resolutions must be adjusted to ensure that the two images are the same size. In this embodiment, the infrared camera acquires the left view, and the visible light camera acquires the right view. However, in other embodiments, the infrared camera may acquire the right view, and the visible light camera may acquire the left view; this disclosure does not limit this. The following will elaborate on the example of an infrared camera acquiring the left view and a visible light camera acquiring the right view.

[0047] Figure 2 A schematic diagram of the lateral resolution calculation of a visible light image according to some embodiments of the present disclosure is shown.

[0048] In some embodiments, refer to Figure 2 Assume the focal length of the infrared camera is f. ir The pixel size is C ir The number of horizontal pixels is X ir And the number of vertical pixels is Y ir The focal length of a visible light camera is f. vis The pixel size is C vis The number of horizontal pixels is X vis And the number of vertical pixels is Y vis Because visible light cameras have a large field of view, the field of view of infrared cameras is first determined... The visible light image is cropped using the intrinsic parameter matrices of the two cameras. Taking the formula for calculating the horizontal cropping size as an example:

[0049]

[0050] Where M is the horizontal resolution of the cropped visible light image.

[0051] In one embodiment, the cropped visible light image can be scaled according to the resolution of the long-wave infrared camera, even if the image sizes are the same, reducing the size of M, i.e., X. ir =M', where M' is the size of the scaled visible light image. Since M decreases during scaling, this step is equivalent to increasing the 'pixel size' of the visible light camera. If the equation still holds when the resolutions of the two images are the same after scaling, then... Where C vis ′ represents the new pixel size of the visible light image. Furthermore, according to the intrinsic parameter calculation formula... It can be seen that after cropping and scaling the visible light image, some elements in the intrinsic parameter matrices of the two cameras can be adjusted to the same value as much as possible. Then, based on u0 and v0 in the intrinsic parameter matrices obtained by calibrating the two cameras respectively, the part of the visible light image to be cropped can be selected, so that the intrinsic parameter matrices of the two cameras can be kept as similar as possible.

[0052] Figure 3 A schematic diagram illustrating the epipolar geometry of an object point p as it is imaged in the left and right cameras of a binocular stereo vision system according to some embodiments of the present disclosure is shown. In one embodiment, such as Figure 3 As shown, when an object point p in three-dimensional space is imaged in the left and right cameras of a binocular stereo vision system, the epipolar geometry relation is satisfied. (Continue referring to...) Figure 3 Let p be the image point formed by a spatial point p on image plane I1. When searching for a matching point p2 of p1 on image plane I2, it can be found along the epipolar line l2. However, the epipolar line l2 is not parallel to the coordinate axes of the pixel coordinate system. Therefore, it is necessary to search for p2 along an inclined line segment on the right view. If all image points on image plane I1 need to be matched by different inclined line segments on image plane I2 in a two-dimensional coordinate system, the search efficiency will be very low. Therefore, it is necessary to perform stereo correction on the left and right views so that the epipolar lines of the two images are parallel and on the same row in the coordinate system in the horizontal direction. In this way, when searching for a matching point on the right image plane, only a one-dimensional search in the horizontal direction is needed, which improves the matching efficiency.

[0053] Figure 4 The diagram illustrates a coplanar alignment of the image points of the same spatial object point on the image planes of two cameras, according to some embodiments of the present disclosure.

[0054] In one embodiment, such as Figure 4 As shown, stereo correction is performed on two images using the calibrated camera intrinsic and extrinsic parameters and the Bouguet algorithm, ensuring that the epipolar lines of the two images are in the same horizontal direction. This corrects two non-coplanar row-aligned images to coplanar row alignment. Coplanar row alignment means that the image points formed by the same spatial point on the image planes of the two cameras are in the same row on both images. Figure 4 The line connecting O1 and O2 shown is called the baseline.

[0055] Figure 5 This shows a schematic diagram of the left and right views after stereoscopic correction according to some embodiments of the present disclosure. (Refer to...) Figure 5In the actual construction of a binocular stereo vision system, achieving coplanar alignment of the left and right views is quite difficult, thus requiring correction of the camera image plane. In one embodiment, the Bouguet algorithm can be used for correction. This method, based on the known rotation matrix R and translation vector T of the two coordinate system transformations, minimizes the number of reprojections for each image while maximizing the observation area, ultimately achieving the desired effect for both the left and right views after stereo correction. Figure 5 The leftmost image (a) shows a schematic diagram before stereo correction using the Bouguet algorithm, while... Figure 5 The diagram on the right (b) shows the stereo correction after using the Bouguet algorithm. This process preprocesses the image before it is input into the deep learning-based stereo matching network for further matching of heterogeneous images.

[0056] In one embodiment, the two heterogeneous images, after being aligned coplanarly, can be input into a deep learning-based stereo matching network model for stereo matching to predict disparity maps associated with the first and second type images. Traditional stereo matching algorithms mainly consist of semi-global matching algorithms like SGBM and local feature matching algorithms like SIFT. These algorithms employ manually designed feature extraction methods, failing to fully utilize the overall image information, and the setting of empirical parameters can significantly impact the final matching effect. Therefore, their performance is unsatisfactory when the left and right views are heterogeneous images. With the research and application of deep learning in stereo matching, it has demonstrated powerful feature extraction and learning capabilities in image processing. The embodiments of this disclosure address the difficulties in achieving stereo matching caused by factors such as inconsistencies between visible light and infrared camera modules, baseline tilt angles with the horizontal direction, and non-ideal correction of left and right views. They attempt to obtain accurate disparity maps by utilizing a deep learning-based heterogeneous image stereo matching network.

[0057] Based on this, using a deep learning-based heterogeneous image stereo matching network for disparity prediction requires training the model network with long-wave infrared-visible light images and corresponding disparity map datasets. Then, the acquired heterogeneous images are input into the trained stereo matching network to obtain the disparity map. The specific implementation schemes can be divided into two categories.

[0058] Figure 6 A schematic diagram illustrating a specific implementation scheme of a heterogeneous image binocular stereo vision ranging method according to some embodiments of the present disclosure is shown. For example... Figure 6As shown, firstly, a long-wave infrared camera and a visible light camera with fixed relative positions are used to acquire left and right views respectively. Then, stereo correction is performed on these two heterogeneous images. Stereo correction is to correct the original images acquired by the camera according to the camera's calibration parameters. The corrected images are parallel in the same image coordinate system, that is, the pixel positions of corresponding feature points are in the same row in the two images. Then, the stereo-corrected left and right views are input into the trained network model to obtain a disparity map. Finally, the disparity map is used to obtain a depth map, thereby obtaining the distance information of the target to be measured. Figure 7 Then it shows according to Figure 6 The diagram shows the prediction effect of the scheme, where (a) is a long-wave infrared image, (b) is a visible light image, and (c) is a disparity map obtained by network prediction. Figure 7 The prediction results can be obtained, for example, through Figure 10 The network model structure shown is obtained, which will be described in more detail below.

[0059] Figure 8 This illustrates a schematic diagram of a specific implementation scheme for a heterogeneous image binocular stereo vision ranging method according to another embodiment of the present disclosure. Figure 6 Compared to the proposed solution, Figure 8 The proposed scheme first converts the second-type image into a pseudo-first-type image. Specifically, it inputs the acquired infrared image into an infrared-to-visible-light image network to obtain a converted pseudo-visible-light image. Then, it performs an AND operation between the acquired real visible-light image and the converted pseudo-visible-light image. Figure 6 The same operation as in [the previous sentence].

[0060] Figure 8 The example scheme shown utilizes a dual-band imaging system, which allows for the complementary advantages of imaging characteristics across different bands. This benefits the system's target recognition and tracking in various environments, a point that is consistent with... Figure 6 The example schemes shown have the same advantages. On the other hand, first converting the infrared image into a visible light image, and then performing subsequent operations such as stereo matching with the converted result and the real visible light image captured by the visible light camera, is equivalent to converting heterogeneous image matching into homogeneous image matching, which helps to improve the matching effect. This retains the advantages of dual-band imaging while improving the matching effect. The network that converts the infrared image into a visible light image can be pre-trained using an open-source dataset. Figure 9 Then it shows according to Figure 8 The diagram shows the prediction effect of the scheme, where (a) is a long-wave infrared image, (b) is a pseudo-visible light image converted from the long-wave infrared image, (c) is a visible light image, and (d) is a disparity map predicted by the network. Figure 9 The prediction results can be obtained, for example, through Figure 11The network model structure shown is obtained, which will be described in more detail below.

[0061] Figure 10 An overall overview of a network model according to some embodiments of the present disclosure is shown, and Figure 11 An overall overview of a network model according to another embodiment of this disclosure is shown. Figure 10 and Figure 11 For example, they can be used to implement such... Figure 6 and Figure 8 The network model shown.

[0062] like Figure 10 As shown, the input to the specific structure of the network model is a pair of long-wave infrared-visible heterogeneous images. Figure 11 The middle part converts the long-wave infrared image into a pseudo-visible light image before inputting it into the network; the rest of the structure is the same as... Figure 10 Similar. See reference. Figure 10 and Figure 11 The network first extracts and fuses features through a common feature extraction and fusion sub-network, using partially coupled filters with different coupling ratios in different convolutional layers. Then, in the semantic information acquisition sub-network, semantic features from heterogeneous images are obtained as the initial hidden states of the ConvGRU. In the multimodal information acquisition sub-network, geometric encoders are obtained through a 3D regularization network, and attention feature volumes are obtained using correlation volumes as attention weights. These two are combined and input into the next-level network. The network also obtains an initial disparity map from the geometric encoders to accelerate the network's update efficiency. In the cascaded convolutional gated recurrent sub-network, each ConvGRU stage receives the joint encoder from the multimodal information acquisition sub-network, the semantic information from heterogeneous images obtained from the semantic information acquisition sub-network, and the disparity update information passed from the previous ConvGRU stage. After multiple ConvGRU calculations, the disparity map is updated. In this way, the network model can make full use of the complementary advantages of the two spectra, and take into account the semantic information, geometric structure information and local matching details contained in the image. Therefore, it can make more robust disparity prediction of the scene contained in the heterogeneous image pair, thereby improving the accuracy of distance prediction of external targets.

[0063] In some embodiments, a deep learning-based heterogeneous image binocular stereo vision ranging method can be implemented in the system. That is, the system is used to execute the deep learning-based heterogeneous image binocular stereo vision ranging method.

[0064] In summary, in this specific embodiment, firstly, single-target calibration is performed on both cameras to obtain parameters such as the intrinsic parameter matrix, extrinsic parameter matrix, and distortion vector of each camera. To ensure that the intrinsic parameter matrices of the two cameras are as similar as possible, the visible light image of the larger scene needs to be cropped and scaled to maintain the same resolution as the long-wave infrared image. Then, stereo calibration is performed on the camera group to obtain the rotation and translation relationships between the camera coordinate systems corresponding to the left and right cameras. Subsequently, stereo correction is performed on the left and right views captured by the two cameras based on the parameters obtained from single-target calibration and stereo calibration. Then, the two heterogeneous images after stereo correction are input into a stereo matching network based on deep learning to obtain a disparity map. Finally, based on the calculated baseline length and focal length of the camera group, the depth information of the scene contained in the image is solved using the principle of triangulation.

[0065] This approach uses only one long-wave infrared camera and one visible light camera to acquire heterogeneous images of the same scene. These images are then input as left and right views into a specially designed deep learning-based stereo matching network to obtain a disparity map. A classic binocular stereo vision pinhole camera model is then used to calculate the distance between objects in the scene and the camera array. This allows for the generation of an accurate disparity map from the two heterogeneous images without increasing the system's structural complexity. The distance to the target is then calculated based on the principle of triangulation. Furthermore, due to the imaging advantages of the heterogeneous images, target detection and tracking at different wavelengths can be achieved simultaneously with ranging. The system, with its lightweight hardware structure, can handle a wider range of visual processing tasks. Its lightweight and stable design allows it to be mounted on drones, automobiles, and intelligent robots, providing them with visual navigation functions such as target ranging and localization.

[0066] Furthermore, most heterogeneous imaging systems use only one camera with a corresponding band response to acquire images in each band, i.e., a heterogeneous image acquisition system consisting of one infrared camera and one visible light camera, which can greatly reduce system costs. The various embodiments of this disclosure can fully utilize the complementary characteristics of heterogeneous images, and can simultaneously acquire relative position information, depth information, and size information of external targets, and realize target identification and tracking in complex environments.

[0067] It should be noted that the above-mentioned embodiments, in which the first type of image is a visible light image, the second type of image is a long-wave infrared image, the first camera is a visible light camera, and the second camera is a long-wave infrared camera, are merely exemplary. Other images and other types of cameras in the art can also be used to realize the inventive concept of this disclosure, and this disclosure does not limit them.

[0068] Figure 12 A block diagram of a computing device 1200 capable of implementing various embodiments of the present disclosure is shown. The device 1200, as an execution system, can be used, for example, to implement... Figure 1 The method shown or one or more operations within the method.

[0069] like Figure 12 As shown, device 1200 includes a computing unit, such as CPU 1201, which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory ROM 1202 or loaded from storage unit 1208 into random access memory RAM 1203. RAM 1203 may also store various programs and data required for the operation of device 1200. CPU 1201, ROM 1202, and RAM 1203 are interconnected via bus 1204. Input / output (I / O) interface 1205 is also connected to bus 1204.

[0070] Multiple components in device 1200 are connected to input / output (I / O) interface 1205, including: input unit 1206, such as a keyboard, mouse, etc.; output unit 1207, such as various types of displays, speakers, etc.; storage unit 1208, such as a disk, optical disk, etc.; and communication unit 1209, such as a network card, modem, wireless transceiver, etc. Communication unit 1209 allows device 1200 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0071] The computing unit, in addition to the central processing unit (CPU), can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Examples include, but are not limited to, a CPU, a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit performs the various methods and processes described above. For example, in some embodiments, the various methods and processes described above can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 1208. In some embodiments, part or all of the computer program can be loaded and / or installed on device 1200 via ROM 1202 and / or communication unit 1209. When the computer program is loaded into RAM 1203 and executed by CPU 1201, one or more steps of the methods described above can be performed. Alternatively, in other embodiments, the computing unit can be configured to perform the methods described above by any other suitable means (e.g., by means of firmware).

[0072] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload programmable logic devices (CPLDs), and so on.

[0073] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0074] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0075] Furthermore, although the operations are described in a specific order, this should be understood as requiring that such operations be performed in the specific order shown or in sequential order, or requiring that all illustrated operations be performed to achieve the desired result. In certain environments, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented individually or in any suitable sub-combination in multiple implementations.

[0076] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A deep learning-based heterogenous image binocular stereo vision ranging method, characterized in that, The method comprises: acquiring a first type image and a second type image, the first type image and the second type image being captured at the same time for the same scene via a first camera and a second camera at preset relative positions, and a field of view of the first type image being greater than a field of view of the second type image; processing at least one of the first type image and the second type image so that a resolution of the first type image is equal to a resolution of the second type image; performing coplanar line alignment on the processed first type image and the processed second type image; inputting the coplanar line aligned first type image and the coplanar line aligned second type image into a deep learning based stereo matching network model to perform stereo matching to predict a disparity map associated with the first type image and the second type image, wherein the network model comprises a common feature extraction and fusion subnetwork, a semantic information acquisition subnetwork, a multi-modal information acquisition subnetwork, and a cascaded convolution gated recurrent subnetwork, the common feature extraction and fusion subnetwork uses partially coupled filters with different coupling ratios in different convolution layers to extract and fuse features, the semantic information acquisition subnetwork acquires semantic features of heterogeneous images as initial hidden states of a ConvGRU, the multi-modal information acquisition subnetwork acquires a geometry encoding volume through a three-dimensional regularization network, and generates an attention feature volume using a correlation volume as attention weights, both of which are input into a next level network and acquire an initial disparity map from the geometry encoding volume to accelerate updating, and the cascaded convolution gated recurrent subnetwork updates the disparity map through multiple levels of ConvGRU based on a joint encoding volume obtained from the multi-modal information acquisition subnetwork, heterogeneous image semantic information obtained from the semantic information acquisition subnetwork, and disparity update information passed from a previous ConvGRU; and based on the disparity map, acquiring distance information of a target to be measured in the scene. The processing of at least one of the first type image and the second type image further comprises:

2. The method of claim 1, wherein, converting the second type image into a pseudo first type image; and processing the first type image and the pseudo first type image so that the resolution of the first type image is equal to the resolution of the pseudo first type image.

3. The method of claim 1 or 2, wherein: the first type image comprises a visible light image; the second type image comprises a long-wave infrared image; the first camera comprises a visible light camera; and the second camera comprises a long-wave infrared camera. The acquiring of the first type image and the second type image comprises:

4. The method of claim 1, wherein, performing single target calibration on the first camera and the second camera respectively to obtain at least the following parameters of the first camera and the second camera: an intrinsic matrix, an extrinsic matrix, and a distortion vector; and capturing the first type image and the second type image using the first camera and the second camera. The coplanar line alignment on the processed first type image and the processed second type image comprises:

5. The method of claim 4, wherein, ​ stereo-calibrating the first camera and the second camera to obtain a rotation and translation relationship between camera coordinate systems corresponding to the first camera and the second camera, respectively; and co-planar line aligning the first type of image and the second type of image based on parameter information associated with the mono-calibration and the stereo-calibration.

6. The method of claim 5, wherein, processing at least one of the first type of image and the second type of image such that a resolution of the first type of image is equal to a resolution of the second type of image includes: cropping and scaling the first type of image such that a resolution of the first type of image is equal to a resolution of the second type of image.

7. The method of claim 6, wherein, co-planar line aligning the processed first type of image and the second type of image further includes: stereo-correcting the cropped and scaled first type of image and the second type of image using a Bouguet algorithm to obtain the co-planar line aligned first type of image and the second type of image.

8. The method of claim 1, wherein, obtaining distance information of a target object in the scene based on the disparity map includes: solving the depth information of the scene using a triangulation principle based on parameter information corresponding to the co-planar line aligned first type of image and the second type of image.

9. The method of claim 4, wherein, mono-calibrating the first camera and the second camera, respectively, includes: in response to determining that the second type of image is a long-wave infrared image, calibrating the second type of image using a calibration board with a heating plate, and wherein the calibration board has a heat conduction capability corresponding to a color grid.

10. A deep learning-based heterogenous image binocular stereo vision ranging system, characterized in that, the system is configured to perform the method according to any one of claims 1 to 9.

Images