Stereoscopic vision system based on bimodal camera and processing method thereof

Abstract

The invention provides a stereoscopic vision system based on bimodal cameras, which is composed of two cameras with different imaging modes, a data storage unit and a data processing unit, the two cameras are a visible light camera and an infrared camera, the two cameras are connected, and the data storage unit is connected with the data processing unit. The two cameras are connected with the same trigger structure. According to the invention, two cameras with different modalities, namely a visible light camera and an infrared camera, are utilized to form a stereoscopic vision system. The imaging principles and imaging characteristics of the two cameras are greatly different, and the resolution ratios, field angles and other main optical properties of the two modules are completely different. A stereoscopic vision system composed of the two cameras can output three different dimensions of data of visible light color images, infrared images and parallax point clouds at the same time. According to the invention, two cameras with different optical performance parameters and imaging effects are utilized, and the algorithm scheme innovation of stereo matching and parallax point cloud data generation is realized.

Description

Technical Field

[0001] The present invention relates to the field of stereoscopic vision measurement, and in particular to a stereoscopic vision system based on a dual-mode camera and a processing method thereof. Background Art

[0002] Stereoscopic vision system is a technology that obtains three-dimensional spatial information by simulating the human binocular vision mechanism. It usually consists of two or more cameras, which are arranged at a certain baseline distance (i.e. relative position). The core idea of ​​the stereoscopic vision system is to calculate the depth information of the object by comparing images taken from different perspectives, thereby reconstructing the three-dimensional scene.

[0003] Traditional stereo vision systems usually consist of two or more cameras, and these sensors work together to achieve the perception and reconstruction of three-dimensional space.

[0004] The prior art "Stereoscopic Camera Device" (CN103163716B) mainly describes a stereoscopic camera device that can automatically obtain clear images at close range. The main innovation of this patent is that it uses full-frame focusing technology to use color sharpness information to control the focusing distance of the two cameras to obtain clear images, and then generate stereoscopic images, namely disparity maps. Among them, the main innovation of the stereoscopic ray device is to control the automatic focusing technology according to color sharpness. It is mainly based on logical definitions to control the camera module to achieve focal length changes to obtain clear images.

[0005] Prior art "Structural modal parameter identification system based on binocular stereo vision"

[0006] (CN103323209B), mainly describes the use of binocular stereo cameras to collect key point three-dimensional motion data of freely moving structures, and recognize the undamped frequency, damping ratio, formation, modal confidence criterion (MAC) and modal phase collinearity (MPC) of the measured structure based on these motion data. The main innovation of binocular stereo vision is to use binocular vision cameras to collect key point three-dimensional motion data of freely moving structures.

[0007] Prior art: Dynamic stereo calibration method for multispectral cameras based on salient features

[0008] (CN110969670B) mainly describes the calibration algorithm of the relative relationship between the visible light camera and the infrared camera. The main purpose is to calibrate the spatial position between the visible light camera and the infrared camera: the rotation matrix R and the translation vector t.

[0009] The sensing devices mentioned in the above patents are all of the same type. In order to ensure the accuracy and stability of the stereo vision system, these visual sensors must be consistent in modality, that is, they use the same type of sensor technology, such as image sensors based on CMOS or CCD. In addition, the matching of hardware parameters is also crucial, including but not limited to resolution, frame rate, field of view, etc. The consistency of these parameters can ensure that the images captured by the two sensors are highly similar in size and content, thereby providing a reliable data basis for subsequent stereo matching and three-dimensional reconstruction. The final data obtained by different visual sensors is not the same;

[0010] Visible light stereo vision systems can usually provide high-precision depth information, especially under uniform lighting conditions, and can accurately identify the shape and position of objects.

[0011] The acquisition of depth information by infrared stereo vision systems may be affected by temperature changes and reflectivity, which may lead to errors in depth estimation.

[0012] In terms of object recognition capabilities, visible light cameras perform well in object recognition and classification, and can use color and texture information for accurate recognition; infrared cameras can effectively recognize objects in specific applications (such as heat source detection), but their capabilities in color and detail are weaker.

[0013] Visible light stereo vision systems are widely used in scenarios such as robot vision, autonomous driving, and augmented reality that require color and detail information.

[0014] Infrared stereo vision systems are often used in security monitoring, night vision equipment, thermal imaging detection and other fields, especially in low-light environments or when temperature changes need to be detected.

[0015] In general, the difference in the results of the two systems is mainly reflected in image features, environmental adaptability, depth information accuracy, object recognition capabilities, and applicable scenarios. Choosing a suitable system depends on specific application requirements and environmental conditions. However, if you want to output three different dimensions of data, namely visible light color images, infrared images, and parallax point clouds, the existing stereo vision system cannot meet the requirements. Summary of the invention

[0016] In view of the above technical problems, the present invention utilizes two cameras with completely different optical performance parameters and imaging effects to achieve stereo matching and generate a disparity point cloud data algorithm.

[0017] The present invention proposes a stereoscopic vision system based on a dual-modal camera, which consists of cameras with two different imaging modes, a data storage unit, and a data processing unit. The two cameras are a visible light camera and an infrared camera. The cameras are connected to each other, and the data storage unit is connected to the data processing unit.

[0018] Both cameras are connected to the same trigger structure.

[0019] The two cameras must be triggered to expose synchronously to ensure the time synchronization of the data. In terms of hardware solution, the same trigger source issues exposure instructions to the ISPs of the two cameras to ensure that the dual-mode cameras collect data of the same scene at the same time, and each follows the adaptive optimal exposure management.

[0020] The processing method of the stereo vision system based on the dual-modal camera is:

[0021] Step 1: Two cameras are triggered to expose synchronously and collect data of the same scene at the same time;

[0022] Step 2: The acquired dual-modal data is stored in a data storage unit, and then read by a data processing unit to perform stereoscopic vision point cloud data calculation;

[0023] Step 3: Correct the distortion of the acquired data;

[0024] Step 4: Perform semantic segmentation on the corrected data;

[0025] Step 5: Merge the structural features and semantic features under the same modality;

[0026] Step 6: Calculate the matching cost of the processed data;

[0027] Step 7: Perform cost aggregation processing on the calculated data;

[0028] Step 8: Perform parallax optimization on the processed data.

[0029] The dual-modal distortion correction goal is:

[0030] (1) Remove the distortion caused by imaging and restore the actual physical structure of the target object in the image;

[0031] (2) The dual-mode camera is aligned with the epipolar line to ensure the smooth progress of subsequent work. Whether it is a visible light camera or an infrared camera, the de-distortion correction scheme is very mature and will not be repeated in this example. Here, only the integrity of the process is maintained.

[0032] The pixel data of visible light camera images represents color intensity, while the pixel data of infrared camera images represents temperature intensity. Therefore, feature data must be extracted from pixel data to correctly describe the same part of the same object after being imaged in two different modality cameras.

[0033] Both cameras have fixed focal length and are not hardware adjustable.

[0034] The specific method for processing the structural features in step 5 is:

[0035] The structural feature encoding is defined as the following formula (1). In fact, the feature encoding here can be expertly designed according to the hardware and imaging effects of different modal cameras, including but not limited to the content of formula (1):

[0036] Strc(x,y)={line (x，y) , corner (x，y) , 2order (x，y) , pts (x，y)}

[0037] Among them, Strc (x, y) represents the pixel point;

[0038] Structural feature encoding of (x, y);

[0039] line (x，y) Represents linear encoding and describes edge features;

[0040] corner (x，y) Represents angle coding and describes angle features;

[0041] 2order (x，y) Represents the second-order moment, describing statistical characteristics;

[0042] pts (x，y) Indicates feature point coding and describes point features;

[0043] According to the above definitions, the features are described from the perspective of set constraints such as point features, line features, and angle features. In addition, statistical features are introduced to describe local distribution. This feature encoding definition can be simultaneously applied to the imaging of camera data of different modalities for feature extraction.

[0044] The specific method of semantic feature processing in step 5 is:

[0045] For image data of different modalities, AI-based semantic segmentation models are trained. The model can output the target-level semantic segmentation results (Seg) based on the modal image and the corresponding pixel-level confidence (Conf) data. Semantic segmentation models for image data of different modalities are generally not interchangeable. Considering the imaging differences between cameras of different modalities, each camera of each modality will correspond to a semantic segmentation model belonging to that modality.

[0046] The design, training, and use of AI-based semantic models are relatively mature technologies in the industry and will not be discussed in detail here. They are only used as a link to ensure the integrity of the process.

[0047] The specific method of the set merging process in step 5 is:

[0048] The structural features and semantic features under the same modality are combined to obtain the following formula (2) to describe the feature encoding of image data of a certain modality:

[0049]

[0050] in, Indicates the feature code of the image data of camera t at position (x, y); strc (x，y) Represents the structural features at the (x, y) position, seg (x，y) Represents the semantic feature at the (x, y) position.

[0051] Step 6:

[0052] The matching cost describes the similarity between two feature sets and is defined as follows:

[0053]

[0054] Among them, f and g represent two feature sets, cost (f，g) Represents the matching cost of f and g.

[0055] Step 7:

[0056] In the cost aggregation stage, it is assumed that the same target object is continuous in space, that is, the feature changes between two adjacent pixels on the same target object are smooth in any camera image. Based on this plane assumption, the following propagation formula (4) is defined:

[0057]

[0058] Among them, L r (p, d) represents the aggregation cost when the disparity value of the point with pixel coordinate p on path r is d;

[0059] c(p, d) represents the matching cost when the disparity value of the point with pixel coordinate p is d;

[0060] pr represents the previous pixel along path r when p is used as the reference point;

[0061] The ⊙ symbol defines a logical operation, as shown in formula (5):

[0062]

[0063] If A and B are the same target, the confidence scores are multiplied; if they are not the same target, 1 is output.

[0064] The final aggregate cost is the average of the aggregate costs on each path, that is:

[0065]

[0066] The above problem is a problem of dynamic programming to select the optimal path. The final optimal path is that the disparity data of each pixel ensures that the aggregation cost L reaches the minimum value.

[0067] The specific processing method of step 8 is:

[0068] The disparity optimization stage mainly includes left-right mutual detection, occlusion processing, hole filling, etc. Based on this, this example introduces the disparity accuracy optimization work, that is, based on the disparity value and cost value of the current pixel, the sub-pixel disparity accuracy of the current pixel position is estimated, as defined in formula (6):

[0069]

[0070] in, It refers to the sub-pixel estimated value at position p on the image and at the moment when the disparity is d;

[0071] L d-1 , L d , L d+1 They represent the cost values ​​when the parallax is d-1, d, and d+1 at position p on the image respectively;

[0072] conf d Represents the semantic confidence of the pixel p position.

[0073] The sub-pixel disparity at position p on the image is defined as:

[0074]

[0075] The beneficial effects of the present invention are as follows: The present invention proposes to use two cameras of different modes: a visible light camera and an infrared camera to form a stereoscopic vision system. Using the image data of the two cameras, the image matching and the generation of the disparity point cloud are realized through an algorithm; the imaging principles and imaging characteristics of the two cameras are quite different, and the main optical properties of the two modules, such as the resolution and field of view, are also completely different. The stereoscopic vision system composed of these two cameras can simultaneously output data of three different dimensions: visible light color image, infrared image, and disparity point cloud. The present application uses two cameras with completely different optical performance parameters and imaging effects to realize stereo matching and generate an innovative algorithm solution for disparity point cloud data.

[0076] This application combines a visible light camera and an infrared camera. The system can simultaneously output data in three different dimensions: visible light color image, infrared image and parallax point cloud, providing users with a more comprehensive and three-dimensional visual experience.

[0077] The imaging principles and imaging characteristics of visible light cameras and infrared cameras are quite different. The present invention cleverly utilizes this difference to achieve complementary imaging in different scenarios, thereby improving the adaptability and reliability of the system.

[0078] The main optical properties of the two cameras, such as resolution and field of view, are completely different. The present invention achieves the goal of expanding the field of view while ensuring high resolution by optimizing algorithms and system design, so that the system can capture objects more comprehensively.

[0079] The present invention proposes an algorithm for achieving stereo matching and generating disparity point cloud data based on two cameras with completely different optical performance parameters and imaging effects. The algorithm innovatively solves the problems of low matching accuracy and difficulty in generating disparity point cloud in traditional stereo vision systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 It is a schematic diagram of the present invention;

[0081] Figure 2 It is a schematic diagram of the processing method of the present invention;

[0082] Figure 3 Schematic diagram of the processing method of the present invention. DETAILED DESCRIPTION

[0083] Example 1

[0084] Dual-mode stereo vision refers to a binocular stereo vision sensor composed of two cameras with different imaging modes. Figure 1 In the figure, camera ① and camera ② are visual sensors with two different imaging modes. In this example, camera ① is a visible light visual sensor and camera ② is an infrared visual sensor.

[0085] Since the two cameras have different imaging modes, it is generally difficult to make the hardware parameters consistent. In this example, we take the common different hardware parameter solutions as an example. Figure 1 As shown in , the FOV and resolution of camera ① and camera ② are different. If they are two cameras of different modalities with exactly the same hardware parameters, they are also fully applicable to the solution described in this example.

[0086] Figure 2 As shown in the figure, the hardware diagram of the dual-modal stereo vision system requires that the two cameras must be triggered and exposed synchronously to ensure the time synchronization of the data. In the hardware solution, the same trigger source issues exposure instructions to the ISPs of the two cameras to ensure that the dual-modal cameras collect data of the same scene at the same time, and each follows the adaptive optimal exposure management.

[0087] The acquired dual-modal data, which in this example respectively corresponds to the visible light image of camera ① and the infrared image of camera ②, are stored in a data storage device and then read by a data processor to perform stereoscopic vision point cloud data calculation.

[0088] 2. Dual-mode distortion correction

[0089] For visible light camera ① and infrared camera ②, distortion image correction is performed based on their respective camera internal parameters and the binocular external parameters of the stereo vision system. The goals of dual-modal distortion correction are: (1) to remove the distortion caused by imaging and restore the actual physical structure of the target object in the image; (2) to make the dual-modal cameras achieve epipolar alignment to ensure the smooth progress of subsequent work. Whether it is a visible light camera or an infrared camera, the distortion correction scheme is very mature and will not be repeated in this example. Here, only the integrity of the process is maintained.

[0090] 3. Structural characteristics

[0091] For dual-mode cameras, the physical meanings of the pixel data of the corresponding images are inconsistent. For example, in this example, the pixel data of the visible light camera image ① represents the color intensity, and the pixel data of the infrared camera image ② represents the temperature intensity. Therefore, feature data must be extracted from pixel data to correctly describe the same part of the same object after imaging in two different modal cameras.

[0092] In this example, the structural feature coding is defined as the following formula (1). In fact, the feature coding here can be expert-designed according to the hardware and imaging effects of different modal cameras, including but not limited to the content of formula (1):

[0093] Strc(x,y)={line (x，y) , corner (x，y) , 2order (x，y) , pts (x，y)}

[0094] Among them, Strc(x, y) represents the structural feature encoding for the pixel point (x, y);

[0095] line (x，y) Represents linear encoding and describes edge features;

[0096] corner (x，y) Represents angle coding and describes angle features;

[0097] 2order (x，y) Represents the second-order moment, describing statistical characteristics;

[0098] pts (x，y)Indicates feature point coding and describes point features;

[0099] According to the above definition, the features are described from the perspective of set constraints such as point features, line features, and angle features; in addition, statistical features are introduced to describe local distribution. This feature encoding definition can be applied to the imaging of different modal visual sensor data for feature extraction. In this example, the above formula (1) is used to extract features from visible light camera image ① and infrared camera image ②.

[0100] 4. Semantic Features

[0101] For image data of different modalities, AI-based semantic segmentation models are trained. The model can output the target-level semantic segmentation results (Seg) based on the modal image and the corresponding pixel-level confidence (Conf) data. Semantic segmentation models for image data of different modalities are generally not interchangeable. Considering the imaging differences between cameras of different modalities, each camera of each modality will correspond to a semantic segmentation model belonging to that modality.

[0102] The design, training, and use of AI-based semantic models are relatively mature technologies in the industry and will not be discussed in detail here. They are only used as a link to ensure the integrity of the process.

[0103] 5. Feature Collection

[0104] The structural features and semantic features under the same modality are combined to obtain the following formula (2) to describe the feature encoding of image data of a certain modality:

[0105]

[0106] in, Indicates the feature code of the image data of camera t at position (x, y); strc (x，y) Represents the structural features at the (x, y) position, seg (x，y) Represents the semantic feature at the (x, y) position.

[0107] 6. Matching cost calculation

[0108] The matching cost describes the similarity between two feature sets and is defined as follows:

[0109]

[0110] Among them, f and g represent two feature sets, cost (f，g) Represents the matching cost of f and g.

[0111] 7. Cost Aggregation

[0112] In the cost aggregation stage, it is assumed that the same target object is continuous in space, that is, the feature changes between two adjacent pixels on the same target object are smooth in any camera image. Based on this plane assumption, the following propagation formula (4) is defined:

[0113]

[0114] Among them, L r (p, d) represents the aggregation cost when the disparity value of the point with pixel coordinate p on path r is d;

[0115] c(p, d) represents the matching cost when the disparity value is d at the point with pixel coordinate p; pr represents the previous pixel along path r when p is used as the reference point;

[0116] The ⊙ symbol defines a logical operation, as shown in formula (5):

[0117]

[0118] If A and B are the same target, the confidence scores are multiplied; if they are not the same target, 1 is output.

[0119] The final aggregate cost is the average of the aggregate costs on each path, that is:

[0120]

[0121] The above problem is a problem of dynamic programming to select the optimal path. The final optimal path is that the disparity data of each pixel ensures that the aggregation cost L reaches the minimum value.

[0122] 8. Parallax Optimization

[0123] The disparity optimization stage mainly includes left-right mutual detection, occlusion processing, hole filling, etc. Based on this, this example introduces the disparity accuracy optimization work, that is, based on the disparity value and cost value of the current pixel, the sub-pixel disparity accuracy of the current pixel position is estimated, as defined in formula (6):

[0124]

[0125] in, It refers to the sub-pixel estimated value at position p on the image and at the moment when the disparity is d;

[0126] L d-1 , L d , L d+1 They represent the cost values ​​when the parallax is d-1, d, and d+1 at position p on the image respectively;

[0127] conf dRepresents the semantic confidence of the pixel p position.

[0128] The sub-pixel disparity at position p on the image is defined as:

[0129]

[0130] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. The various components mentioned in the present invention are common technologies in the existing field. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only to illustrate the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection claimed by the present invention is defined by the attached claims and their equivalents.

Claims

1. A stereo vision system based on a dual-modal camera, characterized in that The device is composed of two cameras with different imaging modes, a data storage unit and a data processing unit. The two cameras are a visible light camera and an infrared camera. The cameras are connected to each other, and the data storage unit is connected to the data processing unit.

2. A stereo vision system based on a dual-modal camera according to claim 1, characterized in that The two cameras are connected to the same trigger structure, and both cameras have fixed focal lengths and cannot be focused by hardware.

3. A stereo vision system based on a dual-modal camera according to claim 1, characterized in that The processing method of the stereo vision system based on the dual-modal camera is: Step 1: Two cameras are triggered to expose synchronously and collect data of the same scene at the same time; Step 2: The acquired dual-modal data is stored in a data storage unit, and then read by a data processing unit to perform stereoscopic vision point cloud data calculation; Step 3: Correct the distortion of the acquired data; Step 4: Perform semantic segmentation on the corrected data; Step 5: Merge the structural features and semantic features under the same modality; Step 6: Calculate the matching cost of the processed data; Step 7: Perform cost aggregation processing on the calculated data; Step 8: Perform parallax optimization on the processed data.

4. A dual-modal camera-based stereo vision system according to claim 3, characterized in that: The dual-modal distortion correction goal is: (1) Remove the distortion caused by imaging and restore the actual physical structure of the target object in the image; (2) The dual-mode camera is aligned with the epipolar line to ensure the smooth progress of subsequent work. Whether it is a visible light camera or an infrared camera, the de-distortion correction scheme is very mature and will not be repeated in this example. Here, only the integrity of the process is maintained.

5. A dual-mode camera-based stereo vision system according to claim 3, characterized in that The specific method for processing the structural features in step 5 is: The structural feature encoding is defined as the following formula (1). In fact, the feature encoding here can be expertly designed according to the hardware and imaging effects of different modal cameras, including but not limited to the content of formula (1): Strc(x,y)={line (x，y) ,corner (x，y) ,2order (x，y) ,pts (x，y) }; Among them, Strc (x, y) represents the pixel point; Structural feature encoding of (x, y); line (x，y) Represents linear encoding and describes edge features; corner (x，y) Represents angle coding and describes angle features; 2order (x，y) Represents the second-order moment, describing statistical characteristics; pts (x，y) Indicates feature point coding and describes point features; According to the above definitions, the features are described from the perspective of set constraints such as point features, line features, and angle features. In addition, statistical features are introduced to describe local distribution. This feature encoding definition can be simultaneously applied to the imaging of camera data of different modalities for feature extraction.

6. A dual-modal camera-based stereo vision system according to claim 3, characterized in that: The specific method of semantic feature processing in step 5 is: For image data of different modalities, an AI-based semantic segmentation model is trained respectively. The model can output the target-level semantic segmentation result (Seg) based on the modal image and the corresponding pixel-level confidence (Conf) data. The semantic segmentation models for image data of different modalities are generally not interchangeable. Considering the imaging differences between cameras of different modalities, each modality camera will correspond to a semantic segmentation model belonging to that modality.

7. A dual-modal camera-based stereo vision system according to claim 3, characterized in that: The design, training, and use of AI-based semantic models are relatively mature technologies in the industry and will not be discussed in detail here. They are only used as a link to ensure the integrity of the process.

8. A dual-modal camera-based stereo vision system according to claim 3, characterized in that: The specific method of the set merging process in step 5 is: The structural features and semantic features under the same modality are combined to obtain the following formula (2) to describe the feature encoding of image data of a certain modality: in, Indicates the feature code of the image data of camera t at position (x, y); strc (x，y) Represents the structural features at the (x, y) position, seg (x，y) Represents the semantic feature at the (x, y) position. Step 6: The matching cost describes the similarity between two feature sets and is defined as follows: Among them, f and g represent two feature sets, cost (f，g) Represents the matching cost of f and g.

9. A dual-modal camera-based stereo vision system according to claim 3, characterized in that: Step 7: In the cost aggregation stage, it is assumed that the same target object is continuous in space, that is, the feature changes between two adjacent pixels on the same target object are smooth in any image of the camera modality. Based on this plane assumption, the following propagation formula (4) is defined: Among them, L r (p, d) represents the aggregation cost when the disparity value of the point with pixel coordinate p on path r is d; c(p, d) represents the matching cost when the disparity value of the point with pixel coordinate p is d; pr represents the previous pixel along path r when p is used as the reference point; The ⊙ symbol defines a logical operation, as shown in formula (5): If A and B are the same target, the confidence scores are multiplied; if they are not the same target, 1 is output. The final aggregate cost is the average of the aggregate costs on each path, that is: The above problem is a problem of dynamic programming to select the optimal path. The final optimal path is that the disparity data of each pixel ensures that the aggregation cost L reaches the minimum value.

10. A dual-modal camera-based stereo vision system according to claim 3, characterized in that: The specific processing method of step 8 is: The disparity optimization stage mainly includes left-right mutual detection, occlusion processing, hole filling, etc. Based on this, this example introduces the disparity accuracy optimization work, that is, based on the disparity value and cost value of the current pixel, the sub-pixel disparity accuracy of the current pixel position is estimated, as defined in formula (6): in, It refers to the sub-pixel estimated value at position p on the image and at the moment when the disparity is d; L d-1 , L d , L d+1 They represent the cost values ​​when the parallax is d-1, d, and d+1 at position p on the image respectively; cof d Represents the semantic confidence of the pixel p position. The sub-pixel disparity at position p on the image is defined as:

Images