Cross-modal contrast learning method for binocular stereoscopic vision

Abstract

The invention belongs to the technical field of computer vision, automatic driving, stereoscopic vision and algorithm design, and discloses a cross-modal contrast learning method for binocular stereoscopic vision. RGB features and event features from a left view and a right view of a camera are input, positive and negative examples are sampled through a cross-view-angle comparison module, a cross-modal comparison module and a cross-time-domain comparison module, a contrast learning loss function is constructed, and binocular cross-modal feature fusion is trained. The method mainly solves the problem of how to improve the quality of an output depth map by combining an event camera, and focuses on training an existing feature fusion module by using a relationship between a binocular view and different modals compared with a modal fusion method which mainly designs a network fusion module in recent years. Binocular RGB and event camera fusion under different visual angles, different modes and different time domains can be considered at the same time, and the calculation burden of the prediction stage is not additionally increased while more accurate matching is obtained.

Description

Technical Field

[0001] The present invention relates to the technical fields of computer vision, autonomous driving, stereoscopic vision and algorithm design, and in particular to a cross-modal contrast learning method for binocular stereoscopic vision. Background Art

[0002] With the growing demand for autonomous driving and 3D reconstruction applications, binocular stereo depth estimation has received increasing attention in the fields of computer vision and robotics research. In recent years, binocular stereo matching methods based on deep learning have achieved great success, but stereo matching may be ambiguous when the lighting conditions are poor and the movement is rapid. Since this is a limitation of the RGB sensor itself, deep learning methods alone cannot completely solve this problem. An effective solution is to fuse other sensors (such as the event camera used in the present invention) to make up for the shortcomings of dynamic range and motion blur, and to improve the representation ability through the fusion of multimodal data, thereby reducing matching ambiguity and depth estimation errors.

[0003] Event cameras (such as dynamic vision sensors DVS) respond to changes in brightness of each pixel and have many potential advantages such as high temporal resolution, low latency, and high dynamic range. Fusion of event cameras and traditional RGB cameras theoretically contains more scene details, is not affected by motion blur, and has high dynamic range and high temporal resolution. In recent years, a large number of methods have used the complementarity of these two sensors and combined with deep learning to invent new binocular stereo matching technology.

[0004] These methods have the following characteristics:

[0005] 1. Deep learning trains network models through massive data-driven training. Compared with traditional methods, it is less restricted by factors such as texture, reflection conditions, and object materials. It can reduce matching ambiguity and thus improve the quality of the output disparity map (or depth map);

[0006] 2. This type of method generally fuses feature maps extracted from RGB images and event data by designing network modules, but lacks cross-modal and cross-perspective comparisons between the two feature maps and corresponding training schemes;

[0007] 3. Most methods ignore the temporal characteristics of event features, and mainly use convolutional neural networks to spatially fuse RGB features and event features.

[0008] Based on the above analysis, a training method for a stereo vision network that simultaneously considers binocular RGB and event camera fusion under different perspectives, different modalities, and different time domains is a technical difficulty to be overcome, which is also the motivation for the invention of the present invention. Summary of the invention

[0009] The purpose of the present invention is to solve the problem that the depth map calculated by the traditional stereo matching method based on RGB camera is restricted by factors such as texture, reflection conditions, and object material. It mainly solves the problem of how to improve the quality of the output depth map by combining with event cameras; it solves the problem that the fusion method of RGB camera and event camera in recent years mainly relies on the design of fusion network module, lacks cross-modal and cross-perspective comparison and corresponding training between the two feature maps, and ignores the timing information of the event camera, resulting in poor fusion effect.

[0010] The technical solution of the present invention is as follows: a cross-modal contrast learning method for binocular stereo vision, the cross-modal contrast learning method comprising three sub-contrast modules: a cross-view contrast module, a cross-modal contrast module and a cross-time domain contrast module;

[0011] The left view features and the right view features are input into three contrast modules. All the positive sample sets and negative sample sets obtained by the three contrast modules are used to train the binocular vision estimation network model by designing a contrast learning loss function to learn and obtain a feature map that contrasts different viewpoints, different modalities, and different time domains.

[0012] The cross-view comparison module regards a certain pixel anchor point in the left view feature as a positive sample, and a non-matching pixel point in the right view feature as a negative sample, and samples the entire feature map to obtain a cross-view positive sample set and a negative sample set.

[0013] The cross-modal comparison module disrupts the RGB feature pixel positions in the left view features so that the event features and RGB features of the left view are no longer pixel-aligned. The disrupted pixel points are regarded as a negative sample, and a cross-modal negative sample set is obtained by sampling on the entire feature map.

[0014] The cross-time domain comparison module fuses the event features at different timestamps with the RGB features. For the fused features at the same timestamp, its pixel point is regarded as a negative sample, and a cross-time domain negative sample set is obtained by sampling on the entire feature map.

[0015] The left view feature includes a left view RGB feature I l , Left view event feature E l , the right view features include the right view RGB features I r and right view event feature E r ; Left view RGB feature I l The pixel at coordinate (u, v) and timestamp t0 is represented as

[0016] The feature fusion module in the binocular vision estimation network model is abstracted into a function F(·), and the feature fusion module fuses the RGB features and event features under the same view into a new fusion feature;

[0017] The sample acquisition of the cross-view comparison module is as follows: Set the left view feature F(I l ,E l ) with coordinates (u, v) and timestamp t0 is the anchor point Use the left view disparity to find the matching pixels in the right view at the same timestamp where u m = ud, where d = D(u, v) is the pixel point matched by the disparity value of the disparity map D at (u, v) in the data set, which is regarded as a positive example for cross-view comparison Non-matching pixel pairs are considered negative samples Negative samples are sampled only in a local window around the matching pixel, where the window is a square with a length and width of w.

[0018] The sample acquisition of the cross-modal comparison module is as follows:

[0019] First, the pixel positions of the RGB features are disrupted so that the event features and RGB features are no longer pixel-aligned; the disrupted event features and RGB features are matched to find the pixel position relative to the anchor point p a Negative examples of Where u′≠u or v′≠v;

[0020] The sample acquisition of the cross-time domain comparison module is as follows:

[0021] The timestamp of the anchor point is t0. Sample the timestamp different from the anchor point in the local time window to create a a Negative examples for cross-temporal comparison where t i ≠t0;

[0022] In each training iteration, all contrast examples are used and the contrastive learning loss function is defined as follows:

[0023]

[0024] Where S is the total number of valid pixels in the feature map, and τ is the temperature coefficient.

[0025] The comparison samples include positive samples for cross-view comparison Negative Examples for Cross-View Comparison Negative Examples for Cross-Modal Comparison Negative examples compared with cross-temporal comparison

[0026] The temperature coefficient is 0.07.

[0027] Beneficial effects of the present invention: The proposed cross-modal contrast learning method for binocular stereo vision is a cross-modal, cross-perspective, and cross-time domain contrast learning method that can be applied to the field of binocular stereo vision that integrates RGB cameras and event cameras. The difference between this method and other methods that fuse two modal feature maps by designing a network model is that the feature discrimination ability is improved by contrastive learning training methods, so it is only used in the training process, and will not increase additional video memory usage during the prediction process. This training method can be easily combined with other methods as a special loss function for joint training. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 Obtain sample schematics for cross-view comparison;

[0029] Figure 2 Obtain sample diagrams for cross-modal comparison;

[0030] Figure 3 A schematic diagram of samples obtained for cross-time domain comparison. DETAILED DESCRIPTION

[0031] The cross-modal contrastive learning method includes three sub-modules: cross-view contrast module, cross-modal contrast module and cross-temporal contrast module.

[0032] 1. For a certain pixel anchor point in the left view feature, the cross-view comparison module regards the matching pixel point in the right view feature as a positive sample and the non-matching pixel point as a negative sample. By sampling on the entire feature map, the cross-view positive sample set and negative sample set can be obtained.

[0033] 2. The cross-modal comparison module disrupts the RGB feature pixel positions of the left view so that the event features of the left view are no longer pixel-aligned with them. Such a pixel point can be regarded as a negative sample. By sampling on the entire feature map, a cross-modal negative sample set can be obtained.

[0034] 3. The cross-time domain comparison module fuses the event features at different timestamps with the RGB features. For the fused features at the same timestamp, its pixel point can be regarded as a negative sample. By sampling on the entire feature map, a cross-time domain negative sample set can be obtained.

[0035] All positive sample sets and negative sample sets obtained through the three comparison modules can be used to train the binocular vision estimation network model by designing a contrast learning loss function to learn and obtain a feature map that compares different perspectives, different modalities, and different time domains, helping to reduce matching ambiguity. The binocular vision estimation network model can use any existing binocular vision estimation network, such as PSMNet, GCNet, StereoNet, etc.

[0036] The input of the present invention includes four parts, namely, the RGB feature I of the left view l , Event feature E of the left view l , RGB features of the right view I r And the event feature E of the right view r , assuming I l The pixel at coordinate (u, v) and timestamp t0 can be expressed as Assume that the feature fusion module in the binocular stereo matching network can be abstracted into a function F(·), which can fuse the RGB features and event features in the same view into a new fusion feature (for example, the fusion feature of the left view can be expressed as ). A specific implementation includes four steps, namely, cross-view comparison sample acquisition, cross-modality comparison sample acquisition, cross-time domain comparison sample acquisition and training. The details are as follows:

[0037] (1) Obtaining cross-view comparison samples

[0038] Set the fusion feature F(I l ,E l ) in the coordinates (u, v) and timestamp t0 is the anchor point The disparity of the left view can be used to find the matching pixels of the right view at the same timestamp. where u m = ud, where d = D(u,v) is the disparity value of the disparity map D at (u,v) in the dataset. This matching pixel point is regarded as a positive example for cross-view comparison Correspondingly, non-matching pixel pairs are considered negative samples. In order to reduce the amount of calculation, negative samples are only selected in the local window around the matching pixel, and the window is a square with a length and width of w. See the attached diagram for a schematic diagram Figure 1 shown.

[0039] (2) Cross-modal comparison examples

[0040] First, the pixel positions of the RGB features are disrupted so that the event features and the light intensity features are no longer pixel-aligned. Then, the disrupted event features and RGB features are matched to find the pixel positions relative to the anchor point p. a Negative examples of The mathematical expression can be expressed as Where u′≠u or v′≠v. See the attached diagram Figure 2 shown.

[0041] (3) Cross-time domain comparison examples

[0042] Assuming that the timestamp of the anchor point is t0, sampling the timestamp different from the anchor point in the local time window can create a a Negative examples for cross-temporal comparison where t i ≠t0. See the attached diagram Figure 3 shown.

[0043] (4) Training

[0044] In each training iteration, all contrast examples (including positive and negative examples for cross-view contrast, negative examples for cross-modal contrast, and negative examples for cross-temporal contrast) are used to define the contrastive learning loss function as follows:

[0045]

[0046] Where S is the total number of valid pixels in the feature map, and τ is the temperature coefficient (for example, the value is 0.07).

[0047] The created contrastive learning loss function can be used to train the feature fusion module in the binocular vision estimation network, or it can be combined with the loss function of supervised depth estimation to perform end-to-end training of the entire network. For example, in the training of PSMNet, based on the loss function of the original method, the contrast module proposed in the present invention samples a set of positive and negative samples and calculates the contrastive learning loss function according to the above formula, which is added to the original loss function and used to train the network. Compared with training using only the original loss function, the final predicted depth map is more accurate.

[0048] The present invention mainly solves the problem of how to improve the quality of the output depth map by combining an event camera. Compared with the modal fusion method in recent years that mainly designs a network fusion module, the present invention focuses on using the relationship between binocular views and different modalities to train the existing feature fusion module, which can simultaneously consider the fusion of binocular RGB and event cameras under different viewing angles, different modalities, and different time domains, while obtaining a more accurate match without adding any additional computational burden to the prediction stage.

Claims

1. A cross-modal contrast learning method for binocular stereo vision, characterized in that: The cross-modal contrast learning method includes three sub-contrast modules: a cross-view contrast module, a cross-modal contrast module, and a cross-temporal contrast module; The left view features and the right view features are input into three contrast modules. All the positive sample sets and negative sample sets obtained by the three contrast modules are used to train the binocular vision estimation network model by designing a contrast learning loss function to learn and obtain a feature map that contrasts different viewpoints, different modalities, and different time domains.

2. The cross-modal contrast learning method for binocular stereo vision according to claim 1, characterized in that: The cross-view comparison module regards a certain pixel anchor point in the left view feature as a positive sample, and a non-matching pixel point in the right view feature as a negative sample, and samples the entire feature map to obtain a cross-view positive sample set and a negative sample set.

3. The cross-modal contrast learning method for binocular stereo vision according to claim 1, characterized in that: The cross-modal comparison module disrupts the RGB feature pixel positions in the left view features so that the event features and RGB features of the left view are no longer pixel-aligned. The disrupted pixel points are regarded as a negative sample, and a cross-modal negative sample set is obtained by sampling on the entire feature map.

4. The cross-modal contrast learning method for binocular stereo vision according to claim 1, characterized in that: The cross-time domain comparison module fuses the event features at different timestamps with the RGB features. For the fused features at the same timestamp, its pixel point is regarded as a negative sample, and a cross-time domain negative sample set is obtained by sampling on the entire feature map.

5. The cross-modal contrast learning method for binocular stereo vision according to any one of claims 1 to 4, characterized in that: The left view feature includes a left view RGB feature I l , Left view event feature E l , the right view features include the right view RGB features I l and right view event feature E r ; Left view RGB feature I l The pixel at coordinate (u, v) and timestamp t0 is represented as The feature fusion module in the binocular vision estimation network model is abstracted into a function F(·), and the feature fusion module fuses the RGB features and event features under the same view into a new fusion feature; The sample acquisition of the cross-view comparison module is as follows: Set the left view feature F(I l ,E l ) with coordinates (u, v) and timestamp t0 is the anchor point Use the left view disparity to find the matching pixels in the right view at the same timestamp where u m = ud, where d = D(u, v) is the pixel point matched by the disparity value of the disparity map D at (u, v) in the data set, which is regarded as a positive example for cross-view comparison Non-matching pixel pairs are considered negative samples. Negative samples are sampled only in a local window around the matching pixel, where the window is a square with a length and width of w. The sample acquisition of the cross-modal comparison module is as follows: First, the pixel positions of the RGB features are disrupted so that the event features and RGB features are no longer pixel-aligned; Match the scrambled event features and RGB features to find the anchor point p a Negative examples of Where u′≠u or v′≠v; The sample acquisition of the cross-time domain comparison module is as follows: The timestamp of the anchor point is t0. Sample the timestamp different from the anchor point in the local time window to create a a Negative examples for cross-temporal comparison where t i ≠t0; In each training iteration, all contrast examples are used and the contrastive learning loss function is defined as follows: Among them, S is the total number of valid pixels in the feature map, and τ is the temperature coefficient.

6. The cross-modal contrast learning method for binocular stereo vision according to claim 5 or 6, characterized in that: The comparison samples include positive samples for cross-view comparison Negative Examples for Cross-View Comparison Negative Examples for Cross-Modal Comparison Negative examples compared with cross-temporal comparison 7. The cross-modal contrast learning method for binocular stereo vision according to claim 6, characterized in that: The temperature coefficient is 0.07.

Images