Binocular disparity map acquisition method, device and system for small targets

Abstract

The application discloses a binocular disparity map acquisition method, device and system for small targets, which is used for improving disparity optimization effect of small targets. The method comprises the following steps: performing hierarchical feature extraction on left and right views to obtain a feature map, compressing the feature map to 32 channels through a convolution operation with structure information, and constructing an initial splicing body; obtaining attention weights of corresponding stereo images of the left and right views, screening the initial splicing body by using the attention weights, constructing a cost volume according to a screening result and an initial disparity loss; fusing convolution context information and intermediate features after preliminary aggregation to aggregate the cost volume, and obtaining an aggregation result; supervising the aggregation result by using an adaptive multi-modal cross-entropy loss function, and performing multi-modal output on the supervised aggregation result by using a multi-modal disparity estimator, so as to obtain binocular disparity maps of the left and right views.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of assisted driving, and in particular to a binocular disparity map acquisition method, device and system for small targets. BACKGROUND

[0002] Stereo matching technology has a wide range of applications in the field of computer vision, such as three-dimensional reconstruction, autonomous driving, etc. In the field of autonomous driving, stereo matching technology obtains depth information of the surrounding environment through binocular cameras, realizes accurate obstacle detection and distance estimation, and provides important support for vehicle navigation and obstacle avoidance. In addition, stereo matching technology is also combined with the road preview system to improve the driving stability and comfort of the vehicle. The road preview system uses binocular stereo vision technology to scan and identify the road conditions in front of the vehicle in real time, and obtains information such as the flatness of the road, obstacles, and the slope of the road. These information is transmitted to the electronic control unit (ECU) of the vehicle, and the ECU adjusts the damping characteristics of the suspension system according to these information to adapt to the road conditions in front. For example, if a speed bump is previewed in front, the ECU will adjust the suspension system in advance to make it stiffer to reduce the impact when the vehicle passes through. This technology can significantly improve the driving quality of the vehicle and the comfort of the passengers.

[0003] However, the existing technology has some limitations in generating disparity maps. Due to the limitations of network structure or algorithm, geometric structure information is easily lost in the disparity generation process, especially at the edge details of the image, and the disparity matching accuracy of small targets is low. This loss of information can cause the model of three-dimensional reconstruction to deviate in the detailed part, affecting the overall accuracy and reliability. For example, in autonomous driving, if the disparity matching of small targets (such as road signs, speed bumps, etc.) is not accurate, it may cause the vehicle to estimate the distance of these targets incorrectly, thereby affecting the safety of driving.

[0004] In view of this, the present application is proposed. SUMMARY

[0005] The main purpose of the present application is to disclose a binocular disparity map acquisition method, device and system for small targets, which solves the problem of low disparity matching accuracy of small targets in the prior art.

[0006] To achieve the above purpose, according to one aspect of the present application, a binocular disparity map acquisition method for small targets is provided, and the following technical solution is adopted:

[0007] A binocular disparity map acquisition method for small targets comprises: performing hierarchical feature extraction on left and right views to obtain feature maps, compressing the feature maps to 32 channels through a feature extraction module incorporating structural information, and thereby constructing an initial stitching body; obtaining attention weights of the left and right views corresponding to a stereo image, screening the initial stitching body using the attention weights, and constructing a cost volume according to a screening result and an initial disparity loss; fusing convolutional context information with intermediate features obtained after preliminary aggregation to perform cost aggregation on the cost volume, and obtaining an aggregation result; supervising the aggregation result using an adaptive multi-modal cross-entropy loss function, and performing multi-modal output on the supervised aggregation result through a multi-modal disparity estimator to obtain binocular disparity maps of the left and right views.

[0008] Further, the hierarchical feature extraction on the left and right views to obtain the feature maps, and the compression of the feature maps to 32 channels through the feature extraction module incorporating the structural information, and thereby the construction of the initial stitching body, comprises: down-sampling the input left and right views with a pre-set size of convolution kernel and corresponding step lengths respectively, to reduce the image size; extracting unary features at 1 / 4 resolution using 16 residual layers, i.e. a 4-layer feature map, a 8-layer feature map and a 16-layer feature map, each containing 3 residual layers, and expanding the receptive field and obtaining semantic information by increasing the number of channels; on the basis of the original convolutional feature layer, each layer incorporates local structural information LCS, and all feature maps at 1 / 4 resolution are fused to generate a feature map with 320 channels, and the feature map is compressed to 32 channels to thereby construct the initial stitching body.

[0009] Further, the extraction of unary features at 1 / 4 resolution using 16 residual layers, i.e. a 4-layer feature map, a 8-layer feature map and a 16-layer feature map, each containing 3 residual layers, and expanding the receptive field and obtaining semantic information by increasing the number of channels, comprises: in each convolutional feature extraction layer, calculating structural information LCS of different dilation rates, increasing the structural information LCS of feature extraction, fusing multiple local structural feature layers with the original convolutional feature layer, and the fused features learn by themselves to balance the appearance information and the structural information; in the structural information LCS, each channel is used to represent the relationship between a center pixel and its adjacent pixels, and the formula is:

[0010] wherein, represents a pixel point on the kth channel of LCS, is the original convolutional feature, offsets representing the k-th neighbor point, is a function for measuring the cosine similarity between two vectors, in the convolutional neural network, the low-level features contain rich texture information, and the high-level features usually reveal semantic clues, by extracting features from different layers, the calculated neighborhood relationship.

[0011] Further, the attention weight of the left and right view corresponding stereo image is obtained, the initial splicing body is screened using the attention weight, and the cost volume is constructed according to the screening result and the initial disparity loss, including: step one: construction of the initial cost volume; the size of the input network stereo image pair is HxWx3, after the feature extraction step, the unary feature maps of the left view and the right view are obtained respectively and ; the size of the feature map ( ) is xH / 4xW / 4, wherein =32, representing the number of channels, H: height, W: width, the initial cost volume is formed by splicing the feature maps and at each disparity level; step two: generation of attention weight; three different levels of feature maps are obtained from the feature extraction module, and the number of channels is 64, 128 and 128 respectively; for each pixel at a certain level, a dilated block with a predefined size and adaptive learning weight is used to calculate the matching cost; by controlling the dilation rate, it is ensured that the range of the window is related to the level of the feature map, and at the same time, the similarity of the two corresponding pixels is the weighted sum of the correlation of the corresponding pixels in the window, in order to keep the same number of pixels in the calculation of the similarity of the center pixel; step three: divide the features into groups and calculate the correlation map according to the groups; divide the channels into groups evenly, wherein the first 8 groups come from layer feature map, the middle 16 groups come from layer feature map, and the last 16 groups come from layer feature map; different levels of feature maps will not interfere with each other, the gth group of features is represented as 、 , the matching cost body of the gth group, the matching weight of the gth group of features, which is adaptively learned during the training process, and the final multi-level block matching cost body is obtained by connecting the matching costs of all levels; the obtained multi-level block matching cost body is applied to two 3D convolutions and one 3D hourglass network to normalize, and then another convolution layer is used to compress the channels to 1, so as to obtain the attention weight.

[0012] Furthermore, the convolution context information and the intermediate features obtained after the preliminary aggregation are combined to perform cost aggregation on the cost body to obtain the aggregation result, which includes: given the context features obtained in the convolution feature extraction step from the left view , and the geometric features obtained by preliminary aggregation of the attention cost body ;Will The size of , recorded as , where N represents the number of channels of feature maps at different levels, and D represents the maximum disparity value; spatial attention weights are generated by exploring the spatial relationship between convolutional context and geometric features, and the attention mechanism is used to fuse features, thereby adaptively selecting important areas of features; an aggregation strategy based on an hourglass structure is used to process the attention cost body, and multi-scale fusion of features is achieved through gradual downsampling and upsampling.

[0013] Furthermore, the aggregation result is supervised by an adaptive multimodal cross entropy loss function, and the supervised aggregation result is multimodally outputted by a multimodal disparity estimator to obtain a binocular disparity map of the left and right views. The network training process includes: stage 1: only the attention weight part is trained, and supervision is performed using modeling based on ground truth probability; stage 2: the attention weight part is frozen so as to focus on training other parts of the network; stage 3: the entire network is trained together to achieve overall performance improvement; in each stage, the aggregation result is supervised by an adaptive multimodal cross entropy loss function, and the adaptive multimodal cross entropy loss is expressed as:

[0014] in, The predicted disparity probability distribution is obtained by the attention weight A, express The coefficient of is the i-th predicted disparity probability distribution output during training, express The coefficient of represents the above ground truth probability modeling.

[0015] Further, the adopting the adaptive multi-modal cross-entropy loss function to supervise the aggregation result, and the multi-modal disparity estimator is used to perform multi-modal output on the supervised aggregation result, to obtain the binocular disparity map of the left and right views further comprises: separating each modality in the multi-modal output, and respectively calculating cumulative probabilities; each modality represents a potential matching object with a specific depth, and the cumulative probability represents the matching possibility of the object; an object-level WTA strategy is adopted to select the modality with the highest cumulative probability as the dominant modality, and the modality is normalized; finally, the weighted average operation is used to estimate the disparity, and the multi-modal output of the network is estimated by the multi-modal disparity estimator to obtain the final disparity map.

[0016] According to another aspect of the present application, a binocular disparity map acquisition device for small targets is provided, and the following technical solutions are adopted:

[0017] A binocular disparity map acquisition device for small targets comprises: an extraction module configured to perform hierarchical feature extraction on left and right views to obtain feature maps, compress the feature maps to 32 channels through a feature extraction module incorporating structural information, and thereby construct an initial stitching body; a construction module configured to obtain attention weights of corresponding stereo images of the left and right views, use the attention weights to screen the initial stitching body, and construct a cost volume according to a screening result and an initial disparity loss; an aggregation module configured to fuse convolutional context information and intermediate features obtained after preliminary aggregation to perform cost aggregation on the cost volume, and obtain an aggregation result; and a supervision module configured to adopt an adaptive multi-modal cross-entropy loss function to supervise the aggregation result, and perform multi-modal output on the supervised aggregation result through a multi-modal disparity estimator, to obtain a binocular disparity map of the left and right views.

[0018] Further, the extraction module comprises: a sampling module configured to downsample the input left and right views with a preset size of convolution kernel and corresponding step lengths respectively, to reduce the image size; an acquisition module configured to extract unary features at 1 / 4 resolution using 16 residual layers, layer feature maps, layer feature maps and layer feature maps respectively contain 3 residual layers, and the receptive field is expanded and semantic information is obtained by increasing the number of channels; and a fusion module configured to fuse all feature maps at 1 / 4 resolution to generate a feature map with 320 channels, and compress the feature map to 32 channels to thereby construct the initial stitching body.

[0019] According to still another aspect of the present application, a binocular disparity map acquisition system for small targets is provided, and the following technical solutions are adopted:

[0020] A binocular disparity map acquisition system for small targets comprises the device described above.

[0021] The application can effectively improve the disparity matching precision of small targets, and further improve the accuracy of three-dimensional reconstruction and the safety of autonomous driving, by adding local structure information (LCS) in the feature extraction module, introducing context information (CCF) in the cost aggregation module, and using adaptive multi-modal cross-entropy loss function (AML) and multi-modal disparity estimator (MDE) for optimization. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments described in the present application, and other drawings can also be obtained by those skilled in the art according to these drawings.

[0023] Figure 1 A flowchart of a binocular disparity map acquisition method for small targets according to an embodiment of the present application;

[0024] Figure 2 A flowchart of another binocular disparity map acquisition method for small targets according to an embodiment of the present application;

[0025] Figure 3 A structural diagram of a binocular disparity map acquisition device for small targets according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] The embodiments of the present application will be described in detail below with reference to the drawings, but the present application can be implemented in various different ways limited and covered by the claims.

[0027] Figure 1 A flowchart of a binocular disparity map acquisition method for small targets according to an embodiment of the present application.

[0028] Referring to Figure 1 A binocular disparity map acquisition method for small targets comprises:

[0029] S101: Layered feature extraction is performed on left and right views to obtain a feature map, the feature map is compressed to 32 channels by a feature extraction module incorporating local structure information, and an initial stitching body is constructed;

[0030] S103: The attention weight of the corresponding stereo image of the left and right views is obtained, the initial stitching body is filtered using the attention weight, and a cost volume is constructed according to the filtering result and the initial disparity loss;

[0031] S105: Fuse the convolutional context information with the intermediate features obtained after the preliminary aggregation to perform cost aggregation on the cost volume to obtain an aggregation result;

[0032] S107: Supervise the aggregation result using an adaptive multi-modal cross-entropy loss function, and perform multi-modal output on the supervised aggregation result through a multi-modal disparity estimator to obtain a binocular disparity map of the left and right views.

[0033] Firstly, the binocular disparity map acquisition method for small targets proposed in the present application includes five parts: feature extraction, cost volume construction, cost aggregation, loss function, and disparity estimation.

[0034] In step S101, hierarchical feature extraction is performed on the left and right views to obtain a feature map, which is compressed to 32 channels through a feature extraction module incorporating local structure information, thereby constructing an initial stitching volume. Specifically, in the convolutional feature extraction part, a three-level ResNet architecture is adopted to divide the features into three levels. First, three 3x3 convolutional kernels are used to downsample the input image with a step size of 2, 1, and 1, respectively, to reduce the image size. Next, 16 residual layers are used to extract unary features at a resolution of 1 / 4, i.e. layer feature map, and layer feature map, which respectively contain 3 residual layers, increase the receptive field and obtain more rich semantic information by increasing the number of channels. Finally, all the layer feature maps ( , , ) at a resolution of 1 / 4 are fused to generate a feature map with 320 channels ( =320) for calculating attention weights. Subsequently, the feature map is compressed to 32 channels through convolutional operation, thereby constructing an initial stitching volume.

[0035] In step S103, the attention weights of the corresponding stereo images of the left and right views are obtained, which are used to filter the initial stitching volume. According to the filtering result and the initial disparity loss, the cost volume is constructed. Specifically, first, the initial cost volume is constructed. The size of the input network's stereo image pair is HxWx3, and after the feature extraction step, the unary feature maps of the left and right images are obtained. and . The size of the feature map ( ) is xH / 4xW / 4 (where =32, represents the number of channels, H: height, W: width). The initial cost volume is obtained by and The stitching is then performed to form. This is followed by the generation of attention weights, which are used to filter the initial stitching to highlight useful information and suppress irrelevant information. To this end, the geometric information in the correlation between stereo image pairs is extracted through multi-level adaptive block matching, thereby generating the attention weights. Three different levels of feature maps 、 and are obtained from the feature extraction module, with their channel numbers being 64, 128 and 128, respectively. For each pixel on a certain level, an expanded block with a predefined size and adaptive learning weights is used to calculate the matching cost. By controlling the expansion rate, the range of the window is ensured to be related to the level of the feature map, while the similarity of two corresponding pixels is the weighted sum of the correlation of corresponding pixels within the window, in order to maintain the same number of pixels in the calculation of the similarity of the center pixel. Next, the grouping idea of GwcNet is adopted to divide the features into groups and calculate the correlation map by group. The channels are evenly divided into groups ( =40), of which the first 8 groups come from the feature map, the middle 16 groups come from the feature map, and the last 16 groups come from the feature map. The feature maps of different levels do not interfere with each other. The gth group of features is denoted as 、 and the matching cost volume of the gth group is denoted as:

[0036]

[0037] wherein, is the inner product, and d represents different disparity levels. Subsequently, the multi-level block matching cost volume k=1,2,3) is calculated as:

[0038]

[0039] wherein, is a nine-point coordinate set that defines the window range on the kth feature map, represents the matching weight of the gth group of features on the kth feature layer, which is adaptively learned during the training process. The final multi-level block matching cost volume is obtained by connecting the matching costs of all levels:

[0040]

[0041] The obtained multi-level block matching cost volume is applied to two 3D convolutions and a 3D hourglass network for normalization, and then another convolution layer is used to compress the channels to 1 to obtain the attention weight. In order to obtain accurate attention weights of different disparities to filter the initial stitching cost volume, the real disparity A is supervised. Specifically, the probability distribution is obtained from A by normalizing through the softmax function, and the adaptive multi-modal cross-entropy loss between the real disparity probability distribution is calculated to guide the network learning process, so as to obtain accurate attention weight A. Finally, after obtaining the attention weight A, it is used to filter the redundant information in the initial stitching volume, thereby enhancing its representation ability. The final attention stitching volume The calculation formula on the channel i is:

[0042]

[0043] Wherein, represents Hadamard product.

[0044] In step S105, the convolution context information is fused with the intermediate feature obtained after preliminary aggregation to perform cost aggregation on the cost volume, and the aggregation result is obtained. Specifically, in order to decode accurate and high-resolution feature information from low-resolution feature information with the aid of context information, the present application proposes to fuse the convolution context information and the intermediate feature after preliminary aggregation (CCF) to realize efficient and flexible cost aggregation. Given the context feature obtained from the left eye image in the convolution feature extraction step, and the geometric feature obtained by preliminary aggregation of the attention cost volume, the size of is expanded to , denoted as . Wherein, N represents the number of channels of different hierarchical feature maps, and D represents the maximum disparity value. The spatial attention weight is generated by exploring the spatial relationship between the convolution context and the intermediate feature, and the attention mechanism is used to fuse the features, thereby adaptively selecting the important area of the features. The formula is as follows:

[0045]

[0046] An aggregation strategy based on a sandglass structure is used to process the attention cost volume, and multi-scale fusion of features is realized by stepwise down-sampling and up-sampling. First, the input cost volume Downsampling is performed through a series of 3D convolutional layers, gradually extracting features at different scales and enhancing feature representation. After downsampling, CCF and an upsampling module are alternately used to decode high-resolution features. CCF adaptively fuses multi-scale features and incorporates contextual information during the cost aggregation phase, enabling the cost volume to better reflect the scene, further enhancing features and recovering edges and weakly textured areas.

[0047] In step S107, an adaptive multimodal cross-entropy loss function is used to supervise the aggregation results, and the supervised aggregation results are multimodally output by a multimodal disparity estimator to obtain a binocular disparity map for the left and right views. Specifically, in stereo matching, a smooth L1 loss is often used to indirectly supervise the cost volume. However, this approach limits the ultimate performance improvement. By converting the stereo matching problem into a classification task, the cross-entropy loss can directly supervise the probability distribution volume. Considering the uncertainty of disparity estimation and the multimodal probability distribution of pixels, this application adopts an adaptive multimodal cross-entropy loss (AML) to provide more direct and effective supervision to the network, thereby significantly improving matching accuracy. The probability distribution of edge pixels should be composed of multiple modes, each representing a specific depth or disparity. To this end, this application uses an adaptive multimodal ground truth modeling method to generate an independent Laplacian distribution for each potential depth of edge pixels, and then fuses these distributions to form a Laplacian mixture model. This task is accomplished by using the neighborhood of each pixel. For each pixel with a true disparity label, an m×n (1×9) local window centered on that pixel is considered. The DBSCAN clustering algorithm is then used to partition all disparity values ​​within the window into K disjoint subsets, each corresponding to a different potential depth. The formula is as follows:

[0048]

[0049]

[0050] The Laplace distribution is discretized and normalized on the disparity candidates d∈{0,1,…,D-1}, and 、 and are the mean, scale and weight parameters of the kth Laplace distribution respectively. Set as cluster The average value of the internal parallax. Definition contains the center pixel to be modeled, is replaced with the true value of the center pixel to ensure the accuracy of supervision.

[0051]

[0052] weights are used to adjust the relative proportion of the obtained multiple modalities, and can be assigned according to the local structure within the window. is the fixed weight of the center pixel. Set to ensure the dominance of the ground truth modality. The rest weights are evenly distributed to the remaining 8 neighboring pixels. With the cardinality of the ground truth as an indicator of local structure, for example, smaller corresponds to finer structures, and thus should have a smaller weight accordingly. For datasets like KITTI that have sparse ground truth values, only the valid disparities within the local window are computed, and the equation becomes:

[0053]

[0054] For non-edge pixels with only one cluster within the window, equals 1, and the equation degenerates to a unimodal Laplacian distribution.

[0055] Therefore, the final loss of the network uses the adaptive multi-modal cross-entropy loss, denoted as:

[0056]

[0057] where, is the predicted disparity probability distribution obtained through the attention weights A, denotes the coefficient of is the i-th predicted disparity probability distribution output during training, denotes the coefficient of represents the above ground truth probability modeling.

[0058] The training process of the network is divided into three stages to ensure effective learning and optimization of each part. In the first stage, only the attention weight part is trained, supervised by the ground truth probability-based modeling. In the second stage, the attention weight part is frozen to focus on training other parts of the network. Finally, in the third stage, the entire network will be trained together to achieve overall performance improvement. In each stage, the adaptive multi-modal cross-entropy loss function is used to ensure the stability and effectiveness of the training process.

[0059] Stereo matching networks trained with cross-entropy loss usually produce more multi-modal outputs than L1 loss, so the soft argmin function cannot be directly used to estimate the disparity. The present application uses a multi-modal disparity estimator (MDE) to separate each modality in the multi-modal output and calculate their cumulative probabilities separately. Each modality represents a potential matching object with a specific depth, and its cumulative probability represents the likelihood of the object matching. Therefore, an object-level WTA strategy is used to select the modality with the highest cumulative probability as the dominant modality. Subsequently, the modality is normalized:

[0060]

[0061] Finally, the weighted average operation is used to estimate the disparity:

[0062]

[0063] The final disparity map can be obtained by estimating the multi-modal output of the network through the MDE.

[0064] The present application significantly improves the performance of stereo matching algorithms in small target disparity matching accuracy and real-time performance through a series of innovative technologies. The present application can more accurately capture the geometric features of small targets in images, such as edges, contours, etc. By optimizing the disparity matching process, the present application reduces the loss of geometric structure information, making the generated disparity map more accurate in detail. This provides higher quality data input for three-dimensional reconstruction, effectively reducing the deviation of the reconstructed three-dimensional model in detail, and significantly improving the overall accuracy and reliability.

[0065] Figure 2 The flowchart of another binocular disparity map acquisition method for small targets according to an embodiment of the present application.

[0066] Referring to Figure 2 , a binocular disparity map acquisition method for small targets includes:

[0067] Step 21: input left and right images;

[0068] Step 22: weight sharing feature extraction;

[0069] Step 221: local structure feature extraction module;

[0070] Step 23: cost volume construction;

[0071] Step 231: initial disparity loss;

[0072] Step 232: context information fusion module;

[0073] Step 24: cost aggregation;

[0074] Step 25: predicting disparity probability distribution;

[0075] Step 251: adaptive multi-modal loss;

[0076] Step 26: outputting the disparity map.

[0077] As a preferred embodiment, in the structure feature extraction part in step 22, local cosine similarity (LCS) is introduced as the structure information. In LCS, each channel is used to represent the relationship between a center pixel and its adjacent pixels. The formula is as follows:

[0078]

[0079] wherein, represents a pixel point the kth channel on LCS, is the original convolutional feature, represents the offset of the kth adjacent point, is a function for measuring the cosine similarity between two vectors. Cosine similarity is used to measure the relationship between two vectors because it has good numerical stability.

[0080] Considering that a smaller neighborhood helps to match at the disparity discontinuity, and a larger neighborhood helps to understand the structure of the textureless region. In order to fully identify objects of different sizes, a square window (3x3) is designed to construct the neighborhood relationship in different ranges. Since the pooling strategy may cause loss of details, an expansion strategy is adopted to expand the receptive field. In the convolutional neural network, the low-level features contain rich texture information, and the high-level features usually reveal semantic clues. By extracting features from different layers, the calculated neighborhood relationship is diverse. Therefore, the present application adopts LCS with different expansion rates at each convolutional feature extraction layer to increase the structure information of feature extraction. The multiple local structure feature layers are fused with the original convolutional feature layer, and the fused features will learn to balance the appearance information and the structure information.

[0081] Specifically, based on the ACVNet, the present application proposes a stereo matching algorithm which is both real-time and high-precision. Compared with the original network, the present application realizes the following innovations:

[0082] On the basis of the traditional convolutional feature extraction of appearance information, the extraction of geometric structure features is introduced, which more accurately reveals the relationship between local pixels.

[0083] In the hourglass structure of cost aggregation, rich context information is integrated, which effectively preserves the detailed features and improves the matching accuracy.

[0084] In the three training stages of the network, the adaptive multi-modal cross-entropy loss is used to effectively guide the model to learn clear pixel distribution patterns, provide direct and efficient supervision for the network, and significantly improve the matching accuracy.

[0085] In the disparity estimation and the acquisition of the disparity map, a multi-modal disparity estimation method is used to obtain more accurate results on the multi-modal output of the network.

[0086] The application applies an adaptive feature enhancement mechanism to dynamically adjust parameters according to the feature distribution of the input image and the requirements of the matching task in the feature extraction and cost aggregation stages.

[0087] Through the fusion of the designed module, the benchmark model ACVNet is greatly surpassed in the comparative experiments of the Scene Flow and KITTI data sets.

[0088] Figure 3 A structure diagram of a binocular disparity map acquisition device for small targets according to an embodiment of the application.

[0089] Referring to Figure 3 As shown in the figure, a binocular disparity map acquisition device for small targets includes an extraction module 30 for performing hierarchical feature extraction on left and right views to obtain feature maps, compressing the feature maps to 32 channels through a feature extraction module that incorporates local structure information, and constructing an initial stitching body; a construction module 32 for obtaining attention weights of corresponding stereo images of the left and right views, using the attention weights to filter the initial stitching body, and constructing a cost volume according to the filtering result and an initial disparity loss; an aggregation module 34 for fusing convolution context information and intermediate features obtained after preliminary aggregation to perform cost aggregation on the cost volume, and obtaining an aggregation result; a supervision module 36 for supervising the aggregation result using an adaptive multi-modal cross-entropy loss function, and performing multi-modal output on the supervised aggregation result through a multi-modal disparity estimator to obtain binocular disparity maps of the left and right views.

[0090] Optionally, the extraction module 30 includes a sampling module (not shown in the figure) for downsampling the input left and right views with a corresponding step size through a preset size of convolution kernel to reduce the image size; an acquisition module (not shown in the figure) for extracting unary features at a resolution of 1 / 4 using 16 residual layers, a 64-layer feature map, a 128-layer feature map, and The layer feature maps respectively contain 3 residual layers, the receptive field is expanded and semantic information is obtained by increasing the number of channels, and each layer feature map integrates local structure information; a fusion module (not shown in the figure) is used to fuse all feature maps at 1 / 4 resolution to generate a feature map with 320 channels, and the feature map is compressed to 32 channels through convolution operation, thereby constructing an initial splicing body.

[0091] The application provides a binocular disparity map acquisition system for small targets.

[0092] The application can effectively improve the disparity matching precision of small targets, and further improve the accuracy of three-dimensional reconstruction and the safety of autonomous driving by adding local structure information (LCS) in the feature extraction module, introducing context information (CCF) in the cost aggregation module, and using adaptive multi-modal cross-entropy loss function (AML) and multi-modal disparity estimator (MDE) for optimization.

[0093] The above only describes certain exemplary embodiments of the present embodiment by way of illustration, and it is needless to say that those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present application. Therefore, the above drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present application.

Claims

1. A binocular disparity map acquisition method for small targets, characterized in that, The left and right views are subjected to hierarchical feature extraction to obtain a feature map, the feature map is compressed to 32 channels through a convolution operation incorporating structural information, and an initial stitching body is constructed; Attention weights of the left and right views corresponding to the stereo image are obtained, the initial stitching body is screened using the attention weights, and a cost volume is constructed according to the screening result and an initial disparity loss; Convolution context information and the preliminarily aggregated intermediate features are fused to aggregate the cost volume, and an aggregation result is obtained; An adaptive multi-modal cross-entropy loss function is used to supervise the aggregation result, and a multi-modal output is obtained through a multi-modal disparity estimator, thereby obtaining a binocular disparity map of the left and right views. The hierarchical feature extraction of the left and right views to obtain a feature map, and the feature map is compressed to 32 channels through a feature extraction module incorporating structural information, thereby constructing an initial stitching body, includes:

2. The method of claim 1, wherein, A convolution kernel of a preset size is used to downsample the input left and right views with corresponding step sizes to reduce the image size; All feature maps at 1 / 4 resolution are fused to generate a feature map with 320 channels, and the feature map is compressed to 32 channels to construct an initial stitching body. The unary feature is extracted at 1 / 4 resolution by using 16 residual layers, i.e. layer feature maps, layer feature maps and layer feature maps respectively contain 3 residual layers, and the receptive field is enlarged and semantic information is obtained by increasing the number of channels. In each convolution feature extraction layer, the structural information LCS of different expansion rates is calculated, the structural information LCS of the feature extraction is increased, the multiple local structural feature layers are fused with the original convolution feature layer, and the fused features learn by themselves to balance the appearance information and the structural information; 3. The method of claim 2, wherein, The 16 residual layers are used to extract unary features at 1 / 4 resolution, i.e. layer feature maps, layer feature maps and The layer feature maps respectively contain 3 residual layers, and the receptive field is expanded and semantic information is obtained by increasing the number of channels. In the structural information LCS, each channel is used to represent the relationship between a center pixel and its adjacent pixels, and the formula is: The attention weights of the left and right views corresponding to the stereo image are obtained, the initial stitching body is screened using the attention weights, and a cost volume is constructed according to the screening result and an initial disparity loss, includes: wherein, represents a pixel point the kth channel on LCS, is the original convolutional feature, represents the offset of the kth neighbor point, is a function for measuring the cosine similarity between two vectors, in a convolutional neural network, low-level features contain rich texture information, while high-level features usually reveal semantic clues, by extracting features from different layers, the neighborhood relationship is calculated.

4. The method of claim 2, wherein, Step 1: construction of an initial cost volume; Step 2: generation of attention weights; The size of the stereoscopic image pair input into the network is HxWx3, and after a feature extraction step, monocular feature maps of the left view and the right view are obtained respectively and ; feature map ( ) is wherein represents the number of channels, H: height, W: width, an initial cost volume is formed by concatenating the feature map and at each disparity level; Three feature maps of different levels are obtained from the feature extraction module, and the number of channels is 64, 128 and 128 respectively; For each pixel at a certain level, a dilated block with a predefined size and adaptive learning weights is used to calculate the matching cost; By controlling the expansion rate, it is ensured that the range of the window is related to the level of the feature map, and the similarity of the two corresponding pixels is the weighted sum of the correlation of the corresponding pixels within the window, so as to maintain the same number of pixels in the calculation of the similarity of the center pixel; The obtained multi-level block matching cost volume is applied to two 3D convolutions and a 3D hourglass network to normalize, and then another convolution layer is used to compress the channels to 1, thereby obtaining the attention weights. Step three: divide the features into groups and calculate the correlation map by group; divide the channels into groups averagely, where the first 8 groups come from layer feature maps, the middle 16 groups come from layer feature maps, and the last 16 groups come from layer feature maps; the feature maps of different levels do not interfere with each other, and the gth group of features is denoted as , , the matching cost body of the gth group, the matching weight of the gth group of features, which is adaptively learned in the training process, and the final multi-level block matching cost body is obtained by connecting the matching costs of all levels; The convolution context information and the preliminarily aggregated intermediate features are fused to aggregate the cost volume, and an aggregation result is obtained, includes:

5. The method of claim 4, wherein, The spatial relationship between the convolution context and the geometric features is explored to generate spatial attention weights, and the attention mechanism is used to fuse the features, so as to adaptively select the important areas of the features; Given the context features obtained at the convolutional feature extraction step from the left view , and the geometry features obtained after the preliminary aggregation of the attention cost volume ; The size of is extended to , denoted as , where N represents the number of channels of different hierarchical feature maps, and D represents the maximum disparity value. An aggregation strategy based on a hourglass structure is used to process the attention cost volume, and multi-scale fusion of the features is realized through stepwise down-sampling and up-sampling. ​ 6. The method of claim 2, wherein, The adaptive multi-modal cross-entropy loss function is used to supervise the aggregation result, and the multi-modal output of the supervised aggregation result is obtained through the multi-modal disparity estimator to obtain the binocular disparity map of the left and right views. The training process of the network includes: Stage one: only the attention weight part is trained, and the modeling based on the ground truth probability is used for supervision; Stage two: the attention weight part is frozen so as to focus on training other parts of the network; Stage three: the whole network is trained together to improve the overall performance; In each stage, the adaptive multi-modal cross-entropy loss function is used to supervise the aggregation result, and the adaptive multi-modal cross-entropy loss is represented as: wherein, is the predicted disparity probability distribution obtained by attention weights A, denotes coefficients of is the i-th predicted disparity probability distribution output during the training process, denotes coefficients of represent the ground truth probability modeling described above.

7. The method of claim 6, wherein, The adaptive multi-modal cross-entropy loss function is used to supervise the aggregation result, and the multi-modal output of the supervised aggregation result is obtained through the multi-modal disparity estimator to obtain the binocular disparity map of the left and right views. Each modality in the multi-modal output is separated and the cumulative probability is calculated respectively; Each modality represents a potential matching object with a specific depth, and the cumulative probability represents the matching possibility of the object; An object-level WTA strategy is used to select the modality with the highest cumulative probability as the dominant modality, and the modality is normalized: Finally, the disparity is estimated by using the weighted average operation, the multi-modal output of the network is estimated by the multi-modal disparity estimator, and the final disparity map is obtained.

8. A binocular disparity map acquisition apparatus for small objects, characterized by, Comprise: The extraction module is used for hierarchical feature extraction of the left and right views to obtain a feature map, and the feature extraction module with structural information is used to compress the feature map to 32 channels, thereby constructing an initial splicing body; The construction module is used to obtain the attention weight of the left and right view corresponding to the stereo image, and the initial splicing body is filtered using the attention weight, and the cost volume is constructed according to the filtering result and the initial disparity loss; The aggregation module is used to fuse the convolution context information and the intermediate features after preliminary aggregation to aggregate the cost volume, and obtain the aggregation result; The supervision module is used to supervise the aggregation result by using the adaptive multi-modal cross-entropy loss function, and the multi-modal output of the supervised aggregation result is obtained through the multi-modal disparity estimator to obtain the binocular disparity map of the left and right views.

9. The apparatus of claim 8, wherein, The extraction module comprises: The sampling module is used to downsample the input left and right views by a preset size of convolution kernel with corresponding step length to reduce the image size; The acquisition module is configured to extract unary features at 1 / 4 resolution by using 16 residual layers, i.e. layer feature maps, layer feature maps and The layer feature maps respectively contain 3 residual layers, and receptive fields are enlarged and semantic information is obtained by increasing the number of channels. The fusion module is used to fuse the local structure information LCS on each layer of the convolution feature, and fuse all feature maps at 1 / 4 resolution to generate a feature map with 320 channels, and compress the feature map to 32 channels, thereby constructing an initial splicing body.

10. A binocular disparity map acquisition system for small objects, characterized by, The device of any one of claims 8-9. The device of any one of claims 8-9.

Images