A real-time binocular stereo matching method based on reparameterization and recursive weighting

Abstract

The application discloses a real-time binocular stereo matching method based on reparameterization and recursive weighting, which comprises the following steps: firstly, the feature details of an input image pair are acquired by using a reparameterization and block convolution based feature extraction module sharing parameters, and a feature map is output; then the obtained feature map is grouped, and the matching confidence of multiple groups is calculated respectively by using the grouped feature map, and a multi-channel cost volume is obtained; then the information connection between the channels is calculated one by one by using a cross-correlation module, and finally a refined single-channel cost volume is obtained; then the cost volume is regularized by using a disparity asymmetric aggregation network, and an aggregated cost volume is obtained; the aggregated cost volume is subjected to disparity regression to obtain a disparity map; after the above steps are repeated, the same but different weight cost aggregation and disparity regression are performed, and finally a high-precision disparity map of a full-resolution non-information redundant cost volume is obtained by interpolation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of re-parameterization structure vision extraction, and particularly relates to a real-time binocular stereo matching method based on re-parameterization and recursive weighting. BACKGROUND

[0002] Depth perception of three-dimensional space is an important branch of computer vision, which perceives the geometric information of objects in a three-dimensional environment through different sensors, and has been widely used in automatic driving, augmented reality and three-dimensional reconstruction. The current mainstream perception method is the time-of-flight method based on laser radar equipment and the structured light method based on optical scanning equipment. These mainstream methods have high perception accuracy, but the cost of hardware equipment is high. The stereo vision algorithm based on ordinary cameras is a new depth perception method in recent years, which is flexible to implement and has low cost. Among them, the binocular stereo vision algorithm has become a representative technology of stereo vision due to its similar underlying principles to human eyes and fast and accurate perception performance.

[0003] The basic principle of binocular stereo matching technology is to obtain the left and right images of the scene through binocular cameras, calculate the disparity map by matching the pixels in the two images, and then calculate the scene depth according to the disparity. The traditional binocular stereo matching method can be roughly divided into global method and local method. The global method calculates the matching confidence of each pixel to estimate the disparity value through the matching algorithm; the local method calculates the local matching confidence to complete the overall disparity value estimation by creating a window and controlling the window movement. With the development of deep learning algorithms and the improvement of computer hardware performance, the binocular stereo matching technology based on deep learning has greatly improved in accuracy and speed compared with traditional global or local matching algorithms. The stereo matching network mainly includes four steps: feature extraction, cost volume construction, cost aggregation, and disparity regression. Among them, feature extraction, cost volume construction and cost aggregation directly determine the matching confidence calculation of each spatial pixel of the image, and then directly determine the prediction accuracy and speed of the network disparity map. However, most of the current advanced stereo matching technology researches only focus on the accuracy optimization of the stereo matching network. These high-precision networks often have complex network structures and a large number of convolution block stacks, which makes the network's computational complexity and memory occupancy high, seriously limiting the flexibility of network deployment (only deployable on high-performance, high-energy computer devices).

[0004] Based on the high difficulty of high-precision network deployment, the coarse-to-fine real-time network represented by AnyNet realizes the real-time deployment of the network on low-power devices. However, this also leads to the decrease of its precision, and the excessive stage correction also affects the reasoning speed. The specific defects include: (1) the basic convolution block of feature extraction is too simple; (2) the cost volume has information redundancy; (3) the cost aggregation network has weak perception of disparity dimension; in summary, the existing real-time algorithm cannot realize the real-time high-precision disparity map output on the device with limited function. Therefore, based on the above three defects, the present application optimizes the network to improve the precision of the output disparity map on the basis of ensuring the real-time of the network, so as to improve the practical application value of the algorithm. SUMMARY

[0005] The present application provides a real-time binocular stereo matching method based on reparameterization and recursive weighting, which solves the problem that the adversarial patch is not natural enough in the target detection of the artificial intelligence security field and the adversarial disturbance area is too large. Specifically, most of the attacks in target detection are still in the digital domain attack stage, and in the physical domain attack, in order to achieve the effect of attacking the target detector, the generated adversarial patch is very unnatural and easy to be detected by the human eye, and the disturbance of the attack target is modified.

[0006] The purpose of the present application is achieved by the following technical solutions:

[0007] A real-time binocular stereo matching method based on reparameterization and recursive weighting comprises the following steps:

[0008] S1: Obtain the feature details of the input image, and output the 1 / 4 resolution feature map and the 1 / 8 resolution feature map of the input image respectively;

[0009] S2: Divide the obtained 1 / 8 resolution feature map into N groups, and use the grouped 1 / 8 resolution feature map to calculate the matching confidence of N groups respectively to obtain a multi-channel cost volume, and calculate the information connection between channels one by one to obtain a refined single-channel cost volume;

[0010] S3: Regularize the cost volume using the disparity asymmetric aggregation network to obtain an aggregated cost volume;

[0011] S4: The aggregated cost volume is subjected to disparity regression to obtain a 1 / 8 resolution disparity map;

[0012] S5: Interpolating the 1 / 8 resolution disparity map to get a coarse 1 / 4 resolution disparity map, and using the disparity map and the 1 / 4 resolution feature map to construct a fine cost volume after fixed disparity correction, and after the same steps as steps S3 and S4 but different weights of cost aggregation and disparity regression, the final full resolution disparity map is obtained by interpolation;

[0013] S6: Using the Smooth_L1 function to calculate the prediction loss of the full resolution disparity map output by S5 and the dataset true value.

[0014] Preferably, in step S1, the feature details of the input image are obtained by a feature extraction module based on reparameterization and a feature extraction module based on block convolution, and the conversion formula of the feature extraction module based on reparameterization is as follows:

[0015]

[0016]

[0017]

[0018]

[0019] wherein μ and σ represent the mean and standard deviation of the input BN layer data respectively; γ and β represent the learnable parameters of the BN layer; w' i and w i represent the converted and unconverted convolution weights of each branch respectively; b' i represent the converted convolution bias of each branch; W and B represent the total weights and total bias of the converted linear convolution, M represents the total number of branches of the module, and i represents a single branch in the M branches.

[0020] More preferably, the feature extraction module based on block convolution uses a convolution block with a step size consistent with the size of the convolution kernel.

[0021] Preferably, in step S2, after obtaining the multi-channel cost volume, in order to reduce the information redundancy between each channel cost, the information correlation between each channel cost is calculated to weight each channel information, and the weight of the weighting is learned by using a convolution block.

[0022] More preferably, the specific implementation method of weighting each channel information is as follows: assuming that there are N channel numbers of cost volumes, the cost volume of the first channel and the cost volume of the second channel are used to calculate the cross-correlation information, that is, the information dependence degree of the two channel cost volumes, and the weight is calculated through a linear convolution block to filter the information and suppress repeated information; the weight is calculated through a linear convolution block to filter the information, and the refined cost volume is output; the cross-correlation information of the refined cost volume and the cost volume of the third channel is calculated and filtered, and so on until the information filtering of all N channels is completed, and the final single-channel cost volume is obtained.

[0023] Preferably, in the step S3, the disparity asymmetric aggregation network comprises a plurality of 5x5xN asymmetric convolution blocks, wherein N is the maximum disparity level of the aggregated cost volume.

[0024] More preferably, the asymmetric convolution block comprises a 5x5xN separated 3D convolution and a 1x1x1 3D convolution.

[0025] Preferably, in the step S6, assuming that the prediction loss is x, the Smooth_L1 function loss calculation formula is as follows:

[0026]

[0027] The gradient of the Smooth_L1 function is calculated and back propagated, and the weights and biases are adjusted through the Adam optimizer.

[0028] The real-time binocular stereo matching method based on reparameterization and recursive weighting provided by the application has the following advantages compared with the prior art:

[0029] (1) The application proposes a feature extraction network with a reparameterization structure. In practical applications, the index reflecting the running efficiency of the network is the inference speed. The reparameterization method can decouple the training structure and the inference structure of the network. This means that the network can use a multi-branch residual structure network to comprehensively extract the detailed features of the image during training, and convert the network structure and parameters during inference to ensure the accuracy unchanged while realizing fast inference. This reparameterization method for stereo matching can effectively improve the accuracy of the network without introducing additional calculations.

[0030] (2) This invention proposes a multi-channel cost volume construction method based on channel cross-correlation. Similar to the grouped correlation multi-channel cost volume construction method, this method groups the feature maps and generates multi-channel disparity candidates for each disparity level. These disparity candidates together constitute the multi-channel cost volume. This multi-channel cost volume effectively alleviates the problem of redundant candidate information in single-channel cost volume feature matching. To further improve the network prediction accuracy, the cross-correlation method is used to calculate the similarity of each channel disparity candidate, further reducing redundant information mapping. 1x1 convolution is used to suppress high-similarity information to achieve efficient information mapping and improve the subsequent network operating efficiency.

[0031] (3) This invention proposes a disparity asymmetric aggregation network. It aggregates the cost volume using convolutional blocks composed of 5x5xN separable 3D convolutions and 1x1x1 3D convolutions, where N represents the maximum disparity energy level of the aggregated cost volume. The asymmetric separable 3D convolutions effectively connect all disparity energy levels of each spatial pixel in each channel to match confidence information, and are integrated and mapped by 1x1x1 3D convolutions, effectively improving the receptive field of the aggregation network in the disparity dimension and enhancing the accuracy of disparity prediction.

[0032] (4) Based on the above three advantages, the network can obtain a high-precision disparity map during the first correction after obtaining the initial disparity map. Therefore, this invention reduces the number of stages in the coarse-to-fine network to two stages, effectively improving the inference efficiency of the network.

[0033] (5) This invention uses the TensorRT optimization framework to accelerate network deployment. Compared with existing technologies, this invention can achieve real-time high-precision disparity map prediction on resource-constrained devices, and has higher practical value. Attached Figure Description

[0034] The present invention will be further described with reference to the accompanying drawings, but the embodiments in the drawings do not constitute any limitation on the present invention. For those skilled in the art, other drawings can be obtained based on the following drawings without creative effort.

[0035] Figure 1 This is a flowchart illustrating the implementation of the real-time binocular stereo matching method based on reparameterization and recursive weighting in an embodiment of the present invention.

[0036] Figure 2 This is an overview diagram of the network in an embodiment of the present invention;

[0037] Figure 3 This is a basic structural diagram of the feature extraction convolutional block according to an embodiment of the present invention;

[0038] Figure 4 This is a flowchart of the cost-volume construction method according to an embodiment of the present invention;

[0039] Figure 5 is a schematic diagram of a correlation cost filtering process of an embodiment of the present application. DETAILED DESCRIPTION

[0040] The real-time binocular stereo matching method based on reparameterization and recursive weighting is further described in detail below in combination with specific embodiments, which are only used for comparison and explanation purposes, and the present application is not limited to these embodiments.

[0041] EMBODIMENT

[0042] As shown in Figure 1 and Figure 2 , the real-time binocular stereo matching method based on reparameterization and recursive weighting provided by the embodiment of the present application specifically comprises the following steps:

[0043] S1: Obtain the feature details of the input image, and respectively output the 1 / 4 resolution feature map and the 1 / 8 resolution feature map of the input image;

[0044] S2: Divide the obtained 1 / 8 resolution feature map into N groups, use the grouped 1 / 8 resolution feature map to respectively calculate the matching confidence of the N groups, obtain a multi-channel cost volume, calculate the information connection between channels one by one, and obtain a refined single-channel cost volume;

[0045] S3: Regularize the cost volume using the disparity asymmetric aggregation network to obtain an aggregated cost volume;

[0046] S4: The aggregated cost volume is subjected to disparity regression to obtain a 1 / 8 resolution disparity map;

[0047] S5: Interpolate the 1 / 8 resolution disparity map to obtain a coarse 1 / 4 resolution disparity map, and use the disparity map and the 1 / 4 resolution feature map to construct a fine cost volume after fixed disparity correction, and then after the same steps as steps S3 and S4 but different weights of cost aggregation and disparity regression, obtain a final full resolution disparity map through interpolation;

[0048] S6: Use the Smooth_L1 function to calculate the prediction loss of the full resolution disparity map output by step S5 and the dataset true value.

[0049] Preferably, in the step S1, the feature details of the input image are obtained by a feature extraction module based on reparameterization and a feature extraction module based on block convolution, which is different from the traditional feature extraction module. The feature extraction module based on reparameterization directly extracts 1 / 4 resolution features by using 4x4 block convolution and reparameterized filters to ensure inference efficiency. Then, 1 / 8 resolution features are extracted for subsequent cost volume construction. The block convolution with parameter learning capability can better retain the feature details in the high-resolution feature map and reduce the loss of feature information caused by downsampling. The reparameterized filter can decouple the training and inference of the network. When training, the reparameterization convolution block is a complex multi-branch structure to obtain a more rich receptive field when perceiving features. When inferring, the module is converted into a simple linear structure to improve the inference efficiency of the network

[0050] The conversion formula of the feature extraction module based on reparameterization is as follows:

[0051]

[0052]

[0053]

[0054]

[0055] Wherein, μ and σ represent the mean and standard deviation of the input BN layer data respectively; γ and β represent the learnable parameters of the BN layer; w i and w i represent the converted and unconverted convolution weights of each branch respectively; b i represent the converted convolution bias of each branch; W and B represent the total weights and total bias of the linear convolution after conversion, M represents the total number of branches of the module, and i represents a single branch in the M branches.

[0056] Through the above design, the feature extraction module of the application can efficiently extract image features and improve the overall inference efficiency of the network.

[0057] More preferably, the feature extraction module based on block convolution uses a convolution block with a step size consistent with the size of the convolution kernel.

[0058] As shown in Figure 3 , it is a basic structure diagram of the feature extraction module based on reparameterization and the feature extraction module based on block convolution. The parameter D determines the downsampling size of the module, and the parameter K determines the receptive field size of the module. The actual application scene is adjusted according to the actual application scene.

[0059] Preferably, after obtaining the multi-channel cost volume in step S2, to reduce the information redundancy between each channel cost, the information correlation between each channel cost is calculated to weight each channel information, and the weight of the weighting is learned by using a convolution block.

[0060] More preferably, the specific implementation method of weighting each channel information is that: assuming that there are N channel numbers of cost volume, the mutual information between the cost volume of the first channel and the cost volume of the second channel is calculated, that is, the information dependence degree of the two channel cost volumes (if the dependence degree of the two cost volumes is too high, it means that the similarity of the two costs is high, and vice versa), and the weight is calculated by a linear convolution block to filter the information and suppress the repeated information; and the weight is calculated by a linear convolution block to filter the information and output the refined cost volume, and then the mutual information between the refined cost volume and the cost volume of the third channel is calculated and filtered, and so on until the information filtering of all N channels is completed, and the final single-channel cost volume is obtained.

[0061] Reference Figure 4 As shown in the figure, if there is a disparity map correction, the left feature map is translated according to the disparity map value, and the disparity energy level is corrected by a fixed value when the matching confidence is calculated.

[0062] Reference Figure 5 As shown in the figure, the mutual correlation cost filtering flowchart of the application is shown, taking a 3-channel cost volume as an example, and the N channels in the actual situation are similar to the recursive calculation of the mutual information between the channels in the above figure, and the refined single-channel cost volume is output.

[0063] Preferably, in step S3, the disparity asymmetric aggregation network includes a plurality of 5x5xN asymmetric convolution blocks, wherein N is the maximum disparity energy level of the aggregated cost volume.

[0064] More preferably, the asymmetric convolution block includes a 5x5xN separated 3D convolution and a 1x1x1 3D convolution.

[0065] The disparity asymmetric aggregation network is constituted as shown in Table 1.

[0066] Table 1: Disparity asymmetric aggregation network constitution table

[0067]

[0068]

[0069] Preferably, in step S6, assuming that the prediction loss is x, the Smooth_L1 function loss calculation formula is:

[0070]

[0071] The gradient of the Smooth_L1 function is calculated and back-propagated, and the weights and biases are adjusted by the Adam optimizer.

[0072] The real-time binocular stereo matching method based on reparameterization and recursive weighting provided by the above embodiments of the application focuses on: first, using a parameter-shared feature extraction module based on reparameterization and block convolution to obtain the feature details of the input image pair, and output a feature map; then grouping the obtained feature map, using the grouped feature map to calculate the matching confidence of multiple groups respectively, and obtaining a multi-channel cost volume; then using a cross-correlation module to calculate the information connection between the channels one by one, and finally obtaining a refined single-channel cost volume; then using a disparity asymmetric aggregation network to regularize the cost volume, obtaining an aggregated cost volume, and then using disparity regression to obtain a disparity map; after repeating the above steps, using the same but different weights for cost aggregation and disparity regression, and finally obtaining a high-precision disparity map of the final full-resolution information-redundancy-free cost volume through interpolation.

[0073] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the application, and are not intended to limit the protection scope of the application. Although the application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the application can be modified or replaced by equivalents without departing from the essence and scope of the technical solutions of the application.

Claims

1. A real-time binocular stereo matching method based on reparameterization and recursive weighting, characterized in that, The method comprises the following steps: S1: obtaining feature details of an input image, and outputting a 1 / 4 resolution feature map and a 1 / 8 resolution feature map of the input image respectively; S2: dividing the obtained 1 / 8 resolution feature map into N groups, calculating the matching confidence of the N groups respectively using the grouped 1 / 8 resolution feature map, obtaining a multi-channel cost volume, calculating the information connection between channels one by one, and obtaining a refined single-channel cost volume; S3: using a disparity asymmetric aggregation network to regularize the cost volume, and obtaining an aggregated cost volume; S4: performing disparity regression on the aggregated cost volume to obtain a 1 / 8 resolution disparity map; S5: interpolating the 1 / 8 resolution disparity map to obtain a rough 1 / 4 resolution disparity map, using the disparity map and the 1 / 4 resolution feature map to construct a fine cost volume after fixed disparity correction, and then performing cost aggregation and disparity regression through the same steps as steps S3 and S4 but with different weights, and finally obtaining a full resolution disparity map through interpolation; S6: using a Smooth_L1 function to calculate the prediction loss of the full resolution disparity map output in step S5 and the true value of the data set; In the step S1, the feature details of the input image are obtained through a feature extraction module based on reparameterization and a feature extraction module using block convolution. ； where μ and σ represent the mean and standard deviation of the input BN layer data, respectively; The conversion formula of the feature extraction module based on reparameterization is as follows: i and μ i represent the mean and standard deviation of the i-th branch in the input BN layer data, respectively; γ and β represent the learnable parameters of the BN layer, σ i and β i represent the learnable parameters of the i-th branch in the BN layer data; and represent the converted and the pre-converted convolution weights of each branch, respectively; represent the converted convolution bias of each branch; W and B represent the total converted linear convolution weights and total bias, M represents the total number of branches of the module, and i represents a single branch in the M branches.

2. The real-time binocular stereo matching method based on reparameterization and recursive weighting according to claim 1, characterized in that, γ 3. The real-time binocular stereo matching method based on reparameterization and recursive weighting according to claim 1, characterized in that, The feature extraction module using block convolution uses a convolution block with a step consistent with the size of the convolution kernel.

4. The real-time binocular stereo matching method based on reparameterization and recursive weighting according to claim 3, characterized in that, In the step S2, after obtaining the multi-channel cost volume, in order to reduce the information redundancy between each channel cost, the information correlation between each channel cost is calculated to weight each channel information, and the weight of weighting is learned through a linear convolution block. The specific implementation method of weighting each channel information is as follows: assuming that there are N channel numbers of cost volume, the mutual correlation information between the cost volume of the first channel and the cost volume of the second channel is calculated, that is, the information dependence degree of the two channel cost volumes, and the weight is calculated through a linear convolution block to filter information and suppress repeated information; 5. The real-time binocular stereo matching method based on reparameterization and recursive weighting according to claim 1, characterized in that, The weight is calculated through a linear convolution block to filter information, and a refined cost volume is output, and the mutual information between the refined cost volume and the cost volume of the third channel is calculated and filtered, and so on until the information filtering of all N channels is completed, and finally a single-channel cost volume is obtained.

6. The real-time binocular stereo matching method based on reparameterization and recursive weighting according to claim 5, characterized in that, In the step S3, the disparity asymmetric aggregation network comprises a plurality of asymmetric convolution blocks of 5x5xN, wherein N is the maximum disparity level of the aggregated cost volume.

7. The real-time binocular stereo matching method based on reparameterization and recursive weighting according to claim 1, characterized in that, The asymmetric convolution block comprises a 5x5xN separated 3D convolution and a 1x1x1 3D convolution. ； In the step S6, assuming that the prediction loss is x, the loss calculation formula of the Smooth_L1 function is: The gradient of the Smooth_L1 function is calculated and back propagated, and the weights and biases are adjusted through an Adam optimizer.

Images