Self-attention multi-scale pyramid binocular stereo matching method and electronic device

Abstract

This application discloses a self-attention multi-scale pyramid binocular stereo matching method and electronic device, belonging to the field of stereo vision technology. The method includes: acquiring standard left and right images from a binocular stereo camera; performing multi-scale feature extraction to obtain multi-scale convolutional features of the left and right images; performing channel feature weighting and spatial feature weighting, followed by fusion to obtain multi-scale fused features of the left and right images; constructing a 4D cost volume; performing multi-scale pyramid aggregation to form a multi-scale 3D aggregate; and based on the aggregated 3D aggregate, performing disparity regression after Soft Max operation to output a disparity map of the left image. This application applies an attention mechanism to stereo matching, which plays a role in feature selection, and designs a multi-scale pyramid aggregation module to effectively aggregate multi-scale features to improve the matching accuracy in weakly textured regions. Results show that the method of this invention effectively improves the accuracy of binocular stereo matching.

Description

Technical Field

[0001] This application belongs to the field of stereo vision technology, specifically relating to a self-attention multi-scale pyramid binocular stereo matching method and electronic device. Background Technology

[0002] Commonly used two-dimensional images lack depth information, which is crucial for our perception of the three-dimensional world we inhabit. Simulating the human binocular perception mechanism, binocular stereo vision acquires left and right images with certain angular and positional deviations. By calculating the disparity of each pixel in the image and then using trigonometric formulas, it reconstructs the depth information of the captured scene. This technology holds significant promise for applications in 3D reconstruction, autonomous driving, self-navigation, and augmented reality.

[0003] Traditional stereo matching methods are fast and easy to implement, but have low accuracy. Learning-based stereo matching methods often achieve high matching accuracy, but many learning-based stereo matching methods treat the channel and spatial features of each pixel as equally important in the feature extraction stage. This is not conducive to the full expression of effective features for stereo matching tasks, and gives invalid features the opportunity to hinder the stereo matching process, resulting in a decrease in the final matching accuracy. Meanwhile, stereo matching in weakly textured regions remains an important research direction for binocular stereo matching.

[0004] Therefore, this application provides a self-attention multi-scale pyramid binocular stereo matching method and electronic device to solve the above problems. Summary of the Invention

[0005] The purpose of this application is to provide a self-attention multi-scale pyramid binocular stereo matching method and electronic device. By weighting channel features and spatial features, it enhances the expression of features that are effective for stereo matching tasks and weakens the hindering effect of features that are ineffective or rarely effective for stereo matching tasks. Through the feature filtering effect of the self-attention mechanism, it improves the final stereo matching accuracy. At the same time, it aggregates multi-scale feature information through a multi-scale pyramid module to improve the matching accuracy of the network for weak texture regions, thereby solving the technical problems involved in the background art.

[0006] To solve the above-mentioned technical problems, this application is implemented as follows:

[0007] In a first aspect, embodiments of this application provide a self-attention multi-scale pyramid binocular stereo matching method, including:

[0008] Acquire standard left and right images from a stereo camera;

[0009] A binocular stereo matching network is constructed to extract multi-scale features from the acquired standard left and right images, thereby obtaining the multi-scale convolutional features of the left and right images.

[0010] The obtained multi-scale convolutional features are weighted by channel features and spatial features, and then the weighted channels and spatial features are fused to obtain the multi-scale fusion features of the left and right images.

[0011] Based on the multi-scale fusion features of the left and right images, features are stitched together along the disparity dimension to construct a 4D cost volume;

[0012] The constructed 4D cost volume is aggregated into a multi-scale 3D aggregate through a multi-scale pyramid;

[0013] Based on the aggregated 3D aggregate, after Soft Max operation, disparity regression is performed to output the disparity map of the left image.

[0014] Optionally, acquiring the standard left and right images from the binocular stereo camera includes:

[0015] The binocular stereo camera was calibrated offline to obtain its extrinsic and intrinsic parameters.

[0016] The original left and right images were acquired online, and after distortion calibration and epipolar alignment of the original left and right images using pre-calibrated offline binocular stereo camera parameters, a resolution of [resolution missing] was obtained. The standard left and right images.

[0017] Optionally, the construction of the binocular stereo matching network involves multi-scale feature extraction of the acquired standard left and right images to obtain multi-scale convolutional features of the left and right images, including:

[0018] A binocular stereo matching network is constructed, using the acquired standard left and right images as input to the network. Through multiple layers of residual convolution, the desired result is obtained. , and Convolutional features at three resolution scales;

[0019] Will Convolution on features of resolution, Convolution to features at resolution Resolution, compared to the original Feature convolution features of different resolutions are concatenated;

[0020] Feature fusion is performed through multi-layer convolution to obtain multi-scale convolutional features of the left and right images.

[0021] Optionally, the spatial feature weighting of the obtained multi-scale convolutional features includes:

[0022] Original feature body Along spatial dimensions Sum and average for each channel, then perform feature weighting along the channel dimension:

[0023]

[0024]

[0025] In the formula, Indicates the first The weighted characteristics of the channel, Indicates the first The weighted value of the channel, and Original feature body Height and width;

[0026] No. Features of the channel after channel feature selection It can be calculated from the following formula:

[0027] ;

[0028] The aforementioned channel feature weighting of the obtained multi-scale convolutional features includes:

[0029] Original feature body Along the channel dimension For each pixel Summing and averaging, then weighting the pixel features along the spatial dimension:

[0030]

[0031]

[0032] In the formula, Represents pixels The weighted features, Represents pixels The weighted value, Original feature body Total number of channels;

[0033] pixel Features after spatial feature selection It is calculated by the following formula:

[0034] ;

[0035] The aforementioned fusion of weighted channels and spatial features includes:

[0036] Channel features and spatial features are fused using a self-attention layer, including:

[0037] Will and After cascading, the resulting size is intermediate feature body Then along respectively Weighted and :

[0038]

[0039]

[0040]

[0041]

[0042] In the formula Features selected for channel features The fusion weighted features, Features selected for spatial features The fusion weighted features, and They are respectively and The fusion weighted value, Original feature bodies Total number of channels, height, and width;

[0043] The final feature body that integrates channel and spatial characteristics It is calculated by the following formula:

[0044] .

[0045] Optionally, the step of constructing a 4D cost volume by stitching together features along the disparity dimension based on the multi-scale fusion features of the left and right images includes:

[0046] Based on shape Multi-scale fusion features of the left and right images, along the disparity dimension The features of the left and right images are concatenated to construct a shape of size [value missing]. The 4D cost body, denoted as Then, the features of the left and right images are stitched together according to the following formula:

[0047]

[0048] In the formula, Represents pixels In parallax Cascade features at the location, and Representing pixels In parallax Multi-scale fusion features of the left and right images at the location, This indicates a cascading operation.

[0049] Optionally, the process of aggregating the constructed 4D cost volume into a multi-scale 3D aggregate using a multi-scale pyramid includes:

[0050] Perform separate operations on the constructed 4D cost volume , and Downsampling to shape size , and The cost body , and ;

[0051] exist Pyramid layers Perform two layers of 3D convolution, and then pass the results down to the 1 / 4 pyramid layer. Perform two layers of 3D convolution, and pass the results down to same layer Perform two convolutions, and then pass the results down to the bottom layer of the pyramid. After performing two 3D convolution layers, a final 3D residual layer is applied to obtain a shape with a size of Multiscale 3D polymers.

[0052] Optionally, the 3D aggregated object, after undergoing Soft Max operation, is subjected to disparity regression to output a left image disparity map, including:

[0053] The parallax level is obtained by performing a Soft Max operation along the parallax dimension on the aggregated multi-scale 3D aggregate. probability body Finally, disparity regression is performed using a probabilistic volume to obtain the final left image disparity map. The disparity regression formula is as follows:

[0054]

[0055] In the formula, Indicates a Soft Max operation, then Indicates the disparity of a certain pixel as The probability of.

[0056] Optionally, after the step of performing disparity regression and outputting the left image disparity map based on the aggregated 3D aggregate through Soft Max operation, the method further includes:

[0057] Training a binocular stereo matching network includes:

[0058] During the training phase, the multi-scale pyramid has four layers, which means... , , and The output disparity maps are divided into four scales, and different training weights are used for fully supervised learning. A smooth L1 loss function is used for end-to-end training of the network. The loss function of the stereo matching network is expressed as follows:

[0059]

[0060] In the formula, This represents the total loss value during training. Indicates the first The weighting coefficients for the impact of the layer pyramid output disparity map on the training loss. Represents the true value of parallax. Indicates smoothness The loss function is defined as follows:

[0061]

[0062] In the formula, For smoothing The input variables of the loss function;

[0063] The trained binocular stereo matching network is used to perform stereo matching on the standard left and right images to obtain a dense and accurate disparity map of the left image.

[0064] In a second aspect, embodiments of this application provide an electronic device including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0065] Thirdly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0066] In this embodiment, by weighting the importance of channel features and spatial features, a feature filtering function is achieved, which can enhance the expression of effective features, weaken the hindering effect of invalid features on stereo matching, and improve the accuracy of the final stereo matching. At the same time, this application uses a multi-scale pyramid aggregation module to aggregate multi-scale feature information on the constructed 4D cost volume. Multi-scale features are useful for matching weak texture regions, thereby improving the matching accuracy of the network for weak texture regions. Attached Figure Description

[0067] Figure 1 This is a flowchart illustrating the method provided in an embodiment of this application;

[0068] Figure 2 This is a test result diagram of the method provided in the embodiments of this application on the Scene Flow dataset;

[0069] Figure 3 This is a structural framework diagram of the electronic device provided in the embodiments of this application;

[0070] Figure 4 This is a hardware structure diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0071] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0072] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0073] The self-attention multi-scale pyramid binocular stereo matching method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0074] Please see Figure 1 This application provides a self-attention multi-scale pyramid binocular stereo matching method, comprising:

[0075] Step S101: Acquire the standard left and right images from the binocular stereo camera;

[0076] Step S102: Construct a binocular stereo matching network to extract multi-scale features from the acquired standard left and right images, thereby obtaining the multi-scale convolutional features of the left and right images;

[0077] Step S103: Perform channel feature weighting and spatial feature weighting on the obtained multi-scale convolution features, and then fuse the weighted channels and spatial features to obtain the multi-scale fusion features of the left and right images;

[0078] Step S104: Based on the multi-scale fusion features of the left and right images, feature stitching is performed along the disparity dimension to construct a 4D cost volume;

[0079] Step S105: The constructed 4D cost volume is aggregated into a multi-scale 3D aggregate using a multi-scale pyramid method;

[0080] Step S106: Based on the aggregated 3D aggregate, after Soft Max operation, perform disparity regression and output the disparity map of the left image.

[0081] In step S101, acquiring the standard left and right images from the stereo camera includes:

[0082] The binocular stereo camera was calibrated offline to obtain its extrinsic and intrinsic parameters.

[0083] The original left and right images were acquired online, and after distortion calibration and epipolar alignment of the original left and right images using pre-calibrated offline binocular stereo camera parameters, a resolution of [resolution missing] was obtained. The standard left and right images.

[0084] In step S102, the construction of the binocular stereo matching network involves multi-scale feature extraction of the acquired standard left and right images to obtain multi-scale convolutional features of the left and right images, including:

[0085] A binocular stereo matching network is constructed, using the acquired standard left and right images as input to the network. Through multiple layers of residual convolution, the desired result is obtained. , and Convolutional features at three resolution scales;

[0086] Will Convolution on features of resolution, Convolution to features at resolution Resolution, compared to the original Feature convolution features of different resolutions are concatenated;

[0087] Feature fusion is performed through multi-layer convolution to obtain multi-scale convolutional features of the left and right images.

[0088] In step S103, the spatial feature weighting of the obtained multi-scale convolutional features includes:

[0089] Original feature body Along spatial dimensions Sum and average for each channel, then perform feature weighting along the channel dimension:

[0090]

[0091]

[0092] In the formula, Indicates the first The weighted characteristics of the channel, Indicates the first The weighted value of the channel, and Original feature body Height and width;

[0093] No. Features of the channel after channel feature selection It can be calculated from the following formula:

[0094] ;

[0095] The aforementioned channel feature weighting of the obtained multi-scale convolutional features includes:

[0096] Original feature body Along the channel dimension For each pixel Summing and averaging, then weighting the pixel features along the spatial dimension:

[0097]

[0098]

[0099] In the formula, Represents pixels The weighted features, Represents pixels The weighted value, Original feature body Total number of channels;

[0100] pixel Features after spatial feature selection It is calculated by the following formula:

[0101] ;

[0102] The aforementioned fusion of weighted channels and spatial features includes:

[0103] Channel features and spatial features are fused using a self-attention layer, including:

[0104] Will and After cascading, the resulting size is intermediate feature body Then along respectively Weighted and :

[0105]

[0106]

[0107]

[0108]

[0109] In the formula Features selected for channel features The fusion weighted features, Features selected for spatial features The fusion weighted features, and They are respectively and The fusion weighted value, Original feature bodies Total number of channels, height, and width;

[0110] The final feature body that integrates channel and spatial characteristics It is calculated by the following formula:

[0111] .

[0112] In step S104, the step of constructing a 4D cost volume by stitching together features along the disparity dimension based on the multi-scale fusion features of the left and right images includes:

[0113] Based on shape Multi-scale fusion features of the left and right images, along the disparity dimension The features of the left and right images are concatenated to construct a shape of size [value missing]. The 4D cost body, denoted as Then, the features of the left and right images are stitched together according to the following formula:

[0114]

[0115] In the formula, Represents pixels In parallax Cascade features at the location, and Representing pixels In parallax Multi-scale fusion features of the left and right images at the location, This indicates a cascading operation.

[0116] In step S105, the process of aggregating the constructed 4D cost body into a multi-scale 3D aggregate using a multi-scale pyramid includes:

[0117] Perform separate operations on the constructed 4D cost volume , and Downsampling to shape size , and The cost body , and ;

[0118] exist Pyramid layers Perform two layers of 3D convolution, and then pass the results down to the 1 / 4 pyramid layer. Perform two layers of 3D convolution, and pass the results down to same layer Perform two convolutions, and then pass the results down to the bottom layer of the pyramid. After performing two 3D convolution layers, a final 3D residual layer is applied to obtain a shape with a size of Multiscale 3D polymers.

[0119] In step S106, the 3D aggregated object, after undergoing a Soft Max operation, is subjected to disparity regression to output a left image disparity map, including:

[0120] The parallax level is obtained by performing a Soft Max operation along the parallax dimension on the aggregated multi-scale 3D aggregate. probability body Finally, disparity regression is performed using a probabilistic volume to obtain the final left image disparity map. The disparity regression formula is as follows:

[0121]

[0122] In the formula, Indicates a Soft Max operation, then Indicates the disparity of a certain pixel as The probability of.

[0123] In step S106, after the disparity regression is performed on the aggregated 3D aggregate through Soft Max operation and the left image disparity map is output, the following steps are also included:

[0124] Training a binocular stereo matching network includes:

[0125] During the training phase, the multi-scale pyramid has four layers, which means... , , and The output disparity maps are divided into four scales, and different training weights are used for fully supervised learning. A smooth L1 loss function is used for end-to-end training of the network. The loss function of the stereo matching network is expressed as follows:

[0126]

[0127] In the formula, This represents the total loss value during training. Indicates the first The weighting coefficients for the impact of the layer pyramid output disparity map on the training loss. Represents the true value of parallax. Indicates smoothness The loss function is defined as follows:

[0128]

[0129] In the formula, For smoothing The input variables of the loss function;

[0130] The trained binocular stereo matching network is used to perform stereo matching on the standard left and right images to obtain a dense and accurate disparity map of the left image.

[0131] The method provided in this application was tested on the Scene Flow test set, and the resulting graph is shown below. Figure 2 The results are shown in Table 1. The comparisons are made with the test results of Gc-Net, DispNetC, CRL, PSMNet, EdgeStereo, and Gwc-Net.

[0132] Table 1. Test results of various methods on the Scene Flow test set.

[0133]

[0134] The sources for the above methods are as follows:

[0135] 1) GCNet: "End-to-End Learning of Geometry and Context for Deep StereoRegression";

[0136] 2) DispNetC: "A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation";

[0137] 3) CRL: "Cascade Residual Learning: A Two-stage Convolutional NeuralNetwork for Stereo Matching";

[0138] 4) PSMNet: "Pyramid Stereo Matching Network";

[0139] 5) iResNet-i2: "Learning for disparity estimation through featureconstancy";

[0140] 6) LRCR: "Left-right comparative recurrent model for stereo matching".

[0141] In Table 1, the EPE index represents the average parallax error, and its unit is pixels. The 3px (%) index represents the percentage of pixels whose absolute parallax error is greater than 3 pixels. Both indices are better the smaller they are.

[0142] As shown in Table 1, the method of this invention achieves significant accuracy improvements in both the EPE (Effective Per Element) and 3px (%) metrics. This is attributed to the feature selection mechanism under the attention mechanism, which strengthens features useful for stereo matching and suppresses features useless for stereo matching. Furthermore, multi-scale pyramid aggregation enables the network to learn features at more scales, improving the matching accuracy in weakly textured regions. Therefore, the combined effect of the channel and spatial feature fusion and multi-scale pyramid aggregation ideas in the method of this invention can effectively improve the accuracy of stereo matching.

[0143] like Figure 3As shown, this application embodiment also provides an electronic device 600, which includes a processor 601, a memory 602, and a program or instructions stored in the memory 602 and executable on the processor 601. When the program or instructions are executed by the processor 601, they implement the various processes of the above-described self-attention multi-scale pyramid binocular stereo matching method embodiment and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0144] It should be noted that the first electronic device in the embodiments of this application includes the mobile electronic device and the non-mobile electronic device described above.

[0145] Figure 4 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0146] The electronic device 700 includes, but is not limited to, components such as: radio frequency unit 701, network module 702, audio output unit 703, input unit 704, sensor 705, display unit 706, user input unit 707, interface unit 708, memory 709, and processor 710.

[0147] Those skilled in the art will understand that the electronic device 700 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 710 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 4 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0148] It should be understood that, in this embodiment, the input unit 704 may include a graphics processing unit (GPU) 7041 and a microphone 7042. The GPU 7041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 706 may include a display panel 7061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit 707 includes a touch panel 7071 and other input devices 7072. The touch panel 7071 is also called a touch screen. The touch panel 7071 may include a touch detection device and a touch controller. Other input devices 7072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here. The memory 709 can be used to store software programs and various data, including but not limited to applications and operating systems. The processor 710 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understandable that the aforementioned modem processor may not be integrated into the processor 710.

[0149] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described self-attention multi-scale pyramid binocular stereo matching method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0150] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0151] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described self-attention multi-scale pyramid binocular stereo matching method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0152] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0153] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0154] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0155] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A self-attention multi-scale pyramid binocular stereo matching method, characterized in that, include: Acquire standard left and right images from a stereo camera; A binocular stereo matching network is constructed to extract multi-scale features from the acquired standard left and right images, thereby obtaining the multi-scale convolutional features of the left and right images. The obtained multi-scale convolutional features are weighted by channel features and spatial features, and then the weighted channels and spatial features are fused to obtain the multi-scale fusion features of the left and right images. Spatial feature weighting is applied to the obtained multi-scale convolutional features. include: original feature bodies along the spatial dimension sum over each channel and average, then feature weighting along the channel dimension: In the formula, Indicates the first The weighted characteristics of the channel, Indicates the first The weighted value of the channel, and Original feature body Height and width; No. Features of the channel after channel feature selection It can be calculated from the following formula: ； The aforementioned method involves weighting the obtained multi-scale convolutional features based on channel features. include: Original feature body Along the channel dimension For each pixel Summing and averaging, then weighting the pixel features along the spatial dimension: In the formula, Represents pixels The weighted features, Represents pixels The weighted value, Original feature body Total number of channels; pixel Features after spatial feature selection It is calculated by the following formula: ； The aforementioned fusion of weighted channels and spatial features includes: Channel features and spatial features are fused using a self-attention layer, including: Will and After cascading, the resulting size is intermediate feature body Then along respectively Weighted and : In the formula Features selected for channel features The fusion weighted features, Features selected for spatial features The fusion weighted features, and They are respectively and The fusion weighted value, Original feature bodies Total number of channels, height, and width; The final feature body that integrates channel and spatial characteristics It is calculated by the following formula: ； Based on the multi-scale fusion features of the left and right images, features are stitched together along the disparity dimension to construct a 4D cost volume; The constructed 4D cost volume is aggregated into a multi-scale 3D aggregate through a multi-scale pyramid; Based on the aggregated 3D aggregate, after Soft Max operation, disparity regression is performed to output the disparity map of the left image.

2. The method according to claim 1, characterized in that, The acquisition of standard left and right images from a stereo camera includes: The binocular stereo camera was calibrated offline to obtain its extrinsic and intrinsic parameters. The original left and right images were acquired online, and after distortion calibration and epipolar alignment of the original left and right images using pre-calibrated offline binocular stereo camera parameters, a resolution of [resolution missing] was obtained. The standard left and right images.

3. The method according to claim 1, characterized in that, The aforementioned construction of a binocular stereo matching network involves multi-scale feature extraction from the acquired standard left and right images to obtain multi-scale convolutional features of the left and right images, including: A binocular stereo matching network is constructed, using the acquired standard left and right images as input to the network. Through multiple layers of residual convolution, the desired result is obtained. , and Convolutional features at three resolution scales; Will Convolution on features of resolution, Convolution of features at resolution down to Resolution, compared to the original Feature convolution features of different resolutions are concatenated; Feature fusion is performed through multi-layer convolution to obtain multi-scale convolutional features of the left and right images.

4. The method according to claim 1, characterized in that, The method of constructing a 4D cost volume by stitching together features along the disparity dimension based on the multi-scale fusion features of the left and right images includes: Based on shape Multi-scale fusion features of the left and right images, along the disparity dimension The features of the left and right images are concatenated to construct a shape of size [value missing]. The 4D cost body, denoted as Then, the features of the left and right images are stitched together according to the following formula: In the formula, Represents pixels In parallax Cascade features at the location, and Representing pixels In parallax Multi-scale fusion features of the left and right images at the location, This indicates a cascading operation.

5. The method according to claim 1, characterized in that, The process of aggregating the constructed 4D cost volume into a multi-scale 3D aggregate using a multi-scale pyramid includes: Perform separate operations on the constructed 4D cost volume , and Downsampling to shape size , and The cost body , and ; exist Pyramid layers Perform two layers of 3D convolution, and then pass the results down to the 1 / 4 pyramid layer. Perform two layers of 3D convolution, and pass the results down to same layer Perform two convolutions, and then pass the results down to the bottom layer of the pyramid. After performing two 3D convolution layers, a final 3D residual layer is applied to obtain a shape with a size of Multiscale 3D polymers.

6. The method according to claim 1, characterized in that, The aforementioned 3D aggregate, after being processed by SoftMax, undergoes disparity regression to output a disparity map of the left image, including: The parallax level is obtained by performing a Soft Max operation along the parallax dimension on the aggregated multi-scale 3D aggregate. probability body Finally, disparity regression is performed using a probabilistic volume to obtain the final left image disparity map. The disparity regression formula is as follows: In the formula, Indicates a Soft Max operation, then Indicates the disparity of a certain pixel as The probability of.

7. The method according to claim 1, characterized in that, After the aforementioned 3D aggregate, following the Soft Max operation and subsequent disparity regression to output the left image disparity map, the process further includes: Training a binocular stereo matching network includes: During the training phase, the multi-scale pyramid has four layers, which means... , , and The output disparity maps are divided into four scales, and different training weights are used for fully supervised learning. A smooth L1 loss function is used for end-to-end training of the network. The loss function of the stereo matching network is expressed as follows: In the formula, This represents the total loss value during training. Indicates the first The weighting coefficients for the impact of the layer pyramid output disparity map on the training loss. Represents the true value of parallax. Indicates smoothness The loss function is defined as follows: In the formula, For smoothing The input variables of the loss function; The trained binocular stereo matching network is used to perform stereo matching on the standard left and right images to obtain a dense and accurate disparity map of the left image.

8. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein when the program or instructions are executed by the processor, they implement the steps of the self-attention multi-scale pyramid binocular stereo matching method as described in any one of claims 1 to 7.

9. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the self-attention multi-scale pyramid binocular stereo matching method as described in any one of claims 1 to 7.

Images