Binocular image processing method, device, equipment and storage medium

Abstract

An embodiment of the present invention discloses a binocular image processing method, apparatus, device and storage medium, the method comprising: obtaining a binocular image to be processed; performing feature extraction on a left image and a right image in the binocular image respectively, and determining a first left feature map corresponding to the left image and a first right feature map corresponding to the right image; inputting the first left feature map and the first right feature map into a preset attention network model, determining a left weight matrix and a right weight matrix based on a feature difference between two feature points corresponding in position in the first left feature map and the first right feature map and a correspondence between a preset feature difference and a weight value, and performing attention processing on the first left feature map and the first right feature map respectively based on the left weight matrix and the right weight matrix; obtaining left and right feature maps after attention processing according to the output of the preset attention network model, thereby suppressing features containing disparity edges and improving the accuracy of disparity estimation.

Description

Technical Field

[0001] The present invention relates to image processing technology, and more particularly to a binocular image processing method, apparatus, device, and storage medium. Background Art

[0002] With the rapid development of computer vision, disparity maps can be estimated through stereo matching to reconstruct the three-dimensional geometric information of the scene.

[0003] Currently, deep convolutional network models can be used to directly predict the dense disparity map of the scene, so that the stereo matching method based on deep learning can achieve more accurate disparity estimation.

[0004] However, in the process of implementing the present invention, the inventors found that there are at least the following problems in the prior art:

[0005] When the captured binocular image contains disparity edge areas, such as the edges of buildings, telephone poles, and the outer contours of objects, feature extraction of the binocular image will often introduce noisy or even erroneous information into the subsequent calculation of the cost aggregate, ultimately reducing the accuracy of the disparity estimation. Summary of the Invention

[0006] Embodiments of the present invention provide a binocular image processing method, apparatus, device, and storage medium to suppress features containing parallax edges and improve the accuracy of parallax estimation.

[0007] In a first aspect, an embodiment of the present invention provides a binocular image processing method, comprising:

[0008] Obtaining binocular images to be processed;

[0009] Performing feature extraction on a left image and a right image in the binocular image respectively to determine a first left feature map corresponding to the left image and a first right feature map corresponding to the right image;

[0010] Inputting the first left feature map and the first right feature map into a preset attention network model for feature attention processing, the preset attention network model being configured to: determine a left weight matrix corresponding to the first left feature map and a right weight matrix corresponding to the first right feature map based on a feature difference between two feature points at corresponding positions in the first left feature map and the first right feature map, and a preset correspondence between the feature difference and the weight value, and perform attention processing on the first left feature map and the first right feature map based on the left weight matrix and the right weight matrix, respectively;

[0011] According to the output of the preset attention network model, the left and right feature maps after attention processing are obtained.

[0012] In a second aspect, an embodiment of the present invention further provides a binocular image processing device, comprising:

[0013] A binocular image acquisition module is used to acquire the binocular image to be processed;

[0014] a feature extraction module, configured to perform feature extraction on the left image and the right image in the binocular image respectively, and determine a first left feature map corresponding to the left image and a first right feature map corresponding to the right image;

[0015] a feature map input module, configured to input the first left feature map and the first right feature map into a preset attention network model for feature attention processing, wherein the preset attention network model is configured to: determine a left weight matrix corresponding to the first left feature map and a right weight matrix corresponding to the first right feature map based on a feature difference between two feature points at corresponding positions in the first left feature map and the first right feature map, and to perform attention processing on the first left feature map and the first right feature map based on the left weight matrix and the right weight matrix;

[0016] The feature map output module is used to obtain the left and right feature maps after attention processing according to the output of the preset attention network model.

[0017] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising:

[0018] one or more processors;

[0019] a memory for storing one or more programs;

[0020] When the one or more programs are executed by the one or more processors, the one or more processors implement the binocular image processing method provided by any embodiment of the present invention.

[0021] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the binocular image processing method provided by any embodiment of the present invention.

[0022] The embodiments of the above invention have the following advantages or beneficial effects:

[0023] By performing feature extraction on the left image and the right image in the binocular image to be processed, a first left feature map corresponding to the left image and a first right feature map corresponding to the right image are determined. The first left feature map and the first right feature map obtained at this time may contain features of a disparity edge. By inputting the first left feature map and the first right feature map into a preset attention network model for feature attention processing, the preset attention network model can determine a left weight matrix corresponding to the first left feature map and a right weight matrix corresponding to the first right feature map based on the feature difference between two feature points corresponding to positions in the first left feature map and the first right feature map, and the correspondence between the preset feature difference and the weight value, and perform attention processing on the first left feature map and the first right feature map based on the left weight matrix and the right weight matrix, respectively, thereby reducing attention to features containing disparity edges and increasing attention to features not containing disparity edges, thereby suppressing features containing disparity edges in the first left feature map and the first right feature map, avoiding the influence of features containing disparity edges on disparity estimation, and improving the accuracy of disparity estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is a flow chart of a binocular image processing method provided by an embodiment of the present invention;

[0025] Figure 2 This is an example of a disparity estimation process provided by an embodiment of the present invention;

[0026] Figure 3 is a flow chart of a binocular image processing method provided by an embodiment of the present invention;

[0027] Figure 4 This is an example of a preset attention network model provided by an embodiment of the present invention;

[0028] Figure 5 This is an example of another preset attention network model provided by an embodiment of the present invention;

[0029] Figure 6 is a flow chart of a binocular image processing method provided by an embodiment of the present invention;

[0030] Figure 7 This is an example of a preset attention network model provided by an embodiment of the present invention;

[0031] Figure 8 1 is a schematic structural diagram of a binocular image processing device provided by an embodiment of the present invention;

[0032] Figure 9 It is a structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The present invention will be further described in detail below with reference to the accompanying drawings and examples. It will be understood that the specific embodiments described herein are intended only to illustrate the present invention and are not intended to limit the present invention. It should also be noted that, for ease of description, the accompanying drawings only illustrate portions relevant to the present invention, not all structures.

[0034] Figure 1 This is a flowchart of a binocular image processing method provided by an embodiment of the present invention. This embodiment is applicable to the case of performing feature attention processing on binocular images. The method can be executed by a binocular image processing device, which can be implemented by software and / or hardware and integrated into an electronic device. Figure 1 As shown, the method specifically includes the following steps:

[0035] S110: Acquire a binocular image to be processed.

[0036] The binocular image may be two scene images captured by a binocular camera consisting of a left camera and a right camera, i.e., a left image and a right image. For example, the binocular image may refer to two images captured in an autonomous driving scenario.

[0037] Specifically, this embodiment can acquire binocular images for stereo matching.

[0038] S120 , performing feature extraction on the left image and the right image in the binocular image respectively, and determining a first left feature map corresponding to the left image and a first right feature map corresponding to the right image.

[0039] Specifically, an existing feature extraction network model can be used to extract features from the left image and the right image in the binocular image respectively. The feature extraction network model in this embodiment can refer to a two-dimensional convolutional neural network model for extracting features. For example, the left image and the right image in the binocular image can be input into the feature extraction network model in sequence for feature extraction, so that based on the output of the feature extraction network model, a first left feature map and a first right feature map can be obtained in sequence; or the left image and the right image in the binocular image can be input into two weight-shared feature extraction network models at the same time for feature extraction, so that based on the output of the two feature extraction network models, a first left feature map and a first right feature map can be obtained at the same time, thereby improving the efficiency of feature map acquisition.

[0040] It should be noted that if disparity edges exist in the left and / or right images, such as building edges, utility pole edges, or object outlines, the feature extraction network model will have a higher response in these areas, and the first left and right feature maps extracted will contain features containing disparity edges. Real objects are typically continuous in space. However, when the disparity gradient in disparity edge regions is greater than 1, i.e., when the disparity is discontinuous, the features containing disparity edges extracted in these regions will affect subsequent disparity estimation, reducing its accuracy.

[0041] S130. Input the first left feature map and the first right feature map into a preset attention network model for feature attention processing. The preset attention network model is used to: determine the left weight matrix corresponding to the first left feature map and the right weight matrix corresponding to the first right feature map based on the feature difference between two feature points corresponding to positions in the first left feature map and the first right feature map and the correspondence between the preset feature difference and the weight value, and perform attention processing on the first left feature map and the first right feature map based on the left weight matrix and the right weight matrix.

[0042] Among them, the preset attention network model can be a pre-set network model for reducing attention to features containing disparity edges. The two feature points corresponding to the positions in the first left feature map and the first right feature map can refer to the two receptive field areas corresponding to the two feature points located in the first left feature map and the first right feature map, respectively, which are the same, or one receptive field area is a sub-area of ​​the other receptive field area. For example, the first receptive field area on the left image corresponding to a feature point x in the first left feature map is the same as the second receptive field area on the right image corresponding to the feature point y corresponding to the position in the first right feature map, or the first receptive field area is a sub-area of ​​the second receptive field area. The feature difference between the two feature points can refer to the difference in feature information corresponding to the two feature points. For example, the size of the feature map is: C×H×W, where C represents the number of channels, H represents the image height, and W represents the image width. For a feature map with a channel number of 2 or greater than 2, the feature information of each feature point in the feature map is a feature vector. In this embodiment, the feature vector X of the feature point x can be used to represent the value of the feature point x, that is, X = f(G), where G represents the receptive field area of ​​the feature point x on the original image, and f() is the feature extraction function corresponding to the feature extraction network model.

[0043] Research has found that features containing parallax edges are continuous in one binocular image but discontinuous in the other. Consequently, the positions of corresponding feature points in the first left feature map and the first right feature map extracted by the same feature extraction network model can deviate significantly. The correspondence between feature differences and weights can be pre-set based on the representation of features containing parallax edges. For example, the feature differences and weights are inversely proportional. A larger feature difference indicates a greater likelihood that the receptive field corresponding to the feature point contains a parallax edge, i.e., a greater likelihood that the feature point is a feature containing a parallax edge. In this case, the weight is set smaller to reduce attention to features containing parallax edges. Conversely, smaller feature differences increase weights to increase attention to features not containing edges. The left weight matrix can be composed of the weights corresponding to each feature point in the first left feature map. The right weight matrix can be composed of the weights corresponding to each feature point in the first right feature map.

[0044] Specifically, by inputting the first left feature map and the first right feature map into a preset attention network model for feature attention processing, the preset attention network model can determine the weight value corresponding to each feature point in the first left feature map, i.e., obtain a left weight matrix, and the weight value corresponding to each feature point in the first right feature map, i.e., obtain a right weight matrix, based on the feature difference between two feature points located at corresponding positions in the first left feature map and the first right feature map, and the feature difference between the pre-set feature difference and the weight value. For example, the right feature point corresponding to the position of each left feature point in the first left feature map can be searched on the first right feature map, and the weight value corresponding to each left feature point in the first left feature map can be determined based on the feature difference between the left and right feature points. Similarly, the left feature point corresponding to the position of each right feature point in the first right feature map can be searched on the first left feature map, and the weight value corresponding to each right feature point in the first right feature map can be determined based on the feature difference between the left and right feature points. By performing element-by-element multiplication of the left weight matrix and the first left feature map, the left feature map after attention processing is obtained and output. By multiplying the right weight matrix by the first right feature map element by element, the right feature map after attention processing is obtained and output.

[0045] S140. Obtain left and right feature maps after attention processing based on the output of the preset attention network model.

[0046] Specifically, the preset attention network model can output the left and right feature maps after attention processing, thereby suppressing the extraction of features containing disparity edges, avoiding the influence of features containing disparity edges on subsequent disparity estimation, and improving the accuracy of disparity estimation.

[0047] It should be noted that the search process for a right feature point corresponding to a left feature point on the first left feature map on the first right feature map is similar to the search process for a left feature point corresponding to a right feature point on the first left feature map on the first right feature map on the first left feature map. This embodiment uses the search process on the first right feature map as an example for explanation. Let x represent a left feature point in the first left feature map, and G represent the receptive field area of ​​the left feature point x on the left image. Based on the characteristics of stereo matching, the following inferences can be drawn:

[0048] When G does not contain the parallax edge area, the area on the right image that represents the same actual area as G is denoted as G′, and the right feature point corresponding to G′ that is located in the same layer and the same row as the left feature point x is denoted as x′. At this time, for any feature extraction network model, the feature vector X′ of x′ is equal to the feature vector X of x, that is, |X′-X|=0. At this time, the right feature point x′ can be called the corresponding feature point whose feature difference with the left feature point x is zero.

[0049] When G contains a disparity edge area, G′ is spatially discontinuous. When features containing a disparity edge on G are extracted, these features are also discontinuous on G′. The union of the area of ​​G′ and the area between the left and right parts of G′ can be called G″, and the right feature point that is in the same layer and row as the left feature point x and whose receptive field area is a sub-area of ​​G″ is recorded as x″. There can be multiple right feature points x″, and for any x″, |X″-X|>0. For a specific range, the receptive field area of ​​the feature points within the range corresponds to the union G″ of the areas in the right image, and there is no corresponding right feature point within this range whose feature difference with the left feature point x is zero.

[0050] As can be seen, lower weights can be assigned to left feature points x within a certain range where there is no zero feature difference, thereby reducing attention to features containing parallax edges. Since features containing parallax edges are likely to result in larger minimum feature differences, and considering certain special cases such as data noise and slight differences between the left and right images, the correspondence between feature differences and weights can be set such that the smaller the minimum feature difference, the larger the weight. This can reduce attention to features containing parallax edges and minimize the impact on other beneficial features.

[0051] In the practical application of the preset attention network model, taking into account the time cost, GPU memory cost and the differences between the layers in the network, for feature maps whose values ​​are not negative after the activation function ReLu, without losing the purpose of the preset attention network model, |X″-X| can be approximated as |wX″-wX|, where w is the parameter to be learned, so that a very lightweight convolutional layer can be used. For example, a convolutional layer of 1×1× input channels calculates wX″ or wX and learns w. Assuming that the weights of n feature points need to be calculated, each feature point is a c-dimensional vector, and the maximum allowable disparity range is d, the time complexity can be reduced from O(ndc) to O(nd+nc), so that a lightweight preset attention network model can be constructed without affecting the efficiency of disparity estimation.

[0052] The technical solution of this embodiment is to extract features from the left and right images in the binocular image to be processed, respectively, to determine a first left feature map corresponding to the left image and a first right feature map corresponding to the right image. The first left feature map and the first right feature map obtained at this time may contain features of a disparity edge. By inputting the first left feature map and the first right feature map into a preset attention network model for feature attention processing, the preset attention network model can determine a left weight matrix corresponding to the first left feature map and a right weight matrix corresponding to the first right feature map based on the feature difference between two feature points at corresponding positions in the first left feature map and the first right feature map, and the correspondence between the preset feature difference and the weight value, and perform attention processing on the first left feature map and the first right feature map based on the left weight matrix and the right weight matrix, respectively. This can reduce attention to features containing disparity edges and increase attention to features not containing disparity edges, thereby suppressing features containing disparity edges in the first left feature map and the first right feature map, preventing features containing disparity edges from affecting disparity estimation, and improving the accuracy of disparity estimation.

[0053] On the basis of the above technical solution, after S140, it may also include: inputting the left feature map and the right feature map output by the preset attention network model into the disparity estimation network model for disparity estimation; based on the output of the disparity estimation network model, obtaining the target disparity map corresponding to the binocular image.

[0054] The disparity estimation network model may be a network model for performing disparity region estimation based on left and right feature maps. For example, the disparity estimation network model may refer to a disparity estimation network model in a PSMNet (Pyramid Stereo Matching Network) pyramid stereo matching network model. Figure 2 An example of a disparity estimation process is given. Figure 2As shown in Figure 1, the disparity estimation network model may include a pyramid pooling structure (SPP module), a cost volume, and a three-dimensional convolutional network (3D CNN). The pyramid pooling structure can be used to extract multi-scale target features from the input left and right feature maps. The cost volume can be used to cost-aggregate the multi-scale target features from the left and right feature maps to obtain three-dimensional feature information. The 3D convolutional network can be used to perform subsequent cost calculations on the three-dimensional feature information to estimate the target disparity map.

[0055] Specifically, the left feature map and the right feature map output by the preset attention network model, which have suppressed the disparity edge features, are input into the disparity estimation network model for disparity estimation, thereby avoiding the influence of the features containing the disparity edge on the disparity estimation and improving the accuracy of the target disparity map estimation.

[0056] For example, two identical preset attention network models can be used for attention processing to further improve the suppression effect of features containing disparity edges and further improve the accuracy of disparity estimation. Figure 2 As shown, the left and right images in the binocular image can be simultaneously input into two weight-shared feature extraction network models for feature extraction, so as to simultaneously obtain a first left feature map and a first right feature map based on the outputs of the two feature extraction network models. The first left feature map and the first right feature map are then input into a first preset attention network model for attention processing, and the left and right feature maps output by the first preset attention network model are again input into a second preset attention network model for re-attention processing. The output left and right feature maps with features that further suppress disparity edges are input into a disparity estimation network model for disparity estimation, thereby further avoiding the influence of features containing disparity edges on disparity estimation and further improving the accuracy of target disparity map estimation.

[0057] Figure 3 A flowchart of a binocular image processing method provided in an embodiment of the present invention. In this embodiment, based on the above embodiments, the preset attention network model may include: a feature fusion sub-model, a weight estimation sub-model and an attention processing sub-model, and on this basis, the attention processing process is described in detail, wherein the explanations of the terms that are the same as or corresponding to the above embodiments are not repeated here.

[0058] See also Figure 3 The binocular image processing method provided in this embodiment specifically includes the following steps:

[0059] S310: Obtain a binocular image to be processed.

[0060] S320: Perform feature extraction on the left image and the right image in the binocular image respectively to determine a first left feature map corresponding to the left image and a first right feature map corresponding to the right image.

[0061] S330: Input the first left feature map into the feature fusion sub-model, fuse the feature information of each channel in the first left feature map, and determine a fused second left feature map.

[0062] The feature fusion sub-model can be a network model for fusing feature information on each channel in a feature map to obtain a feature map with a reduced number of channels. For example, the feature fusion sub-model can be used to obtain a feature map with a single channel or a feature map with two channels.

[0063] S340: Input the first right feature map into the feature fusion sub-model, fuse the feature information of each channel in the first right feature map, and determine a fused second right feature map.

[0064] Specifically, in this embodiment, the first left feature map and the first right feature map can be successively input into the feature fusion sub-model for feature fusion, or the first left feature map and the first right feature map can be respectively input into two weight-shared feature fusion sub-models for feature fusion, so as to obtain the second left feature map and the second right feature map at the same time, thereby improving the acquisition efficiency. Figure 4 shown.

[0065] It should be noted that the execution order of step S340 is not limited here. For example, step S340 can be executed sequentially after step S330, or before step S330, or simultaneously with step S330.

[0066] Exemplarily, the feature fusion sub-model may include: a first convolutional layer and a second convolutional layer. Figure 5 Another example of a preset attention network model is given. Figure 5 As shown, S330 may include: inputting the first left feature map into the first convolutional layer for convolution operation to determine a fourth left feature map of the same size as the first left feature map; inputting the fourth left feature map into the second convolutional layer for convolution operation to determine a fused second left feature map. S340 may include: inputting the first right feature map into the first convolutional layer for convolution operation to determine a fourth right feature map of the same size as the first right feature map; inputting the fourth right feature map into the second convolutional layer for convolution operation to determine a fused second right feature map.

[0067] Specifically, the first left feature map C×H×W can be further extracted through the first convolution layer to obtain a fourth left feature map C×H×W with the same size as the first left feature map, and the fourth left feature map C×H×W is input to the second convolution layer for convolution operation to reduce the number of channels to obtain the second left feature map. For example, the number of channels is reduced to 1, that is, the second left feature map is obtained. Figure 1 ×H×W. Similarly, the first convolution layer can further extract features from the first right feature map C×H×W to obtain a fourth right feature map C×H×W of the same size as the first right feature map, and the fourth right feature map C×H×W is input to the second convolution layer for convolution operation to reduce the number of channels to obtain the second right feature map. For example, the number of channels is reduced to 1 to obtain the second right feature map. Figure 1 ×H×W.

[0068] For example, the first convolution layer may include: two two-dimensional convolutions with a convolution kernel size of 3×3. The second convolution layer may include: a two-dimensional convolution with a convolution kernel size of 1×1. Each convolution kernel can act on the entire feature map by rolling. The number of channels of the 1×1 two-dimensional convolution in the second convolution layer can be equal to the number of channels of the fourth right feature map, so that the convolution operation of the second convolution layer can make the output channel number of the second convolution layer 1, that is, the second left feature map with a channel number of 1 is obtained. Figure 1 ×H×W and the second right feature Figure 1 ×H×W. In addition to the 3×3 convolution operation, each two-dimensional convolution in the first convolution layer can also include a batch normalization operation Batch-Norm and an activation function ReLu.

[0069] It should be noted that if the sizes of the first left feature map of the input and the final output feature map are different, the first two-dimensional convolution in the first convolutional layer and an additional two-dimensional convolution with a convolution kernel size of 1×1 can be used to adjust the size, such as by increasing the number of channels or downsampling, to ensure that the sizes of the input feature map and the output feature map are consistent.

[0070] S350. Input the second left feature map and the second right feature map into the weight estimation sub-model, and determine the left weight matrix corresponding to the second left feature map and the right weight matrix corresponding to the second right feature map based on the feature difference between the two feature points corresponding in position in the first left feature map and the first right feature map and the correspondence between the pre-set feature difference and the weight value.

[0071] The weight estimation submodel may refer to a model for estimating the left weight matrix and the right weight matrix. Specifically, Figure 4As shown, the second left feature map and the second right feature map can be input into the weight estimation submodel for weight estimation. The weight estimation submodel can determine the left weight value corresponding to each left feature point in the second left feature map by searching for feature points with corresponding positions on the second right feature map, thereby obtaining a left weight matrix. Similarly, by searching for feature points with corresponding positions on the second left feature map, the right weight value corresponding to each right feature point in the second right feature map can be determined, thereby obtaining a right weight matrix.

[0072] Exemplarily, the weight estimation sub-model may include: a left weight estimation unit and a right weight estimation unit. Figure 5 As shown, S350 may include: inputting the second left feature map and the second right feature map into the left weight estimation unit, searching for a target right feature point corresponding to the current left feature point position on the second right feature map for each left feature point in the second left feature map, and determining a left weight value corresponding to the current left feature point based on a feature difference value between the current left feature point and the target right feature point and a pre-set correspondence between the feature difference value and the weight value. Inputting the second left feature map and the second right feature map into the right weight estimation unit, searching for a target left feature point corresponding to the current right feature point position on the second left feature map for each right feature point in the second right feature map, and determining a right weight value corresponding to the current right feature point based on a feature difference value between the current right feature point and the target left feature point and a pre-set correspondence between the feature difference value and the weight value.

[0073] Specifically, the determination process of the left weight matrix and the right weight matrix is ​​similar. This embodiment is described by taking the determination process of the left weight matrix as an example. In the left weight estimation unit, the left weight value corresponding to each left feature point can be determined one by one for each left feature point in the second left feature map. The left weight estimation unit can determine the left weight value corresponding to each left feature point through the following steps:

[0074] Within the preset search range on the second right feature map, search for each target right feature point corresponding to the current left feature point position; determine the feature difference between the current left feature point and each target right feature point, and take the feature difference with the smallest absolute value as the target feature difference; determine the left weight value corresponding to the current left feature point based on the target feature difference and the correspondence between the preset feature difference and weight value.

[0075] The current left feature point may refer to the currently targeted left feature point. The preset search range may refer to the range in which corresponding right feature points may exist on the second right feature map and have a feature difference of zero with the left feature point x, that is, the receptive field area of ​​the right feature points in this range corresponds to the union G″ of the areas in the right image.

[0076] Specifically, the left and right images in this embodiment differ only in the horizontal direction, so that the search can be performed only in the horizontal direction. For example, each target right feature point that is in the same row as the current left feature point can be searched within the preset search range of the second right feature map. The feature vector of the current left feature point is subtracted from the feature vector of each target right feature point to obtain each feature difference, and the absolute value of each feature difference is taken to obtain the minimum absolute value. If the minimum absolute value, that is, the target feature difference value is equal to zero, it indicates that there is no disparity edge area in the receptive field area of ​​the current left feature point, and the maximum left weight value can be set at this time. If the minimum absolute value, that is, the target feature difference value is greater than zero, it indicates that there may be a disparity edge area in the receptive field area of ​​the current left feature point, and the probability of existence is proportional to the absolute value of the target feature difference value. At this time, the corresponding left weight value can be set based on the correspondence between the feature difference value and the weight value.

[0077] Exemplarily, searching for each target right feature point corresponding to the current left feature point position within a preset search range on the second right feature map may include: searching for each target right feature point on the second right feature map that is in the same row as the current left feature point and whose disparity is less than a preset disparity threshold.

[0078] The preset disparity threshold can be a preset maximum disparity value allowed by two feature points with corresponding positions, that is, the maximum deviation value of the positions of two feature points with corresponding positions. For example, the preset disparity threshold can be set to 192, which can be synchronously downsampled with the feature map size. Specifically, the current left feature point x in the i-th row and j-th column on the second left feature map can be (i,j) , search for each target right feature point x′ whose difference between column number k and j is less than the preset disparity threshold maxdisp on the i-th row of the second right feature map (i,k) , where |kj|<maxdisp. By setting the search range using a preset disparity threshold, search results can be obtained more quickly, improving search efficiency.

[0079] For example, the left weight value corresponding to the current left feature point may be determined based on the following formula:

[0080]

[0081] Among them, r is the target feature difference corresponding to the current left feature point; σ(r) is the left weight value corresponding to the current left feature point. The left weight estimation unit obtains the current left feature point x through the above activation function σ(r). (i,j) The corresponding left weight value, that is, f(x (i,j) )=σ(r).

[0082] Similarly, the right weight estimation unit can determine the right weight value corresponding to each right feature point through the following steps:

[0083] Within the preset search range on the second left feature map, search for each target left feature point corresponding to the current right feature point position; determine the feature difference between the current right feature point and each target left feature point, and take the feature difference with the smallest absolute value as the target feature difference; determine the right weight value corresponding to the current right feature point based on the target feature difference and the correspondence between the preset feature difference and weight value.

[0084] Exemplarily, searching for each target left feature point corresponding to the current right feature point position within a preset search range on the second left feature map may include: searching for each target left feature point on the second left feature map that is in the same row as the current right feature point and whose disparity is less than a preset disparity threshold.

[0085] For example, the right weight value corresponding to the current right feature point may be determined based on the following formula:

[0086]

[0087] Among them, r is the target feature difference corresponding to the current right feature point; σ(r) is the right weight value corresponding to the current right feature point.

[0088] S360: Input the left weight matrix and the first left feature map into the attention processing sub-model for attention processing, determine the third left feature map, and output the third left feature map.

[0089] Specifically, if Figure 4 As shown, the attention processing sub-model can repeat the input left weight matrix 1×H×W C times, that is, superimpose C left weight matrices 1×H×W to obtain a left weight matrix body C×H×W with the same size as the first left feature map C×H×W, and multiply the first left feature map and the left weight matrix body element-wise to obtain the third left feature map C×H×W after attention processing and output it.

[0090] For example, Figure 5 As shown, S360 may include: inputting the left weight matrix and the fourth left feature map into the attention processing sub-model for attention processing, and determining the third left feature map.

[0091] Specifically, the fourth left feature map output by the first convolutional layer can be input into the attention processing sub-model for attention processing, that is, the fourth left feature map for further feature extraction is multiplied pixel by pixel with the left weight matrix to obtain the third left feature map, thereby further improving the accuracy of feature extraction.

[0092] S370. Input the right weight matrix and the first right feature map into the attention processing sub-model for attention processing, determine the third right feature map, and output the third right feature map.

[0093] Specifically, if Figure 4 As shown, the attention processing sub-model can repeat the input right weight matrix 1×H×W C times, that is, superimpose C right weight matrices 1×H×W to obtain a right weight matrix C×H×W of the same size as the first right feature map C×H×W, and multiply the first right feature map and the right weight matrix element-wise to obtain the third right feature map C×H×W after attention processing and output it.

[0094] For example, Figure 5 As shown, S370 may include: inputting the right weight matrix and the fourth right feature map into the attention processing sub-model for attention processing, and determining the third right feature map.

[0095] Specifically, the fourth right feature map output by the first convolutional layer can be input into the attention processing sub-model for attention processing, that is, the fourth right feature map for further feature extraction is multiplied pixel by pixel with the right weight matrix to obtain the third right feature map, thereby further improving the accuracy of feature extraction.

[0096] It should be noted that the execution order of step S370 is not limited here. For example, step S370 can be executed sequentially after step S360, or before step S360, or simultaneously with step S360.

[0097] S380. Obtain left and right feature maps after attention processing based on the output of the preset attention network model.

[0098] The technical solution of this embodiment performs attention processing on the first left feature map and the first right feature map by utilizing a preset attention network model including a feature fusion sub-model, a weight estimation sub-model and an attention processing sub-model. This can reduce the attention to features containing disparity edges and increase the attention to features not containing disparity edges, thereby suppressing the features containing disparity edges in the first left feature map and the first right feature map, avoiding the influence of features containing disparity edges on disparity estimation, and improving the accuracy of disparity estimation.

[0099] Figure 6This is a flowchart of a binocular image processing method provided by an embodiment of the present invention. Based on the above embodiments, this embodiment presets an attention network model that includes not only a feature fusion sub-model, a weight estimation sub-model and an attention processing sub-model, but also a residual construction sub-model. On this basis, the attention processing process is described in detail, and the explanations of the terms that are the same or corresponding to the above embodiments are not repeated here.

[0100] See also Figure 6 The binocular image processing method provided in this embodiment specifically includes the following steps:

[0101] S610: Acquire a binocular image to be processed.

[0102] S620: Perform feature extraction on the left image and the right image in the binocular image respectively to determine a first left feature map corresponding to the left image and a first right feature map corresponding to the right image.

[0103] S630: Input the first left feature map into the feature fusion sub-model, fuse the feature information of each channel in the first left feature map, and determine a fused second left feature map.

[0104] Specifically, Figure 7 An example of a preset attention network model is given. Figure 7 As shown, the first left feature map can be input into the first convolutional layer for convolution operation to determine a fourth left feature map of the same size as the first left feature map; the fourth left feature map can be input into the second convolutional layer for convolution operation to determine the fused second left feature map.

[0105] S640: Input the first right feature map into the feature fusion sub-model, fuse the feature information of each channel in the first right feature map, and determine a fused second right feature map.

[0106] Specifically, if Figure 7 As shown, the first right feature map can be input into the first convolutional layer for convolution operation to determine a fourth right feature map of the same size as the first right feature map; the fourth right feature map is input into the second convolutional layer for convolution operation to determine the fused second right feature map.

[0107] S650. Input the second left feature map and the second right feature map into the weight estimation sub-model, and determine the left weight matrix corresponding to the second left feature map and the right weight matrix corresponding to the second right feature map based on the feature difference between two feature points corresponding in position in the first left feature map and the first right feature map and the correspondence between the pre-set feature difference and the weight value.

[0108] Specifically, if Figure 7As shown, the second left feature map and the second right feature map can be input into the left weight estimation unit to determine the left weight matrix, and the second left feature map and the second right feature map can be input into the right weight estimation unit to determine the right weight matrix.

[0109] S660: Input the left weight matrix and the first left feature map into the attention processing sub-model for attention processing to determine the third left feature map.

[0110] Specifically, if Figure 7 As shown, the left weight matrix and the fourth left feature map can be input into the attention processing sub-model for attention processing to determine the third left feature map.

[0111] S670: Input the first left feature map and the third left feature map into the residual construction sub-model for residual construction, determine the fifth left feature map, and output the fifth left feature map.

[0112] Among them, the residual construction sub-model can be used to construct a model of the residual structure. Specifically, Figure 7 As shown, the residual construction sub-model can add the first left feature map (C×H×W) and the third left feature map (C×H×W) of the same size bitwise, and output the result as the fifth left feature map. By adding a residual construction sub-model, a residual block structure can be constructed, further improving the training effect of the preset attention network model, further suppressing features containing disparity edges, and further improving the accuracy of disparity estimation.

[0113] S680: Input the right weight matrix and the first right feature map into the attention processing sub-model for attention processing to determine the third right feature map.

[0114] Specifically, if Figure 7 As shown, the right weight matrix and the fourth right feature map can be input into the attention processing sub-model for attention processing to determine the third right feature map.

[0115] S690: Input the first right feature map and the third right feature map into the residual construction sub-model to perform residual construction, determine the fifth right feature map, and output the fifth right feature map.

[0116] Specifically, if Figure 7 As shown, the residual construction sub-model can add the first right feature map C×H×W and the third right feature map C×H×W of the same input size bitwise, and output the result as the fifth right feature map. By adding a residual construction sub-model, a residual block structure can be constructed, further improving the training effect of the preset attention network model, further suppressing features containing disparity edges, and further improving the accuracy of disparity estimation.

[0117] S691. According to the output of the preset attention network model, obtain the left and right feature maps after attention processing.

[0118] The technical solution of this embodiment, by adding a residual construction sub-model, can construct a residual block structure, further improve the training effect of the preset attention network model, further suppress features containing disparity edges, and further improve the accuracy of disparity estimation.

[0119] The following is an embodiment of a binocular image processing device provided by an embodiment of the present invention. This device and the binocular image processing methods of the above embodiments are based on the same inventive concept. For details not fully described in the embodiment of the binocular image processing device, reference can be made to the above embodiments of the binocular image processing method.

[0120] Figure 8 This is a schematic diagram of the structure of a binocular image processing device provided by an embodiment of the present invention. This embodiment is applicable to the case where feature attention processing is performed on binocular images, such as Figure 8 As shown, the device includes: a binocular image acquisition module 810, a feature extraction module 820, a feature map input module 830 and a feature map output module 840.

[0121] Among them, the binocular image acquisition module 810 is used to acquire the binocular image to be processed; the feature extraction module 820 is used to perform feature extraction on the left image and the right image in the binocular image respectively, and determine the first left feature map corresponding to the left image and the first right feature map corresponding to the right image; the feature map input module 830 is used to input the first left feature map and the first right feature map into the preset attention network model for feature attention processing, and the preset attention network model is used to: based on the feature difference between the two feature points corresponding to the positions in the first left feature map and the first right feature map and the correspondence between the preset feature difference and the weight value, determine the left weight matrix corresponding to the first left feature map and the right weight matrix corresponding to the first right feature map, and perform attention processing on the first left feature map and the first right feature map respectively based on the left weight matrix and the right weight matrix; the feature map output module 840 is used to obtain the left and right feature maps after attention processing according to the output of the preset attention network model.

[0122] Optionally, the preset attention network model includes: a feature fusion sub-model, a weight estimation sub-model and an attention processing sub-model;

[0123] The feature map input module 830 includes:

[0124] A first input unit is used to input the first left feature map into the feature fusion sub-model, fuse the feature information of each channel in the first left feature map, and determine a fused second left feature map;

[0125] The second input unit is used to input the first right feature map into the feature fusion sub-model, fuse the feature information of each channel in the first right feature map, and determine a fused second right feature map;

[0126] a third input unit, configured to input the second left feature map and the second right feature map into the weight estimation submodel, and determine a left weight matrix corresponding to the second left feature map and a right weight matrix corresponding to the second right feature map based on a feature difference between two feature points at corresponding positions in the first left feature map and the first right feature map and a preset correspondence between the feature difference and the weight value;

[0127] a fourth input unit, configured to input the left weight matrix and the first left feature map into the attention processing sub-model for attention processing, determine the third left feature map, and output the third left feature map;

[0128] The fifth input unit is used to input the right weight matrix and the first right feature map into the attention processing sub-model for attention processing, determine the third right feature map, and output the third right feature map.

[0129] Optionally, the feature fusion sub-model includes: a first convolutional layer and a second convolutional layer;

[0130] The first input unit is specifically used to: input the first left feature map into the first convolutional layer for convolution operation to determine a fourth left feature map of the same size as the first left feature map; input the fourth left feature map into the second convolutional layer for convolution operation to determine the fused second left feature map.

[0131] Optionally, the fourth input unit is specifically configured to: input the left weight matrix and the fourth left feature map into the attention processing sub-model for attention processing, and determine the third left feature map;

[0132] The fifth input unit is specifically used to: input the right weight matrix and the fourth right feature map into the attention processing sub-model for attention processing, and determine the third right feature map.

[0133] Optionally, the weight estimation sub-model includes: a left weight estimation unit and a right weight estimation unit;

[0134] The third input unit is specifically used for:

[0135] Inputting the second left feature map and the second right feature map into the left weight estimation unit, searching for a target right feature point corresponding to the position of the current left feature point on the second right feature map for each left feature point in the second left feature map, and determining the left weight value corresponding to the current left feature point based on the feature difference between the current left feature point and the target right feature point and the pre-set correspondence between the feature difference and the weight value;

[0136] The second left feature map and the second right feature map are input into the right weight estimation unit. For each right feature point in the second right feature map, a target left feature point corresponding to the position of the current right feature point is searched on the second left feature map, and the right weight value corresponding to the current right feature point is determined based on the feature difference between the current right feature point and the target left feature point and the correspondence between the pre-set feature difference and the weight value.

[0137] Optionally, the left weight estimation unit searches for a target right feature point corresponding to the position of the current left feature point on the second right feature map, and determines a left weight value corresponding to the current left feature point based on a feature difference between the current left feature point and the target right feature point and a preset correspondence between the feature difference and the weight value, including:

[0138] Within the preset search range on the second right feature map, search for each target right feature point corresponding to the current left feature point position; determine the feature difference between the current left feature point and each target right feature point, and take the feature difference with the smallest absolute value as the target feature difference; determine the left weight value corresponding to the current left feature point based on the target feature difference and the correspondence between the preset feature difference and weight value.

[0139] Optionally, searching for each target right feature point corresponding to the current left feature point position within a preset search range on the second right feature map includes:

[0140] Search the second right feature map for target right feature points that are in the same row as the current left feature point and whose disparity is less than a preset disparity threshold.

[0141] Optionally, the left weight value corresponding to the current left feature point is determined based on the following formula:

[0142]

[0143] Among them, r is the target feature difference corresponding to the current left feature point; σ(r) is the left weight value corresponding to the current left feature point.

[0144] Optionally, the preset attention network model further includes: a residual construction sub-model;

[0145] The fourth input unit is further used to: input the first left feature map and the third left feature map into the residual construction submodel to perform residual construction, determine the fifth left feature map, and output the fifth left feature map;

[0146] The fifth input unit is further used to: input the first right feature map and the third right feature map into the residual construction submodel for residual construction, determine the fifth right feature map, and output the fifth right feature map.

[0147] Optionally, the device further comprises:

[0148] The disparity estimation module is used to obtain the left and right feature maps after attention processing according to the output of the preset attention network model, and then input the left feature map and the right feature map output by the preset attention network model into the disparity estimation network model for disparity estimation; based on the output of the disparity estimation network model, obtain the target disparity map corresponding to the binocular image.

[0149] The binocular image processing device provided in the embodiment of the present invention can execute the binocular image processing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the binocular image processing method.

[0150] It is worth noting that in the embodiment of the above-mentioned binocular image processing device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other and are not used to limit the scope of protection of the present invention.

[0151] Figure 9 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. Figure 9 A block diagram of an exemplary electronic device 12 suitable for implementing embodiments of the present invention is shown. Figure 9 The electronic device 12 shown is only an example and should not limit the functionality and scope of use of the embodiments of the present invention.

[0152] like Figure 9 As shown, electronic device 12 is implemented as a general-purpose computing device. Components of electronic device 12 may include, but are not limited to, one or more processors or processing units 16, system memory 28, and a bus 18 that connects various system components (including system memory 28 and processing unit 16).

[0153] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures. Examples of these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0154] The electronic device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 12, including volatile and non-volatile media, removable and non-removable media.

[0155] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The electronic device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be configured to read and write non-removable, non-volatile magnetic media ( Figure 9 Not shown, often called a "hard drive"). Although Figure 9 Not shown, a magnetic disk drive for reading and writing to a removable non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. System memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of various embodiments of the present invention.

[0156] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may include an implementation of a network environment. Program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0157] The electronic device 12 can also communicate with one or more external devices 14 (e.g., a keyboard, pointing device, display 24, etc.), one or more devices that enable a user to interact with the electronic device 12, and / or any device that enables the electronic device 12 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication can occur via an input / output (I / O) interface 22. Furthermore, the electronic device 12 can communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with other modules of the electronic device 12 via a bus 18. It should be understood that, although not shown, other hardware and / or software modules can be used in conjunction with the electronic device 12, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0158] The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, such as implementing a binocular image processing method provided in an embodiment of the present invention, which includes:

[0159] Obtaining binocular images to be processed;

[0160] Perform feature extraction on the left image and the right image in the binocular image respectively to determine a first left feature map corresponding to the left image and a first right feature map corresponding to the right image;

[0161] Inputting the first left feature map and the first right feature map into a preset attention network model for feature attention processing, the preset attention network model is used to: determine a left weight matrix corresponding to the first left feature map and a right weight matrix corresponding to the first right feature map based on a feature difference between two feature points corresponding to positions in the first left feature map and the first right feature map and a preset correspondence between the feature difference and the weight value, and perform attention processing on the first left feature map and the first right feature map based on the left weight matrix and the right weight matrix respectively;

[0162] According to the output of the preset attention network model, the left and right feature maps after attention processing are obtained.

[0163] Of course, those skilled in the art will appreciate that the processor may also implement the technical solution of the binocular image processing method provided by any embodiment of the present invention.

[0164] This embodiment provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the steps of the binocular image processing method provided in any embodiment of the present invention are implemented. The method includes:

[0165] Obtaining binocular images to be processed;

[0166] Perform feature extraction on the left image and the right image in the binocular image respectively to determine a first left feature map corresponding to the left image and a first right feature map corresponding to the right image;

[0167] Inputting the first left feature map and the first right feature map into a preset attention network model for feature attention processing, the preset attention network model is used to: determine a left weight matrix corresponding to the first left feature map and a right weight matrix corresponding to the first right feature map based on a feature difference between two feature points corresponding to positions in the first left feature map and the first right feature map and a preset correspondence between the feature difference and the weight value, and perform attention processing on the first left feature map and the first right feature map based on the left weight matrix and the right weight matrix respectively;

[0168] According to the output of the preset attention network model, the left and right feature maps after attention processing are obtained.

[0169] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device.

[0170] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0171] Program code embodied on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0172] Computer program code for performing the operations of the present invention can be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0173] Those skilled in the art will appreciate that the modules or steps of the present invention described above can be implemented using a general-purpose computing device. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Alternatively, they can be implemented using program code executable by a computer device, which can then be stored in a storage device and executed by the computing device. Alternatively, they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module. Thus, the present invention is not limited to any specific combination of hardware and software.

[0174] Note that the above are only preferred embodiments of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments, and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments and may include many other equivalent embodiments without departing from the concept of the present invention. The scope of the present invention is determined by the scope of the appended claims.

Claims

1. A binocular image processing method, characterized in that: include: Obtaining binocular images to be processed; Performing feature extraction on a left image and a right image in the binocular image respectively to determine a first left feature map corresponding to the left image and a first right feature map corresponding to the right image; Inputting the first left feature map and the first right feature map into a preset attention network model for feature attention processing, the preset attention network model being configured to: determine a left weight matrix corresponding to the first left feature map and a right weight matrix corresponding to the first right feature map based on a feature difference between two feature points at corresponding positions in the first left feature map and the first right feature map, and a preset correspondence between the feature difference and the weight value, and perform attention processing on the first left feature map and the first right feature map based on the left weight matrix and the right weight matrix, respectively; According to the output of the preset attention network model, the left and right feature maps after attention processing are obtained.

2. The method according to claim 1, characterized in that The preset attention network model includes: a feature fusion sub-model, a weight estimation sub-model and an attention processing sub-model; Inputting the first left feature map and the first right feature map into a preset attention network model for feature attention processing includes: Inputting the first left feature map into the feature fusion sub-model, fusing feature information of each channel in the first left feature map, and determining a fused second left feature map; Inputting the first right feature map into the feature fusion sub-model, fusing feature information of each channel in the first right feature map, and determining a fused second right feature map; Inputting the second left feature map and the second right feature map into the weight estimation submodel, and determining a left weight matrix corresponding to the second left feature map and a right weight matrix corresponding to the second right feature map based on a feature difference between two feature points at corresponding positions in the first left feature map and the first right feature map and a preset correspondence between the feature difference and the weight value; Inputting the left weight matrix and the first left feature map into the attention processing sub-model for attention processing, determining a third left feature map, and outputting the third left feature map; The right weight matrix and the first right feature map are input into the attention processing sub-model for attention processing, a third right feature map is determined, and the third right feature map is output.

3. The method according to claim 2, characterized in that The feature fusion sub-model includes: a first convolutional layer and a second convolutional layer; Inputting the first left feature map into the feature fusion sub-model, fusing feature information of each channel in the first left feature map, and determining a fused second left feature map, includes: Inputting the first left feature map into the first convolutional layer for performing a convolution operation, and determining a fourth left feature map having the same size as the first left feature map; The fourth left feature map is input into the second convolutional layer for convolution operation to determine the fused second left feature map.

4. The method according to claim 3, characterized in that Inputting the left weight matrix and the first left feature map into the attention processing sub-model for attention processing to determine a third left feature map includes: Inputting the left weight matrix and the fourth left feature map into the attention processing sub-model for attention processing to determine a third left feature map; Inputting the right weight matrix and the first right feature map into the attention processing sub-model for attention processing to determine a third right feature map includes: The right weight matrix and the fourth right feature map are input into the attention processing sub-model for attention processing to determine a third right feature map, wherein the fourth right feature map is a feature map having the same size as the first right feature map determined by inputting the first right feature map into the first convolutional layer for a convolution operation.

5. The method according to claim 2, characterized in that The weight estimation sub-model includes: a left weight estimation unit and a right weight estimation unit; Inputting the second left feature map and the second right feature map into the weight estimation submodel, and determining a left weight matrix corresponding to the second left feature map and a right weight matrix corresponding to the second right feature map based on a feature difference between two feature points at corresponding positions in the first left feature map and the first right feature map and a preset correspondence between the feature difference and the weight value, includes: Inputting the second left feature map and the second right feature map into the left weight estimation unit, searching for a target right feature point corresponding to the position of the current left feature point on the second right feature map for each left feature point in the second left feature map, and determining a left weight value corresponding to the current left feature point based on a feature difference between the current left feature point and the target right feature point and a preset correspondence between the feature difference and the weight value; The second left feature map and the second right feature map are input into the right weight estimation unit. For each right feature point in the second right feature map, a target left feature point corresponding to the position of the current right feature point is searched on the second left feature map, and based on the feature difference between the current right feature point and the target left feature point and the correspondence between the pre-set feature difference value and the weight value, the right weight value corresponding to the current right feature point is determined.

6. The method according to claim 5, characterized in that Searching for a target right feature point corresponding to the position of the current left feature point on the second right feature map, and determining a left weight value corresponding to the current left feature point based on a feature difference between the current left feature point and the target right feature point and a preset correspondence between the feature difference and the weight value, including: Searching for each target right feature point corresponding to the current left feature point position within a preset search range on the second right feature map; Determine a feature difference between the current left feature point and each of the target right feature points, and use the feature difference with the smallest absolute value as the target feature difference; The left weight value corresponding to the current left feature point is determined based on the target feature difference value and the correspondence between the preset feature difference value and the weight value.

7. The method according to claim 6, characterized in that Searching for each target right feature point corresponding to the current left feature point position within a preset search range on the second right feature map includes: Search the second right feature map for target right feature points that are in the same row as the current left feature point and whose disparity is less than a preset disparity threshold.

8. The method according to claim 6, characterized in that Determine the left weight value corresponding to the current left feature point based on the following formula: Among them, r is the target feature difference corresponding to the current left feature point; σ(r) is the left weight value corresponding to the current left feature point.

9. The method according to claim 2, characterized in that The preset attention network model further includes: a residual construction sub-model; Outputting the third left feature map includes: Inputting the first left feature map and the third left feature map into the residual construction submodel to perform residual construction, determining a fifth left feature map, and outputting the fifth left feature map; Outputting the third right feature map includes: The first right feature map and the third right feature map are input into the residual construction submodel for residual construction, a fifth right feature map is determined, and the fifth right feature map is output.

10. The method according to any one of claims 1 to 9, characterized in that: After obtaining the left and right feature maps after attention processing according to the output of the preset attention network model, the method further includes: Input the left feature map and the right feature map output by the preset attention network model into the disparity estimation network model for disparity estimation; Based on the output of the disparity estimation network model, a target disparity map corresponding to the binocular image is obtained.

11. A binocular image processing device, characterized in that: include: A binocular image acquisition module is used to acquire the binocular image to be processed; a feature extraction module, configured to perform feature extraction on the left image and the right image in the binocular image respectively, and determine a first left feature map corresponding to the left image and a first right feature map corresponding to the right image; a feature map input module, configured to input the first left feature map and the first right feature map into a preset attention network model for feature attention processing, wherein the preset attention network model is configured to: determine a left weight matrix corresponding to the first left feature map and a right weight matrix corresponding to the first right feature map based on a feature difference between two feature points at corresponding positions in the first left feature map and the first right feature map, and to perform attention processing on the first left feature map and the first right feature map based on the left weight matrix and the right weight matrix; The feature map output module is used to obtain the left and right feature maps after attention processing according to the output of the preset attention network model.

12. An electronic device, characterized in that: The electronic device comprises: one or more processors; a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the binocular image processing method according to any one of claims 1 to 10.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the binocular image processing method according to any one of claims 1 to 10 is implemented.

Images