Binocular stereo matching method and system based on multi-scale iterative optimization and related equipment

Abstract

The invention relates to the field of binocular stereo vision, and discloses a binocular stereo matching method and system based on multi-scale iterative optimization and related equipment. The method comprises the steps of performing semantic structure feature extraction on a corrected left view and a corrected right view through a feature extraction network; determining a cost space pyramid and a probability matrix according to the left view feature map and the right view feature map output by the feature extraction network; extracting a multi-scale context feature of the left view through a context sensing network, and initializing the multi-scale context feature to obtain a hidden state and input of a recursive network; during recursive network iteration, according to the probability matrix, a multi-range search strategy is adopted to index local cost from the cost space pyramid; and inputting the local cost into the recursive network, combining the hidden state obtained by the context features and the input, and carrying out iterative updating on the parallax value to obtain a parallax map, thereby improving the accuracy of parallax calculation.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of binocular stereo vision, in particular to a binocular stereo matching method based on multi-scale iterative optimization, a system and related equipment. BACKGROUND

[0002] Binocular stereo vision is a classic research topic in the field of computer vision. It captures left and right views of an object from different angles by two cameras, and finally obtains the depth information of the object to be measured according to the principle of triangulation. This technology is widely used in automatic driving, unmanned vehicle navigation and three-dimensional measurement.

[0003] The implementation process of binocular stereo vision includes the following steps: camera calibration, stereo rectification, stereo matching and depth calculation. Among them, stereo matching is the most core and difficult step in the whole process. It aims to analyze the rectified image pair and calculate the displacement between the corresponding pixel points in the two images, i.e. "disparity", so as to estimate the depth of each pixel point. Stereo matching algorithms can be roughly divided into traditional methods and deep learning-based methods. The traditional method is divided into four steps: matching cost calculation, cost aggregation, disparity calculation and disparity optimization. The traditional method mainly relies on manually designed features for matching, which limits its generalization ability in complex real scenes. Especially in pathological areas such as occlusion, textureless and repetitive texture, the error rate of traditional method matching is high. In recent years, stereo matching methods based on deep learning have shown great potential. Some deep learning-based methods use CNN to extract features to construct 3D / 4D cost volume, and then use 2D / 3D convolution to filter the cost volume to aggregate context information, and finally use regression method to directly predict the disparity value of each pixel. Compared with traditional methods, deep learning-based methods significantly improve the accuracy of disparity calculation.

[0004] However, such methods still encounter difficulties in dealing with pathological areas with insufficient matching clues in the real world, especially in occlusion and textureless areas. In the above pathological areas, the disparity map calculated by them often has problems such as loss of details and blurring of object edges. SUMMARY

[0005] The technical problem to be solved by the present application is to provide a binocular stereo matching method based on multi-scale iterative optimization, a system and related equipment to solve the problems in the prior art.

[0006] In a first aspect, the present application provides a binocular stereo matching method based on multi-scale iterative optimization, comprising: extracting semantic structural features from the rectified left view and right view through a feature extraction network; The cost space pyramid and probability matrix are determined based on the left and right view feature maps output by the feature extraction network. Multi-scale contextual features of the left view are extracted through a context-aware network. After initialization of the multi-scale contextual features, the hidden state and input of the recursive network are obtained. The recursive network is composed of multi-scale attention convolutional gated recurrent units as iterators. During recursive network iteration, a multi-range search strategy is used based on the probability matrix to index local costs from the cost space pyramid. The local cost is input into the recurrent network, and the disparity map is obtained by iteratively updating the disparity value by combining the hidden state obtained from the context features with the input.

[0007] Secondly, this application provides a binocular stereo matching system based on multi-scale iterative optimization, comprising: The feature extraction module is used to extract semantic structure features from the corrected left and right views through a feature extraction network; The cost space determination module is used to determine the cost space pyramid and probability matrix based on the left-view feature map and the right-view feature map output by the feature extraction network. The context feature extraction module is used to extract multi-scale context features of the left view through a context-aware network. After the multi-scale context features are initialized, the hidden state and input of the recursive network are obtained. The recursive network is composed of multi-scale attention convolutional gated recurrent units as iterators. The local cost determination module is used to index local costs from the cost space pyramid based on the probability matrix during recursive network iteration using a multi-range search strategy. The iterative update module is used to input the local cost into the recurrent network, combine the hidden state obtained from the context features with the input, and iteratively update the disparity value to obtain the disparity map.

[0008] Thirdly, this application provides a computer-readable storage medium including instructions that, when executed on a computer, cause the computer to perform the binocular stereo matching method based on multi-scale iterative optimization as provided in the above technical solution.

[0009] Fourthly, this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the binocular stereo matching method based on multi-scale iterative optimization provided by the above technical solution.

[0010] Fifthly, this application also provides a computer program product comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the multi-scale iterative optimization-based binocular stereo matching method provided in the above-described scheme.

[0011] The beneficial effects of this invention are as follows: an iterator is constructed using a multi-scale attention convolution gated recurrent unit (MAC-GRU) to iteratively update the disparity value. The MAC-GRU can effectively capture the edge and detail texture of the foreground object in the occluded area while also taking into account the low-frequency information in the textureless area, thereby improving the accuracy of disparity calculation in occluded and textureless areas. A multi-range search strategy is adopted during the iteration process, which can accelerate the convergence speed of disparity in the early stage of iteration and improve the accuracy of disparity update in the later stage of iteration.

[0012] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0013] Figure 1 This is a flowchart illustrating a binocular stereo matching method based on multi-scale iterative optimization, as shown in an exemplary embodiment of this application. Figure 2 This is a structural diagram of a MAC-GRU provided in an exemplary embodiment of this application; Figure 3 This is a structural diagram of an iterator provided in an exemplary embodiment of this application; Figure 4 This is a flowchart illustrating a binocular stereo matching method based on multi-scale iterative optimization, as shown in another exemplary embodiment of this application; Figure 5 This is a block diagram of a binocular stereo matching system based on multi-scale iterative optimization, as illustrated in an exemplary embodiment of this application. Figure 6 This is a schematic diagram of a binocular stereo matching system model based on multi-scale iterative optimization, as illustrated in an exemplary embodiment of the application. Detailed Implementation

[0014] The following specific examples illustrate the implementation of this disclosure. Those skilled in the art can easily understand other advantages and effects of this disclosure from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. This disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this disclosure. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0015] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.

[0016] The solutions provided in this invention can be executed by any electronic device, such as a terminal device, or by a terminal device and a server. The server can be a standalone server, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. No restrictions are imposed here.

[0017] Figure 1 This is a flowchart illustrating a binocular stereo matching method based on multi-scale iterative optimization, as shown in an exemplary embodiment of this application. Figure 1 As shown in an exemplary embodiment, the binocular stereo matching method based on multi-scale iterative optimization may include steps S1 to S5, which are described in detail below: S1, semantic structure features are extracted from the corrected left and right views using a feature extraction network; S2, determine the cost space pyramid and probability matrix based on the left and right view feature maps output by the feature extraction network; S3, extract multi-scale context features of the left view through a context-aware network, and obtain the hidden state and input of the recursive network after initialization of the multi-scale context features; wherein, the recursive network is based on the multi-scale attention convolution gated recurrent unit MAC-GRU (Multi-Scale Attention ConvGRU as iterator). S4, during recursive network iteration, a multi-range search strategy is used to index local costs from the cost space pyramid based on the probability matrix; S5 inputs the local cost into the recurrent network, and combines the hidden state obtained from the context features with the input to iteratively update the disparity value to obtain the disparity map.

[0018] In this embodiment of the invention, a MAC-GRU is used to construct an iterator to iteratively update the disparity value. The MAC-GRU can effectively capture the edge and detail texture of the foreground object in the occluded area while also taking into account the low-frequency information in the textureless area, thereby improving the accuracy of disparity calculation in the occluded and textureless areas. During the iteration process, a multi-range search strategy is adopted, which can accelerate the convergence speed of disparity in the early stage of iteration and improve the accuracy of disparity update in the later stage of iteration.

[0019] Optionally, the feature extraction network in this embodiment of the invention may consist of the first two stages of ConvNeXt (an improved version of convolutional neural network that retains the advantages of traditional CNNs while drawing on the design ideas of Transformer to improve the performance of image recognition and understanding) and a feature fusion module, which are pre-trained on ImageNet (a very large dataset for object recognition). Figure 1 In an exemplary embodiment, step S1 in the illustrated embodiment may specifically include steps S11 to S13, through which feature maps of the left view and the right view are obtained, as detailed below: S11, input the left view into the feature extraction network, and after the first two stages of ConvNeXt pre-trained on ImageNet, obtain feature maps of two different resolutions; S12, use transposed convolution to upsample the feature map with a larger resolution so that it is the same size as the feature map with a smaller resolution. S13, the feature fusion module fuses two feature maps of the same size to obtain the left view feature map.

[0020] Similarly, the right view is processed in the same way as the left view to obtain the feature map of the right view.

[0021] In this embodiment of the invention, after the original view is input into the feature extraction network, and then processed through the first two stages of ConvNeXt pre-trained on ImageNet, a channel count of 128 and a resolution of the original can be obtained.Figure 1 Feature maps of two sizes, 1 / 4 and 1 / 8, are then used. The 1 / 8 resolution feature map is then upsampled using a transposed convolution to make it the same size as the 1 / 4 resolution feature map. The feature fusion module then merges the two feature maps to obtain a fused feature map with 256 channels.

[0022] This invention upsamples features at 1 / 8 resolution and fuses them with features at 1 / 4 resolution. This restores higher spatial resolution while preserving semantic information, combining high-resolution detail with global semantic information to enhance the representational power of the feature map. Furthermore, the channel dimension expands to 256 after feature fusion, which helps the model integrate multi-level feature information and improves its adaptability to complex scenes and texture variations.

[0023] Optionally, Figure 1 In an exemplary embodiment, step S2 in the illustrated example may specifically include steps S21 to S23, through which the cost space pyramid and probability matrix are obtained, as detailed below: S21. Determine the initial cost space matrix based on the inner product of the eigenvectors of the left view feature map and the right view feature map that are located on the same epipolar line. S22, perform one-dimensional average pooling on the disparity dimension of the initial cost space matrix to obtain a two-layer cost space pyramid. S23, normalize the initial cost space matrix of the cost space pyramid along the disparity dimension to obtain the probability matrix.

[0024] In this embodiment of the invention, the left view feature map and the right view feature map output by the feature extraction network are respectively (H, W, and C represent the height, width, and number of channels of the feature map, respectively). The maximum disparity is set to... Calculate the inner product between eigenvectors located on the same epipolar line in the two feature maps to obtain the initial cost space matrix. (d represents disparity, and x and y represent the x and y coordinates of the feature vector's position in the feature map), where Disparity represents the dot product of vectors. Initial cost space matrix Then, one-dimensional average pooling (kernel size 2×1, stride 2) is performed on the disparity dimension of the initial cost space matrix to obtain a two-layer cost space pyramid. The initial cost space matrix A probability matrix is ​​obtained by performing Softmax (number normalization) along the disparity dimension. Each position in the probability matrix represents the "confidence" that the pixel is correctly matched under the current disparity value.

[0025] Figure 1In an exemplary embodiment, step S3 of the illustrated example involves extracting multi-scale context features from the left view. After initializing the context features, the hidden states and inputs of the recurrent network are obtained. Specifically, the left view is input into the context-aware network to obtain multi-scale context features with resolutions of 1 / 4, 1 / 8, and 1 / 16 of the original image. These multi-scale context features are used to initialize the hidden states of the recurrent network and serve as the input to the ConvGRU at the current time step during each iteration. Given three scales of context features... The initial hidden state is obtained as follows: The input to ConvGRU in each iteration is .

[0026] Optionally, Figure 1 In an exemplary embodiment, step S4 in the illustrated example may specifically include steps S41 to S42, through which a local cost is obtained, as detailed below: S41, find the disparity value with the highest confidence for each pixel in the probability matrix, and use this disparity value as the initial disparity value at the start of the iteration; S42, based on the initial disparity value, a multi-range search strategy is used to search in the cost space pyramid, and the local cost is obtained by indexing.

[0027] In this embodiment of the invention, the disparity value with the highest "confidence" corresponding to each pixel is found in the probability matrix, and this disparity value is used as the initial disparity value at the start of the iteration. Based on the initial disparity values, the search operator in RAFT-Stereo is used in the cost space pyramid. The search is performed within the cost space pyramid. Local cost is obtained from the index. .

[0028] In this embodiment of the invention, the disparity value with the highest "confidence" is selected using a probability matrix instead of random initialization. This provides a starting point closer to the true solution for subsequent iterations, reducing the risk of getting trapped in erroneous local optima. Simultaneously, searching within a local region centered on the initial disparity (rather than searching the entire range) significantly reduces the range of disparities that need to be considered in each iteration, decreasing computational load and improving search speed.

[0029] In this embodiment of the invention, when indexing local cost, three different search operators with search radii are used to search in the cost space pyramid based on the initial disparity value; the search operator with the largest search radius is used in the early stage of the iteration; the search operator with the medium search radius is used in the middle stage of the iteration; and the search operator with the smallest search radius is used in the later stage of the iteration.

[0030] Local cost of finding operator index During the process, a multi-range search strategy is adopted. Three different search radii are used throughout the entire disparity iterative update process (…). In the early stages of the iteration, the search operator with the largest search radius is used. It can accelerate the convergence of disparity in the early stages of iteration and avoid the disparity getting stuck near a local optimum peak point due to a small search radius in the early stages of iteration. In the middle stages of iteration, a search operator with a medium search radius is used. This further converges the disparity. In the later stages of the iteration, the disparity values ​​of most points are near the optimal value. To eliminate the influence of interference terms outside the maximum disparity range, the search operator with the smallest search radius is used. This allows for further improvement in the accuracy of disparity updates in later iterations. The search radii for the three search operators are as follows: Where iter represents the iteration number. The search operator is used in the first 8 iterations. The search operator is used in iterations 9 through 16. The search operator is used starting from the 17th iteration. Meanwhile, different search operators correspond to their own specific iterative updaters during the iteration process.

[0031] In the iterative process of this invention, a multi-range search strategy is adopted, dividing the disparity update into three stages: for iterations 1 to 8, the search radius of the search operator is set to 8; for iterations 9 to 16, the search radius is set to 4; and for iterations 17 to 32, the search radius is set to 2. The multi-range search strategy can accelerate the convergence speed of disparity in the early stages of iteration and improve the accuracy of disparity updates in the later stages of iteration.

[0032] Optionally, Figure 1 In an exemplary embodiment, step S5 in the illustrated embodiment may specifically include steps S51 to S53, through which a disparity map is obtained, as detailed below: S51, the context features at different scales output by the context-aware network are input into different iterators of the recurrent network; wherein, the hidden states of each other are passed between iterators of adjacent resolutions using upsampling and downsampling. S52, the local cost input indexed from the cost space pyramid is fed into the minimum resolution iterator, and its hidden state is combined to generate the disparity increment; S53, add the increment of the disparity value to the current disparity value to obtain the updated disparity value.

[0033] The search operator obtains the local cost from the index. The input is fed into the iterator. The iterator consists of a ConvGRU (MAC-GRU) based on multi-scale attention. For example... Figure 2 As shown, MAC-GRU uses two ConvGRU kernels with kernel sizes of 1×1 and 3×3. The hidden states of the two ConvGRU outputs are concatenated along the channel dimension. and After passing through a convolutional layer, learnable pixel-level weight coefficients are generated. Furthermore, the range of the weight coefficients is restricted to [0,1] by using a sigmoid activation layer. Weighting coefficients The resolution is the same as the hidden state of ConvGRU, which determines the hidden state of the two ConvGRU outputs in the final hidden state. The proportion of [something] in the final output of MAC-GRU. It can be represented as: A 1×1 ConvGRU kernel can better capture the edge and detail texture of foreground objects in occluded areas, while a 3×3 ConvGRU kernel can also capture low-frequency information in textureless areas. (Weight coefficients) A larger value is needed in smooth, textureless areas to increase... The output scale. And at the edges of the foreground object in the occluded area, The value will decrease, and vice versa. It will increase. The output ratio will increase. The proportion of output will decrease. In this way, MAC-GRU can dynamically fuse information from different frequencies and receptive fields.

[0034] like Figure 3 As shown, the iterator contains three MAC-GRUs. Feature maps at resolutions of 1 / 4, 1 / 8, and 1 / 16 from the context-aware network are input into the three MAC-GRUs respectively. Adjacent MAC-GRUs use upsampling and downsampling to pass their hidden states to each other. From the cost space pyramid... Local cost indexed in The input is fed into a 1 / 4 resolution MAC-GRU, and the disparity increment is generated by combining it with the hidden state. ,Will Adding this value to the current disparity value yields the updated disparity value. The disparity value with the highest "confidence" in the probability matrix is ​​used as the initial disparity value at the start of the iteration. Parallax from Start iterative update, then the first The disparity value obtained from the next iteration update is: .

[0035] Optionally, in an exemplary embodiment, the binocular stereo matching method based on multi-scale iterative optimization provided in this invention may further include the step of performing disparity optimization on the disparity map using a disparity refinement method to obtain an optimized disparity map. Specifically, as Figure 4 As shown, S1, semantic structure features are extracted from the corrected left and right views using a feature extraction network.

[0036] S2, determine the cost space pyramid based on the left and right view feature maps output by the feature extraction network; S3, extract multi-scale contextual features of the left view through a context-aware network, and obtain the hidden state and input of the recursive network after initialization of the multi-scale contextual features; wherein, the recursive network is constructed by using MAC-GRU based on multi-scale attention as an iterator; S4, during recursive network iteration, a multi-range search strategy is used to index local costs from the cost space pyramid; S5, input the local cost into the recurrent network, combine the hidden state obtained from the context features with the input, and iteratively update the disparity value to obtain the disparity map; S6. The disparity map is optimized by using the disparity refinement method to obtain the optimized disparity map.

[0037] Optionally, in an exemplary embodiment, the disparity map is optimized using a disparity refinement method to obtain the optimized disparity map, as detailed below: S61, Move the points in the original right view according to the disparity value of each point in the disparity map to obtain the reconstructed left view; S62, Subtract the reconstructed left view from the original left view to obtain the reconstruction error map of the left view; S63, the reconstructed error map, the original left view and the disparity map are stitched together along the channel dimension and then input into the U-Net network; S64, the U-Net network outputs the residual value of disparity. The original disparity value of the disparity map is added to the residual value to obtain the optimized disparity map.

[0038] In this embodiment of the invention, the disparity map output by the recursive network is obtained. Then, based on the disparity map By using the disparity value of each point in the original right view to move the points, a left view can be reconstructed. Then reconstruct the left view. With the actual left view By subtracting the values, we obtain the reconstruction error map of the left view. ,Right now Then the error plot will be... Original left view The disparity map is concatenated along the channel dimension and then input into the U-Net network. The U-Net network outputs the residual values ​​of the disparity. The original disparity value With residual The summation yields the optimized disparity map. This invention employs a disparity refinement method to further optimize the disparity map output by the recursive network, correcting some points with incorrect disparity calculations, thereby improving the accuracy of the optimized disparity map.

[0039] In this embodiment of the invention, a MAC-GRU is used to construct an iterator to iteratively update the disparity values. MAC-GRU can effectively capture the edges and detailed textures of foreground objects in occluded areas while also taking into account low-frequency information in textureless areas. This improves the accuracy of disparity calculation in occluded and textureless areas. A multi-range search strategy is employed during the iteration process, dividing the disparity update into three stages: for iterations 1-8, the search radius of the search operator is set to 8; for iterations 9-16, the search radius is set to 4; and for iterations 17-32, the search radius is set to 2. The multi-range search strategy accelerates the convergence speed of disparity in the early stages of iteration and improves the accuracy of disparity updates in the later stages. A disparity refinement method is used to further optimize the disparity map output by the recursive network, correcting some points with incorrect disparity calculations, resulting in a more accurate optimized disparity map.

[0040] Figure 5 This is a block diagram of a binocular stereo matching system based on multi-scale iterative optimization, as illustrated in an exemplary embodiment of this application. Figure 5 As shown in an exemplary embodiment, the binocular stereo matching system based on multi-scale iterative optimization may include a feature extraction module, a cost space determination module, a context feature extraction module, a local cost determination module, and an iterative update module, which are described in detail below: The feature extraction module extracts semantic structure features from the corrected left and right views using a feature extraction network. The cost space determination module determines the cost space pyramid and probability matrix based on the feature maps of the left and right views output by the feature extraction network. The context feature extraction module extracts multi-scale context features of the left view using a context-aware network. After initializing the multi-scale context features, the hidden state and input of the recursive network are obtained. The recursive network is constructed using a MAC-GRU multi-scale attention convolutional gated recurrent unit as an iterator. The local cost determination module indexes the local cost from the cost space pyramid using a multi-range search strategy based on the probability matrix during recursive network iteration. The iterative update module inputs the local cost into the recursive network and iteratively updates the disparity value by combining the hidden state and input obtained from the context features to obtain the disparity map.

[0041] Figure 6 This is a schematic diagram of a binocular stereo matching system model based on multi-scale iterative optimization, as illustrated in an exemplary embodiment of the application. Figure 6 As shown, the system model includes a feature extraction network, a context-aware network, a cost space pyramid determination module, a recursive network based on multi-scale attention, a reconstruction module, and an optimization module. The recursive network based on multi-scale attention includes multiple iterators, and the search operators during iteration include L1, L2, and L3.

[0042] This invention also provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the binocular stereo matching method based on multi-scale iterative optimization as provided in any of the above embodiments.

[0043] This invention also provides a computer program product comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the multi-scale iterative optimization-based binocular stereo matching method provided in the various optional embodiments described above.

[0044] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described apparatus and unit can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0045] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0046] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of the present invention, depending on actual needs.

[0047] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0048] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0049] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A binocular stereo matching method based on multi-scale iterative optimization, characterized in that, The method comprises the following steps: performing semantic structure feature extraction on the corrected left view and right view through a feature extraction network; determining a cost space pyramid and a probability matrix according to the left view feature map and the right view feature map output by the feature extraction network; extracting multi-scale context features of the left view through a context perception network, and initializing the multi-scale context features to obtain hidden states and inputs of a recurrent network; wherein the recurrent network is composed of a multi-scale attention convolution gate recurrent unit as an iterator; when the recurrent network is iterated, a multi-range lookup strategy is used to index a local cost from the cost space pyramid according to the probability matrix; inputting the local cost into the recurrent network, combining the hidden states and inputs obtained from the context features, and iteratively updating the disparity value to obtain a disparity map.

2. The method of claim 1, wherein, The feature extraction on the corrected left view and right view through the feature extraction network comprises the following steps: inputting the left view into the feature extraction network, and obtaining two feature maps with different resolutions through the first two stages of a pre-trained ConvNeXt on ImageNet; using transposed convolution to upsample the feature map with a larger resolution to make it consistent in size with the feature map with a smaller resolution; fusing the two feature maps consistent in size through a feature fusion module to obtain a left view feature map; Similarly, the right view is processed in the same way as the left view to obtain a right view feature map.

3. The method of claim 1, wherein, The determination of the cost space pyramid and the probability matrix according to the left view feature map and the right view feature map output by the feature extraction network comprises the following steps: determining an initial cost space matrix according to the inner product of the feature vectors of the left view feature map and the right view feature map located on the same epipolar line; performing one-dimensional average pooling on the disparity dimension of the initial cost space matrix to obtain a two-layer cost space pyramid; performing numerical normalization on the initial cost space matrix along the disparity dimension to obtain a probability matrix.

4. The method according to any one of claims 1 to 3, characterized in that, When the recurrent network is iterated, a multi-range lookup strategy is used to index a local cost from the cost space pyramid according to the probability matrix, which comprises the following steps: finding the disparity value with the highest confidence corresponding to each pixel in the probability matrix, and taking the disparity value as an initial disparity value at the beginning of iteration; using a multi-range lookup strategy to find in the cost space pyramid according to the initial disparity value to index a local cost.

5. The method of claim 4, wherein, The use of a multi-range lookup strategy to find in the cost space pyramid according to the initial disparity value to index a local cost comprises the following steps: using three different lookup radius lookup operators to find in the cost space pyramid according to the initial disparity value; using the lookup operator with the largest lookup radius to find in the early stage of iteration; using the lookup operator with the medium lookup radius to find in the middle stage of iteration; and using the lookup operator with the smallest lookup radius to find in the late stage of iteration.

6. The method of claim 5, wherein, The inputting of the local cost into the recurrent network, the combination of the hidden states and inputs obtained from the context features, and the iteratively updating of the disparity value to obtain a disparity map comprise the following steps: The different scale context features output by the context perception network are respectively input into different iterators of the recurrent network; wherein the hidden states of adjacent resolution iterators are transmitted to each other using up-sampling and down-sampling; The local cost indexed from the cost space pyramid is input into the iterator of the minimum resolution to generate an increment of the disparity in combination with the hidden state thereof; The increment of the disparity value is added to the current disparity value to obtain an updated disparity value.

7. The method according to any one of claims 1 to 3, characterized in that, Further comprising disparity optimization on the disparity map by a disparity refinement method to obtain an optimized disparity map, specifically comprising: Moving the points in the original right view according to the disparity values of each point in the disparity map to obtain a reconstructed left view; Subtracting the reconstructed left view from the original left view to obtain a reconstruction error map of the left view; Concatenating the reconstruction error map, the original left view and the disparity map in the channel dimension and inputting into a U-Net network; The U-Net network outputs a residual value of the disparity, and the original disparity value of the disparity map is added to the residual value to obtain an optimized disparity map.

8. A binocular stereo matching system based on multi-scale iterative optimization, characterized in that, Comprise: A feature extraction module for performing semantic structure feature extraction on the corrected left view and right view by a feature extraction network; A cost space determination module for determining a cost space pyramid and a probability matrix according to the left view feature map and the right view feature map output by the feature extraction network; A context feature extraction module for extracting multi-scale context features of the left view by a context perception network, the multi-scale context features being initialized to obtain the hidden state and input of the recurrent network; wherein the recurrent network is composed of multi-scale attention convolution gate recurrent units as iterators; A local cost determination module for indexing local cost from the cost space pyramid according to the probability matrix using a multi-range lookup strategy when the recurrent network iterates; An iterative update module for inputting the local cost into the recurrent network, combining the hidden state and input obtained from the context features to iteratively update the disparity value and obtain a disparity map.

9. A computer-readable storage medium comprising instructions, characterized in that, When the instructions run on the computer, the computer executes the binocular stereo matching method based on multi-scale iterative optimization of any one of claims 1 to 7.

10. An electronic device comprising: A memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the binocular stereo matching method based on multi-scale iterative optimization of any one of claims 1 to 7 when executing the program.

Images