Lightweight binocular stereo matching method and device based on deep learning

Abstract

The invention provides a lightweight binocular stereo matching method and device based on deep learning, and relates to the technical field of image data processing, and the method comprises the steps: obtaining a to-be-matched left image and a to-be-matched right image, calling a convolutional neural network to respectively extract a left image feature map of the to-be-matched left image and a right image feature map of the to-be-matched right image, and constructing a to-be-aggregated cost cube; and calling a plurality of inverse residual processing modules in the lightweight network model to sequentially carry out cost aggregation processing on the to-be-aggregated cost cubes to obtain a final cost cube, and constructing according to the final cost cube to obtain a fine disparity map. According to the method and the device, the technical problems that the processing speed of the binocular stereo matching model is reduced and the energy consumption is increased due to the fact that the model calculation amount is large and the memory requirement is high in the cost aggregation process of the binocular stereo matching model in the prior art can be solved.

Description

Technical Field

[0001] The present invention relates to the technical field of image data processing, and in particular to a lightweight binocular stereo matching method and device based on deep learning. Background Art

[0002] Binocular stereo matching technology is an important computer vision technology that simulates the principle of human binocular vision and uses two cameras to take pictures of the same scene from slightly different perspectives to reconstruct three-dimensional spatial information. The core of this technology is to find the corresponding pixels in the two perspectives and calculate the depth information of the objects in the scene through these corresponding relationships.

[0003] However, although the existing binocular stereo matching algorithm provides an effective three-dimensional visual perception solution through the binocular stereo matching network, in order to extract rich features and accurately calculate disparity, the network used by the binocular stereo matching network is usually multi-layered and complex, requiring a large number of matrix multiplications and other intensive operations. Secondly, since most of the current algorithms use 4D cost cubes and 3D convolutions for cost aggregation, the cost aggregation calculation process requires not only a large amount of computing resources, but also a large amount of memory to store a large number of weights and intermediate layer outputs (feature maps), making it difficult to quickly adapt to environmental changes on lightweight devices. Furthermore, the high computational complexity of deep learning models not only affects the processing speed, but also leads to high energy consumption, limiting its application in environments where frequent charging is not possible. Summary of the invention

[0004] The present invention provides a lightweight binocular stereo matching method and device based on deep learning, which are used to solve the defects in the prior art that, during the cost aggregation process of the binocular stereo matching model, the processing speed of the binocular stereo matching model is reduced and the energy consumption is increased due to the large amount of model calculation and memory requirements.

[0005] The present invention provides a lightweight binocular stereo matching method based on deep learning, comprising the following steps: Obtain a left image to be matched and a right image to be matched, and call a convolutional neural network to perform feature extraction on the left image to be matched and the right image to be matched, so as to obtain a left image feature map of the left image to be matched and a right image feature map of the right image to be matched, respectively; Constructing a cost cube to be aggregated according to the left image feature map and the right image feature map; Calling multiple inverse residual processing modules in the lightweight network model to perform cost aggregation processing on the cost cube to be aggregated to obtain a final cost cube; A fine disparity map is constructed according to the final cost cube.

[0006] In some embodiments, constructing a cost cube to be aggregated according to the left image feature map and the right image feature map includes: For each first target pixel in the left image feature map, determine a second target pixel at a corresponding disparity position from the right image feature map, and form a pixel pair with the first target pixel and the second target pixel; According to the similarity data of the pixel features between each of the pixel pairs, a cost cube to be aggregated is constructed.

[0007] In some embodiments, the lightweight network model is composed of a plurality of inverse residual processing modules connected together, and the calling of the plurality of inverse residual processing modules in the lightweight network model sequentially performs cost aggregation processing on the cost cube to be aggregated to obtain a final cost cube, including: Calling the first inverse residual processing module in the lightweight network model to perform cost aggregation processing on the cost cube to be aggregated to obtain an aggregated cost cube, and inputting the aggregated cost cube into the N-1th inverse residual processing module in the lightweight network model; Inputting the result obtained by cost aggregation processing of the N-1th inverse residual processing module into the Nth inverse residual processing module in the lightweight network model; The result obtained by cost aggregation processing of the Nth inverse residual processing module is used as the final cost cube, wherein N is the total number of inverse residual processing modules in the lightweight network model, and N is an integer greater than or equal to 3.

[0008] In some embodiments, the inverse residual processing module includes an inverse residual block and a multi-scale feature aggregation module, and calling the first inverse residual processing module in the lightweight network model to perform cost aggregation processing on the cost cube to be aggregated to obtain the aggregated cost cube includes: Calling the inverse residual block to perform cost aggregation processing on the cost cube to be aggregated to obtain a residual cost cube, and inputting the residual cost cube into the multi-scale feature aggregation module; Obtain a size feature map of the left image to be matched, wherein the size feature map is obtained by calling a convolutional neural network to perform feature extraction on the left image to be matched; The size feature map and the residual cost cube are feature aggregated through a multi-scale feature aggregation module to obtain an aggregated cost cube.

[0009] In some embodiments, the inverse residual block includes an extended convolution layer, a depth-separable convolution layer, and a projection layer, and calling the inverse residual block to perform cost aggregation processing on the cost cube to be aggregated to obtain a residual cost cube includes: Calling the extended convolutional layer to perform channel expansion processing on the feature map of the cost cube to be aggregated to obtain an extended feature map; Calling the depth-separable convolution layer to perform channel convolution processing on the extended feature map to obtain a depth-combined convolution feature; The projection layer is called to perform channel reduction processing on the deep combined convolutional features to obtain a residual cost cube corresponding to the cost cube to be aggregated.

[0010] In some embodiments, constructing a fine disparity map according to the final cost cube includes: Calling the context feature network to perform feature extraction processing on the left image to be matched, to obtain a global feature map of the left image to be matched; Performing disparity regression calculation on the final cost cube to obtain a rough disparity map corresponding to the final cost cube; The global feature map and the coarse disparity map are upsampled to construct a fine disparity map.

[0011] The present invention also provides a lightweight binocular stereo matching device based on deep learning, which includes the following modules: An acquisition module, used to acquire a left image to be matched and a right image to be matched, and call a convolutional neural network to perform feature extraction on the left image to be matched and the right image to be matched, so as to obtain a left image feature map of the left image to be matched and a right image feature map of the right image to be matched respectively; A construction module, used to construct a cost cube to be aggregated according to the left image feature map and the right image feature map; An aggregation module, used for calling multiple inverse residual processing modules in the lightweight network model to perform cost aggregation processing on the cost cube to be aggregated in turn to obtain a final cost cube; A matching module is used to construct a fine disparity map according to the final cost cube, and match the left image to be matched and the right image to be matched according to the fine disparity map.

[0012] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a lightweight binocular stereo matching method based on deep learning as described above is implemented.

[0013] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program implements any of the above-described lightweight binocular stereo matching methods based on deep learning.

[0014] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned lightweight binocular stereo matching methods based on deep learning.

[0015] The lightweight binocular stereo matching method based on deep learning provided by the present invention optimizes the final cost cube by calling multiple inverse residual processing modules in the lightweight network model to perform cost aggregation processing on the cost cube to be aggregated in turn, and then constructs a fine disparity map. Since the lightweight network model can greatly reduce the number of parameters and complexity of the model, and the inverse residual processing module therein can efficiently extract features in the feature extraction process, further improving the information feature expression ability of the model, thereby improving the performance and processing speed of the lightweight network model and reducing energy consumption. This can overcome the defects of the prior art in that the processing speed of the binocular stereo matching model is reduced and the energy consumption is increased due to the large amount of model calculation and very high memory requirements during the cost aggregation process. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced one by one below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 It is a flowchart of a lightweight binocular stereo matching method based on deep learning provided by the present invention.

[0018] Figure 2 This is a schematic diagram of the framework of the lightweight binocular stereo matching method based on deep learning provided by the present invention.

[0019] Figure 3 It is a schematic diagram of extracting features of the inverse residual block provided by the present invention.

[0020] Figure 4 It is a structural schematic diagram of a lightweight binocular stereo matching device based on deep learning provided by the present invention.

[0021] Figure 5 It is a schematic diagram of the physical structure of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] Combine the following Figure 1 and Figure 2 Describe the lightweight binocular stereo matching method based on deep learning provided by the present invention, Figure 1 is a flow chart of a lightweight binocular stereo matching method based on deep learning provided by the present invention, such as Figure 1 As shown, the method includes the following steps.

[0024] Step 101: obtain a left image to be matched and a right image to be matched, and call a convolutional neural network to extract features of the left image to be matched and the right image to be matched, and obtain a left image feature map of the left image to be matched and a right image feature map of the right image to be matched, respectively.

[0025] First, obtain the left image and the right image to be matched for binocular matching. These two images are pictures of the same scene taken by two cameras from slightly different perspectives. The original sizes of these two images are the same. For details, see Figure 2 , Figure 2 is a schematic diagram of the framework of a lightweight binocular stereo matching method based on deep learning provided by the present invention. Figure 2 The left image to be matched and the right image to be matched are motorcycle pictures taken from two perspectives. The next step is the feature extraction stage, which calls the convolutional neural network to extract features from the left image to be matched and the right image to be matched, and obtains the left image feature map of the left image to be matched and the right image feature map of the right image to be matched.

[0026] Here, features are extracted from two images independently and input into the convolutional neural network for convolution processing. Figure 2 As shown in Figure 1, the convolutional neural network consists of multiple convolutional layers. Figure 2 Five convolutional layers are given as an example, and each convolutional layer outputs feature maps of different sizes, which are 1 / 4, 1 / 8, and 1 / 16 of the original size respectively.

[0027] Step 102: construct a cost cube to be aggregated according to the left image feature map and the right image feature map.

[0028] In order to determine the matching cost of the left image to be matched and the right image to be matched, the embodiment of the present invention here constructs a cost cube to be aggregated based on the feature map of the left image and the feature map of the right image. The cost cube is a three-dimensional data structure based on pixel matching, which is used to store the similarity of features between pixel pairs of two perspective images within the disparity range, and provides basic data for the generation of disparity maps. Therefore, the construction of the cost cube to be aggregated requires the calculation of the similarity of pixel pairs, which is explained in detail below.

[0029] First, for each first target pixel in the left image feature map, determine the second target pixel at the corresponding disparity position from the right image feature map, and form a pixel pair with the first target pixel and the second target pixel. By traversing all pixel pairs corresponding to possible disparity values, the similarity data between pixel features is calculated to construct a cost cube to be aggregated, denoted as Cost. Among them, the similarity matching calculation is to calculate the similarity of two pixel features, and the calculation method can be to calculate the Euclidean distance or cosine similarity. By mapping the similarity data of each pixel pair into a three-dimensional data structure, the cost cube to be aggregated can be constructed.

[0030] Continue to see Figure 1 , step 103, calling multiple inverse residual processing modules in the lightweight network model to perform cost aggregation processing on the cost cubes to be aggregated in turn to obtain the final cost cube.

[0031] After constructing the cost cube to be aggregated, the next step is the cost aggregation process, which is an important step in binocular stereo matching. In order to reduce the amount of calculation and memory in the cost aggregation process, the embodiment of the present invention sequentially performs cost aggregation processing on the cost cube to be aggregated by calling multiple inverse residual processing modules in the lightweight network model to obtain the final cost cube.

[0032] Here, the lightweight network model is specifically a two-dimensional (2D) regularized network, which may be a Mobilenet model or a MobilenetV2 model. The present invention will illustrate the cost aggregation process of the cost cube Cost to be aggregated based on the MobilenetV2 model.

[0033] First, if Figure 2 As shown in the figure, the lightweight network model (MobilenetV2) is composed of multiple inverse residual processing modules. The number of inverse residual processing modules can be determined according to actual needs (such as the size of the image to be matched). After the cost cube Cost to be aggregated is input into the lightweight network model, multiple inverse residual processing modules perform cost aggregation processing on the cost cube to be aggregated in turn. First, call the first inverse residual processing module in the lightweight network model to perform cost aggregation processing on the cost cube Cost to be aggregated, and obtain the aggregated cost cube.

[0034] Next, the process of cost aggregation in the inverse residual processing module is explained. The inverse residual processing module includes two parts: the inverse residual block and the multi-scale feature aggregation module. First, the inverted residual block (InvertedResidual Blocks) in the inverse residual processing module is called to perform cost aggregation processing on the aggregated cost cube to obtain the residual cost cube. The core structure inside the inverted residual block includes three parts: the extended convolution layer, the depth-separable convolution layer and the projection layer. Among them, the extended convolution layer and the projection layer are used to expand and enlarge the channels of the feature map and reduce and restore them. In the process of expanding and enlarging the feature map, the depth-separable convolution layer is used to extract features from the feature map when the channel is expanded to further extract more feature information.

[0035] See also Figure 3 , Figure 3 Schematic diagram of the inverse residual block extraction feature provided by the present invention, such as Figure 3 As shown in the figure, the extended convolution layer is first called to perform channel expansion processing on the feature map of the aggregated cost cube to obtain an extended feature map. The channel expansion processing is to expand the number of channels through the 1×1 convolution kernel of the extended convolution layer (usually the expansion multiple can be 6 times, that is, if the number of input channels is 32, then the number after expansion is 192). The main function of the extended convolution layer is to increase the nonlinearity of the network and the expression ability of features, so as to facilitate the subsequent further extraction of features through the depthwise separable convolution layer under the condition of channel expansion.

[0036] Next, the depthwise separable convolution layer is called to perform channel convolution on the extended feature map to obtain the depthwise combined convolution feature. The convolution steps of the depthwise separable convolution layer are depthwise convolution and pointwise convolution. The depthwise convolution uses a 3×3 depthwise separable convolution kernel to perform convolution on each channel separately, thereby greatly reducing the amount of calculation and extracting more feature information. For pointwise convolution, a 1×1 convolution kernel is used to combine the feature information after the depthwise convolution and further fuse the feature information to finally obtain the depthwise combined convolution feature.

[0037] The deep combined convolutional features are then input into the projection layer, which performs channel reduction processing on the deep combined convolutional features by calling the projection layer to obtain the residual cost cube corresponding to the cost cube to be aggregated. The projection layer uses a layer of 1×1 convolution kernel to reduce the number of channels after the expansion of the extended convolutional layer, so that the feature restoration is reduced to the size dimension before the input inverse residual module. The main purpose of the projection layer is to reduce the feature dimension to reduce the computational burden of the lightweight network model.

[0038] In the inverse residual processing module, after the three parts of the inverse residual block are processed, the residual cost cube is still output in the end, and then the residual cost cube is input into the multi-scale feature aggregation module. Here, the multi-scale feature aggregation module includes a multi-scale convolutional attention (MSCA) model, which uses stacked 1×n and n×1 convolution layers to simulate a large convolution kernel, which is used to integrate image feature information into the cost cube after the inverse residual block is processed. In the present invention, the image feature information uses the image feature information extracted from the left image to be matched. As mentioned above, the convolutional neural network includes multiple convolutional layers ( Figure 2 In the example, five convolutional layers are given), each convolutional layer outputs feature maps of different sizes, such as 1 / 4 size feature map, 1 / 8 size feature map, and 1 / 16 size feature map. Therefore, the size feature map of the left image to be matched is obtained here, where the size feature map is obtained by calling the convolutional neural network (CNN) to extract features from the left image to be matched. Each size feature map will be aggregated into the cost cube after the inverse residual block processing through the MSCA module to help optimize the cost cube.

[0039] like Figure 2 As shown in the figure, the 1 / 4 size feature map and the cost cube obtained by processing the inverse residual block in the first inverse residual processing module are feature aggregated through the MSCA module to obtain the aggregated cost cube. This completes the introduction of the processing process of the first inverse residual processing module in the lightweight network model (MobilenetV2).

[0040] Since MobilenetV2 is composed of multiple inverse residual processing modules, the size of the cost cube feature map processed by each inverse residual processing module is different. The MSCA module integrates the size feature map of the corresponding left image to be matched according to the size of the processed feature map. Figure 2As shown, the aggregated cost cube is then input into the second adjacent inverse residual processing module in MobilenetV2 to continue cost aggregation (and integrated into the 1 / 8 size feature map of the left image to be matched through the MSCA module). By analogy, when the third inverse residual processing module continues to perform cost aggregation, the 1 / 16 size feature map of the left image to be matched is integrated through the MSCA module. Thus, cost aggregation processing (including residual processing of inverse residual blocks and image feature fusion of MSCA) is performed in sequence through multiple inverse residual processing modules until the aggregated cost cube is input into the N-1th inverse residual processing module. The cost aggregation processing of the aggregated cost cube is continued through the N-1th inverse residual processing module, and then the result obtained by the cost aggregation processing of the N-1th inverse residual processing module is input into the Nth inverse residual processing module in MobilenetV2 (i.e., the last inverse residual processing module). Finally, the result obtained by the cost aggregation processing of the Nth inverse residual processing module is used as the final cost cube, where N is the total number of inverse residual processing modules in the MobilenetV2 model, and N is an integer greater than or equal to 3.

[0041] Through the embodiment of the present invention, in the cost aggregation process of the MobilenetV2 model, by stacking multiple inverse residual blocks of different scales, a multi-scale feature aggregation (MSCA) module is used between the inverse residual blocks to integrate the intermediate feature information of the feature extraction process of the left image to be matched into the cost aggregation process, thereby helping to optimize the cost cube. By using multiple inverse residual blocks of different scales to perform cost aggregation at the same time, the MobilenetV2 model can have a larger receptive field and extract more feature detail information, generate attention weights for the cost (i.e., similarity data) of each pixel in the cost cube, and help optimize the cost cube. The MSCA module enables the MobilenetV2 model to understand various structures and patterns in the image from subtle to coarse, which helps to improve the MobilenetV2 model's ability to understand and process complex scenes, especially in the edges, weak textures, and complex texture areas of the image.

[0042] Continue to see Figure 1 , step 104, constructing a fine disparity map based on the final cost cube.

[0043] After the MobilenetV2 model cost aggregation step in step 103, the final cost cube is obtained, and finally a fine disparity map is constructed based on the final cost cube. Since the final cost cube is low-resolution, it needs to be refined and integrated with high-resolution image features to calculate a fine disparity map. The construction process is described in detail below.

[0044] First, we need to obtain high-resolution image features, such as Figure 2 As shown, the embodiment of the present invention calls the context feature network to perform feature extraction processing on the left image to be matched, and obtains the global feature map of the left image to be matched. Here, the context feature network can also use a convolutional neural network. The context feature network can directly extract the global feature map for the left image to be matched. The resolution of this global feature map is the same as the original resolution of the left image to be matched, and can be used as a high-resolution image feature.

[0045] Next, the disparity regression method (SoftArgmax) commonly used in deep learning stereo matching methods is used for disparity calculation. This method can calculate a low-resolution disparity map from the output low-resolution final cost cube. Therefore, the disparity regression method is used here to perform disparity regression calculation on the final cost cube to obtain a rough disparity map corresponding to the final cost cube. Next, the global feature map and the rough disparity map are upsampled to construct a fine disparity map. Here, during the upsampling process, according to the global feature map of the left image to be matched, the low-resolution coarse disparity map is upsampled to the same resolution as the global feature map (that is, the original resolution of the left image to be matched), and finally a high-quality fine disparity map is generated as shown in the figure. Figure 2 The final disparity map shown is used for subsequent research and application of the depth information of the motorcycle in the scene.

[0046] Through the embodiments of the present invention, the global feature map is integrated into the resolution adjustment when constructing the disparity map, which not only improves the accuracy of the disparity map and the performance of the model in the occluded area, but also enhances the visual effect of the disparity map, making the final fine disparity map clearer and more realistic.

[0047] The lightweight binocular stereo matching device based on deep learning provided by the present invention is described below. The lightweight binocular stereo matching device based on deep learning described below and the lightweight binocular stereo matching method based on deep learning described above can refer to each other.

[0048] Figure 4 is a schematic diagram of the structure of a lightweight binocular stereo matching device based on deep learning provided by the present invention, such as Figure 4As shown, the present invention provides a lightweight binocular stereo matching device based on deep learning, and the device specifically includes: an acquisition module 401, a construction module 402, an aggregation module 403, and a matching module 404. Among them, the acquisition module 401 is used to acquire the left image to be matched and the right image to be matched, and call the convolutional neural network to extract the features of the left image to be matched and the right image to be matched, and respectively obtain the left image feature map of the left image to be matched and the right image feature map of the right image to be matched; the construction module 402 is used to construct a cost cube to be aggregated according to the left image feature map and the right image feature map; the aggregation module 403 is used to call multiple inverse residual processing modules in the lightweight network model to perform cost aggregation processing on the cost cube to be aggregated, and obtain the final cost cube; the matching module 404 is used to construct a fine disparity map according to the final cost cube.

[0049] Specifically, the above-mentioned lightweight binocular stereo matching device based on deep learning provided by the present invention can implement all the method steps implemented by the above-mentioned lightweight binocular stereo matching method embodiment based on deep learning, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as the method embodiment will not be described in detail here.

[0050] Figure 5 is a schematic diagram of the physical structure of an electronic device provided by the present invention, such as Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute a lightweight binocular stereo matching method based on deep learning, the method comprising: obtaining a left image to be matched and a right image to be matched, and calling a convolutional neural network to extract features of the left image to be matched and the right image to be matched, respectively obtaining a left image feature map of the left image to be matched and a right image feature map of the right image to be matched; constructing a cost cube to be aggregated according to the left image feature map and the right image feature map; calling multiple inverse residual processing modules in the lightweight network model to perform cost aggregation processing on the cost cube to be aggregated to obtain a final cost cube; and constructing a fine disparity map according to the final cost cube.

[0051] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0052] On the other hand, the present invention also provides a computer program product, which includes a computer program, and the computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the lightweight binocular stereo matching method based on deep learning provided by the above methods, and the method includes: obtaining a left image to be matched and a right image to be matched, and calling a convolutional neural network to perform feature extraction on the left image to be matched and the right image to be matched, respectively obtaining a left image feature map of the left image to be matched and a right image feature map of the right image to be matched; constructing a cost cube to be aggregated according to the left image feature map and the right image feature map; calling multiple inverse residual processing modules in the lightweight network model to perform cost aggregation processing on the cost cube to be aggregated, and obtaining a final cost cube; and constructing a fine disparity map according to the final cost cube.

[0053] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the lightweight binocular stereo matching method based on deep learning provided by the above-mentioned methods, the method comprising: obtaining a left image to be matched and a right image to be matched, and calling a convolutional neural network to perform feature extraction on the left image to be matched and the right image to be matched, respectively obtaining a left image feature map of the left image to be matched and a right image feature map of the right image to be matched; constructing a cost cube to be aggregated according to the left image feature map and the right image feature map; calling multiple inverse residual processing modules in the lightweight network model to perform cost aggregation processing on the cost cube to be aggregated, to obtain a final cost cube; and constructing a fine disparity map according to the final cost cube.

[0054] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0055] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0056] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A lightweight binocular stereo matching method based on deep learning, characterized in that: The method comprises: Obtain a left image to be matched and a right image to be matched, and call a convolutional neural network to perform feature extraction on the left image to be matched and the right image to be matched, so as to obtain a left image feature map of the left image to be matched and a right image feature map of the right image to be matched, respectively; Constructing a cost cube to be aggregated according to the left image feature map and the right image feature map; Calling multiple inverse residual processing modules in the lightweight network model to sequentially perform cost aggregation processing on the cost cube to be aggregated to obtain a final cost cube; A fine disparity map is constructed according to the final cost cube.

2. The lightweight binocular stereo matching method based on deep learning according to claim 1, characterized in that: The step of constructing a cost cube to be aggregated according to the left image feature map and the right image feature map includes: For each first target pixel in the left image feature map, determine a second target pixel at a corresponding disparity position from the right image feature map, and form a pixel pair with the first target pixel and the second target pixel; A cost cube to be aggregated is constructed based on the similarity data of the pixel features between each pair of pixels.

3. The lightweight binocular stereo matching method based on deep learning according to claim 1, characterized in that: The lightweight network model is composed of a plurality of inverse residual processing modules connected together, and the plurality of inverse residual processing modules in the lightweight network model are called to sequentially perform cost aggregation processing on the cost cube to be aggregated to obtain a final cost cube, including: Calling the first inverse residual processing module in the lightweight network model to perform cost aggregation processing on the cost cube to be aggregated to obtain an aggregated cost cube, and inputting the aggregated cost cube into the N-1th inverse residual processing module in the lightweight network model; Inputting the result obtained by cost aggregation processing of the N-1th inverse residual processing module into the Nth inverse residual processing module in the lightweight network model; The result obtained by cost aggregation processing of the Nth inverse residual processing module is used as the final cost cube, wherein N is the total number of inverse residual processing modules in the lightweight network model, and N is an integer greater than or equal to 3.

4. The light-weight binocular stereo matching method based on deep learning according to claim 3, characterized in that: The inverse residual processing module includes an inverse residual block and a multi-scale feature aggregation module, and the calling of the first inverse residual processing module in the lightweight network model performs cost aggregation processing on the cost cube to be aggregated to obtain the aggregated cost cube, including: Calling the inverse residual block to perform cost aggregation processing on the cost cube to be aggregated to obtain a residual cost cube, and inputting the residual cost cube into the multi-scale feature aggregation module; Obtain a size feature map of the left image to be matched, wherein the size feature map is obtained by calling a convolutional neural network to perform feature extraction on the left image to be matched; The size feature map and the residual cost cube are feature aggregated through a multi-scale feature aggregation module to obtain an aggregated cost cube.

5. The light-weight binocular stereo matching method based on deep learning according to claim 4, characterized in that: The inverse residual block includes an extended convolution layer, a depth-separable convolution layer and a projection layer. The inverse residual block is called to perform cost aggregation processing on the cost cube to be aggregated to obtain a residual cost cube, including: Calling the extended convolutional layer to perform channel expansion processing on the feature map of the cost cube to be aggregated to obtain an extended feature map; Calling the depth-separable convolution layer to perform channel convolution processing on the extended feature map to obtain a depth-combined convolution feature; The projection layer is called to perform channel reduction processing on the deep combined convolutional features to obtain a residual cost cube corresponding to the cost cube to be aggregated.

6. The lightweight binocular stereo matching method based on deep learning according to claim 1, characterized in that: According to the final cost cube, a fine disparity map is constructed, including: Calling the context feature network to perform feature extraction processing on the left image to be matched, to obtain a global feature map of the left image to be matched; Performing disparity regression calculation on the final cost cube to obtain a rough disparity map corresponding to the final cost cube; The global feature map and the coarse disparity map are upsampled to construct a fine disparity map.

7. A lightweight binocular stereo matching device based on deep learning, characterized in that: The device comprises: An acquisition module, used to acquire a left image to be matched and a right image to be matched, and call a convolutional neural network to perform feature extraction on the left image to be matched and the right image to be matched, so as to obtain a left image feature map of the left image to be matched and a right image feature map of the right image to be matched respectively; A construction module, used to construct a cost cube to be aggregated according to the left image feature map and the right image feature map; An aggregation module, used for calling multiple inverse residual processing modules in the lightweight network model to perform cost aggregation processing on the cost cube to be aggregated in turn to obtain a final cost cube; The matching module is used to construct a fine disparity map according to the final cost cube.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the lightweight binocular stereo matching method based on deep learning as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the lightweight binocular stereo matching method based on deep learning as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the lightweight binocular stereo matching method based on deep learning as described in any one of claims 1 to 6 is implemented.

Images