Binocular depth estimation method and device based on automatic network compression

Abstract

The application discloses a binocular depth estimation method and device based on automatic network compression, and the method comprises the following steps: acquiring a left view and a right view and corresponding left and right parallax maps; constructing a binocular depth estimation model, wherein the binocular depth estimation model comprises a NASCNet network and a NASSNet network, a residual block is added in a feature extraction module of the NASCNet network to replace a convolution layer in DispNetC, and a point correlation layer is used to optimize feature extraction operation in DispNetC; the binocular depth estimation model is trained by using a progressive search strategy, first, a network with the largest model structure is trained, the convolution kernel size, the model depth, the model width and the search scale are set to the maximum values in the optional range, and then the automatic search is sequentially performed on the convolution kernel size, the model depth and the model width; and the binocular depth estimation is performed on the image to be detected based on the trained binocular depth estimation model. The binocular depth estimation model and the automatic search strategy are used to improve the estimation accuracy and speed.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and particularly relates to a binocular depth estimation method and device based on automatic network compression. BACKGROUND

[0002] Binocular depth estimation: current algorithms are mainly divided into traditional stereo matching algorithms and stereo matching algorithms based on deep learning. The traditional stereo matching algorithm mainly predicts the time difference based on a manual feature extraction algorithm or an optimization function, and generally includes four steps: cost calculation, cost aggregation, disparity calculation, and disparity post-processing. The cost calculation method includes the normalized cross-correlation method, the mutual information-based cost calculation method, and the transformation-based method. The traditional matching algorithm can be generally divided into three categories: local matching algorithm, global matching algorithm, and semi-global matching algorithm. The local algorithm usually only focuses on the local area, and most of them have low computational complexity and fast running speed. Compared with the global algorithm, the local algorithm has poor accuracy, especially in areas with repetitive textures, discontinuous disparity, or occlusions. The global matching algorithm mainly uses the theory of global optimization to estimate the disparity by constructing a global energy function including data items and smoothing items, which can better solve the matching problem in areas with repetitive textures and occlusions. The semi-global matching algorithm (SGM) balances the relationship between accuracy and time complexity, and has higher accuracy than the local algorithm and faster speed than the global algorithm. The algorithm based on deep learning relies on the powerful feature extraction and model expression ability of neural networks, and can learn more robust and effective features based on a large amount of data sets, so that its accuracy is much higher than that of traditional algorithms. With the increase of available training data sets and the improvement of computer performance in recent years, the stereo matching method based on deep learning has developed rapidly and gradually become the mainstream of stereo matching technology research. The end-to-end training deep learning method can directly generate the final disparity, reducing human intervention and better utilizing the powerful learning ability of the network. This type of method can be roughly divided into two categories: 2D convolution with encoder-decoder structure and 3D convolution based on cost volume matching. In 2D convolution, the decoding part is used for feature extraction of the input image pair, the decoding part is used to predict the disparity based on the generated disparity map, and the regression method is used for disparity optimization. In order to enhance the learning ability of the deep neural network, the existing work introduces the correlation layer and the residual block structure, so the calculation efficiency of the method based on 2D convolution is greatly improved, but the accuracy is still not very high. Compared with 2D convolution, 3D convolution changes the input dimension to three dimensions, and the cost feature is calculated through the 3D convolution layer, so the calculation accuracy is very high, but the memory occupation is very large. The method based on 3D convolution needs to balance the accuracy and computational efficiency before being used to solve real problems.

[0003] Neural Architecture Search (NAS): The main research problems of NAS can be generally divided into three parts: building search space, optimization algorithm and model evaluation. NAS configures the structure of neural network by building a suitable search space, and a funny search space will directly affect the performance of the model. At present, the main building ideas of search space are chain construction, hierarchical construction, etc. Optimization algorithm optimizes the structure of neural network by constraint, such as reinforcement learning, evolutionary algorithm, Bayesian optimization, etc. Model evaluation occupies most of the time of NAS, and generally uses the training subset to train and sort, or uses the method of proxy model. NAS can let the model choose the appropriate hyperparameters by itself, which is more reasonable than manual adjustment.

[0004] At present, the binocular depth estimation algorithm based on deep learning has basically replaced the traditional method and become the mainstream method, but due to the increasing requirement of the accuracy of the model, the size and the calculation cost of the model increase explosively, which will be difficult to deploy on edge computing devices, how to balance the model accuracy and memory occupation (or calculation speed) becomes the key problem, the existing work cannot guarantee that the model is superior in both accuracy and speed, such as 2D network speed is fast but accuracy is low, 2D network accuracy is high but calculation speed is slow; Due to the different performance constraints (such as delay, energy consumption, etc.) of different hardware computing devices, the model for each device needs to be trained and deployed, and it usually takes a long GPU time to train a network model, which is undoubtedly a huge waste in terms of time and resources, how to reduce the training cost becomes a challenge. SUMMARY

[0005] The main purpose of the present application is to overcome the shortcomings and deficiencies of the prior art, and to provide a binocular depth estimation method and device based on automatic network compression, which improves the estimation accuracy and speed through the constructed binocular depth estimation model and automatic search strategy.

[0006] In order to achieve the above purpose, the technical scheme adopted by the present application is as follows:

[0007] In the first aspect, the present application provides a binocular depth estimation method based on automatic network compression, comprising the following steps:

[0008] Obtaining left view and right view, and obtaining left disparity map and right disparity map corresponding to left view and right view respectively;

[0009] Construct a binocular depth estimation model, which includes a NASCNet network and a NASSNet network. A residual block is added to the feature extraction module of the NASCNet network to replace the convolution layer in DispNetC, and the point correlation layer is used to optimize the feature extraction operation in DispNetC. The predicted disparity map generated by NASCNet is added to the input part of the NASSNet network. The input of the NASCNet network is an image pair of left view and right view, and the NASSNet network has an additional set of disparity maps generated by the NASCNet network. The final disparity map groups generated by the NASCNet network and the NASSNet network are added to the target disparity map respectively, and the loss loss is calculated and then added as the final loss. Finally, the largest and true values ​​in the disparity map group generated by NASSNet are used to calculate EPE.

[0010] A progressive search strategy is used to train the binocular depth estimation model. First, the network with the largest model structure is trained. The search convolution kernel size, model depth, model width, and search scale are set to the maximum value within the optional range. Then, the convolution kernel size, model depth, and model width are automatically searched in sequence.

[0011] Perform binocular depth estimation on the image to be detected based on the trained binocular depth estimation model.

[0012] As a preferred technical solution, the point correlation layer is used to optimize the feature extraction operation in DispNetC, specifically:

[0013] Set the constrained moving range D, and each patch only needs to be calculated with the nearby 2D+1 patches. The calculation formula is shown in Equation (1), and the patch in DispNetC is removed;

[0014]

[0015] Where c represents the point-related cost, function f represents the pixel value of the point, and “<>” represents the element-wise multiplication calculation of the points in the left and right images on the x-axis (-D, D). This operation is named point-related calculation, and the corresponding layer is named point-related layer. The number of multiplications required by NASCNet is C×K×H×W×(2D+1), thereby reducing the number of multiplications.

[0016] As a preferred technical solution, the binocular depth estimation model also includes an improvement to the loss function, specifically:

[0017] At the beginning of the training of the binocular depth estimation model, the loss of all disparity maps is enabled, and the disparity is predicted. The disparity d from the true value s The loss function between is defined as formula (3);

[0018]

[0019] where N is the number of pixels in each disparity map, denotes the i-th element of the s-th disparity map, denotes the i-th element of the s-th ground truth disparity map, L s denotes the pixel-wise smooth L1 loss between the predicted disparity map and the ground truth, as shown in equation (4).

[0020]

[0021] The disparity maps generated by the model correspond to different losses The final loss is calculated according to equation (5).

[0022]

[0023] where w s denotes the weight value of the s-th disparity map loss.

[0024] As a preferred technical solution, when performing convolution kernel search training, the binocular depth estimation model loads the model of the hypernetwork that has been trained. Since the size of the convolution kernel changes during training, the weights within the convolution kernel have different distributions and sizes in different cases. Therefore, when sharing the convolution kernel weight values, a kernel transformation matrix is introduced to map the weight values of a larger kernel to the weight values of a smaller kernel, thereby making more use of the weight values within the convolution kernel. Different layers have different convolution kernel sizes, so a corresponding kernel transformation matrix should be used for each layer. The convolution kernel transformation matrix in each layer can be shared between different channels.

[0025] As a preferred technical solution, when performing model depth search, it is assumed that the maximum depth in the hypernetwork is N, and it is assumed that the depth of the current search block is set to D during training. The model retains the first D layers and skips the last N-D layers. The model weights of the first D layers are shared between the hypernetwork and the subnetwork.

[0026] As a preferred technical solution, when performing model width search, the channels are reorganized according to their importance, i.e., the channels are sorted. The importance of the channels is calculated according to the L1-norm algorithm.

[0027] As a preferred technical solution, five automatic search blocks are designed in the progressive search strategy, wherein the first three automatic search blocks constitute a feature extraction part of the NASCNet network, and the last two automatic search blocks replace the first two residual blocks in the encoder of the NASCNet network; each automatic search block includes a plurality of automatic residual blocks, the automatic residual block is the smallest automatic search module, DynamicConv2d and DynamicTransposeConv2d modules are used for feature extraction in the automatic residual block, DynamicSeparableConv2d and DynamicSeparableTransposeConv2d modules dynamically change the convolution kernel size and width, and a lightweight DynamicSE module is used to improve the sensitivity of the model to channel features.

[0028] In a second aspect, the application provides a binocular depth estimation system based on automatic network compression, which is applied to the binocular depth estimation method based on automatic network compression and includes an image acquisition module, a model construction module, a model training module and a depth estimation module.

[0029] The image acquisition module is configured to acquire left and right views and acquire left and right parallax maps corresponding to the left and right views, respectively.

[0030] The model construction module is configured to construct a binocular depth estimation model, wherein the binocular depth estimation model includes a NASCNet network and a NASSNet network, a residual block is added to the feature extraction module of the NASCNet network to replace the convolution layer in DispNetC, and a point correlation layer is used to optimize the feature extraction operation in DispNetC; a predicted parallax map generated by the NASCNet network is added to the input part of the NASSNet network; the input of the NASCNet network is an image pair of the left and right views, and the NASSNet network has an additional parallax map generated by the NASCNet network; the final parallax maps generated by the NASCNet network and the NASSNet network are added to a target parallax map to calculate a loss, and then the loss is added as a final loss; the parallax map with the largest size in the parallax map group generated by the NASSNet network is used to calculate an EPE with a true value.

[0031] The model training module is configured to train the binocular depth estimation model by using a progressive search strategy, wherein a network with the largest model structure is first trained, the convolution kernel size, the model depth, the model width and the search scale are set to the maximum values in the optional range, and then the automatic search is performed on the convolution kernel size, the model depth and the model width in sequence.

[0032] The depth estimation module is configured to perform binocular depth estimation on the image to be detected based on the trained binocular depth estimation model.

[0033] In a third aspect, the present application provides an electronic device, comprising:

[0034] at least one processor; and

[0035] a memory in communication with the at least one processor; wherein

[0036] The memory stores computer program instructions executable by the at least one processor, and the computer program instructions are executed by the at least one processor to enable the at least one processor to perform the binocular depth estimation method based on automatic network compression.

[0037] In a fourth aspect, the present application provides a computer readable storage medium storing a program, and the program is executed by a processor to implement the binocular depth estimation method based on automatic network compression.

[0038] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0039] 1. Large models with excellent performance often have large space occupation, large number of parameters, slow inference speed and other problems. The present application provides a solution to the problem that large models cannot be deployed on edge computing devices. The model of the prior art has excellent precision performance, but at the cost of memory occupation and calculation speed. Such a model cannot be landed (applied in the industrial field). The binocular depth estimation model constructed by the present application can make the network deeper. This has the advantage that the model precision is more practically guaranteed. Automatic search can ensure that the optimal substructure is extracted in a deeper model, reducing the memory occupation of the model.

[0040] 2. The progressive shrinking algorithm used in the present application can ensure that the model is more robust under different hardware limitation conditions. That is, different optimal model substructures correspond to different hardware conditions. Each substructure is trained by the super network according to the progressive shrinking algorithm, ensuring that the model can be deployed in different application environments by selecting different sub-network structures after being trained only once, greatly saving training resources and time cost. BRIEF DESCRIPTION OF DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0042] Figure 1 Structure diagram of binocular depth estimation model of the embodiment of the application;

[0043] Figure 2 Structure diagram of NASCNet feature extraction module of the embodiment of the application;

[0044] Figure 3 Matching diagram based on region block and point correlation of the embodiment of the application;

[0045] Figure 4 Structure diagram of residual block in NAS of the embodiment of the application;

[0046] Figure 5 Flowchart of automatic search of the embodiment of the application;

[0047] Figure 6 Diagram of weight transformation matrix of the embodiment of the application;

[0048] Figure 7 Diagram of depth selection strategy of the embodiment of the application;

[0049] Figure 8 Diagram of channel ordering of the embodiment of the application;

[0050] Figure 9 Structure diagram of binocular depth estimation system based on automatic network compression of the application;

[0051] Figure 10 Structure diagram of electronic device of the embodiment of the application. DETAILED DESCRIPTION

[0052] In order to enable persons skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without creative work fall within the scope of protection of the present application.

[0053] In the present application, the phrase "embodiment" means that the specific features, structures or characteristics described in combination with the embodiment can be contained in at least one embodiment of the present application. The appearance of this phrase at various places in the specification does not necessarily mean the same embodiment, nor is it an independent or alternative embodiment to other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in the present application can be combined with other embodiments.

[0054] Stereo Depth Estimation: Also called Stereo Matching or Disparity Estimation, the input is a pair of rectified left and right images captured at the same time, then use stereo matching algorithm to find the matching pixel pairs in left and right views, finally use triangulation to restore the 3D physical point.

[0055] Neural Architecture Search: The meaning of NAS is to solve the parameter tuning problem of deep learning model, which is a cross research combining optimization and machine learning, and the appropriate model structure is selected to make the model optimal.

[0056] Rectification: Only align the camera mathematically to the unified viewing surface before obtaining the left and right images, so that the pixel rows of the camera are strictly aligned.

[0057] Disparity: The column coordinate of the same point pair in the left view minus the column coordinate in the right view, used in the rectified image pair.

[0058] Disparity Map: A two-dimensional image that stores the disparity values of all pixels in the rectified view. Each position of the disparity map saves the disparity value of the pixel in that position in units of pixels. For example, in the left view disparity map, the disparity value at pixel position p is equal to the column coordinate of the matching point of the pixel in the right image minus its column coordinate in the left image.

[0059] Cost Calculation: Used to measure the correlation between matching pixels and candidate pixels. The smaller the cost value, the greater the correlation.

[0060] Cost Aggregation: Cost calculation only considers local information, while cost aggregation establishes the connection between adjacent pixels and optimizes the cost matrix according to certain criteria, such as adjacent pixels should have continuous disparity values.

[0061] Disparity Calculation: Disparity calculation determines the optimal disparity value of each pixel through the cost matrix after cost aggregation, which can accurately reflect the correlation between pixels.

[0062] Disparity Post-processing: Further optimization of the disparity map obtained in the previous step to improve the quality of the disparity map.

[0063] Edge Computing Device: Lightweight devices with low computing power, small storage, and limited hardware configuration, such as robots, cameras, and smart car sensors.

[0064] Point Correlation Layer: Mainly used to compare the relationship between two feature maps.

[0065] Example 1

[0066] Embodiment 1 of the present application provides a binocular depth estimation method based on automatic network compression, comprising the following steps:

[0067] S1-1. Obtain a left view and a right view, and obtain a left disparity map and a right disparity map corresponding to the left view and the right view, respectively.

[0068] S1-2, build a binocular depth estimation model;

[0069] Furthermore, the binocular depth estimation model NASNet (a combination of NASCNet and NASSNet) constructed in this application is improved on the basis of DispNetC. Specifically, a residual block is added to the NASCNet network and the relevant feature operations in DispNetC are optimized using the point-wise correlation layer. The predicted disparity map generated by NASCNet is added to the input part of the NASSNet network, and there is an additional cross-link between the Encoder-Decoder of the two networks. The binocular depth estimation model in this application assigns weight loss to disparity maps of all resolutions, instead of only using the weight loss of the lowest resolution at the beginning of training like DispNet.

[0070] Furthermore, the residual block and point-related layer are as follows:

[0071] S1-2-1. After adding neural architecture search, the network depth will inevitably increase, and a deeper network may cause the gradient to disappear. Therefore, the binocular depth estimation model adds a residual block to the NASCNet feature extraction module instead of the convolutional layer in DispNetC, such as Figure 2 As shown in the figure, this allows for training deeper networks without increasing the training difficulty and effectively avoids the vanishing gradient problem. Due to the residual structure, compared to DispNetC's 5 layers of feature extraction and downsampling layers, this stereo depth estimation model has 7 layers of feature extraction and downsampling layers. The corresponding decoder part of DispNet outputs 6 intermediate eigenvalues, while this model increases the number of eigenvalues ​​to 7.

[0072] S1-2-2, the operation of the point correlation layer optimizes the method of calculating the cost in DispNetC. The purpose of Correlation in DispNetC is to find the correspondence between the left and right images. When calculating, a patch is taken from each of the left and right feature maps based on block matching and the cost is calculated using a convolution-like method, such as Figure 3 As shown in part (a), when the number of feature map channels is C and the cost matching kernel patch size is K, the number of multiplications required for DispNetC to calculate the cost is C×K 2×H 2 ×W 2 , where H, W represent the height and width of the feature map respectively. However, this method has high computational complexity, since the image has been rectified by epipolar line, the feature map in the vertical direction is not parallax, only the horizontal cost needs to be calculated, as shown in part (b) of FIG. 1. In addition, the embodiment sets a constraint moving range D, each patch only needs to calculate with 2D+1 nearby patches, and the calculation formula is shown in formula (1), which removes the patch in DispNetC. Figure 3

[0073]

[0074] , where c represents the point correlation cost, the function f represents the pixel value of the point, and the element point multiplication of the point in the x axis (-D, D) of the left and right images is represented by “<>”, so the embodiment names this operation as point correlation calculation, and the corresponding layer is named as point correlation layer. The number of multiplications required by NASCNet is CxKxHxWx(2D+1), which can be seen that the number of multiplications is obviously reduced, and the experimental verification shows that the model accuracy is improved.

[0075] S1-2-3, processing of input and loss function design;

[0076] In the input part of NASSNet, the last predicted disparity map of NASCNet is additionally added. According to the concept of disparity, the left and right images obtained by observing an object from two viewpoints can be calculated into a disparity map. Then, similarly, the left image can be calculated inversely from the predicted disparity map of NASCNet and the input right image, which is recorded as a pseudo left image, as shown in the NASSNet part of FIG. 1. The pseudo left image and the input are spliced together as the features for learning feedback, so that NASSNet also has a learning disparity map instead of only predicting the disparity residual according to the image pair. The experiment proves that this operation makes the accuracy higher. Figure 1

[0077] In the improvement of the loss function, since the residual structure is introduced, the network becomes deeper, and the number of intermediate feature maps of the upsampling output of the model increases to 7, so the predicted disparity of the model is , as shown in formula (2).

[0078]

[0079] , where c s represents the predicted disparity map of NASCNet, and r s ​​Disparity maps predicted by NASSNet. The loss function in DispNet uses a loss weight schedule, only using the loss corresponding to the lowest resolution disparity at the beginning of training, then gradually increasing the loss weight of high resolution disparity maps while removing the loss of low resolution disparity maps. Unlike it, this model enables the loss of all 7 disparity maps at the beginning of training, and predicts the disparity and the loss function between ground truth disparity d s is defined as equation (3).

[0080]

[0081] where N is the number of pixels in each disparity map, is the i-th element of the s-th disparity map, is the i-th element of the s-th disparity map of the ground truth, L s is the pixel-wise smooth L1 loss between the predicted disparity map and the ground truth, as shown in equation (4).

[0082]

[0083] The 7 disparity maps generated by the model correspond to 7 different losses The final loss is calculated according to equation (5).

[0084]

[0085] where w s is the weight value of the s-th disparity map loss. According to the above improvement, the model can learn features from coarse to fine from the beginning, and can also consider all disparity maps, and the feedback of the model is better. Experiments show that assigning larger weight values w s to feature maps with higher resolution can allow the network to better learn fine features, and in future improvements, different groups of weight scheduling strategies will be adopted in a gradual and progressive manner according to the training round number.

[0086] S1-3, training the binocular depth estimation model using a progressive search strategy, first training the network with the largest model structure, setting the search convolution kernel size, model depth, model width and search size to the maximum value in the optional range, then sequentially performing automatic search on the convolution kernel size, model depth and model width.

[0087] It can be understood that in the prior art, there is a network Once for All (OFA) which can select only a part of the network without additional training, and the model can be directly deployed under different computing power and memory limitations, but this part of the work only studies sparse prediction tasks based on feature matching such as image classification, and the binocular depth estimation of the embodiment uses matching based on region correlation, which belongs to dense matching, and the two are very different. Therefore, applying neural architecture search to the field of binocular depth estimation, and letting the model automatically search for a suitable structure configuration to adapt to the hardware device according to different hardware characteristics, is a feasible route. The automatic search of the embodiment mainly studies the definition of the search space, the development of the search strategy and the model evaluation of the three aspects.

[0088] S1-3-1, modular search space

[0089] The embodiment designs five automatic search blocks, of which the first three automatic search blocks constitute the feature extraction part of the NASCNet, and the last two automatic search blocks replace the first two residual blocks in the encoder of the NASCNet. Each search block includes a plurality of automatic residual blocks, and the automatic residual block is the smallest automatic search module. The structure of the automatic residual block is as shown in Figure 4 The DynamicConv2d module in the figure performs feature extraction, the DynamicSeparableConv2d module dynamically changes the convolution kernel size and width, and the lightweight DynamicSE module is used to improve the sensitivity of the model to channel features. The four dimensions of the convolution kernel size, the model depth, the model width and the search scale are searched. The model depth determines the number of automatic residual blocks in each search block, and the search scale determines the number of search blocks (it is verified that the search scale is fixed to 3+2, that is, five search blocks, and the model performs better in this way).

[0090] S1-3-2, progressive search strategy

[0091] The progressive search refers to the training sequence from a large network to a small subnetwork. The advantage of this is that the model can directly use the model parameters obtained after the last training to continue training. Compared with other search strategies, the progressive shrinking search strategy can not start training again, greatly reducing the training cost. The automatic search process is as shown in Figure 5 First, the largest network (called super network) is trained, that is, the convolution kernel size, the model depth, the model width and the search scale are set to the maximum value in the selectable range. Then, the automatic search of the convolution kernel size, the model depth and the model width is performed in turn.

[0092] a. Convolution kernel / deconvolution kernel search:

[0093] The present case learns from the dynamic search of the convolution kernel and realizes the dynamic search of the deconvolution kernel, which is collectively referred to as the convolution kernel below. When performing convolution kernel search training, the model loads the model of the hypernetwork that has been trained. Since the convolution kernel size changes during training, the weights within the convolution kernel have different distributions and sizes under different conditions. When the convolution kernel changes from 7 to 5 or from 5 to 3, directly forcing the smaller kernel from the larger kernel will reduce the performance of the large subnet, so a kernel transformation matrix is introduced when sharing the convolution kernel weight values, as shown in formula (1). Figure 6 In this way, the weight values of the larger kernel can be mapped to the weight values of the smaller kernel according to the transformation matrix, and the weight values within the convolution kernel are more utilized. Different layers have different convolution kernel sizes, so a corresponding kernel transformation matrix should be used for each layer, and the convolution kernel transformation matrix in each layer can be shared between different channels.

[0094] B. Depth search;

[0095] After performing convolution kernel search training, the depth is searched. Assuming that the maximum depth in the hypernetwork is N, and the depth of the current search block is set to D during training, the model retains the first D layers and skips the last N-D layers, as shown in formula (2). Figure 7 The model weights of the first D layers are shared between the hypernetwork and the subnet.

[0096] C. Width search;

[0097] After performing depth search training, the width is searched. Each layer of the model can flexibly select different channel expansion ratios during the search. According to the scheme of the present embodiment, after loading the model of the hypernetwork, the channels are reorganized according to their importance, i.e., the channels are sorted and processed. The importance of the channels is calculated according to the L1-norm algorithm. For example, when the number of channels is reduced to four, the channels of the large subnet are sorted according to the L1-norm and the top four channels are selected in descending order of importance, as shown in formula (3). Figure 8 This ensures that the subnet retains more accuracy of the larger network.

[0098] S1-4. Based on the trained binocular depth estimation model, the binocular depth of the image to be detected is estimated.

[0099] It can be understood that model evaluation occupies most of the time of neural architecture search. In order to reduce time consumption, the embodiment uses a subset of the dataset to evaluate in the model evaluation stage, because only the relative performance ranking of the subnetwork is needed for all the subnetworks sampled in each round, and the accurate accuracy is not needed to traverse the entire dataset. After training the model, the embodiment sets different network structures to verify the accuracy of the model on purpose, and records the models with good performance in the table, and continues to feed data to train the models with poor performance.

[0100] After obtaining the evaluated binocular depth estimation model, the binocular image can be processed by using the evaluated model.

[0101] Embodiment 2

[0102] In a specific embodiment, the embodiment 2 is a binocular depth estimation method based on automatic network compression, including the following steps:

[0103] S2-1, obtaining left view, right view and corresponding target disparity map from the database; ColorJitter(brightness=0.4,contrast=0.4,saturation=0.4,);{'mean':[0.485,0.456,0.406],'std':[0.229,0.224,0.225]};input picture height and width are [h,w]=[384,768];

[0104] S2-2, constructing a binocular depth estimation model as shown in Figure 1 The model includes two parts of NASCNet network and NASSNet network. The inputs of the two network structures are different, and the internal structure will be dynamically changed during training. The input of NASCNet is the image pair in step S2-1, and the input of NASSNet network is a set of disparity maps generated by NASCNet network. The final disparity map set generated by the two parts of network is added to the target disparity map to calculate the loss, and then added as the final loss. Finally, the largest size in the disparity map set generated by NASSNet is used to calculate EPE with Groundtruth (true value);

[0105] S2-3, the embodiment uses NAS and progressive search strategy during training. The macroscopic process is to first train the super network, which is the network with the largest parameters {convolution kernel=7, search depth=4, search width=8}. After the super network training is completed, the algorithm of progressive shrinkage is used to train the convolution kernel size, depth and width in turn. Each kind of training has 80 epochs, and the loss weight is updated according to the following table 1.

[0106] Table 1

[0107] Round w0 w1 w2 w3 w4 w5 w6 1 0.4 0.20 0.10 0.05 0.02 0.01 0.005 2 0.6 0.30 0.04 0.02 0.02 0.01 0.005 3 0.8 0.08 0.04 0.10 0.01 0.005 0.0025 4 1.0 0 0 0 0 0 0

[0108] S2-4, first in the decoding structure of NASCNet ( Figure 1 In the feature extraction in [1], the input is extracted through 6 search blocks, each search block contains (search depth) residual blocks, and the residual block structure is as follows Figure 4 As shown, the search block completes the dynamic training of the convolution kernel, depth and width. After each search block, the feature map will generate two feature maps (left_fea and right_fea). The point correlation block will extract the relationship between the two feature maps and then splice them with left_fea. After that, the feature extraction process is performed by the ResBlock residual block. The final generated intermediate value corr_feas will be used in the subsequent decoding structure ( Figure 1 The resolution recovery structure in the algorithm is involved in the calculation. (A search block, a point correlation block, and a residual block can be considered as a group, with a total of six groups. Each group will generate left_fea and right_fea for the next group to use, and will also generate corr_feas for the decoding structure to use).

[0109] S2-5, Similarly, the resolution extraction in the NASSNet network contains 6 deconvolution search blocks ( Figure 4 The bottom_fea generated by each search block will be concatenated with the corr_fea and then passed through a deconvolution layer with a step size of 1 as the input of the next block. Each search block will perform a convolution layer on the corr_feas to generate the final disp. The last search block will perform a convolution on the result to get last_disp. In this way, NASCNet generates a total of 7 disp with different sizes ( Figure 1 Center left disparity map C).

[0110] S2-6: The NASSNet network receives the feature values ​​from the NASCNet network, inverts them with the right view, and then concatenates them with the left view as the input to the NASSNet. Following the same steps as for the NASCNet, a left disparity map S is generated. The loss is calculated as in step 2-2. This completes the training of the supernet and saves the model for subsequent progressive training.

[0111] S2-7, preferentially search the convolution kernel, {convolution kernel\deconvolution kernel = [3, 5, 7], depth = 4, width = 8}, the training process is the same as the above steps, except that in each iteration of each epoch, the size of the convolution kernel / deconvolution kernel is randomly changed, selected from [3, 5, 7], the change of the network structure is reflected in the DynamicSeparableConv2d in the search block, this module can support the operation of changing the size of the convolution kernel, when the convolution kernel changes, the stride, padding, etc. will change to ensure that the size of the output feature map is unchanged, the three different convolution kernels (3, 5, 7) of each layer are registered to the Parameter to become trainable parameters. Load the model of the super network, so that after each epoch training is completed, the accuracy is compared, and the model with good performance is saved.

[0112] S2-8, after the convolution kernel search training is completed, the depth training is performed, the depth value is [2, 3, 4], that is, in each iteration of each epoch, the size of the depth is randomly changed on the basis of the convolution kernel search, that is, the selectable range is {convolution kernel\deconvolution kernel = [3, 5, 7], depth = [2, 3, 4], width = 8}, the depth change is reflected in the number of residual blocks in the search block, which is at least 2 and at most 4. Load the model saved after the convolution kernel search training, and after each epoch training is completed, the accuracy is compared, and the model with good performance is saved.

[0113] S2-9, search the width, here the width refers to the channel multiplier, such as when the initial channel is 32 and the width is 8, the number of channels in the search block is 32*8. The width value is [2, 4, 6, 8], the selectable range is {convolution kernel\deconvolution kernel = [3, 5, 7], depth = [2, 3, 4], width = [2, 4, 6, 8]}, load the model saved after the depth search training, and after each epoch training is completed, the accuracy is compared, and the model with good performance is saved.

[0114] S2-10, finally, according to the difference of the edge device, select a suitable structure of the sub network model for inference verification. The selection here is according to the network structure kernel accuracy obtained in the training stage to make a lookup table, different network structures represent different computing power edge devices. When a specific sub model is obtained, the inference accuracy will be higher after fine tuning.

[0115] Embodiment 3

[0116] This embodiment 3 uses SceneFlow dataset, SceneFlow is proposed in CVPR 2016, the purpose is to build a large-scale synthetic dataset for training binocular depth estimation network, SceneFlow dataset is captured from real scenes and includes various indoor and outdoor scenes. The image pair is captured using a multi-camera system, including 39824 pairs of images (35454 pairs of training images and 4370 test images) with a resolution of 960x540 pixels, each pair of images includes a left view, a right view and a corresponding disparity / flow map, for each image pair, SceneFlow dataset provides pixel-level disparity and flow annotations. Disparity represents the horizontal displacement of pixels in the left view relative to the right view, and flow represents the pixel motion between the two views.

[0117] GPUs: Tesla V100x4; Due to the power and time limit, it takes 15 gpuh to train the entire super network on the full dataset for 20 epochs, so this experiment data is randomly taken from SceneFlow, 6000 for training and 1000 for testing, and each training runs for 10 epochs.

[0118] Evaluation index: If the disparity or flow endpoint error |d est -d gt |<3px or is considered correct. The flow endpoint error EPE of the entire region is:

[0119] For building binocular depth estimation model hyperparameters:

[0120] This embodiment 3 is implemented on PyTorch, the optimization method is Adam (β1=0.9, β2=0.999), the learning rate is 0.004, the learning rate is halved every 10 rounds, the color normalization is set to mean=[0.485, 0.456, 0.406], variation=[0.229, 0.224, 0.225]. When training, randomly crop the picture H=384, W=768, batchsize=4, the loss weight is set to [w0, w1, w2, w3, w4, w5, w6, w7]=[0.40, 0.20, 0.10, 0.05, 0.02, 0.01, 0.005], wherein the weight value is assigned to the corresponding disparity map from left to right, the resolution decreases in turn, 4 Tesla V100 GPUs are used, and the training round is 10 epochs.

[0121] Table 2 shows the experimental results of applying residual structure and point correlation layer, without automatic search, wherein searchrange D represents the disparity search threshold when the cost is aggregated, and it can be seen that when the disparity search threshold is fixed at 20, the EPE of the model of the present embodiment 3 is lower than the EPE of DispNetC, indicating that the model of the present embodiment performs better than DispNetC, and when different disparity search thresholds are tried, it is found that there is no positive correlation between the EPE and the disparity search threshold, for example, in the table, when D = 10 and D = 20, the EPE is lower than when D = 30, which is presumably because the model after training is over-fitted or under-fitted, and therefore finding a suitable disparity search threshold can improve the accuracy of the model.

[0122] Table 2 Model accuracy improved by residual structure and point correlation

[0123]

[0124] Table 3 shows the accuracy and memory occupation of the present model and the current SOTA binocular depth estimation model on SceneFlow, including 2D and 3D based networks, and it can be seen from Table 3 that other SOTA models cannot balance model accuracy and model size, and only one of the two can be satisfied. The model performance of DispNetC is slightly better than that of the model of the present embodiment, but the present embodiment model has only run for 10 epochs, and in the future, after optimizing the loss weight distribution and increasing the training number of rounds, the accuracy can exceed that of DispNetC.

[0125] Table 3 Comparison of model accuracy and memory occupation

[0126]

[0127] The experimental results of the present model and DispNetC on SceneFlow are compared as shown in Table 4.

[0128] Table 4 Performance of the model

[0129]

[0130] The model built in the present embodiment 3 has a memory occupation of 2+ times that of DispNetC, and a reasoning time of 2.5+ times that of DispNetC, but the EPE is lower than that of DispNetC. The experimental results show that the model accuracy has been significantly improved, and the preliminary requirements have been met.

[0131] The hyperparameters for model training are set as follows:

[0132] After adding automatic search, the model supports searching in four dimensions of convolution kernel size, model depth, model width, and model size. The candidate values of the model convolution kernel size are [3, 5, 7], the candidate values of the model depth are [2, 3, 4], the candidate values of the model width are [2, 4, 6, 8], and the candidate values of the model size are [2, 3, 4, 5]. The model first trains the super network, and the structure configuration of the super network selects the maximum value in the above candidate values (it is verified through experiments that the model size of 3 is the most appropriate), and after training the super network, the search training of the convolution kernel size, the search training of the model depth, and the search training of the model width are performed in turn. Specifically, when the convolution kernel is automatically searched and trained, all candidate values of the convolution kernel size are opened, but the network depth, the network width, and the kernel size remain consistent with the super network configuration. When the network depth is automatically searched, the candidate values of the convolution kernel size and the network depth are all opened, and the network width and the size remain consistent with the automatic search of the convolution kernel size. In this way, the search training of the network width is performed. In addition, when performing automatic search training, a number of sub-networks are randomly sampled before each iteration, and their average loss is updated. This has the benefit of increasing the robustness of the sub-networks. After training, the model purposefully selects a number of sub-networks for verification, and records the models with good performance in the table. The optimization method is selected as Adam (β1 = 0.9, β2 = 0.999), the learning rate is 0.004, the learning rate is halved every 10 rounds, the color normalization is set to mean = [0.485, 0.456, 0.406], and variation = [0.229, 0.224, 0.225]. When training, the picture is randomly cropped to H = 384, W = 768, the batch size is 4, the loss weight is set to [w0, w1, w2, w3, w4, w5, w6, w7] = [0.40, 0.20, 0.10, 0.05, 0.02, 0.01, 0.005], wherein the weight values are assigned to the corresponding resolution disparity maps from left to right in order of decreasing resolution, and 4 Tesla V100 GPUs are used for training, and the training rounds are 10 epochs.

[0133] Table 5 shows the model accuracy after super network search, wherein the convolution kernel is set to 7, the network depth is set to 4, the network width is set to 8, and the search size is set to 3. From the table, it can be seen that the super network has very good prediction ability, and as the number of training rounds increases, the super network can achieve lower accuracy than in the table.

[0134] Table 5 shows the model accuracy after super network search, wherein the convolution kernel is set to 7, the network depth is set to 4, the network width is set to 8, and the search size is set to 3. From the table, it can be seen that the super network has very good prediction ability, and as the number of training rounds increases, the super network can achieve lower accuracy than in the table.

[0135]

[0136] Table 6 shows the automatic search after training the super network, search training of the convolution kernel size. The convolution kernel size setting range is [3, 5, 7], and a random sampling search strategy is applied before each iteration of each round, that is, a random convolution kernel candidate value is set for each dynamic layer of the search block. The resulting model convolution kernel sequence is completely random. When the batchszie is set to 4 and the epoch is set to 10, a total of 3750 subnetworks can be sampled for training. This can increase the robustness of the subnetwork.

[0137] Table 6 Elastic kernel search experiment results

[0138]

[0139] Where config represents the network configuration, and the values from left to right represent the convolution kernel, network depth, network width, and search size setting (the same below). After n times of training, it is found that when the convolution kernel is set to 7, the accuracy is best, and when the convolution kernel is 3, the model EPE is high. It is guessed that this may be because the network with a convolution kernel of 3 is not trained during sampling, so it does not have good performance.

[0140] Table 7 shows the automatic search network depth training results after automatic search. The depth of the search block is set to [2, 3, 4], and the method is the same as when training the elastic convolution kernel. Only part of the subnetwork configuration after automatic search of the network depth is listed in the table. From the table, it can be seen that the accuracy of the super network structure configuration is best, and the change of the network depth has little effect on the experimental results, but this conclusion still needs to be verified, because only 10 epochs are trained and the loss function weight scheduling can be optimized. Increasing the loss weight of the larger size disparity map will make the model have better performance.

[0141] Table 7 Elastic depth search experiment results

[0142]

[0143] Table 8 shows the training results of the automatic search of the network width after the automatic search of the network depth. The width of the search block is set to [2, 4, 6, 8], and before the width search, the channels are sorted according to importance by L1-smooth, then searched and trained. From the table, it can be seen that after the automatic search of the network width, the model accuracy is further improved.

[0144] Table 8 Elastic width search experiment results

[0145]

[0146] It is verified by experiments that the NAS effect is best when the scale is fixed as 3, so the search for scale is not described again. Finally, this embodiment samples several groups of optimal structure configurations as shown in Table 9.

[0147] Table 9 NAS optimal configuration and experimental results

[0148]

[0149] Finally, the precision of the model of embodiment 3 and the SOTA model is shown in Table 10. Among them, the precision of the model of the embodiment has been far ahead of other models except GANet, which is a 3D convolution method, and the cost is to occupy a large amount of memory.

[0150] Table 10

[0151]

[0152]

[0153] The final result is shown in Table 11, where OOM (Out of Memory) refers to memory overflow, and runtimes refers to the average time used for 50 times of inference. It can be seen that the method proposed in the case can well balance the precision and speed, and also takes into account the memory occupation, solving the problem that large models cannot be deployed on edge computing devices.

[0154] Table 11

[0155]

[0156] It should be noted that for each of the above method embodiments, in order to facilitate description, it is expressed as a combination of a series of actions, but those skilled in the art should know that the present application is not limited by the order of the described actions, because according to the present application, certain steps can be performed in other order or simultaneously.

[0157] Based on the same idea as the automatic network compression based binocular depth estimation method in the above embodiment, the present application also provides an automatic network compression based binocular depth estimation system, which can be used to execute the above automatic network compression based binocular depth estimation method. For the convenience of description, in the structure diagram of the embodiment of the automatic network compression based binocular depth estimation system, only the part related to the embodiment of the present application is shown, and those skilled in the art can understand that the structure diagram does not constitute a limitation on the device, and can include more or fewer components than the diagram, or combine certain components, or different component arrangements.

[0158] Please refer to Figure 9In another embodiment of the present application, a binocular depth estimation system 100 based on automatic network compression is provided, which comprises an image acquisition module 101, a model construction module 102, a model training module 103 and a depth estimation module 104;

[0159] The image acquisition module 101 is configured to acquire left and right views and acquire left and right disparity maps corresponding to the left and right views respectively.

[0160] The model construction module 102 is configured to construct a binocular depth estimation model, wherein the binocular depth estimation model comprises a NASCNet network and a NASSNet network, a residual block is added to the feature extraction module of the NASCNet network instead of the convolution layer in DispNetC, and a point correlation layer is used to optimize the feature extraction operation in DispNetC; a predicted disparity map generated by the NASCNet network is added to the input part of the NASSNet network; the input of the NASCNet network is an image pair of the left and right views, and the NASSNet network has an additional set of disparity maps generated by the NASCNet network; the final disparity map set generated by the NASCNet network and the NASSNet network is added to the target disparity map to calculate the loss loss and then added as the final loss, and the largest size in the disparity map set generated by the NASSNet network is used to calculate the EPE with the true value.

[0161] The model training module 103 is configured to train the binocular depth estimation model by using a progressive search strategy, first train the network with the largest model structure, set the search kernel size, model depth, model width and search scale to the maximum value in the optional range, and then sequentially search the kernel size, model depth and model width.

[0162] The depth estimation module 104 is configured to perform binocular depth estimation on the image to be detected based on the trained binocular depth estimation model.

[0163] It should be noted that the binocular depth estimation system based on automatic network compression of the present application corresponds to the binocular depth estimation method based on automatic network compression of the present application, and the technical features and advantages described in the embodiment of the above binocular depth estimation method based on automatic network compression are applicable to the embodiment of the binocular depth estimation based on automatic network compression, and the specific content can be referred to the description in the method embodiment of the present application. Here, no further description is given, and this is declared.

[0164] Further, in the implementation of the binocular depth estimation system based on automatic network compression of the above-mentioned embodiments, the logical division of each program module is only illustrative. In actual applications, the above-mentioned functions can be completed by different program modules according to needs, for example, for the configuration requirements of corresponding hardware or the convenience of software implementation. That is, the internal structure of the binocular depth estimation system based on automatic network compression is divided into different program modules to complete all or part of the above-described functions.

[0165] Please refer to Figure 10 In one embodiment, an electronic device implementing a binocular depth estimation method based on automatic network compression is provided. The electronic device 200 can include a first processor 201, a first memory 202, and a bus. The electronic device 200 can also include a computer program, such as a binocular depth estimation program 203 based on automatic network compression, stored in the first memory 202 and executable on the first processor 201.

[0166] The first memory 202 includes at least one type of readable storage medium, including a flash memory, a mobile hard disk, a multimedia card, a card-type memory (e.g., an SD or DX memory, etc.), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the first memory 202 can be an internal storage unit of the electronic device 200, such as a mobile hard disk of the electronic device 200. In other embodiments, the first memory 202 can also be an external storage device of the electronic device 200, such as a plug-in mobile hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the first memory 202 can include both an internal storage unit and an external storage device of the electronic device 200. The first memory 202 can be used not only to store application software and various data installed in the electronic device 200, such as the code of the binocular depth estimation program 203 based on automatic network compression, but also to temporarily store data that has been or will be output.

[0167] The first processor 201 may, in some embodiments, be composed of integrated circuits, for example, may be composed of a single packaged integrated circuit, or may be composed of multiple packaged integrated circuits of the same function or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips, etc. The first processor 201 is the control unit of the electronic device, connects various components of the entire electronic device through various interfaces and lines, and executes programs or modules stored in the first memory 202 and calls data stored in the first memory 202 to perform various functions and process data of the electronic device 200.

[0168] Figure 10 Only the electronic device with components is shown, and those skilled in the art can understand that, Figure 10 The structure shown does not constitute a limitation on the electronic device 200, and can include fewer or more components than shown, or combine certain components, or different component arrangements.

[0169] The automatic network compression-based binocular depth estimation program 203 stored in the first memory 202 in the electronic device 200 is a combination of multiple instructions, which, when running in the first processor 201, can achieve:

[0170] Obtaining left and right views, and obtaining left and right parallax maps corresponding to the left and right views, respectively;

[0171] Building a binocular depth estimation model, the binocular depth estimation model including a NASCNet network and a NASSNet network, a residual block is added to the feature extraction module of the NASCNet network instead of the convolution layer in DispNetC, and a point correlation layer is used to optimize the feature extraction operation in DispNetC; the NASSNet network has an input part that is added to the predicted parallax map generated by the NASCNet network; the input of the NASCNet network is an image pair of left and right views, and the NASSNet network has an additional set of parallax maps generated by the NASCNet network; the final parallax map sets generated by the NASCNet network and the NASSNet network are added to the target parallax map to calculate the loss loss and then added as the final loss, and the largest size in the parallax map set generated by the NASSNet is used to calculate the EPE with the true value;

[0172] The progressive search strategy is used to train the binocular depth estimation model. First, the network with the largest model structure is trained. The convolution kernel size, model depth, model width and search scale are set to the maximum value in the optional range. Then, the automatic search is performed on the convolution kernel size, model depth and model width in turn.

[0173] Based on the trained binocular depth estimation model, binocular depth estimation is performed on the image to be detected.

[0174] Further, the modules / units of the electronic device 200, if implemented in the form of software function units and sold or used as independent products, can be stored in a non-volatile computer readable storage medium. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM).

[0175] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware. The program can be stored in a non-volatile computer readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiments. In the embodiments provided in the present application, any reference to memory, storage, database or other medium can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM) and memory bus dynamic RAM (RDRAM).

[0176] The technical features of the above embodiments can be combined in any way. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of the technical features does not exist, it should be considered as the scope of the present application.

[0177] The above embodiments are the preferred embodiments of the present application, but the embodiments of the present application are not limited to the above embodiments, and any changes, modifications, substitutions, combinations, simplifications, etc. made without departing from the spirit and principles of the present application should be equivalent replacement manners and should be included in the protection scope of the present application.

Claims

1. A binocular depth estimation method based on automatic network compression, characterized in that: The steps include: Obtain the left view and the right view, and obtain the left disparity map and the right disparity map corresponding to the left view and the right view respectively; A binocular depth estimation model was constructed, comprising a NASCNet network and a NASSNet network. Residual blocks were added to the feature extraction module of the NASCNet network to replace the convolutional layers in DispNetC, and point correlation layers were used to optimize the feature extraction operations in DispNetC. A predicted disparity map generated by NASCNet was added to the input of the NASSNet network. The input of the NASCNet network is an image pair of the left view and the right view, and the NASSNet network has an additional set of disparity maps generated by the NASCNet network; the final disparity map groups generated by the NASCNet network and the NASSNet network are added to the target disparity map respectively to calculate the loss loss and then added as the final loss. Finally, the largest size and the true value of the disparity map group generated by NASSNet are used to calculate EPE; A progressive search strategy is used to train the binocular depth estimation model. First, the network with the largest model structure is trained. The search convolution kernel size, model depth, model width, and search scale are set to the maximum value within the optional range. Then, the convolution kernel size, model depth, and model width are automatically searched in sequence. Perform binocular depth estimation on the image to be detected based on the trained binocular depth estimation model; The point correlation layer is used to optimize the feature extraction operation in DispNetC, specifically: Set the constrained moving range D, and each patch only needs to be calculated with the nearby 2D+1 patches. The calculation formula is shown in Equation (1), and the patch in DispNetC is removed; Where c represents the point-related cost, function f represents the pixel value of the point, and "<>" represents the element-wise multiplication of the points in the left and right images on the x-axis (-D, D). This operation is named point-related calculation, and the corresponding layer is named point-related layer. The number of multiplications required by NASCNet is C×K×H×W×(2D+1), thereby reducing the number of multiplications. A total of five automatic search blocks are designed in the progressive search strategy. The first three automatic search blocks constitute the feature extraction part of the NASCNet network, and the last two automatic search blocks replace the first two residual blocks in the encoder of the NASCNet network. Each automatic search block includes several automatic residual blocks. The automatic residual block is the smallest automatic search module. The DynamicConv2d and DynamicTransposeConv2d modules are used in the automatic residual block for feature extraction. The DynamicSeparableConv2d and DynamicSeparableTransposeConv2d modules dynamically change the convolution kernel size and width. The lightweight DynamicSE module is used to improve the model's sensitivity to channel features.

2. The binocular depth estimation method based on automatic network compression according to claim 1, characterized in that The binocular depth estimation model also includes improvements to the loss function, specifically: At the beginning of the training of the binocular depth estimation model, the loss of all disparity maps is enabled, and the disparity is predicted. The disparity d from the true value s The loss function between is defined as formula (3); Where N is the number of pixels in each disparity map, Represents the i-th element of the s-th disparity map, The i-th element of the s-th disparity map representing the true value, L s represents the pixel-by-pixel smoothing smoothL1 loss between the predicted disparity map and the true value, as shown in formula (4); The disparity map generated by the model corresponds to different losses The final loss is calculated according to formula (5): where w s Represents the weight value of the loss of the s-th disparity map.

3. The binocular depth estimation method based on automatic network compression according to claim 1, characterized in that During convolution kernel search training, the binocular depth estimation model loads the trained hypernetwork model. Since the convolution kernel size changes during training, the weights within the convolution kernel have different distributions and sizes in different situations. Therefore, a kernel transformation matrix is ​​introduced when sharing the convolution kernel weight values. The weight values ​​of the larger kernel are mapped to the weight values ​​of the smaller kernel according to the transformation matrix, making better use of the weight values ​​within the convolution kernel. The convolution kernel sizes of different layers are different, so the corresponding kernel transformation matrix should be used for each layer. The convolution kernel transformation matrix in each layer can be shared between different channels.

4. The binocular depth estimation method based on automatic network compression according to claim 1, characterized in that When performing model depth search, assuming that the maximum depth in the super network is N, and assuming that the depth of the current search block is set to D during training, the model retains the first D layers and skips the following ND layers. The model weights of the first D layers are shared between the super network and the sub-network.

5. The binocular depth estimation method based on automatic network compression according to claim 1, characterized in that: When searching for model width, channels are reorganized according to their importance, that is, they are sorted. The importance of channels is calculated using the L1-norm algorithm.

6. Binocular depth estimation system based on automatic network compression, characterized by: A binocular depth estimation method based on automatic network compression applied to any one of claims 1-5, comprising an image acquisition module, a model construction module, a model training module and a depth estimation module; The image acquisition module is used to obtain a left view and a right view, and obtain a left disparity map and a right disparity map corresponding to the left view and the right view respectively; The model construction module is used to build a binocular depth estimation model, which includes a NASCNet network and a NASSNet network. A residual block is added to the feature extraction module of the NASCNet network to replace the convolution layer in DispNetC, and a point correlation layer is used to optimize the feature extraction operation in DispNetC; the predicted disparity map generated by NASCNet is added to the input part of the NASSNet network; The input of the NASCNet network is an image pair of the left view and the right view, and the NASSNet network has an additional set of disparity maps generated by the NASCNet network; the final disparity map groups generated by the NASCNet network and the NASSNet network are added to the target disparity map respectively to calculate the loss loss and then added as the final loss. Finally, the largest size and the true value of the disparity map group generated by NASSNet are used to calculate EPE; The model training module is used to train the binocular depth estimation model using a progressive search strategy. First, the network with the largest model structure is trained. The convolution kernel size, model depth, model width, and search scale are set to the maximum values ​​within the optional range. Then, the convolution kernel size, model depth, and model width are automatically searched in sequence. The depth estimation module is used to perform binocular depth estimation on the image to be detected based on the trained binocular depth estimation model.

7. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor so that the at least one processor can execute the binocular depth estimation method based on automatic network compression as described in any one of claims 1-5.

8. A computer-readable storage medium storing a program, characterized in that: When the program is executed by a processor, the binocular depth estimation method based on automatic network compression according to any one of claims 1 to 5 is implemented.

Images