Lightweight depth estimation method and system

Abstract

The invention discloses a lightweight depth estimation method and system, and belongs to the technical field of binocular system depth estimation, and the method comprises the steps: obtaining a depth estimation model, inputting a binocular depth image into the depth estimation model, obtaining a depth estimation result, determining the binocular depth image as an input parameter when the depth estimation model is obtained, and obtaining the depth estimation result; the method comprises the following steps: carrying out feature extraction on an input parameter based on two-dimensional depth separable convolution, obtaining a cost body based on the extracted feature parameter, carrying out stacking hourglass processing on the cost body based on three-dimensional point-by-point convolution, obtaining multi-depth feature information, and determining the multi-depth feature information as a depth estimation result, namely, an output parameter. According to the method, lightweight feature extraction modes such as two-dimensional depth separable convolution and three-dimensional point-by-point convolution are fused, the number of layers of feature extraction can be reduced, the parameter quantity of a depth estimation model can be reduced, and the technical problems that a binocular depth estimation network is difficult to deploy in an actual application scene, the memory utilization rate is low, and hardware resources are wasted are solved.

Description

Technical Field

[0001] The present application belongs to the technical field of binocular system depth estimation, and specifically relates to a lightweight depth estimation method and system. Background Art

[0002] With the development of artificial intelligence, application fields such as autonomous driving, drones, and robots have received more and more attention, and environmental perception is the core technology. Depth estimation obtains depth information by reconstructing the three-dimensional information of the environment, thereby realizing environmental perception, and is an important research direction in the field of computer vision. Traditional methods of obtaining depth information come from laser radar, monocular camera imaging, etc., which have problems such as insufficient image features and only applicable to close-range scenes. In recent years, convolutional neural networks, as an important part of deep learning, have made major breakthroughs in image fields such as target detection and target recognition. They can extract rich feature information from images and overcome the shortcomings of traditional algorithms.

[0003] In the past few years, researchers have proposed some lightweight models by improving the convolutional layers of neural networks. In 2016, a team proposed the MobileNet model (mobile network model), which selected depthwise separable convolution instead of traditional convolution, decomposed the convolution calculation into two steps: depthwise convolution and pointwise convolution, and introduced the width multiplier as a hyperparameter to select a model of appropriate size according to the actual application scenario. Compared with VGG16 (Visual Geometry Group Network with 16 layers), this new convolution method of lightweight network reduces the amount of multiplication and addition calculations by 27 times and the number of parameters by 33 times while losing less accuracy, thereby effectively compressing the model size. However, the network model has a simple structure and does not reuse image features for feature fusion. Subsequently, a more efficient network model was proposed. Chang et al. proposed the PSMNet (Pyramid Stereo Matching Network) network. In the feature extraction part, a deep residual neural network (ResNet) was used as the backbone network, and then the SPP (Spatial Pyramid Pooling) spatial pyramid structure was used to obtain feature information under receptive fields of different scales, and the global information and local information were combined to form a matching cost volume.

[0004] However, as the requirements for accuracy in actual application scenarios become higher and higher, the large-scale parameters and high-complexity calculations of neural networks pose challenges to the storage, power consumption and computing power of devices. In resource-constrained mobile applications such as smart wearable devices, drones and smart robots, neural networks will quickly exhaust their storage, memory, battery and computing units. Therefore, reducing the parameter scale and computational complexity of neural networks is of great significance for deploying neural networks in mobile applications. Summary of the invention

[0005] Purpose of the invention: This application develops a lightweight depth estimation method and system, aiming to solve the technical problems in the prior art of the binocular depth estimation network, such as the difficulty in deploying in actual application scenarios, low memory utilization, and wasted hardware resources.

[0006] Technical solution: In a first aspect, an embodiment of the present application provides a lightweight depth estimation method, comprising:

[0007] Acquire a depth estimation model, input the binocular depth image into the depth estimation model, and obtain a depth estimation result, wherein the step of acquiring the depth estimation model includes:

[0008] Determining input parameters, wherein the input parameters include a binocular depth image;

[0009] Performing feature extraction on the input parameter based on two-dimensional depth-separable convolution to obtain feature parameters;

[0010] Obtaining a cost volume through the characteristic parameters;

[0011] Performing stacked hourglass processing on the cost volume based on three-dimensional point-by-point convolution to obtain multi-depth feature information;

[0012] An output parameter is determined, where the output parameter includes multiple depth feature information, and the multiple depth feature information is used to characterize a depth estimation result.

[0013] In some embodiments, the step of extracting features from the input parameters based on two-dimensional depth-separable convolution to obtain feature parameters includes:

[0014] Performing a multi-layer two-dimensional depth-separable convolution operation on the binocular depth image, and performing nonlinear processing using a rectified linear unit activation function after each layer of the two-dimensional depth-separable convolution operation to obtain a first feature map;

[0015] Processing the first feature map through a multi-layer residual network to obtain a second feature map;

[0016] Processing the second feature map through a fast spatial pyramid pooling layer to obtain a multi-channel feature map that integrates multiple scale features as a third feature map;

[0017] The information of the third feature map is fused through a pixel-by-pixel convolution layer to obtain feature parameters.

[0018] In some embodiments, the binocular depth image includes a left view and a right view, and the step of obtaining a cost volume through the feature parameter includes:

[0019] The difference data of the feature parameters of the left view and the right view corresponding to each pixel are obtained, and matching is performed by moving the window, and the left view and the right view are respectively cropped and matched with another view to obtain the cost volume.

[0020] In some embodiments, the step of performing stacked hourglass processing on the cost volume based on three-dimensional point-by-point convolution to obtain multi-depth feature information includes:

[0021] Multiple hourglass networks are stacked, in which:

[0022] Performing a lightweight three-dimensional point-by-point convolution operation on the cost volume to reduce the resolution of the cost volume and obtain a preliminary prediction result;

[0023] A three-dimensional deconvolution operation is performed on the preliminary prediction result to perform upsampling to obtain the depth feature information.

[0024] In some embodiments, the depth estimation method further comprises:

[0025] Performing model training on the depth estimation model based on the depth estimation result, wherein the model training includes a classification task and a regression task;

[0026] Obtaining a depth map label, and obtaining a classification label and a regression label based on the depth map label;

[0027] Based on the classification label and the classification task, obtaining a classification loss;

[0028] Obtaining a regression loss based on the regression label and the regression task;

[0029] Determine a total loss function based on the classification loss and the regression loss;

[0030] The depth estimation model is optimized based on the total loss function.

[0031] In some embodiments, the step of obtaining a classification label based on the depth map label includes:

[0032] Performing clustering processing on the depth map labels to obtain cluster centers;

[0033] Based on the cluster centers and the disparity classification, obtaining a category map;

[0034] The classification label is obtained based on the category map.

[0035] In some embodiments, the steps of the classification task include:

[0036] Processing the depth estimation result by multi-layer pyramid three-dimensional lightweight convolution and multi-layer two-dimensional convolution to obtain a disparity estimation map;

[0037] The steps of the regression task include:

[0038] Channel fitting is performed on the disparity estimation map through multi-layer pyramid three-dimensional lightweight convolution and multi-layer two-dimensional convolution, and a continuous disparity map is obtained based on disparity regression.

[0039] In some embodiments, the stacked hourglass processing is performed by a multi-layer serially stacked hourglass network, and the step of determining the total loss function based on the classification loss and the regression loss includes:

[0040] In the multi-layer hourglass network,

[0041] The output of each hourglass network is determined using a smoothed absolute error loss function, and a weight coefficient is assigned; wherein the output of the hourglass network with a greater depth is assigned a greater weight coefficient;

[0042] The output of each hourglass network is subjected to a cross entropy loss function to determine the classification loss and to assign a weight coefficient; wherein the output of the hourglass network with a greater depth is assigned a greater weight coefficient;

[0043] For the weight coefficient allocation between classification loss and regression loss, dynamic coefficient allocation is used. When the number of optimization iterations is less than the iteration threshold, only the classification loss is used as the total loss function; when the number of iterations is greater than or equal to the iteration threshold, the total loss function is determined based on the function after the weight coefficients are allocated based on the classification loss and regression loss.

[0044] In some embodiments, the lightweight depth estimation method further includes: performing linear quantization, weight clipping optimization, and activation clipping optimization on the depth estimation model to optimize model storage requirements, and the linear quantization step includes:

[0045] The single-precision floating-point data is linearly mapped to the data of the specified bit. The characterization formula includes:

[0046] ;

[0047] in, For floating point data that needs to be quantized; For quantitative scale; To quantify zero; The bit width of the data is specified in bits; For rounding operations; is the truncation operation; is the maximum clipping threshold; is the minimum clipping threshold; q is the integer after quantizing the initial target parameter;

[0048] The steps of weight clipping optimization include:

[0049] Determining a target convolution kernel in the depth estimation model;

[0050] Initializing particles in the depth estimation model based on a particle swarm algorithm and determining position range parameters of the particles;

[0051] Determining the position and velocity of the particle based on the target convolution kernel and the position range parameter;

[0052] Determine the quantization deviation of the position, and determine that the position corresponding to the minimum quantization deviation is the optimal position;

[0053] In response to the quantization deviation being within the deviation threshold, determining the optimal position corresponding to the quantization deviation as the optimal weight clipping threshold;

[0054] In response to the quantized deviation being outside a deviation threshold, updating the velocity based on the optimal position;

[0055] The step of activating the cropping optimization comprises:

[0056] Get the activation distribution histogram of each channel in the same interval, which is the channel attention;

[0057] Compress the original feature map matrix based on principal component analysis to obtain single-layer spatial attention;

[0058] Sum and normalize the single-layer spatial attention to obtain global spatial attention;

[0059] Based on the single-layer spatial attention, the global spatial attention and the channel attention, obtaining activation clipped attention;

[0060] The step of compressing the original activation based on principal component analysis to obtain a single-layer spatial attention comprises:

[0061] Get the decentralized matrix of the original feature map matrix;

[0062] Performing singular decomposition on the decentralized matrix, and performing column vector compression on the decomposed data to obtain the projection of the original feature map matrix on the first column vector of the velocity feature;

[0063] A linear rectification unit and a normalization operation are applied to the projection to obtain the activated spatial attention, which is the single-layer spatial attention.

[0064] In a second aspect, an embodiment of the present application provides a lightweight depth estimation system, including:

[0065] A model building module, the model building module is used to obtain a depth estimation model, input the binocular depth image into the depth estimation model, and obtain a depth estimation result, wherein the model building module includes:

[0066] An input submodule, the input submodule is used to determine an input parameter, the input parameter includes a binocular depth image;

[0067] A feature extraction submodule, wherein the feature extraction submodule is used to extract features of the input parameters based on a two-dimensional depth-separable convolution to obtain feature parameters;

[0068] A cost body acquisition submodule, wherein the cost body acquisition submodule is used to acquire a cost body through the feature parameter;

[0069] A depth feature acquisition module, wherein the depth feature acquisition submodule is used to perform stacked hourglass processing on the cost volume based on three-dimensional point-by-point convolution to obtain multiple depth feature information;

[0070] An output submodule, wherein the output submodule is used to determine an output parameter, wherein the output parameter includes multiple depth feature information, and the multiple depth feature information is used to characterize a depth estimation result.

[0071] Beneficial effect: Compared with the prior art, the embodiment of the present application provides a lightweight depth estimation method, which obtains a depth estimation model, inputs a binocular depth image into the depth estimation model, obtains a depth estimation result, and when obtaining the depth estimation model, determines the binocular depth image as an input parameter, performs feature extraction on the input parameter based on two-dimensional depth separable convolution, obtains a cost body based on the extracted feature parameter, performs stacked hourglass processing on the cost body based on three-dimensional point-by-point convolution, obtains multi-depth feature information, and determines the multi-depth feature information as a depth estimation result, i.e., an output parameter. The present application integrates lightweight feature extraction methods such as two-dimensional depth separable convolution and three-dimensional point-by-point convolution, which can reduce the number of feature extraction layers, reduce the number of parameters of the depth estimation model, and solve the technical problems of the difficulty in deploying the binocular depth estimation network in actual application scenarios, low memory utilization, hardware resource waste, and performance loss. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0073] Figure 1 A flowchart of the steps of obtaining a depth estimation model in a lightweight depth estimation method provided in an embodiment of the present application;

[0074] Figure 2 A flowchart of the steps for obtaining characteristic parameters in the lightweight depth estimation method provided in an embodiment of the present application;

[0075] Figure 3 A flowchart of the steps of optimizing a depth estimation model in a lightweight depth estimation method provided in an embodiment of the present application;

[0076] Figure 4 A flowchart of the steps of obtaining classification labels in the lightweight depth estimation method provided in an embodiment of the present application;

[0077] Figure 5 A module connection diagram of a lightweight depth estimation system provided in an embodiment of the present application;

[0078] Figure 6 It is a module flow chart of the depth estimation method;

[0079] Figure 7 The module flow chart for obtaining cluster centers using Kmeans clustering algorithm;

[0080] Figure 8 Flowchart of the module for extracting deep features from the input left and right views;

[0081] Fig. 9 Schematic diagram for cost body construction;

[0082] Fig.10 Flowchart of hourglass network and loss function calculation module;

[0083] Fig.11 To construct a schematic diagram of channel attention;

[0084] Figure numerals: 10, model building module; 11, input submodule; 12, feature extraction submodule; 13, cost volume acquisition submodule; 14, deep feature acquisition module; 15, output submodule. DETAILED DESCRIPTION

[0085] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0086] With the development of artificial intelligence, application fields such as autonomous driving, drones, and robots have received more and more attention, and environmental perception is the core technology. Depth estimation obtains depth information by reconstructing the three-dimensional information of the environment, thereby realizing environmental perception, and is an important research direction in the field of computer vision. Traditional methods of obtaining depth information come from laser radar, monocular camera imaging, etc., which have problems such as insufficient image features and only applicable to close-range scenes. In recent years, convolutional neural networks, as an important part of deep learning, have made major breakthroughs in image fields such as target detection and target recognition. They can extract rich feature information from images and overcome the shortcomings of traditional algorithms.

[0087] In the past few years, researchers have proposed some lightweight models by improving the convolutional layers of neural networks. In 2016, a team proposed the MobileNet model (mobile network model), which selected depthwise separable convolution instead of traditional convolution, decomposed the convolution calculation into two steps: depthwise convolution and pointwise convolution, and introduced the width multiplier as a hyperparameter to select a model of appropriate size according to the actual application scenario. Compared with VGG16 (Visual Geometry Group Network with 16 layers), this new convolution method of lightweight network reduces the amount of multiplication and addition calculations by 27 times and the number of parameters by 33 times while losing less accuracy, thereby effectively compressing the model size. However, the network model has a simple structure and does not reuse image features for feature fusion. Subsequently, a more efficient network model was proposed. Chang et al. proposed the PSMNet (Pyramid Stereo Matching Network) network. In the feature extraction part, a deep residual neural network (ResNet) was used as the backbone network, and then the SPP (Spatial Pyramid Pooling) spatial pyramid structure was used to obtain feature information under receptive fields of different scales, and the global information and local information were combined to form a matching cost volume.

[0088] However, as the requirements for accuracy in actual application scenarios become higher and higher, the large-scale parameters and high-complexity calculations of neural networks pose challenges to the storage, power consumption and computing power of devices. In resource-constrained mobile applications such as smart wearable devices, drones and smart robots, neural networks will quickly exhaust their storage, memory, battery and computing units. Therefore, reducing the parameter scale and computational complexity of neural networks is of great significance for deploying neural networks in mobile applications.

[0089] Researchers have proposed a series of methods for neural network compression and acceleration, such as distillation, pruning, and quantization. Among these technologies, model quantization shows significant advantages. Current quantization methods include training-based quantization and post-training quantization. Training-based quantization generally inserts quantization computing nodes into the original network or directly designs low-bit computing units, and then uses backpropagation to learn network weights and related quantization parameters. It can significantly improve quantization performance, but the time cost is huge. Post-training quantization has been more widely used in mobile model deployment due to its convenience and efficiency. However, it will cause the network to suffer greater performance loss, especially for lightweight networks. It has been proven that pruning optimization can effectively improve the performance of post-training quantization, such as TensortRt, EasyQuant, MSE, ACIQ, etc., which aims to select the optimal pruning threshold to minimize quantization deviation. However, these methods ignore the fact that the network pays different attention to each weight or activation. Therefore, a new pruning optimization method is needed to find the optimal search step size and search interval.

[0090] In view of this, an embodiment of the present application provides a lightweight depth estimation method, which obtains a depth estimation model, inputs a binocular depth image into the depth estimation model, obtains a depth estimation result, and when obtaining the depth estimation model, determines the binocular depth image as an input parameter, performs feature extraction on the input parameter based on two-dimensional depth separable convolution, obtains a cost body based on the extracted feature parameter, performs stacked hourglass processing on the cost body based on three-dimensional point-by-point convolution, obtains multi-depth feature information, and determines the multi-depth feature information as a depth estimation result, i.e., an output parameter. The present application integrates lightweight feature extraction methods such as two-dimensional depth separable convolution and three-dimensional point-by-point convolution, which can reduce the number of feature extraction layers, reduce the number of parameters of the depth estimation model, and solve the technical problems of the difficulty in deploying the binocular depth estimation network in actual application scenarios, low memory utilization, and waste of hardware resources.

[0091] In some embodiments, see Figure 1 , Figure 1A flowchart of the steps of obtaining a depth estimation model in a lightweight depth estimation method provided in an embodiment of the present application. After obtaining the depth estimation model, the binocular depth image is input into the depth estimation model to obtain a depth estimation result. Specifically, the method for obtaining the depth estimation model is implemented by steps 100 to 500:

[0092] Step 100: Determine input parameters, where the input parameters include a binocular depth image.

[0093] Step 200: Perform feature extraction on input parameters based on two-dimensional depth-separable convolution to obtain feature parameters.

[0094] In some embodiments, see Figure 2 , Figure 2 A flowchart of the steps for obtaining feature parameters in a lightweight depth estimation method provided in an embodiment of the present application. Specifically, feature extraction is performed on input parameters based on two-dimensional depth separable convolution, and the method for obtaining feature parameters is implemented through steps 210 to 240:

[0095] Step 210: Perform multiple layers of two-dimensional depth-separable convolution operations on the binocular depth image, and use a rectified linear unit activation function to perform nonlinear processing after each layer of two-dimensional depth-separable convolution operation to obtain a first feature map.

[0096] Step 220: Process the first feature map through a multi-layer residual network to obtain a second feature map.

[0097] Step 230: Process the second feature map through a fast spatial pyramid pooling layer to obtain a multi-channel feature map that integrates multiple scale features, which is a third feature map.

[0098] Step 240: Fusing the information of the third feature map through a pixel-by-pixel convolution layer to obtain feature parameters.

[0099] Step 300: Obtain the cost volume through feature parameters.

[0100] In some embodiments, the binocular depth image includes a left view and a right view. When constructing the cost volume, the difference data of the feature parameters of the left view and the right view corresponding to each pixel are obtained, and matching is performed by moving the window. The left view and the right view are respectively cropped and matched with the other view to obtain the cost volume.

[0101] Step 400: Perform stacked hourglass processing on the cost volume based on three-dimensional point-by-point convolution to obtain multi-depth feature information.

[0102] In some embodiments, stacked hourglass processing is achieved by stacking multiple hourglass networks, performing a lightweight three-dimensional point-by-point convolution operation on the cost body in each hourglass network to reduce the resolution of the cost body and obtain a preliminary prediction result; performing a three-dimensional deconvolution operation on the preliminary prediction result, upsampling it, and obtaining deep feature information.

[0103] Step 500: Determine output parameters, where the output parameters include multiple depth feature information, and the multiple depth feature information is used to characterize the depth estimation result.

[0104] In some embodiments, after the depth estimation model is constructed, it is necessary to train and optimize the depth estimation model. Figure 3 , Figure 3 This is a flowchart of the steps for optimizing the depth estimation model in the lightweight depth estimation method provided in an embodiment of the present application. The method for optimizing the depth estimation model is specifically implemented through steps 610 to 660:

[0105] Step 610: Perform model training on the depth estimation model based on the depth estimation result, where the model training includes a classification task and a regression task.

[0106] Step 620: Obtain a depth map label, and obtain a classification label and a regression label based on the depth map label.

[0107] In some embodiments, see Figure 4 , Figure 4 This is a flowchart of the steps for obtaining classification labels in the lightweight depth estimation method provided in an embodiment of the present application. The method for obtaining classification labels is specifically implemented through steps 621 to 623:

[0108] Step 621: cluster the depth map labels to obtain cluster centers.

[0109] Step 622: Based on the cluster center and disparity classification, obtain a category map.

[0110] Step 623: Obtain classification labels based on the category mapping.

[0111] Step 630: Obtain classification loss based on the classification label and the classification task.

[0112] In some embodiments, the classification task processes the depth estimation result by multi-layer pyramid three-dimensional lightweight convolution and multi-layer two-dimensional convolution to obtain a disparity estimation map.

[0113] Step 640: Obtain regression loss based on the regression label and the regression task.

[0114] In some embodiments, the regression task performs channel fitting on the disparity estimation map through multi-layer pyramid three-dimensional lightweight convolution and multi-layer two-dimensional convolution, and obtains a continuous disparity map based on disparity regression.

[0115] Step 650: Determine a total loss function based on the classification loss and the regression loss.

[0116] In some embodiments, stacked hourglass processing is performed by stacking multiple layers of hourglass networks in series, and the output of each hourglass network is determined using a smoothed absolute error loss function to determine the regression loss and assign a weight coefficient; wherein, the output of the hourglass network with a greater depth is assigned a greater weight coefficient;

[0117] The cross entropy loss function is used to determine the classification loss of the output of each hourglass network, and a weight coefficient is assigned; the deeper the hourglass network, the greater the weight coefficient assigned to its output;

[0118] For the weight coefficient allocation between classification loss and regression loss, dynamic coefficient allocation is used. When the number of optimization iterations is less than the iteration threshold, only the classification loss is used as the total loss function; when the number of iterations is greater than or equal to the iteration threshold, the total loss function is determined based on the function after the weight coefficients are allocated based on the classification loss and regression loss.

[0119] Step 660: Optimize the depth estimation model based on the total loss function.

[0120] In some embodiments, after the depth estimation model is constructed or optimized, the present application further optimizes and prunes the depth estimation model through linear quantization, weight pruning optimization and / or activation pruning optimization to further reduce the memory consumption of the mobile terminal.

[0121] In some embodiments, the step of linear quantization includes:

[0122] The FP32 (Floating Point 32, 32-bit floating point number, also known as single-precision floating point number) data is linearly mapped to the specified bit data. The characterization formula includes:

[0123] ;

[0124] in, For floating point data that needs to be quantized; For quantitative scale; To quantify zero; The bit width of the data is specified in bits; For rounding operations; is the truncation operation; is the maximum clipping threshold; is the minimum clipping threshold; q is the integer after quantizing the initial target parameter;

[0125] In some embodiments, the steps of weight clipping optimization include:

[0126] Determine the target convolution kernel in the depth estimation model.

[0127] Based on the particle swarm algorithm, particles are initialized in the depth estimation model and the position range parameters of the particles are determined.

[0128] Determine the position and velocity of the particle based on the target convolution kernel and the position range parameters.

[0129] Determine the quantization deviation of the position, and determine that the position corresponding to the minimum quantization deviation is the optimal position.

[0130] In response to the quantization deviation being within the deviation threshold, an optimal position corresponding to the quantization deviation is determined as an optimal weight clipping threshold.

[0131] In response to the quantized deviation being outside of the deviation threshold, the speed is updated based on the optimal position.

[0132] In some embodiments, the step of activating crop optimization includes:

[0133] Get the activation distribution histogram of each channel in the same interval, which is the channel attention.

[0134] Based on principal component analysis, the original feature map matrix is ​​compressed to obtain single-layer spatial attention.

[0135] Multiple single-layer spatial attentions are summed and normalized to obtain the global spatial attention.

[0136] Obtain attention for activation clipping based on single-layer spatial attention, global spatial attention, and channel attention.

[0137] In some embodiments, the step of compressing the original feature map matrix based on principal component analysis to obtain a single-layer spatial attention includes:

[0138] Get the decentralized matrix of the original feature map matrix.

[0139] The decentralized matrix is ​​singularly decomposed, and the decomposed data is column-vector compressed to obtain the projection of the original feature map matrix on the first column vector of the velocity feature.

[0140] Apply the linear rectification unit and normalization operation to the projection to obtain the activated spatial attention, which is a single-layer spatial attention.

[0141] It can be understood that the embodiments of the present application provide a lightweight depth estimation method, obtain a depth estimation model, input a binocular depth image into the depth estimation model, obtain a depth estimation result, and when obtaining the depth estimation model, determine the binocular depth image as an input parameter, perform feature extraction on the input parameter based on two-dimensional depth separable convolution, and obtain a cost body based on the extracted feature parameters, perform stacked hourglass processing on the cost body based on three-dimensional point-by-point convolution, obtain multi-depth feature information, and determine the multi-depth feature information as a depth estimation result, i.e., an output parameter. The present application integrates lightweight feature extraction methods such as two-dimensional depth separable convolution and three-dimensional point-by-point convolution, which can reduce the number of feature extraction layers, reduce the number of parameters of the depth estimation model, and solve the technical problems of the difficulty of deploying the binocular depth estimation network in actual application scenarios, low memory utilization, and waste of hardware resources.

[0142] Exemplarily, an embodiment of the present application further provides a specific implementation scheme of a lightweight depth estimation method, including:

[0143] (1) Please refer to Figure 6 , Figure 6 It is a module flow chart of the depth estimation method. When the depth estimation steps are specifically implemented, they include:

[0144] 1) Dataset Clustering:

[0145] The purpose of clustering the depth map labels is to divide the different depth values ​​into 20 preset classes and find a "center" value for each class.

[0146] First, initialize the category center. Divide the depth value range [0,192] into 20 categories, and use the uniform distribution method to initialize the center value of each category: centers = [9,19,29,39,...,189].

[0147] Then, for each pixel, we match the center of the corresponding category. For each pixel’s depth, we calculate its distance from each category center. Then, we assign each pixel to the category center with the smallest distance.

[0148] See also Figure 7 , Figure 7 This is a flow chart of the module for obtaining cluster centers using the Kmeans clustering algorithm. When the Kmeans clustering algorithm is used to process a data set, the binocular depth image is first read, and the two-dimensional depth image is flattened into a one-dimensional array using the flatten() method. A Kmeans clustering model is created, and the depth values ​​are divided into 20 clusters. The cluster center of each category is returned, and the distance between each disparity value and the cluster center is calculated:

[0149] (1.1)

[0150] in, is the distance between the disparity value and the cluster center; is the disparity value; is the cluster center; is the number of categories; is the ordinal number of the category. For each disparity value, it is assigned to the cluster center that is closest to it, and the pixel is classified into the category .

[0151] Finally, the cluster center is calculated. For each category, the average depth value of all pixels in the category is calculated, which ensures that the center value of each category represents the "typical" depth value of the category:

[0152] (1.2)

[0153] in, It is The number of all pixels in the class; It is The set of all pixels of the class; Is it the first The depth value of the pixel.

[0154] 2) Extract deep features from the input left and right views:

[0155] See also Figure 8 , Figure 8 The flowchart of the module for extracting deep features from the left and right views of the input is as follows: First, three layers of two-dimensional depth-separable convolution (DSConv) are used to extract the features of the image, reducing the number of parameters while maintaining the effect of extracting features. The ReLU activation function is used after each layer of convolution to increase the nonlinearity of the network.

[0156] Then, more advanced features are gradually extracted through residual blocks. In layer1, three basic blocks are used to process input features and capture finer-grained features. Layer2 increases the depth of the feature map and reduces the spatial size through downsampling. The main function of this stage is to gradually improve the representation ability of the network and keep the information smooth through jump connections. In layer3, the network continues to extract more advanced features from the features output by layer2 while keeping the spatial size unchanged. Layer4 introduces dilated convolution (dilation=2) to increase the receptive field while keeping the feature map resolution unchanged, allowing the network to capture a wider range of contextual information. Through residual connections, the network can effectively utilize lower-level features while improving information fluidity and gradient transfer.

[0157] Then, the fast spatial pyramid pooling layer SPPF is used to process different scales simultaneously, effectively enhancing the multi-scale perception ability of the network. Multiple branches are created for different pooling scales, and each branch performs pooling and convolution operations on the input feature map at different scales. This application sets four pooling windows of different scales: 8*8, 16*16, 32*32, and 64*64, to capture local and global information in a balanced manner. The feature map after pooling is upsampled in the spatial dimension to restore it to the original image size, and is concatenated with the original feature map in the channel dimension to obtain a multi-channel feature map that integrates features of different scales.

[0158] Finally, a 1*1 convolution layer (pixel-by-pixel convolution layer) is used to fuse information of different scales to extract features of the left and right views.

[0159] 3) Cost body construction:

[0160] See also Fig. 9 , Fig. 9 The schematic diagram for cost volume construction (where H is the height of the image, that is, the number of pixels in the vertical direction of the image; W is the width of the image, that is, the number of pixels in the horizontal direction of the image; C is the number of channels of the image) is first initialized with a cost volume tensor of all zeros to store the cost volumes of the left and right views for subsequent matching calculations. Then, the difference in the features of the left and right views is calculated pixel by pixel, and the matching is performed by moving the window. The left and right views are respectively cropped and matched to the other view to obtain the cost volume. The matching costs of the left and right views are quantified by their features, and the cost volume represents the possibility of different disparities.

[0161] 4) Stacking hourglass processing cost body:

[0162] Three layers of convolution and deconvolution operations are used to process image data, combining top-down and bottom-up convolution operations to capture feature information at different depths.

[0163] In the first hourglass network, the input cost volume undergoes two steps of lightweight P3D convolution (three-dimensional point-by-point convolution) operations to reduce the image resolution to 1 / 16H×1 / 16W×1 / 16D, while outputting the first preliminary prediction result, gradually capturing global information while extracting features while reducing the spatial size. Then, upsampling is performed using a 3D deconvolution (transposed convolution) operation to restore the spatial size of the image. This stage helps the network refine the depth information in the image.

[0164] Three hourglass networks are stacked to gradually optimize the feature representation. Each unit outputs a preliminary prediction result for calculating the loss. This recursive process makes the output of each stage gradually more accurate. By stacking the outputs of multiple hourglass networks, the network can obtain high-quality depth estimation results.

[0165] 5) Classification and regression mixed tasks:

[0166] For the classification task, the input is the three results of the three-layer hourglass network output, and two layers of P3D convolution (3D point-by-point convolution) and two layers of 2D depth separable convolution (DSConv) are used to output 20 types of disparity estimation maps, corresponding to 20 channels. For the regression task, two layers of P3D convolution (3D point-by-point convolution) and two layers of 2D depth separable convolution (DSConv) are used to fit the disparity in one channel, and disparity regression is used to estimate the continuous disparity map. The probability of each disparity d is calculated by softmax operation. From the predicted cost Get the predicted disparity is calculated as the sum of each disparity d weighted by its probability, and its characterization formula includes:

[0167] (1.3)

[0168] in, is the predicted disparity value; The parallax is The matching cost at , which is used to characterize the prediction cost;

[0169] 6) Loss function calculation:

[0170] The loss function of this model is divided into two parts: classification loss and regression loss.

[0171] See also Fig.10 , Fig.10 The figure is a flow chart of the hourglass network and the loss function calculation module. For the regression loss, the present invention adopts the smooth absolute error loss (smooth L1 loss) to characterize the regression loss. The smooth L1 loss is widely used in the bounding box regression of object detection due to its robustness and low sensitivity to outliers. The calculation formula is as follows:

[0172] (1.4)

[0173] in:

[0174] (1.5)

[0175] in, It is the smooth L1 loss, which is used to characterize the regression loss; is the distance between the disparity value and the cluster center, which is used to characterize the actual disparity; is the sorting subscript; N is the number of labeled pixels; is the predicted disparity; is the difference between the predicted value and the true value.

[0176] For the three outputs of the three hourglass networks in the regression task, the smoothed absolute error loss function is used to determine the regression loss, and weight coefficients of 0.5, 0.7 and 1 are assigned to the three outputs. The greater the depth, the more refined the information obtained, and the larger the weight coefficient assigned.

[0177] For the classification task loss, the cross entropy loss function is used to calculate the error between the predicted category and the true category. First, the data set is constructed based on the regression task and belongs to a continuous depth map, so it needs to be processed into a label for the classification task. For the label disp_L (depth map) in the regression task, the disparity value is first mapped to the nearest category center according to the predefined category center value (center array). Then the disparity value (disp_L) of each pixel is converted to the corresponding category label according to the category center mapping. In this way, the regression label disp_L becomes the label of the classification task. After obtaining the label of the classification task, the loss function of the classification task is calculated. The present invention uses a cross entropy loss function to calculate the classification loss. For a multi-classification task with C categories, the cross entropy loss is defined as follows:

[0178] (1.6)

[0179] in, is the cross entropy loss; C is the number of categories; Is the real category One-hot encoding of, if the true category is the kth category, then , the rest of the categories ; is the model for the category The predicted probability of It is used to measure the degree of match between the predicted probability and the true probability. Similarly, for the three hourglass outputs of the classification task, weight coefficients of 0.5, 0.7, and 1 are also assigned.

[0180] For the weight coefficient allocation between classification tasks and regression tasks, dynamic coefficient allocation is used. When the number of iterative optimization iterations is less than 5, only classification loss is used. When the number of optimization iterations is greater than or equal to 5, weight coefficients of 1 and 0.7 are allocated to classification loss and regression loss.

[0181] (2) Clipping optimization method:

[0182] Ordinary post-training quantization methods perform well on mobile deployments, but can lead to performance loss. Clipping optimization is an advanced technique for improving post-training quantization. However, existing methods balance quantization deviations, limiting further improvements in quantization accuracy. Therefore, this application uses a quantization deviation algorithm that can give different attention to each parameter and prioritize reducing important parameters. The specific steps are as follows:

[0183] 1) First, linear quantization is performed to linearly map FP32 (Floating Point 32, 32-bit floating point numbers, also known as single-precision floating point numbers) data to low-bit data, which is defined as:

[0184] (1.7)

[0185] in, For floating point data that needs to be quantized; For quantitative scale; To quantify zero; The bit width of the data is specified in bits; For rounding operations; is the truncation operation; is the maximum clipping threshold; is the minimum clipping threshold; q is the integer after quantizing the initial target parameter. Truncation operation, quantized data is limited to [ − 2 n − 1 , 2 n − 1 − 1 ] ,so Map to ,at the same time Map to .

[0186] 2) Weight clipping optimization:

[0187] A particle swarm algorithm is added to the weight clipping to find the optimal weight clipping threshold. , the positions and velocities of several particles are initialized. The position of the particle , and Represent the minimum clipping threshold and the maximum clipping threshold respectively. Its movement speed is , and Represent the search direction and step length respectively. The initial position and initial velocity of the particle are defined as formula (1.8), where It is a parameter that controls the range of the initial position of the particles.

[0188] (1.8)

[0189] in, is uniformly distributed; yes The difference between the maximum and minimum values ​​of Calculate the quantization deviation at the position and record the minimum quantization deviation, which is defined as the optimal position In addition, the optimal position of each particle is recorded , the particle velocity is based on and change.

[0190] (1.9) in, and is the learning factor, when The larger the value, the particle tends to learn from its own experience and approach its optimal position. The larger it is, the particles tend to learn from the experience of the entire particle group and move toward the global optimal position; is the inertia factor of the original velocity, indicating that the particle will retain the initial motion trend to a certain extent. Defined as:

[0191] (1.10)

[0192] (1.11)

[0193] in, and They are respectively represented as the total number of iterations of the particle swarm and the current iteration cycle. After the particle swarm moves to the new position at a new speed that conforms to (1.10), the quantization deviation of each particle is recalculated to obtain the new and , and re-determine the speed. Finally, most particles will converge to the global optimal position. The optimal position of the particle swarm is the optimal weight clipping threshold G best → [ w min , w max ] .

[0194] 3) Activate cropping optimization:

[0195] Specifically, see Fig.11 , Fig.11 To construct the schematic diagram of channel attention. First, calculate the same interval [ min( A ),max( A ) ] The activation distribution histogram of each channel in . Information entropy of the channel Calculated by formula (1.12), where Represents the distribution histogram The frequency with which the group data occurs.

[0196] (1.12)

[0197] Then, based on principal component analysis, the original feature map matrix is ​​compressed to remove redundant information and obtain its principal components. Specifically, the original feature map matrix The eigenvector in is first subtracted from its own mean value to obtain the decentralized matrix , then Perform singular value decomposition, where X is given by The matrix of group eigenvectors, is the activation height, is the activation width.

[0198] (1.13)

[0199] is an orthogonal matrix whose column vectors are left singular vectors. is a diagonal matrix with n singular values ​​on its diagonal, is also an orthogonal matrix, and its column vectors are called right singular vectors. By compressing ,get At speed Finally, ReLu and normalization are applied to the projection to obtain the activated spatial attention , that is, single-layer spatial attention:

[0200] (1.14)

[0201] According to the change of spatial attention from shallow to deep layers in MobileNet (Mobile Network V2), in the shallow layers, attention is focused on the boundaries of the image, which shows that the shallow network layer focuses on extracting local details of the input. As the network goes deeper, the receptive field gradually becomes larger, and the spatial attention is focused on the core part of the foreground object.

[0202] Since the quantization bias of shallow layer activations will be propagated to deep layers, it is necessary to consider the spatial attention of the target layer and other network layers simultaneously when quantizing activations. Therefore, the spatial attention of each layer activation is summed and normalized to construct the global spatial attention .

[0203] Based on the above single-layer spatial attention, global spatial attention, and channel attention, we can get the activated attention:

[0204] (1.15)

[0205] in, is the second weight of the initial feature map parameter; and is the balance factor between single-layer spatial attention and global attention; It is the activation distribution histogram of the channel, which is used to characterize the channel attention; It is the activated spatial attention, which is used to represent the single-layer spatial attention; is the global spatial attention.

[0206] In summary, the optimization goals of activation clipping are as follows:

[0207] (1.16)

[0208] in, Indicates The standard deviation of the channel; C is the number of categories; h ’ is the height of the feature map; is the width of the feature map; A q is the quantitative feature map parameter; A is the initial feature map parameter; are the minimum and maximum values ​​of the activation clipping threshold, respectively. It suppresses the adverse effects of fluctuations in the values ​​of different channels. Because the activation amount is much larger than the weight, the particle swarm algorithm takes too long, and because the activation data comes from a small number of samples, it may cause overfitting problems. Therefore, grid search is used to solve the activation clipping threshold.

[0209] Accordingly, the embodiment of the present application also provides a lightweight depth estimation system, see Figure 5 , Figure 5 This is a module connection diagram of a lightweight depth estimation system provided in an embodiment of the present application. The depth estimation system provided in an embodiment of the present application includes:

[0210] The model building module 10 is used to obtain a depth estimation model, input the binocular depth image into the depth estimation model, and obtain a depth estimation result, wherein the model building module 10 includes:

[0211] An input submodule 11, the input submodule 11 is used to determine input parameters, and the input parameters include a binocular depth image;

[0212] A feature extraction submodule 12, the feature extraction submodule 12 is used to extract features of input parameters based on two-dimensional depth separable convolution to obtain feature parameters;

[0213] The cost body acquisition submodule 13 is used to acquire the cost body through the feature parameters;

[0214] A depth feature acquisition module 14, a depth feature acquisition submodule is used to perform stacked hourglass processing on the cost volume based on three-dimensional point-by-point convolution to obtain multi-depth feature information;

[0215] The output submodule 15 is used to determine output parameters, and the output parameters include multiple depth feature information, and the multiple depth feature information is used to characterize the depth estimation result.

[0216] In some embodiments, the feature extraction submodule 12 is specifically used for:

[0217] Performing a multi-layer two-dimensional depth-separable convolution operation on the depth image, and performing nonlinear processing using a rectified linear unit activation function after each layer of the two-dimensional depth-separable convolution operation to obtain a first feature map;

[0218] Processing the first feature map through a multi-layer residual network to obtain a second feature map;

[0219] The second feature map is processed by a fast spatial pyramid pooling layer to obtain a multi-channel feature map that integrates features of multiple scales, which is the third feature map;

[0220] The information of the third feature map is fused through a pixel-by-pixel convolution layer to obtain feature parameters.

[0221] In some embodiments, the cost volume acquisition submodule 13 is specifically used for:

[0222] The feature difference data of the left view and the right view corresponding to each pixel are obtained, and the matching is performed by moving the window. The left view and the right view are respectively cropped to match the other view to obtain the cost volume.

[0223] In some embodiments, the depth feature acquisition module 14 is specifically used to:

[0224] Stack multiple hourglass networks. In the hourglass network:

[0225] Perform lightweight three-dimensional point-by-point convolution operations on the cost volume to reduce the resolution of the cost volume and obtain preliminary prediction results;

[0226] Perform three-dimensional deconvolution operation on the preliminary prediction results, perform upsampling, and obtain deep feature information.

[0227] The present application has introduced in detail a lightweight depth estimation method and system provided by the embodiments of the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A lightweight depth estimation method, characterized in that: include: Acquire a depth estimation model, input the binocular depth image into the depth estimation model, and obtain a depth estimation result, wherein the step of acquiring the depth estimation model includes: Determining input parameters, wherein the input parameters include a binocular depth image; Performing feature extraction on the input parameter based on two-dimensional depth-separable convolution to obtain feature parameters; Obtaining a cost volume through the characteristic parameters; Performing stacked hourglass processing on the cost volume based on three-dimensional point-by-point convolution to obtain multi-depth feature information; An output parameter is determined, where the output parameter includes multiple depth feature information, and the multiple depth feature information is used to characterize a depth estimation result.

2. The lightweight depth estimation method according to claim 1, characterized in that: The step of extracting features from the input parameters based on two-dimensional depth-separable convolution to obtain feature parameters includes: Performing a multi-layer two-dimensional depth-separable convolution operation on the binocular depth image, and performing nonlinear processing using a rectified linear unit activation function after each layer of the two-dimensional depth-separable convolution operation to obtain a first feature map; Processing the first feature map through a multi-layer residual network to obtain a second feature map; Processing the second feature map through a fast spatial pyramid pooling layer to obtain a multi-channel feature map that integrates multiple scale features as a third feature map; The information of the third feature map is fused through a pixel-by-pixel convolution layer to obtain feature parameters.

3. The lightweight depth estimation method according to claim 1, characterized in that: The binocular depth image includes a left view and a right view, and the step of obtaining a cost volume through the feature parameter includes: The difference data of the feature parameters of the left view and the right view corresponding to each pixel are obtained, and matching is performed by moving the window, and the left view and the right view are respectively cropped and matched with another view to obtain the cost volume.

4. The lightweight depth estimation method according to claim 1, characterized in that: The step of performing stacked hourglass processing on the cost volume based on three-dimensional point-by-point convolution to obtain multi-depth feature information includes: Multiple hourglass networks are stacked, in which: Performing a lightweight three-dimensional point-by-point convolution operation on the cost volume to reduce the resolution of the cost volume and obtain a preliminary prediction result; A three-dimensional deconvolution operation is performed on the preliminary prediction result to perform upsampling to obtain the depth feature information.

5. The lightweight depth estimation method according to claim 1, characterized in that: The depth estimation method further comprises: Performing model training on the depth estimation model based on the depth estimation result, wherein the model training includes a classification task and a regression task; Obtaining a depth map label, and obtaining a classification label and a regression label based on the depth map label; Based on the classification label and the classification task, obtaining a classification loss; Obtaining a regression loss based on the regression label and the regression task; Determine a total loss function based on the classification loss and the regression loss; The depth estimation model is optimized based on the total loss function.

6. The lightweight depth estimation method according to claim 5, characterized in that: The step of obtaining a classification label based on the depth map label comprises: Performing clustering processing on the depth map labels to obtain cluster centers; Based on the cluster centers and the disparity classification, obtaining a category map; The classification label is obtained based on the category map.

7. The lightweight depth estimation method according to claim 5, characterized in that: The steps of the classification task include: Processing the depth estimation result by multi-layer pyramid three-dimensional lightweight convolution and multi-layer two-dimensional convolution to obtain a disparity estimation map; The steps of the regression task include: Channel fitting is performed on the disparity estimation map through multi-layer pyramid three-dimensional lightweight convolution and multi-layer two-dimensional convolution, and a continuous disparity map is obtained based on disparity regression.

8. The lightweight depth estimation method according to claim 5, characterized in that: The stacked hourglass processing is performed by an hourglass network with multiple layers stacked in series, and the step of determining the total loss function based on the classification loss and the regression loss includes: In the multi-layer hourglass network, The output of each hourglass network is determined using a smoothed absolute error loss function, and a weight coefficient is assigned; wherein the output of the hourglass network with a greater depth is assigned a greater weight coefficient; The output of each hourglass network is subjected to a cross entropy loss function to determine the classification loss and to assign a weight coefficient; wherein the output of the hourglass network with a greater depth is assigned a greater weight coefficient; For the weight coefficient allocation between classification loss and regression loss, dynamic coefficient allocation is used. When the number of optimization iterations is less than the iteration threshold, only the classification loss is used as the total loss function; when the number of iterations is greater than or equal to the iteration threshold, the total loss function is determined based on the function after the weight coefficients are allocated based on the classification loss and regression loss.

9. The lightweight depth estimation method according to claim 1, characterized in that: The lightweight depth estimation method further includes: performing linear quantization, weight clipping optimization and / or activation clipping optimization on the depth estimation model, and the linear quantization step includes: The single-precision floating-point data is linearly mapped to the data of the specified bit. The characterization formula includes: ； in, For floating point data that needs to be quantized; For quantitative scale; To quantify zero; The bit width of the data is specified in bits; is the rounding operation; is the truncation operation; is the maximum clipping threshold; is the minimum clipping threshold; q is the integer after quantizing the initial target parameter; The steps of weight clipping optimization include: Determining a target convolution kernel in the depth estimation model; Initializing particles in the depth estimation model based on a particle swarm algorithm and determining position range parameters of the particles; Determining the position and velocity of the particle based on the target convolution kernel and the position range parameter; Determine the quantization deviation of the position, and determine that the position corresponding to the minimum quantization deviation is the optimal position; In response to the quantization deviation being within the deviation threshold, determining the optimal position corresponding to the quantization deviation as the optimal weight clipping threshold; In response to the quantized deviation being outside a deviation threshold, updating the velocity based on the optimal position; The step of activating the cropping optimization comprises: Get the activation distribution histogram of each channel in the same interval, which is the channel attention; Compress the original feature map matrix based on principal component analysis to obtain single-layer spatial attention; Sum and normalize the single-layer spatial attention to obtain global spatial attention; Based on the single-layer spatial attention, the global spatial attention and the channel attention, obtaining activation clipped attention; The step of compressing the original activation based on principal component analysis to obtain a single-layer spatial attention comprises: Get the decentralized matrix of the original feature map matrix; Performing singular decomposition on the decentralized matrix, and performing column vector compression on the decomposed data to obtain the projection of the original feature map matrix on the first column vector of the velocity feature; A linear rectification unit and a normalization operation are applied to the projection to obtain the activated spatial attention, which is the single-layer spatial attention.

10. A lightweight depth estimation system, characterized in that: include: A model building module (10), the model building module (10) is used to obtain a depth estimation model, input the binocular depth image into the depth estimation model, and obtain a depth estimation result, wherein the model building module (10) includes: An input submodule (11), the input submodule (11) is used to determine an input parameter, the input parameter comprising a binocular depth image; A feature extraction submodule (12), the feature extraction submodule (12) being used to extract features from the input parameter based on a two-dimensional depth-separable convolution to obtain a feature parameter; A cost body acquisition submodule (13), wherein the cost body acquisition submodule (13) acquires a cost body through the feature parameter; A depth feature acquisition module (14), wherein the depth feature acquisition submodule is used to perform stacked hourglass processing on the cost volume based on three-dimensional point-by-point convolution to obtain multiple depth feature information; An output submodule (15), the output submodule (15) is used to determine an output parameter, the output parameter includes multiple depth feature information, and the multiple depth feature information is used to characterize the depth estimation result.

Images