A model pruning method based on a BGNet stereo matching network

Abstract

The present application relates to a kind of model pruning methods based on BGNet stereo matching network, belong to computer vision field.The method includes building BGNet stereo matching network model, and pre-training is carried out to obtain the model to be pruned, determine the convolution layer needing to participate in pruning, estimate the average rank of feature map in the convolution layer to be pruned, set adaptive pruning rate for each convolution layer to be pruned, i.e.the pruning threshold of each convolution layer is different, the model after pruning is fine-tuned to recover accuracy, finally the most accurate model is generalized on real dataset, obtain the final BGNet stereo matching network model.The present application is compressed to BGNet stereo matching network by the way of pruning, a light-weight BGNet stereo matching network is obtained, and from the comprehensive point of view of speed, accuracy, parameter amount, the performance of BGNet stereo matching network after pruning is superior to BGNet stereo matching network before pruning.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of computer vision, and particularly relates to a model pruning method based on a BGNet stereo matching network. BACKGROUND

[0002] Stereo matching technology has a wide range of applications in automatic driving, augmented reality and other fields. In recent years, due to the excellent performance of convolutional neural networks in various computer vision tasks, convolutional neural networks have also been applied to stereo matching tasks to improve their performance, showing strong application potential. However, the stereo matching network model has a large number of parameters and high computational complexity, requiring high device computing power, which greatly hinders the productization of the stereo matching network method. Therefore, the traditional stereo matching algorithm is still used in the internal algorithm of the depth camera on the market, and therefore it is urgent to solve the problems of large number of parameters and high model complexity of the stereo matching network and to accelerate the productization process of the stereo matching network algorithm.

[0003] There are currently three methods to solve the problem of large number of parameters of the stereo matching network and high device computing power during deployment. Method one is to design a lightweight stereo matching network structure, such as the invention patent application publication document “An Progressive Real-time Stereo Matching Method Based on Three-dimensional Convolutional Neural Network” (CN112200853A) which uses a lightweight backbone neural network to extract features, ensuring real-time algorithm while providing multi-scale features for other modules. This method achieves real-time stereo matching, but its accuracy is not very high. Method two is to design a special hardware to accelerate it, which has the disadvantage of high cost. Method three is to compress the stereo matching network using model compression technology, which includes model quantization, model pruning, knowledge distillation and other technologies. Model pruning is the simplest and most efficient of the many model compression technologies, but existing model technologies are based on classification networks as the research object, and cannot be directly applied to stereo matching networks. In addition, improper setting of the pruning rate of some pruning technologies will lead to insufficient network pruning, and the accuracy and speed cannot be balanced.

[0004] In summary, the current stereo matching network and model pruning technology still has the following problems:

[0005] 1. The stereo matching network has a large number of parameters and cannot be deployed on devices with limited resources.

[0006] 2. Some existing model pruning technologies are not suitable for stereo matching networks.

[0007] 3. Some model pruning technologies cannot achieve sufficient pruning due to improper setting of the pruning rate. SUMMARY

[0008] The technical problem to be solved by the present application is to overcome the defects of the prior art. Specifically, the present application provides a model pruning method based on a BGNet stereo matching network, which can solve the problems of insufficient pruning and excessive parameter quantity in the prior art, balance accuracy and speed, and adapt to the pruning method of the stereo matching network.

[0009] The object of the present application is achieved by providing a model pruning method based on a BGNet stereo matching network, comprising the following steps:

[0010] Step 1: download a synthetic data set from a public database, the synthetic data set comprising a synthetic training set and a synthetic test set, the synthetic training set comprising Q1 groups of training samples, the synthetic test set comprising Q2 groups of test samples, and each group of training samples and each group of test samples containing one left image, one right image and one real disparity image;

[0011] Each left image and right image in the synthetic training set is subjected to image enhancement using ψ data enhancement methods, and the images obtained by each data enhancement method are retained, i.e. Q1×(ψ+1) pairs of images are obtained, and the Q1×(ψ+1) pairs of images form an enhanced synthetic training set;

[0012] Download a real data set from a public database, the real data set comprising a real training set and a real test set, the real training set comprising Q3 groups of training samples, and each pair of training samples containing one left image, one right image and one real disparity image, the real test set comprising Q4 pairs of test samples, and each pair of test samples containing one left image and one right image;

[0013] Step 2: build a BGNet stereo matching network model, denoted as model A, the module containing convolution layers in the model A comprising a feature extraction module, a guided filtering module and a cost aggregation module;

[0014] The feature extraction module includes the following structures connected in series along the input-output direction of network A: a first convolutional structure unit, a residual structure unit, a second convolutional structure unit, a first deconvolutional structure unit, a third convolutional structure unit, and a second deconvolutional structure unit. The residual structure unit comprises four residual structures connected in series along the input-output direction of network A, wherein the output of the third residual structure is feature map F1, and the output of the fourth residual structure is feature map F2. The second convolutional structure unit comprises four two-dimensional convolutional layers connected in series along the input-output direction of network A, wherein the output of the first two-dimensional convolutional layer is feature map F3. Both the first and second deconvolutional structure units comprise three deconvolutional modules connected in series along the input-output direction of network A, and their final outputs are feature map F4 and feature map F5, respectively.

[0015] The guided filtering module includes two two-dimensional convolutional layers connected in series along the input-output direction of network A; the cost aggregation module includes a first three-dimensional convolutional structure unit, a first three-dimensional deconvolutional structure unit, and a three-dimensional convolutional layer connected in series along the input-output direction of network A.

[0016] Step 3: Train model A to obtain an optimal pre-trained BGNet stereo matching network model, which is the BGNet stereo matching network model to be pruned, and is denoted as model B.

[0017] Let the number of iterations be G. The images from the augmented synthesis training set are fed into model A in G iterations for training, resulting in G trained BGNet stereo matching network models, denoted as pre-trained BGNet stereo matching network models. These G pre-trained BGNet stereo matching network models are then grouped into a pre-trained model set M1, where M1 = [M11, M12, ..., M1...]. β ,...,M1 G ], of which M1 β The pre-trained BGNet stereo matching network model obtained after the βth training iteration is β=1,2,...,G;

[0018] The pre-trained endpoint errors of G pre-trained BGNet stereo matching network models are calculated on the synthetic test set, resulting in G pre-trained endpoint errors. These G pre-trained endpoint errors are then combined into a pre-trained endpoint error set O1, where O1 = [EPE11, EPE12, ..., EPE1...]. β ,...,EPE1 G ], of which EPE1 β The pre-trained BGNet stereo matching network model M1 obtained after the βth training is β The corresponding pre-training endpoint error;

[0019] The minimum value in the pre-training endpoint error set O1 is recorded as the optimal pre-training endpoint error EPE1 min , and the optimal endpoint error EPE1 min corresponds to the optimal pre-training BGNet stereo matching network model.

[0020] Step 4, set the convolutional layers not participating in pruning, and calculate the rank of the feature map in the convolutional layer of the model B obtained in step 3, and the implementation process is as follows:

[0021] Step 4.1, set the following three kinds of convolutional layers not participating in pruning:

[0022] The convolutional layers of the feature maps F1, F2, F3, F4 and F5 generated in the feature extraction module do not participate in pruning;

[0023] All convolutional layers in the guided filtering module do not participate in pruning;

[0024] The last convolutional layer in the cost aggregation module does not participate in pruning;

[0025] In addition to the above 3 kinds of convolutional layers, all convolutional layers in the model B participate in pruning, and the convolutional layers participating in pruning are recorded as pruning convolutional layers;

[0026] Step 4.2, randomly select N groups of training samples from the synthesized training set in step 1, and record the set of the N groups of training samples as sample set P, P = [P1, P2, …, P E ,...,P N ], P E is any one pair of samples in the sample set P, E = 1, 2, …, N;

[0027] Input the N groups of training samples into the model B, and each pruning convolutional layer in the model B generates N groups of feature maps, and record the N groups of feature maps as feature map set T, T = [T1, T2, … T E ...,T N ], T E is any one group of feature maps in the feature map set T;

[0028] Step 4.3, when the pruning convolutional layer is a two-dimensional convolutional layer and a two-dimensional deconvolutional layer, the resolution of the feature map T E is recorded as Res1, Res1 = X × H × W, wherein X is the number of channels in the feature map T E , H is the height dimension of the feature map T E , and W is the width dimension of the feature map T E , at this time, the mathematical expression of each channel in the feature map T E is a two-dimensional matrix Y;

[0029] When the parameter-pruned convolutional layer is a 3D convolutional layer and a 3D deconvolutional layer, the feature map T E The resolution is denoted as Res2, where Res2 = X × Z × H × W, and Z is the feature map T. E The disparity dimension, at this time, feature map T E The mathematical expression for each channel is a three-dimensional matrix; the height dimension H and width dimension W in the channel are merged into one dimension, that is, the three-dimensional matrix is ​​transformed into a two-dimensional matrix Y;

[0030] Feature map T E Let any one of the channels in be denoted as the J-th channel, and let the rank of the J-th channel be denoted as . The rank of the J-th channel The calculation formula is as follows:

[0031]

[0032] Among them, matrix_rank() is a library function in the deep learning library for solving the rank of a two-dimensional matrix Y;

[0033] Feature map T E The rank of X channels is denoted as the rank set R. E ,

[0034] Step 4.4, repeat step 4.3, to obtain the rank set of N sets of feature maps in the feature map set T, and the rank of the J-th channel in the N sets of feature maps. Calculate the average value and denote it as the channel average rank r of the J-th channel. J The calculation formula is as follows:

[0035]

[0036] For each channel of N feature maps, calculate the channel average rank to obtain X channel average ranks, and write them into a queue of channel average ranks of the parametric pruning convolutional layer [r]. 1 ,r 2 ,...,r J ,...,r X ];

[0037] Step 5, pruning, the process is as follows:

[0038] Step 5.1, Confirmation of pruning threshold

[0039] The average rank queue of the channels of the parametric convolutional layer [r] 1 ,r 2 ,...,r J ,...,r XThe channel average ranks in the array are rearranged in ascending order to obtain the rearranged parameter-pruned convolutional layer average rank queue [r'1, r'2, ..., r']. i ,r' I ,...,r' X ], where r′ i Let r' be the channel average rank of any channel in the rearranged parameter-pruned convolutional layer average rank queue R', where i is the index of the rearranged channel average rank, i = 1, 2, ..., X, r' I Let I = i + 1 be the pruning threshold to be confirmed, and r′ be the pruning threshold to be confirmed. I The qualifier is as follows:

[0040]

[0041] In the formula, α is the scaling factor;

[0042] Substituting I starting from 2 into the above constraint: if the pruning threshold r′ to be confirmed... I If the constraint is satisfied, then the pruning threshold r′ to be determined is... I It is identified as the pruning threshold; if the pruning threshold r′ to be identified is... I If the constraint is not met, gradually increase I until the pruning threshold r′ to be confirmed. I Until the above equation is satisfied;

[0043] Step 5.2 When the parametric convolutional layer is not in the residual module, rearrange the channel average queue [r'1,r'2,...,r'] i ,r' I ,...,r' X In the [r'1,r'2,...,r'] i Remove the convolution kernels from the corresponding channels of the segment;

[0044] Step 5.3: When the pruned convolutional layer is in the residual module, add a constraint: the number of channels in the convolutional layer in the residual module is the same before and after pruning.

[0045] Let the existing residual module be б, the convolutional layer before the residual module б be conv1, the number of channels before pruning conv1 is C1, the number of channels after pruning is C1', the convolutional layer after the residual module б be conv2, the number of channels before pruning conv2 is C2, the number of channels after pruning is C2', where C1=C2;

[0046] Prune according to the following rules:

[0047] When C1' = C2', the channel average queue [r'1, r'2, ..., r'] will be rearranged. i ,r' I ,...,r'X ]th of the channels in the feature map generated by conv1, and denote it as r i ]th of the channels in the feature map generated by conv1, and denote it as r

[0048] When C1’≠C2’, first, the pruning threshold of all channels of the feature map generated by conv1 is calculated in the manner of step 5.1, and denoted as r s The position of the convolution kernel corresponding to the channel that needs to be pruned in the feature map generated by conv1 is marked by mask1, which is a 1×C1 vector, wherein the position of the convolution kernel corresponding to the channel with a rank less than r s is represented by 1, i.e., the current position convolution kernel is pruned, and the position of the convolution kernel corresponding to the remaining channels is represented by 0, i.e., the current position convolution kernel is not pruned.

[0049] The pruning threshold of all channels of the feature map generated by conv2 is calculated in the manner of step 5.1, and denoted as r t The position of the convolution kernel corresponding to the channel that needs to be pruned in the feature map generated by conv2 is marked by mask2, which is a 1×C2 vector, wherein the position of the convolution kernel corresponding to the channel with a rank less than r t is represented by 1, i.e., the current position convolution kernel is pruned, and the position of the convolution kernel corresponding to the remaining channels is represented by 0, i.e., the current position convolution kernel is not pruned.

[0050] The vector mask is obtained by taking the union of mask1 and mask2, and the pruning positions of the convolution kernels in conv1 and conv2 are marked by mask, and the convolution kernels marked for pruning are pruned.

[0051] After pruning model B in the manner of step 5, the pruned optimal BGNet stereo matching network model is obtained, and denoted as model C.

[0052] Step 6: Model C is fine-tuned and trained on the enhanced synthetic training set in step 1 to restore the precision of model B, and an optimal BGNet stereo matching network model after pruning and fine-tuning is obtained, and denoted as model D.

[0053] Step 7: Model D obtained in step 6 is generalized and trained on the real training set in step 1, and an optimal BGNet stereo matching network model after generalization is obtained, which is the final BGNet stereo matching network model.

[0054] Preferably, the data enhancement method comprises random cropping, vertical disparity enhancement, brightness enhancement, and contrast enhancement.

[0055] Preferably, the pre-training endpoint error EPE1 β is calculated according to the following formula:

[0056]

[0057] wherein: d est (x, y) is the pixel value at the horizontal coordinate x and the vertical coordinate y in the real disparity image provided in the real training set β The pixel value at the horizontal coordinate x and the vertical coordinate y in the output disparity image, d gt (x, y) is the pixel value at the horizontal coordinate x and the vertical coordinate y in the real disparity image provided in the real training set; K is the pre-trained BGNet stereo matching network model M1 obtained after the βth training β The number of all pixel points in the output disparity image.

[0058] Preferably, the implementation process of step 6 is specifically as follows:

[0059] The images in the enhanced synthetic training set are sent into the model C for iterative training G times, and G trained fine-tuned BGNet stereo matching network models are obtained. The G trained fine-tuned BGNet stereo matching network models form a fine-tuned model set M2, M2 = [M21, M22, …, M2ε, …, M2G], wherein M2ε is the fine-tuned BGNet stereo matching network model obtained after the εth training, and ε = 1, 2, 3, …, G. G

[0060] The G trained BGNet stereo matching network models are respectively used to calculate the fine-tuned end point error of the synthetic test set, and G fine-tuned end point errors are obtained. The G fine-tuned end point errors form a fine-tuned end point error set O2, O2 = [EPE21, EPE22, …, EPE2ε, …, EPE2G], wherein EPE2ε is the end point error corresponding to the fine-tuned BGNet stereo matching network model M2ε obtained after the εth training. G

[0061] The minimum value in the fine-tuned end point error set O2 is recorded as the optimal fine-tuned end point error EPE2 min , and the fine-tuned BGNet stereo matching network model corresponding to the optimal fine-tuned end point error EPE2 min is the pruned and fine-tuned optimal BGNet stereo matching network model.

[0062] Preferably, the implementation process of step 7 is specifically as follows:

[0063] Randomly select Q5 groups of training samples in the real training set as training samples for use, and record them as a generalization real training set. The remaining (Q3-Q5) groups of training samples are used for verification, and are recorded as a generalization verification set, Q5 < Q3. ​​

[0064] Set the number of iterations of the generalization training as V, and send the images in the generalization real training set into the model D for iterative training V times, obtain V trained generalization BGNet stereo matching network models, and group the V trained generalization BGNet stereo matching network models to form a generalization model set M3, M3 = [M31, M32,..., M3 е ,...,M3 V ] Wherein M3 е is the generalization BGNet stereo matching network model obtained by the e-th training, e = 1, 2,..., V.

[0065] Calculate the generalization end point error of the V generalization BGNet stereo matching network models on the generalization verification set respectively, obtain V generalization end point errors, and group the V generalization end point errors to form a generalization end point error set O3, O3 = [EPE31, EPE32,..., EPE3 е ,...,EPE3 V ] Wherein EPE3 е is the end point error corresponding to the e-th trained generalization BGNet stereo matching network model M3 е .

[0066] Take the minimum value in the generalization end point error set O3 as the optimal generalization end point error EPE3 min , and the generalization BGNet stereo matching network model corresponding to the optimal generalization end point error EPE3 min is the optimal BGNet stereo matching network model after generalization.

[0067] The beneficial effects of the present application are as follows:

[0068] (1) The model pruning technology is used to realize the lightweight of the stereo matching network, and the balance between speed and accuracy is achieved.

[0069] (2) The present application fuses the dimensions of the three-dimensional feature map in the stereo matching network, so that the pruning method according to the feature map rank can also be applied to the BGNet and other 3D stereo matching networks.

[0070] (3) Since the distribution of the feature map rank in each convolution layer is different, the present application sets a self-adaptive pruning rank threshold for each convolution layer, so that each convolution layer can be fully pruned while the accuracy is guaranteed. BRIEF DESCRIPTION OF DRAWINGS

[0071] Figure 1 is the flowchart of the model pruning method of the present application.

[0072] Figure 2 is the flowchart of the pruning step in the embodiment of the present application. DETAILED DESCRIPTION

[0073] The application will be further described in conjunction with the embodiments and the accompanying drawings.

[0074] Figure 1 is a flow chart of the model pruning method of the application, Figure 2 is a flow chart of the pruning step in the embodiment of the application. The pruning step comprises the following steps: Figure 1 and Figure 2 It can be seen that the application provides a model pruning method based on a BGNet stereo matching network, which comprises the following steps:

[0075] Step 1: downloading a synthetic dataset from a public database, the synthetic dataset comprising a synthetic training set and a synthetic test set, the synthetic training set comprising Q1 groups of training samples, the synthetic test set comprising Q2 groups of test samples, and each group of training samples and each group of test samples containing one left image, one right image and one real disparity image;

[0076] performing image enhancement on each left image and right image in the synthetic training set by using ψ kinds of data enhancement methods, and retaining the images obtained by each kind of data enhancement method, i.e., obtaining Q1×(ψ+1) pairs of images in total, and grouping the Q1×(ψ+1) pairs of images to form an enhanced synthetic training set;

[0077] Step 2: downloading a real dataset from a public database, the real dataset comprising a real training set and a real test set, the real training set comprising Q3 groups of training samples, and each group of training samples containing one left image, one right image and one real disparity image, the real test set comprising Q4 pairs of test samples, and each pair of test samples containing one left image and one right image.

[0078] In this embodiment, Q1=35454, Q2=4370, Q3=394 and Q4=395.

[0079] In this embodiment, the data enhancement methods include random cropping, vertical disparity enhancement, brightness enhancement and contrast enhancement, i.e., ψ=4 kinds.

[0080] Step 2: building a BGNet stereo matching network model, and denoted as model A, the modules containing convolution layers in the model A including a feature extraction module, a guided filtering module and a cost aggregation module;

[0081] The feature extraction module includes the following structures connected in series along the input-output direction of network A: a first convolutional structure unit, a residual structure unit, a second convolutional structure unit, a first deconvolutional structure unit, a third convolutional structure unit, and a second deconvolutional structure unit. The residual structure unit comprises four residual structures connected in series along the input-output direction of network A, wherein the output of the third residual structure is feature map F1, and the output of the fourth residual structure is feature map F2. The second convolutional structure unit comprises four two-dimensional convolutional layers connected in series along the input-output direction of network A, wherein the output of the first two-dimensional convolutional layer is feature map F3. Both the first and second deconvolutional structure units comprise three deconvolutional modules connected in series along the input-output direction of network A, and their final outputs are feature map F4 and feature map F5, respectively.

[0082] The guided filtering module includes two two-dimensional convolutional layers connected in series along the input-output direction of network A; the cost aggregation module includes a first three-dimensional convolutional structure unit, a first three-dimensional deconvolutional structure unit, and a three-dimensional convolutional layer connected in series along the input-output direction of network A.

[0083] Step 3: Train model A to obtain an optimal pre-trained BGNet stereo matching network model, which is the BGNet stereo matching network model to be pruned, and is denoted as model B.

[0084] Let the number of iterations be G. The images from the augmented synthesis training set are fed into model A in G iterations for training, resulting in G trained BGNet stereo matching network models, denoted as pre-trained BGNet stereo matching network models. These G pre-trained BGNet stereo matching network models are then grouped into a pre-trained model set M1, where M1 = [M11, M12, ..., M1...]. β ,...,M1 G ], of which M1 β The pre-trained BGNet stereo matching network model obtained after the βth training iteration is β=1,2,...,G;

[0085] The pre-trained endpoint errors of G pre-trained BGNet stereo matching network models are calculated on the synthetic test set, resulting in G pre-trained endpoint errors. These G pre-trained endpoint errors are then combined into a pre-trained endpoint error set O1, where O1 = [EPE11, EPE12, ..., EPE1...]. β ,...,EPE1 G ], of which EPE1 β The pre-trained BGNet stereo matching network model M1 obtained after the βth training is β The corresponding pre-training endpoint error;

[0086] The minimum value in the pre-training endpoint error set O1 is recorded as the optimal pre-training endpoint error EPE1 min , and the optimal endpoint error EPE1 min corresponds to the optimal pre-training BGNet stereo matching network model.

[0087] In this embodiment, the current training is pre-training, and the iteration number G = 100.

[0088] In this embodiment, the pre-training endpoint error EPE1 β is calculated according to the following formula:

[0089]

[0090] wherein d est (x,y) is the pixel value at the horizontal coordinate x and the vertical coordinate y in the disparity image output by the pre-training BGNet stereo matching network model M1 β obtained after the βth training, d gt (x,y) is the pixel value at the horizontal coordinate x and the vertical coordinate y in the real disparity image provided in the synthesis test set; and K is the number of all pixel points in the disparity image output by the pre-training BGNet stereo matching network model M1 β obtained after the βth training.

[0091] Step 4, setting the convolution layers not participating in pruning, and calculating the rank of the feature map in the model B obtained in step 3, the implementation process is as follows:

[0092] Step 4.1, setting the following three kinds of convolution layers not participating in pruning:

[0093] The convolution layers of the feature map F1, the feature map F2, the feature map F3, the feature map F4 and the feature map F5 generated in the feature extraction module do not participate in pruning;

[0094] All convolution layers in the guided filtering module do not participate in pruning;

[0095] The last convolution layer in the cost aggregation module does not participate in pruning;

[0096] In addition to the above 3 kinds of convolution layers, all convolution layers in the model B participate in pruning, and the convolution layers participating in pruning are recorded as pruned convolution layers.

[0097] Step 4.2, randomly selecting N groups of training samples from the synthesis training set in step 1, and recording the set of the N groups of training samples as a sample set P, P = [P1, P2, …, P E ,...,P N ] P ELet E be any set of samples in the sample set P, where E = 1, 2, ..., N;

[0098] N sets of training samples are input into model B. Each parameterized convolutional layer in model B generates N sets of feature maps. These N sets of feature maps are denoted as the feature map set T, where T = [T1, T2, ..., T]. E ...,T N ], T E Let T be any set of feature maps in the feature map set.

[0099] Step 4.3, when the parameter-clipping convolutional layer is a two-dimensional convolutional layer and a two-dimensional deconvolutional layer, the feature map T E The resolution is denoted as Res1, where Res1 = X × H × W, and X is the feature map T. E The number of channels in the feature map T, H is the number of channels in the feature map T. E The height dimension, W, is the feature map T. E The width dimension, at this time, feature map T E The mathematical expression for each channel in the matrix is ​​a two-dimensional matrix Y;

[0100] When the parameter-pruned convolutional layer is a 3D convolutional layer and a 3D deconvolutional layer, the feature map T E The resolution is denoted as Res2, where Res2 = X × Z × H × W, and Z is the feature map T. E The disparity dimension, at this time, feature map T E The mathematical expression for each channel is a three-dimensional matrix; the height dimension H and width dimension W in the channel are merged into a single dimension, that is, the three-dimensional matrix is ​​transformed into a two-dimensional matrix Y.

[0101] Feature map T E Let any one of the channels in be denoted as the J-th channel, and let the rank of the J-th channel be denoted as . The rank of the J-th channel The calculation formula is as follows:

[0102]

[0103] Among them, matrix_rank() is a library function in the deep learning library for calculating the rank of a two-dimensional matrix Y.

[0104] Feature map T E The rank of X channels is denoted as the rank set R. E ,

[0105] Step 4.4, repeat step 4.3, to obtain the rank set of N sets of feature maps in the feature map set T, and the rank of the J-th channel in the N sets of feature maps. Calculate the average value and denote it as the channel average rank r of the J-th channel.J The calculation formula is as follows:

[0106]

[0107] For each channel of N feature maps, calculate the channel average rank to obtain X channel average ranks, and write them into a queue of channel average ranks of the parametric pruning convolutional layer [r]. 1 ,r 2 ,...,r J ,...,r X ].

[0108] Step 5, pruning, the process is as follows:

[0109] Step 5.1, Confirmation of pruning threshold

[0110] The average rank queue of the channels of the parametric convolutional layer [r] 1 ,r 2 ,...,r J ,...,r X The channel average ranks in the array are rearranged in ascending order to obtain the rearranged parameter-pruned convolutional layer average rank queue [r'1, r'2, ..., r']. i ,r' I ,...,r' X ], where r′ i Let r' be the channel average rank of any channel in the rearranged parameter-pruned convolutional layer average rank queue R', where i is the index of the rearranged channel average rank, i = 1, 2, ..., X, r' I Let I = i + 1 be the pruning threshold to be confirmed, and r′ be the pruning threshold to be confirmed. I The qualifier is as follows:

[0111]

[0112] In the formula, α is the scaling factor;

[0113] Substituting I starting from 2 into the above constraint: if the pruning threshold r′ to be confirmed... I If the constraint is satisfied, then the pruning threshold r′ to be determined is... I It is identified as the pruning threshold; if the pruning threshold r′ to be identified is... I If the constraint is not met, gradually increase I until the pruning threshold r′ to be confirmed. I Continue until the above equation is satisfied.

[0114] Step 5.2 When the parametric convolutional layer is not in the residual module, rearrange the channel average queue [r'1,r'2,...,r'] i ,r' I ,...,r' X In the [r'1,r'2,...,r']i The convolution kernels in the corresponding channels of the segment are pruned.

[0115] Step 5.3: When the pruned convolutional layer is in the residual module, add a constraint: the number of channels in the convolutional layer in the residual module is the same before and after pruning.

[0116] Let the existing residual module be... The convolutional layer preceding the residual module б is conv1, where conv1 has C1 channels before pruning and C1' channels after pruning. The next convolutional layer is conv2. The number of channels before pruning conv2 is C2, and the number of channels after pruning is C2', where C1 = C2.

[0117] Prune according to the following rules:

[0118] When C1' = C2', the channel average queue [r'1, r'2, ..., r'] will be rearranged. i ,r' I ,...,r' X In the [r'1,r'2,...,r'] i The convolution kernels in the corresponding channels of the segment are pruned.

[0119] When C1'≠C2', first calculate the pruning thresholds for all channels of the feature map generated by conv1 in step 5.1, and denot them as the pruning threshold r of the previous convolutional layer. s We use `mask1` to mark the positions of the convolutional kernels corresponding to the channels that need to be clipped in the feature map generated by `conv1`. `mask1` is a 1×C1 vector, where the kernels with rank less than r are... s The position of the convolution kernel corresponding to the current channel is represented by 1, which means that the convolution kernel at the current position is pruned; the position of the convolution kernel corresponding to the other channels is represented by 0, which means that the convolution kernel at the current position is not pruned.

[0120] Following step 5.1, the threshold for the rank of all channels of the feature map generated by conv2 during pruning is obtained, and denoted as the pruning threshold r of the next convolutional layer. t We use `mask2` to mark the positions of the convolutional kernels corresponding to the channels that need to be clipped in the feature map generated by `conv2`. `mask2` is a 1×C2 vector, where the kernels with rank less than r are... t The position of the convolution kernel corresponding to the current channel is represented by 1, which means that the convolution kernel at the current position is pruned; the position of the convolution kernel corresponding to the other channels is represented by 0, which means that the convolution kernel at the current position is not pruned.

[0121] The vector mask is obtained by taking the union of mask1 and mask2. The mask is used to mark the positions where the convolution kernels in conv1 and conv2 are pruned. The convolution kernels marked with pruning positions are then pruned.

[0122] After pruning model B according to the method of step 5, a pruned optimal BGNet stereo matching network model is obtained, and is denoted as model C.

[0123] In this embodiment, it is assumed that the number of channels of conv1 and conv2 is 16, mask1 is [0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0], that is, the convolution kernels corresponding to the second feature map, the fifth feature map and the eleventh feature map of conv1 need to be pruned, and mask2 is [1, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0], that is, the convolution kernels corresponding to the first feature map, the third feature map, the tenth feature map and the eleventh feature map of conv2 need to be pruned. The result of taking the union of mask1 and mask2 is mask, and mask is [1, 1, 1, 0, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0]. At this time, the convolution kernels corresponding to the first feature map, the second feature map, the third feature map, the fifth feature map, the tenth feature map and the eleventh feature map of conv1 and conv2 need to be pruned.

[0124] Step 6, fine-tuning training of model C on the enhanced synthetic training set in step 1 to restore the accuracy of model B to obtain a pruned and fine-tuned optimal BGNet stereo matching network model, and is denoted as model D.

[0125] In this embodiment, the fine-tuning training process is as follows:

[0126] The images in the enhanced synthetic training set are sent into model C for iterative training G times to obtain G trained fine-tuned BGNet stereo matching network models, and the G trained fine-tuned BGNet stereo matching network models are combined to form a fine-tuned model set M2, M2 = [M21, M22, …, M2ε, …, M2G], where M2ε is the fine-tuned BGNet stereo matching network model obtained by the εth training, and ε = 1, 2, 3, …, G. G

[0127] The G trained BGNet stereo matching network models are respectively used to calculate the fine-tuned endpoint errors of the synthetic test set to obtain G fine-tuned endpoint errors, and the G fine-tuned endpoint errors are combined to form a fine-tuned endpoint error set O2, O2 = [EPE21, EPE22, …, EPE2ε, …, EPE2G], where EPE2ε is the endpoint error corresponding to the fine-tuned BGNet stereo matching network model M2ε obtained by the εth training. G

[0128] The minimum value in the fine-tuned endpoint error set O2 is denoted as the optimal fine-tuned endpoint error EPE2​​min the optimal fine-tuned end-point error EPE2 min The corresponding fine-tuned BGNet stereo matching network model is the pruned and fine-tuned optimal BGNet stereo matching network model.

[0129] In this embodiment, the current training is the continued training of the pre-training, so the same iteration number G is taken, G = 100.

[0130] Step 7, generalization training of the model D obtained in step 6 on the real training set in step 1 to obtain a generalization optimal BGNet stereo matching network model, which is the final BGNet stereo matching network model.

[0131] In this embodiment, the generalization training process is as follows:

[0132] Randomly select Q5 groups of training samples in the real training set as training samples for use, and mark them as a generalization real training set, and the remaining (Q3-Q5) groups of training samples are used as verification, and are marked as a generalization verification set, Q5 < Q3;

[0133] Suppose the iteration number of the generalization training is V, and the images in the generalization real training set are sent into the model D for iteration training for V times, to obtain V generalization BGNet stereo matching network models trained, and the V generalization BGNet stereo matching network models trained are combined to form a generalization model set M3, M3 = [M31, M32, …, M3V], wherein M3e is the generalization BGNet stereo matching network model obtained by the e-th training, e = 1, 2, …, V; е V е

[0134] Calculate the generalization end-point error of the V generalization BGNet stereo matching network models on the generalization verification set respectively, to obtain V generalization end-point errors, and the V generalization end-point errors are combined to form a generalization end-point error set O3, O3 = [EPE31, EPE32, …, EPE3V], wherein EPE3e is the generalization end-point error corresponding to the e-th generalization BGNet stereo matching network model M3e, e = 1, 2, …, V; е З е е

[0135] Take the minimum value in the generalization end-point error set O3 as the optimal generalization end-point error EPE3opt, and the corresponding generalization BGNet stereo matching network model is the generalization optimal BGNet stereo matching network model. min min ​​​​​​​​​

[0136] In the embodiment, Q5=345, Q3-Q5=40, that is, 345 groups of training samples are randomly selected from the real training set as training samples for use, and are recorded as a generalization real training set, and the remaining 40 groups of training samples are used as verification, and are recorded as a generalization verification set.

[0137] In the embodiment, the number of iterations V of the generalization training is 1040.

[0138] In the embodiment, the module containing the convolution layer in the model A includes a feature extraction module, a guided filtering module and a cost aggregation module. The specific structures of the feature extraction module, the guided filtering module and the cost aggregation module are as follows:

[0139] The feature extraction module includes the following structures connected in series along the input-output direction of the network A: a first convolution structure unit, a residual structure unit, a second convolution structure unit, a first deconvolution structure unit, a third convolution structure unit and a second deconvolution structure unit.

[0140] The first convolution structure unit is composed of three two-dimensional convolution layers with a convolution kernel size of 3×3 and a channel number of 32 connected in series along the input-output direction of the network A.

[0141] The residual structure unit is composed of a first residual structure, a second residual structure, a third residual structure and a fourth residual structure connected in series along the input-output direction of the network A. The first residual structure is composed of two two-dimensional convolution layers with a convolution kernel size of 3×3 and a channel number of 32 connected in series along the input-output direction of the network A. The second residual structure is composed of two two-dimensional convolution layers with a convolution kernel size of 3×3 and a channel number of 64 connected in series along the input-output direction of the network A. The third residual structure is composed of two two-dimensional convolution layers with a convolution kernel size of 3×3 and a channel number of 128 connected in series along the input-output direction of the network A. The fourth residual structure is composed of two two-dimensional convolution layers with a convolution kernel size of 3×3 and a channel number of 128 connected in series along the input-output direction of the network A.

[0142] The second convolution structure unit includes the following four two-dimensional convolution layers connected in series along the input-output direction of the network A: a two-dimensional convolution layer with a convolution kernel size of 3×3 and a channel number of 32, a two-dimensional convolution layer with a convolution kernel size of 3×3 and a channel number of 48, a two-dimensional convolution layer with a convolution kernel size of 3×3 and a channel number of 64, and a two-dimensional convolution layer with a convolution kernel size of 3×3 and a channel number of 96.

[0143] The first deconvolution structural unit is composed of a first deconvolution module, a second deconvolution module and a third deconvolution module connected in series along the input-output direction of network A. The first deconvolution module is composed of a two-dimensional deconvolution layer and a two-dimensional convolution layer connected in series along the input-output direction of network A: the two-dimensional deconvolution layer has a convolution kernel size of 4x4 and a channel number of 64, and the two-dimensional convolution layer has a convolution kernel size of 3x3 and a channel number of 64. The second deconvolution module is composed of a two-dimensional deconvolution layer and a two-dimensional convolution layer connected in series along the input-output direction of network A: the two-dimensional deconvolution layer has a convolution kernel size of 4x4 and a channel number of 48, and the two-dimensional convolution layer has a convolution kernel size of 3x3 and a channel number of 48. The third deconvolution module is composed of a two-dimensional deconvolution layer and a two-dimensional convolution layer connected in series along the input-output direction of network A: the two-dimensional deconvolution layer has a convolution kernel size of 4x4 and a channel number of 32, and the two-dimensional convolution layer has a convolution kernel size of 3x3 and a channel number of 32.

[0144] The third convolution structural unit is composed of a first convolution module, a second convolution module and a third convolution module connected in series along the input-output direction of network A. The first convolution module is composed of two two-dimensional convolution layers with a convolution kernel size of 3x3 and a channel number of 48 connected in series along the input-output direction of network A. The second convolution module is composed of two two-dimensional convolution layers with a convolution kernel size of 3x3 and a channel number of 64 connected in series along the input-output direction of network A. The third convolution module is composed of two two-dimensional convolution layers with a convolution kernel size of 3x3 and a channel number of 96 connected in series along the input-output direction of network A.

[0145] The second deconvolution structural unit is composed of a fourth deconvolution module, a fifth deconvolution module and a sixth deconvolution module connected in series along the input-output direction of network A. The structures of the fourth deconvolution module, the fifth deconvolution module and the sixth deconvolution module are the same as those of the first deconvolution module, the second deconvolution module and the third deconvolution module of the first deconvolution structural unit, respectively.

[0146] The guided filtering module is composed of two two-dimensional convolution layers connected in series: a two-dimensional convolution layer with a convolution kernel size of 1x1 and a channel number of 16, and a two-dimensional convolution layer with a convolution kernel size of 1x1 and a channel number of 1.

[0147] The cost aggregation module includes a first three-dimensional convolution structural unit, a first three-dimensional deconvolution structural unit and a three-dimensional convolution layer with a convolution kernel size of 3x3x3 and a channel number of 32 connected in series along the input-output direction of network A.

[0148] The first three-dimensional convolution structure unit is composed of the following eight three-dimensional convolution layers connected in series along the input-output direction of network A: a three-dimensional convolution layer with a convolution kernel size of 3*3*3 and a channel number of 32, a three-dimensional convolution layer with a convolution kernel size of 3*3*3 and a channel number of 16, a three-dimensional convolution layer with a convolution kernel size of 3*3*3 and a channel number of 32, a three-dimensional convolution layer with a convolution kernel size of 3*3*3 and a channel number of 32, a three-dimensional convolution layer with a convolution kernel size of 3*3*3 and a channel number of 64, a three-dimensional convolution layer with a convolution kernel size of 3*3*3 and a channel number of 64, a three-dimensional convolution layer with a convolution kernel size of 3*3*3 and a channel number of 128, and a three-dimensional convolution layer with a convolution kernel size of 3*3*3 and a channel number of 128.

[0149] The first three-dimensional deconvolution structure unit is composed of the following three three-dimensional deconvolution layers connected in series along the input-output direction of network A: a three-dimensional deconvolution layer with a convolution kernel size of 3*4*4 and a channel number of 64, a three-dimensional deconvolution layer with a convolution kernel size of 3*4*4 and a channel number of 32, and a three-dimensional deconvolution layer with a convolution kernel size of 3*4*4 and a channel number of 16.

[0150] In order to demonstrate the effect of the present application, the model pruning method of the present application and other three model pruning methods are compared, and the comparison results are shown in the following table.

[0151] Model Parameter quantity / 10 6 ]]> EPE2 min ]] BGNet 5.31 0.99 BGNet-Net-slim 2.13 1.12 BGNet-FPGM 2.11 1.10 BGNet-Taylor 2.14 1.09 The present invention 2.01 1.06

[0152] The first column is the BGNet model after using different pruning methods, BGNet is the original model before pruning, BGNet-Net-slim is the pruning model obtained after using the Net-slim pruning method, BGNet-FPGM is the pruning model obtained after using the FPGM pruning method, and BGNet-Taylor is the pruning model obtained after using the Taylor pruning method; the second column is the parameter size of each model; and the third column is the optimal fine-tuning endpoint error of each model on the synthetic test set.

[0153] The pruning experiment results of the present application method and other three pruning methods on the BGNet stereo matching network show that the other three pruning methods compress the parameters of the BGNet stereo matching network by 60%, and the EPE2 min only increased by 0.1 to 0.13; while the present application method reduces the EPE2 min 0.03 compared with the best Taylor pruning method of the three pruning methods; and the endpoint EPE2 minis 0.99, which is already very low, no matter what pruning method is used, the performance of the model after pruning must be worse than the original model before pruning, especially in the case where the compression rate of model parameters is as high as 62%, while the EPE2 of the model after pruning by the method of the application is only 0.07 higher than that of the original model before pruning, which has reached a very small range. min The EPE2 of the model after pruning is only 0.07 higher than that of the original model before pruning, which has reached a very small range.

[0154] The above shows the advantages and effectiveness of the method of the application.

Claims

1. A model pruning method based on a BGNet stereo matching network, characterized in that, The method comprises the following steps: Step 1, downloading a synthetic data set from a public database, the synthetic data set comprising a synthetic training set and a synthetic test set, the synthetic training set comprising Q1 groups of training samples, the synthetic test set comprising Q2 groups of test samples, and each group of training samples and each group of test samples containing a left image, a right image and a real disparity image; Step 2, building a BGNet stereo matching network model, denoted as model A, the model A comprising a convolutional layer module including a feature extraction module, a guided filtering module and a cost aggregation module; The feature extraction module comprises the following structures connected in series along the input-output direction of the network A: a first convolutional structure unit, a residual structure unit, a second convolutional structure unit, a first deconvolutional structure unit, a third convolutional structure unit and a second deconvolutional structure unit; the residual structure unit comprises four residual structures connected in series along the input-output direction of the network A, wherein the output of the third residual structure is a feature map F1, and the output of the fourth residual structure is a feature map F2; the second convolutional structure unit comprises four two-dimensional convolutional layers connected in series along the input-output direction of the network A, wherein the output of the first two-dimensional convolutional layer is a feature map F3; the first deconvolutional structure unit and the second deconvolutional structure unit each comprise three deconvolutional modules connected in series along the input-output direction of the network A, and the final outputs thereof are feature maps F4 and F5, respectively; The guided filtering module comprises two two-dimensional convolutional layers connected in series along the input-output direction of the network A; the cost aggregation module comprises a first three-dimensional convolutional structure unit, a first three-dimensional deconvolutional structure unit and a three-dimensional convolutional layer connected in series along the input-output direction of the network A; Step 3, training the model A to obtain an optimal pre-train BGNet stereo matching network model, which is the BGNet stereo matching network model to be pruned, denoted as model B; Step 4, setting the convolutional layers not participating in pruning, and calculating the ranks of the feature maps in the convolutional layers of the model B obtained in step 3, the implementation process being as follows: Step 4.1, setting the following three kinds of convolutional layers not participating in pruning: Let the number of iterations be G. The images from the augmented synthesis training set are fed into model A in G iterations for training, resulting in G trained BGNet stereo matching network models, denoted as pre-trained BGNet stereo matching network models. These G pre-trained BGNet stereo matching network models are then grouped into a pre-trained model set M1, where M1 = [M11, M12, ..., M1...]. β M1 G ], of which M1 β The pre-trained BGNet stereo matching network model obtained after the βth training iteration, where β = 1, 2, ..., G; G pre-training BGNet stereo matching network models are respectively used to calculate the pre-training endpoint errors of the synthetic test set, G pre-training endpoint errors are obtained, and the G pre-training endpoint errors are combined into a pre-training endpoint error set O1, O1 = [EPE11, EPE12,..., EPE1G], wherein EPE1 β is the pre-training endpoint error corresponding to the pre-training BGNet stereo matching network model M1 β obtained after the βth training. G β β ​ The minimum value in the pre-training endpoint error set O1 is recorded as the optimal pre-training endpoint error EPE1 min The optimal endpoint error EPE1 min The corresponding pre-training BGNet stereo matching network model is the optimal pre-training BGNet stereo matching network model; The convolutional layers of the feature maps F1, F2, F3, F4 and F5 generated in the feature extraction module do not participate in pruning; All the convolutional layers in the guided filtering module do not participate in pruning; ​ ​ The last convolutional layer in the cost aggregation module is not pruned; All the convolutional layers in the model B are pruned except the above three convolutional layers, and the convolutional layers which are pruned are recorded as pruned convolutional layers; Step 4.2, randomly select N groups of training samples from the synthetic training set of step 1, and denote the set of the N groups of training samples as sample set P, P = [P1, P2,..., PN], P E is any one group of samples in sample set P, E = 1, 2,..., N; N E is any one group of samples in sample set P, E = 1, 2,..., N;​ inputting the N groups of training samples into the model B, each pruned convolutional layer in the model B generating N groups of feature maps, denoting the N groups of feature maps as a feature map set T, T = [T1, T2,... T E ..., T N ], T E is any one group of feature maps in the feature map set T; Step 4.3, when the parametric convolution layer is a two-dimensional convolution layer and a two-dimensional deconvolution layer, the feature map T E The resolution is recorded as Res1, Res1 = X × H × W, wherein X is the number of channels in the feature map T E , H is the height dimension of the feature map T E , and W is the width dimension of the feature map T E , at this time, the mathematical expression of each channel in the feature map T E is a two-dimensional matrix Y; When the parametric convolution layer is a three-dimensional convolution layer and a three-dimensional deconvolution layer, the resolution of the feature map T E is denoted as Res2, Res2=X×Z×H×W, Z is a parallax dimension of the feature map T E , at this time, the mathematical expression of each channel in the feature map T E is a three-dimensional matrix; the height dimension H and the width dimension W in the channel are fused into one dimension, i.e., the three-dimensional matrix is transformed into a two-dimensional matrix Y; Any one channel in the feature map T E is recorded as the Jth channel, and the rank of the Jth channel is recorded as The rank of the Jth channel The calculation formula is as follows: Wherein, matrix_rank() is a library function for solving the rank of a two-dimensional matrix Y in a deep learning library; The feature map T E The rank of X channels is denoted as a rank set R E , Step 4.4, repeat step 4.3 to obtain a rank set of N groups of feature maps in the feature map set T, and the rank of the Jth channel in the N groups of feature maps Average and record as the channel average rank r of the Jth channel J , the calculation formula is as follows: For each channel of N feature maps, calculate the channel average rank to obtain X channel average ranks, and write them into a queue of channel average ranks of the parametric pruning convolutional layer [r]. 1 r 2 ,...,r J ,...,r X ]; Step 5, pruning, the implementation process is as follows: Step 5.1, confirmation of the pruning threshold rearrange the channel average ranks in the channel average rank queue [r 1 , r 2 ,..., r J ,..., r X ] in ascending order to obtain a rearranged channel average rank queue [r' i , r' I ,..., r' x ], wherein r' i is the channel average rank of any one channel in the rearranged channel average rank queue R', i is the rearranged channel average rank number, i = 1, 2,..., X, and r' I is a to-be-confirmed pruning threshold, I = i + 1, and the limit formula of the to-be-confirmed pruning threshold r' I is as follows: In the formula, α is a scaling factor; Substituting I starting from 2 into the above constraint: if the pruning threshold r′ to be confirmed... I If the constraint is satisfied, then the pruning threshold r′ to be determined is... I It is identified as the pruning threshold; if the pruning threshold r′ to be identified is... I If the constraint is not met, gradually increase I until the pruning threshold r′ to be confirmed. I Until the above equation is satisfied; Step 5.2 When the clipping convolution layer is not in the residual module, the convolution kernel in the channel corresponding to the [r'1, r'2,..., r' ] segment in the rearranged channel average value queue [r'1, r'2,..., r' ] is clipped. i I x i ​​​​ Step 5.3, when the pruned convolutional layer is in the residual module, a limitation condition is added: the number of channels of the convolutional layer in the residual module before and after pruning is the same; The existing residual module is Residual module The front convolution layer of the residual module is conv1, the channel number of conv1 before pruning is C1, the channel number of conv1 after pruning is C1', the rear convolution layer of the residual module 6 is conv2, the channel number of conv2 before pruning is C2, the channel number of conv2 after pruning is C2', and C1=C2. Pruning is performed according to the following rules: When C1' = C2', the channel average queue [r'1, r'2, ..., r'] will be rearranged. i ,r' I ,...,r' x In the [r'1, r'2, ..., r'] i Remove the convolution kernels from the corresponding channels of the segment; When C1'≠C2', first, the pruning threshold of all channels of the feature map generated by conv1 is obtained respectively in the manner of step 5.1, and is recorded as the pruning threshold r of the previous convolution layer s The position of the convolution kernel corresponding to the channel to be pruned in the feature map generated by conv1 is marked with mask1, and mask1 is a 1xC1 vector, wherein the position of the convolution kernel corresponding to the channel with a rank less than r s is expressed by 1, that is, the current position convolution kernel is pruned, and the position of the convolution kernel corresponding to the remaining channel is expressed by 0, that is, the current position convolution kernel is not pruned. The threshold of the rank of all channel pruning of the feature map generated by conv2 is obtained according to step 5.1, and is recorded as the pruning threshold r of the next convolution layer t The position of the convolution kernel corresponding to the channel to be cropped in the feature map generated by conv2 is marked with mask2, which is a 1xC2 vector, wherein the position of the convolution kernel corresponding to the channel with a rank less than r is represented by 1, that is, the current position convolution kernel is cropped, and the position of the convolution kernel corresponding to the remaining channel is represented by 0, that is, the current position convolution kernel is not cropped. t The position of the convolution kernel corresponding to the channel to be cropped in the feature map generated by conv2 is marked with mask2, which is a 1xC2 vector, wherein the position of the convolution kernel corresponding to the channel with a rank less than r is represented by 1, that is, the current position convolution kernel is cropped, and the position of the convolution kernel corresponding to the remaining channel is represented by 0, that is, the current position convolution kernel is not cropped. The mask1 and the mask2 are taken as a set to obtain a vector mask, and the mask is used to mark the pruning position of the convolution kernel in the conv1 and the conv2 at the same time, and the convolution kernel marked with the pruning position is pruned; After the model B is pruned according to the method of step 5, an optimal BGNet stereo matching network model after pruning is obtained, and is recorded as a model C; Step 6, fine-tuning training of the model C on the enhanced synthetic training set in step 1 to restore the accuracy of the model B, to obtain a pruned and fine-tuned optimal BGNet stereo matching network model, and is recorded as a model D; Step 7, generalization training of the model D obtained in step 6 on the real training set in step 1, to obtain a generalization optimal BGNet stereo matching network model, which is the final BGNet stereo matching network model.

2. The model pruning method based on the BGNet stereo matching network according to claim 1, wherein, The data enhancement method includes random cropping, vertical disparity enhancement, brightness enhancement and contrast enhancement.

3. The model pruning method based on the BGNet stereo matching network according to claim 1, wherein, The pre-trained endpoint error EPE1 β The calculation formula is as follows: wherein: d est (x, y) is the pixel value at the horizontal coordinate x and the vertical coordinate y in the real disparity image provided in the synthesis test set; K is the pre-trained BGNet stereo matching network model M1 obtained after the βth training β The pixel value at the horizontal coordinate x and the vertical coordinate y in the output disparity image is d gt (x, y) is the pixel value at the horizontal coordinate x and the vertical coordinate y in the real disparity image provided in the synthesis test set; K is the pre-trained BGNet stereo matching network model M1 obtained after the βth training β The number of all pixel points in the output disparity image.

4. The model pruning method based on the BGNet stereo matching network according to claim 1, wherein, The implementation process of step 6 is specifically as follows: The enhanced images in the training set are sent into the model C for iterative training G times, to obtain G trained fine-tuned BGNet stereo matching network models, and the G trained fine-tuned BGNet stereo matching network models are combined to form a fine-tuned model set M2, M2 = [M21, M22,..., M2ε,..., M2G], wherein M2ε is a fine-tuned BGNet stereo matching network model obtained by the εth training, and ε = 1, 2, 3,..., G. i , M22,..., M2ε,..., M2 G G], wherein M2ε is a fine-tuned BGNet stereo matching network model obtained by the εth training, and ε = 1, 2, 3,..., G. G trained BGNet stereo matching network models are respectively used to calculate fine-tuning end point errors of the synthetic test set, G fine-tuning end point errors are obtained, and the G fine-tuning end point errors are combined to form a fine-tuning end point error set O2, O2=[EPE21, EPE22,..., EPE2ε,..., EPE2G], wherein EPE2ε is an end point error corresponding to a fine-tuning BGNet stereo matching network model M2e obtained by the εth training. G ] The minimum value in the fine-tuning endpoint error set O2 is recorded as the optimal fine-tuning endpoint error EPE2 min The optimal fine-tuning endpoint error EPE2 min The corresponding fine-tuning BGNet stereo matching network model is the pruned and fine-tuned optimal BGNet stereo matching network model.

5. The model pruning method based on the BGNet stereo matching network according to claim 4, characterized in that, The implementation process of step 7 is specifically as follows: In the real training set, Q5 groups of training samples are randomly selected as training samples, and are recorded as a generalization real training set, and the remaining (Q3-Q5) groups of training samples are used as verification, and are recorded as a generalization verification set, Q5 < Q3; Set the number of iterations of the generalization training as V, and send the images in the generalization real training set into the model D for iterative training V times, to obtain V trained generalization BGNet stereo matching network models, and group the V trained generalization BGNet stereo matching network models to form a generalization model set M3, M3 = [M31, M32, …, M3 e , …, M3 v ], wherein M3 e is the generalization BGNet stereo matching network model obtained through the e-th training, and e = 1, 2, …, V. V generalization BGNet stereo matching network models are respectively used to calculate the generalization endpoint error of the generalization verification set, V generalization endpoint errors are obtained, and the V generalization endpoint errors are combined into a generalization endpoint error set O3, O3 = [EPE31, EPE32,..., EPE3 e ,..., EPE3 v ], wherein EPE3 e is the endpoint error corresponding to the generalization BGNet stereo matching network model M3 e obtained by the e-th training. The minimum value in the generalization endpoint error set O3 is recorded as the optimal generalization endpoint error EPE3 min , the optimal generalization endpoint error EPE3 min The corresponding generalization BGNet stereo matching network model is the optimal BGNet stereo matching network model after generalization.

Images