Method and system for 3D reconstruction of garbage dump based on wide baseline light field imaging technology

Abstract

The invention discloses a method and system for 3D reconstruction of a landfill based on wide baseline light field imaging technology. The method comprises the following steps: S1, collecting landfill images using a multi-camera array; S2, restoring full light field images from the collected sparse camera array images; S3, selecting images of multiple view streams from the full light field images as input data; S4, using a wide baseline light field public data set LLF and a light field image sequence collected from the landfill as model training samples; S5, using the model training samples to train a deep neural network to obtain corresponding weights and biases, bringing the obtained weights and biases into a validation data set, calculating the prediction error of the validation data set, and saving the weights and biases that minimize the prediction error of the training set; S6, bringing the weights and biases that minimize the prediction error of the training set in S5 into a prediction set, and obtaining a predicted depth value by inference calculation. The invention has the advantages of low cost, high precision, good real-time performance, etc.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and in particular relates to a method and system for three-dimensional reconstruction of a garbage dump based on wide-baseline light field imaging technology. Background Art

[0002] With the rapid development of my country's economy and the acceleration of urbanization, industrial and domestic waste has exploded. Scientific and effective waste treatment is an important issue in modern urban environmental governance and sustainable development. Waste incineration power generation is the main way of urban waste treatment. It plays an important role in harmless and resource-based waste treatment and is a key link in achieving the country's major strategy of "carbon neutrality and carbon peak". Before incineration, garbage must undergo 5-7 days of fermentation and dehydration at the dumpsite. Therefore, the realization of digital twin intelligent management of garbage dumps has important guiding significance for the research on garbage calorific value prediction and garbage traceability tracking and monitoring technology in various areas of the city, and the realization of efficient and stable power generation. It is the focus and difficulty of the current urban waste management operation. The key link is the three-dimensional scene reconstruction of the garbage dump.

[0003] Three-dimensional reconstruction technology is a hot research topic in the field of machine vision, but research on three-dimensional reconstruction for landfills is relatively scarce. At present, the common three-dimensional reconstruction methods mainly include: passive three-dimensional reconstruction technology and active three-dimensional reconstruction technology. Passive three-dimensional reconstruction technology generally uses the reflection of the surrounding environment such as natural light, uses a camera to obtain images, and then calculates the three-dimensional spatial information of the object through a specific algorithm, mainly including texture recovery shape method, shadow recovery shape method and stereo vision method; active three-dimensional reconstruction technology refers to the use of light sources or energy sources such as lasers, sound waves, electromagnetic waves, etc. to transmit to the target object, and obtain the depth information of the object by receiving the returned light waves. It mainly includes four methods: moiré fringe method, time of flight method, structured light method and triangulation ranging method.

[0004] The landfill scene itself has the characteristics of large area, uneven surface, many occluded areas, and fast changing speed. Most existing passive 3D reconstruction methods are easily affected by surface texture and lighting conditions and cannot effectively solve the existing problems of the landfill. At the same time, the use of active 3D modeling technology also faces problems such as high equipment cost and easy corrosion, which is not suitable for this scene in terms of economic cost. Summary of the invention

[0005] The technical problem to be solved by the present invention is: in view of the problems existing in the prior art, the present invention provides a method and system for 3D reconstruction of a garbage dump based on wide-baseline light field imaging technology with low cost, high precision and good real-time performance.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0007] A method for 3D reconstruction of a landfill based on wide baseline light field imaging technology comprises the following steps:

[0008] Step S1, using a multi-camera array to collect images of a garbage dump;

[0009] Step S2, using deep compressed sensing technology to restore the full light field image from the acquired sparse camera array image;

[0010] Step S3, selecting images of multiple view streams from the reconstructed full light field image as input data;

[0011] Step S4, using the wide baseline light field public dataset LLF and the light field image sequences collected from the landfill as model training samples;

[0012] Step S5, using the model training samples to train the deep neural network to obtain corresponding weights and biases, bringing the obtained weights and biases into the verification data set, calculating the prediction error of the verification data set, and saving the weights and biases that minimize the prediction error of the training set;

[0013] Step S6, bringing the weights and biases that minimize the prediction error of the training set in step S5 into the prediction set, and obtaining the predicted depth value by inference calculation.

[0014] Preferably, step S2 comprises:

[0015] Step S2.1, constructing a full light field restoration network model;

[0016] Step S2.2, performing feature extraction operation;

[0017] Step S2.3, performing nonlinear mapping operation;

[0018] Step S2.4, perform feature aggregation operation;

[0019] Step S2.5, adding a long connection.

[0020] Preferably, in step S2.1, the full light field restoration network model includes an initial restoration model and a depth restoration model, which can be generally described as:

[0021] R(y)=D(I(y)) (1)

[0022] Where y is the sampling result obtained by the camera array in step S1, I is the initial recovery process, D is the depth recovery process, and R is the overall recovery process, as follows:

[0023] The initial recovery process I(y) is described as: Among them, lB 2 ×nB matrix is the sampling matrix ΦB The inverse matrix of For recovered data;

[0024] In the initial recovery model, a bias-free convolutional layer is used to simulate this process, which can be expressed as W int For IB 2 The size is 1×1×n B The filter with a step size of Each column is of size 1×1×lB 2 The reconstruction vectors are deformed and linked to obtain the initial reconstructed light field data

[0025] In step S2.2, the feature extraction operation is used to generate high-dimensional features from the regional receptive field. The process consists of a convolutional layer connected to an activation layer. The convolutional layer is assumed to consist of d filters of size f×f×l; this process is expressed as Where W is a series of filters, B e is a bias term of size d×1, Act(·) is an activation operation, using the ReLU activation function;

[0026] In step S2.3, the nonlinear mapping operation is used to increase the network nonlinearity and the related receptive field to ensure the reconstruction quality; the operation process consists of residual blocks, convolutional layers and activation layers, which can be expressed as:

[0027]

[0028]

[0029] In the formula, i∈{1,2,3···,n} is the residual block, and Both contain d filters of size f×f×d. and is an offset term of size ×1,

[0030] In step S2.4, feature aggregation operation is used to recover light field data from high-dimensional features. The operation process can be expressed as Where W a is a filter of size f×f×d, B a is a bias term of size l×1;

[0031] In step S2.5, the light field data is initially reconstructed Output of the model with deep reconstruction Long connections are added between them to accelerate network aggregation. The final deep reconstruction process can be expressed as

[0032] Preferably, the process of step S5 is:

[0033] Step S5.1, determine the error threshold ε of the neural network back propagation and set the hyperparameters;

[0034] Step S5.2, each branch extracts features from the input image;

[0035] Step S5.3, performing cost volume generation operation on the extracted feature elements;

[0036] Step S5.4, adaptively cross-scale aggregation is performed on the multi-scale fusion cost volume generated in the above steps.

[0037] Step S5.5, cost regression;

[0038] Step S5.6, disparity regression.

[0039] Step S5.7, calculating the loss function for parameter optimization of model training;

[0040] Step S5.8, return.

[0041] Preferably, in step S5.1, the number of initialization iterations I=0 is set, and the maximum number of iterations is IMAT; the weight matrix w and the deviation β of each network layer are randomly initialized;

[0042] In step S5.2, the feature extraction operation is used to generate high-dimensional features from the regional receptive field. The process consists of three feature extraction convolution modules, each of which contains a strided convolution layer, a 2D convolution layer, a batch normalization layer, and an activation layer. The stride of the former convolution layer is set to 2 for downsampling input, while the latter is set to 1, using the ReLU activation function.

[0043] In step S5.3, the cost volume is generated by using operations such as shift interpolation, cost calculation and fusion on the four generated feature element streams;

[0044] In step S5.4, the final cost volume is obtained by adaptively combining the results of cost aggregation performed at different scales; the algorithm is approximated as

[0045]

[0046] in, is the resulting cost volume after cross-scale cost aggregation, is the aggregate cost within the scale k. kis a general function used to achieve adaptive combination between cost quantities at each scale; specifically, for the cost quantity For:

[0047]

[0048] Where I represents the unit function, (sk) stride-2 3×3 convolution is used for 2 s-k times downsampled to achieve the same resolution, while It means that the same resolution is first bilinearly upsampled, followed by a 1×1 convolution to align the number of channels;

[0049] In step S5.5, two types of 3D attention networks, view attention and flow attention, are used. The view attention module is used to exploit the interdependence between feature channels in the same angular direction, while the flow attention module is used to exploit the interdependence between feature channels in different angular directions.

[0050] In step S5.6, the expected value of the weighted disparity is calculated using the differential soft argmin operator to regress the continuous disparity The final disparity map is obtained as shown in formula (8):

[0051]

[0052] In step S5.7, a smooth L1 loss function is used in the model, which calculates the difference between the predicted disparity value and the true disparity value, as shown in formulas (9)-(10):

[0053]

[0054]

[0055] in is the predicted disparity value, g i is the actual disparity value.

[0056] Step S5.8, when Loss>ε or I≤IMAX, return to step S5.1.

[0057] Preferably, the specific process of step S5.3 is:

[0058] Step S5.31, shift interpolation;

[0059] Step S5.32, cost calculation;

[0060] Step S5.33, fusion.

[0061] Preferably, in step S5.31, the feature element map of the central view is used as a reference, and other feature maps along the flow direction are used as target feature maps F t , through each hypothesized disparity in the disparity range Target View F t The feature map is shifted to the reference view, and then bilinear interpolation is used to calculate the appropriate value for each pixel at the sub-pixel position:

[0062]

[0063] where d min d max and L represent the minimum disparity, maximum disparity and discrete disparity in the range respectively; the size of the feature map of each target view and reference view is (L×H 1 ×W 1 ×C 1 ), where L represents the scale level and C represents the channel;

[0064] In step S5.32, after obtaining the feature map at scale l from the target view stream, the matching cost is calculated using the connection between the reference view and the target view feature map to form a 4D cost volume, as shown in formula (4) and formula (5):

[0065]

[0066] c i represents all feature channels of view I;

[0067] In step S5.33, the computational costs between the views and streams in the previous step are fused, and the fusion module includes segmentation, splicing fusion and fusion operations;

[0068] Specifically, the cross-line light field input is divided into (N-1) / 2 groups, where N is the total number of baselines. Each group has the same view baseline and contains five views, including a reference view and its four adjacent target views. For each group of light field images, all feature maps are first cascaded and fused across channel dimensions, and then all groups are fused. The final fusion output consists of a horizontal part CV_0° and a vertical part CV_90°, with a size of L l ×H l ×W l ×6C.

[0069] The present invention also discloses a garbage dump 3D reconstruction system based on wide baseline light field imaging technology, comprising:

[0070] The first module is used to collect images of the garbage dump using a multi-camera array;

[0071] The second module uses deep compressed sensing technology to restore the full light field image from the acquired sparse camera array image;

[0072] The third module selects images of multiple view streams from the reconstructed full light field image as input data;

[0073] The fourth module uses the light field image sequences collected from the wide baseline light field public dataset LLF and the landfill as model training samples;

[0074] The fifth module uses the model training samples to train the deep neural network to obtain the corresponding weights and biases, and then brings the obtained weights and biases into the validation data set to calculate the prediction error of the validation data set, and save the weights and biases that minimize the prediction error of the training set;

[0075] In the sixth module, the weights and biases in the fifth module that minimize the prediction error of the training set are brought into the prediction set, and the predicted depth value is obtained by inference calculation.

[0076] The present invention further discloses a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the method for three-dimensional reconstruction of a landfill based on wide-baseline light field imaging technology as described above are executed.

[0077] The present invention also discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is run by the processor, the steps of the three-dimensional reconstruction method of a landfill based on wide baseline light field imaging technology are executed as described above.

[0078] Compared with the prior art, the advantages of the present invention are:

[0079] The overall idea of ​​the present invention is to propose a multi-scale wide baseline light field deep neural network model (ALFNet) based on a combination of a deep convolutional neural network (CNN) and a wide baseline light field to extract multi-scale depth features of the landfill for coupling and then predicting the depth map of the central view. In order to fully obtain the depth of each perspective and improve the prediction accuracy, images of four perspectives of 0°, 90°, 45° and 135° are selected as input data. In the first network module, the Resnet residual network structure is adopted to extract multi-scale depth features and associate them; in the second network module, given four mixed depth feature streams, a series of operations such as shift interpolation, cost calculation and fusion are used to generate the cost volume; in the third network module, six cross-scale aggregation modules are introduced to further improve the depth prediction accuracy by fusing depth features of different scales; in the fourth network module, a 3D attention module is used to eliminate partial occlusion problems, and a 3D codec is used to regularize the output of the attention network in the disparity dimension; finally, a soft argmin operator is introduced to calculate the expected value of the weighted difference to regress the continuous disparity. By constructing these five modules, the disparity of the central view of the landfill is inferred to obtain richer depth feature information, further improving the modeling accuracy of the constructed ALFNet.

[0080] One of the main advantages of the present invention is that it uses deep compressed sensing technology to construct a full light field image restoration network. By inputting 3×3 multi-camera array images collected at a landfill, a 9×9 full light field image can be restored, which solves the difficulties of existing light field camera equipment, such as high cost and strong scene dependence.

[0081] The second main advantage of the present invention is that a cross-scale aggregation algorithm is used to target the large garbage dump scene and many similar scenes. When downsampling the image, the texture information will be more distinguishable at the same block size. In low-texture or textureless areas, searching for correspondences at a coarse scale is beneficial to the reconstruction quality. Therefore, the present invention introduces multi-scale interaction into the traditional cross-scale cost aggregation algorithm, further enhancing the texture recognition of similar scenes in the garbage dump and reducing the probability of mismatching.

[0082] The third main advantage of the present invention lies in the use of the attention mechanism. Three-dimensional real-world points visible in the reference view may be occluded by foreground objects in the target view, which makes it difficult to find the correspondence. Using the idea that "the target point is occluded in the horizontal view stream, but may be less occluded or not occluded in the vertical view stream. For any stream composed of multiple views, the target point is not visible in some target views, but may be visible in other views", a 3D attention module is designed, which includes a view attention module and a stream attention module, which makes full use of the interdependence between feature channels in the same angular direction and the interdependence between feature channels in different angular directions (cross directions), thereby improving the depth reasoning accuracy of occluded objects.

[0083] The present invention utilizes the principle of light field imaging to design a data acquisition method in which a wide baseline camera array collects light field images of a landfill at different angles and uses deep compressed sensing technology to restore the full light field image as the input of the model; in view of the characteristics of the landfill such as strong occlusion and large scene, a 3D reconstruction model based on deep learning is proposed to realize real-time 3D reconstruction of the landfill, which has stronger generalization performance and better real-time reasoning performance than the traditional light field 3D reconstruction method; different from the traditional 3D reconstruction method of stereo vision matching, this method adopts cross-scale coupling and 3D convolution blocks to improve the real-time reasoning of 3D reconstruction in view of the characteristics of the landfill scene such as large size and fast change speed; at the same time, through the use of 3D attention mechanism, the difficulty of strong occlusion of the landfill is further overcome. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] Figure 1 It is a flow chart of the method of the present invention in an embodiment.

[0085] Figure 2 It is a flow chart of restoring the full light field image using the deep compressed sensing technology in the present invention.

[0086] Figure 3 This is a diagram of the network structure for restoring the full light field image in the present invention.

[0087] Figure 4 This is the ALFNet network structure diagram in the present invention.

[0088] Figure 5 It is a detailed structural diagram of the cost fusion module in the ALFNet network of the present invention.

[0089] Figure 6 It is a detailed structural diagram of the attention module in the ALFNet network of the present invention.

[0090] Figure 7 This is the depth prediction map of the landfill 3D reconstruction based on this method. DETAILED DESCRIPTION

[0091] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0092] The purpose of the present invention is to provide a three-dimensional reconstruction method of a garbage dump based on wide-baseline light field imaging technology, so as to realize real-time three-dimensional modeling and imaging of the garbage dump in the garbage incineration plant, thereby laying the foundation for the subsequent construction of a digital twin system and the realization of the garbage traceability technical solution.

[0093] like Figure 1 As shown, the method for 3D reconstruction of a landfill based on wide baseline light field imaging technology in this embodiment specifically includes the following steps:

[0094] Step S1, using a 3×3 multi-camera array to collect images of the garbage dump;

[0095] Step S2, using deep compressed sensing technology to restore a 9×9 full light field image from the acquired sparse camera array image;

[0096] Step S3, selecting images of four viewing angle streams of 0°, 90°, 45° and 135° from the reconstructed 9×9 full light field image as input data;

[0097] Step S4, using the wide baseline light field public dataset LLF and 1000 light field image sequences collected from the landfill as model training samples, divided into 75% training dataset, 20% validation dataset, and 5% test dataset;

[0098] Step S5, using the training data set to train the deep neural network to obtain corresponding weights and biases, bringing the obtained weights and biases into the verification data set, calculating the prediction error of the verification data set, and saving the weights and biases that minimize the prediction error of the training set;

[0099] Step S6, bringing the weights and biases that minimize the prediction error of the training set in step S5 into the prediction set to infer and calculate the predicted depth value, thereby completing the reconstruction of the garbage dump.

[0100] In a specific embodiment, if Figure 2 As shown, step S2 specifically includes:

[0101] Step S2.1, constructing a full light field restoration network model; the network model includes an initial restoration model and a depth restoration model, which can be abstractly described as follows:

[0102] R(y)=D(I(y)) (1)

[0103] Where y is the sampling result obtained by the 3×3 camera array in step S1, I is the initial recovery process, D is the deep recovery process (which also includes feature extraction, nonlinear mapping and feature aggregation), and R is the overall recovery process. The specific explanations of each part are as follows:

[0104] The initial recovery process I(y) can be described as: Among them, lB 2 ×n B matrix is the sampling matrix Φ B The inverse matrix of For recovered data;

[0105] In the initial recovery model, a bias-free convolutional layer is used to simulate this process, which can be expressed as W int For IB 2 The size is 1×1×n B The filter has a step size of 1×1. Each column is of size 1×1×lB 2 The reconstruction vectors of these vectors can be deformed and linked to obtain the initial reconstructed light field data

[0106] Step S2.2, perform feature extraction operation. This operation is used to generate high-dimensional features from the regional receptive field. The process consists of a convolution layer connected to an activation layer. The convolution layer is set to consist of d filters of size f×f×l. This process can be expressed as Where W e is a series of filters, B e is a bias term of size d×1, Act(·) is an activation operation, using the ReLU activation function;

[0107] Step S2.3, perform nonlinear mapping operation. This operation is used to increase the network nonlinearity and related receptive field to ensure the reconstruction quality. The operation process consists of residual blocks, convolutional layers and activation layers, which can be expressed as:

[0108]

[0109]

[0110] In the formula, i∈{1,2,3···,n} is a residual block (which allows the input and output of the convolutional layer to be temporarily connected). and Both contain d filters of size f×f×d. and is an offset term of size d×1,

[0111] Step S2.4, perform feature aggregation operation. This operation is used to recover light field data from high-dimensional features. The operation process can be described as Where W a is a filter of size f×f×d, B a is a bias term of size l×1;

[0112] Step S2.5, initially reconstructing the light field data Output of the model with deep reconstruction The purpose of adding long connections between is to accelerate network aggregation. The final deep reconstruction process can be expressed as

[0113] In a specific embodiment, if Figure 3 As shown, step S5 specifically includes:

[0114] Step S5.1, determine the error threshold ε of the neural network back propagation and set the hyperparameters. Set the initialization iteration number I = 0, and the maximum iteration number is IMAX; randomly initialize the weight matrix w and bias β of each network layer;

[0115] In step S5.2, each branch performs feature extraction on the input image. This operation is used to generate high-dimensional features from the regional receptive field. The process consists of three feature extraction convolution modules, each of which contains a strided convolution layer, a 2D convolution layer, a batch normalization layer, and an activation layer. The stride of the former convolution layer is set to 2 for downsampling input, while the latter is set to 1, using the ReLU activation function.

[0116] Step S5.3, performing cost volume generation operation on the extracted feature elements. The four feature element streams generated in the above steps are subjected to operations such as shift interpolation, cost calculation and fusion to generate a cost volume;

[0117] The specific process of step S5.3 is as follows:

[0118] Step S5.31, shift interpolation. The feature feature map of the center view is used as a reference, and other feature maps along the flow direction are used as target feature maps F t , through each hypothesized disparity in the disparity range Target View F t The feature map of is shifted to the reference view (see formula (4)), and then the appropriate value is calculated for each pixel at the sub-pixel position using bilinear interpolation:

[0119]

[0120] where d min d maxand L represent the minimum disparity, maximum disparity, and discrete disparity in the range, respectively. The size of the feature map of each target view and reference view is (L×H 1 ×W 1 ×C 1 ), where L represents the scale level and C represents the channel.

[0121] Step S5.32, cost calculation. After obtaining the feature map at scale l from the target view stream, the matching cost is calculated using the connection between the reference view and the target view feature map to form a 4D cost volume, as shown in formula (4) and formula (5):

[0122]

[0123] represents all feature channels of view I.

[0124] Step S5.33, fusion. The computational costs between the views and streams in the previous step are fused. The fusion module includes segmentation, splicing fusion and fusion operations, which aims to fuse the costs of multiple benchmark cost volumes. Its detailed structure is as follows: Figure 3 and Figure 5 As shown. In order to eliminate image redundancy between adjacent baselines, cross-baseline fusion is adopted. Specifically, the cross-line light field input is divided into (N-1) / 2 groups, where N is the total number of baselines. Each group has the same view baseline and contains five views, including a reference view and its four adjacent target views. For each group of light field images, all feature maps are first cascaded and fused across channel dimensions, and then all groups are fused. The final fusion output consists of a horizontal part CV_0° and a vertical part CV_90°, with a size of L l ×H l ×W l ×6C.

[0125] Step S5.4, adaptively cross-scale aggregate the multi-scale fusion cost volume generated in the above steps. The final cost volume is obtained by adaptively combining the results of cost aggregation performed at different scales. The algorithm is approximated as

[0126]

[0127] in, is the resulting cost volume after cross-scale cost aggregation, is the aggregate cost within the scale k. k is a general function that implements adaptive combination of cost quantities at each scale.

[0128] Specifically, for the cost For:

[0129]

[0130] Where I represents the unit function, and (sk) stride-23×3 convolution is used for 2 s-k times downsampled to achieve the same resolution, while It means first bilinear upsampling to the same resolution, followed by 1×1 convolution to align the number of channels.

[0131] Step S5.5, cost regression. Two types of 3D attention networks are used: view attention and flow attention. Figure 6 As shown in , the view attention module is used to exploit the interdependence between feature channels in the same angular direction, while the stream attention module is used to exploit the interdependence between feature channels in different angular directions. Figure 4 As shown in Figure 1, global average pooling is applied to capture channel-level statistics of the cost volume, and then a simple gating mechanism is used to capture the channel correlation of the cost volume. In this gating mechanism, the Softmax operator in the view attention block is responsible for outputting an attention map containing multiple different weights for features in each stream; the Sigmoid operator in the stream attention block assigns these two weights to features from the four streams respectively. Therefore, interdependent features of the cost volume are assigned larger weights, and vice versa. After the attention network, a 3D encoder-decoder network is used to regularize the output of the attention network on the disparity dimension, which consists of three 3D convolutions and two transposed convolutions. Finally, bilinear interpolation is used to resize the input back to the same spatial resolution, and the output has a dimension of L×H×W×1.

[0132] Step S5.6, parallax regression. Use the differential soft argmin operator to calculate the expected value of the weighted parallax to regress the continuous parallax. The final disparity map is obtained as shown in formula (8):

[0133]

[0134] Step S5.7, calculate the loss function for parameter optimization of model training. In this model, the smooth L1 loss function is used, which calculates the difference between the predicted disparity value and the true disparity value, as shown in formulas (9)-(10):

[0135]

[0136]

[0137] in is the predicted disparity value, g i is the actual disparity value.

[0138] Step S5.8, when Loss>ε or I≤IMAX, return to step S5.1.

[0139] like Figure 7 As shown, the above method can realize the three-dimensional reconstruction depth prediction of the landfill.

[0140] The overall idea of ​​the present invention is to propose a multi-scale wide baseline light field deep neural network model (ALFNet) based on the combination of deep convolutional neural network (CNN) and wide baseline light field. Figure 4 As shown in the figure, the multi-scale depth features of the garbage dump are extracted and coupled to predict the depth map of the central view. In order to fully obtain the depth of each perspective and improve the prediction accuracy, images of four perspectives of 0°, 90°, 45° and 135° are selected as input data. In the first network module, the Resnet residual network structure is adopted to extract and associate multi-scale depth features; in the second network module, given four mixed depth feature streams, a series of operations such as shift interpolation, cost calculation and fusion are used to generate the cost volume; in the third network module, six cross-scale aggregation modules are introduced to further improve the depth prediction accuracy by fusing depth features of different scales; in the fourth network module, the 3D attention module is used to eliminate partial occlusion problems, and the 3D codec is used to regularize the output of the attention network in the disparity dimension; finally, the soft argmin operator is introduced to calculate the expected value of the weighted difference to regress the continuous disparity. By constructing these five modules, the disparity of the central view of the garbage dump is inferred, richer depth feature information is obtained, and the modeling accuracy of the constructed ALFNet is further improved.

[0141] One of the main advantages of the present invention is that it uses deep compressed sensing technology to construct a full light field image restoration network. By inputting 3×3 multi-camera array images collected at a landfill, a 9×9 full light field image can be restored, which solves the difficulties of existing light field camera equipment, such as high cost and strong scene dependence.

[0142] The second main advantage of the present invention is that a cross-scale aggregation algorithm is used to target the large garbage dump scene and many similar scenes. When downsampling the image, the texture information will be more distinguishable at the same block size. In low-texture or textureless areas, searching for correspondences at a coarse scale is beneficial to the reconstruction quality. Therefore, the present invention introduces multi-scale interaction into the traditional cross-scale cost aggregation algorithm, further enhancing the texture recognition of similar scenes in the garbage dump and reducing the probability of mismatching.

[0143] The third main advantage of the present invention lies in the use of the attention mechanism. Three-dimensional real-world points visible in the reference view may be occluded by foreground objects in the target view, which makes it difficult to find the correspondence. Using the idea that "the target point is occluded in the horizontal view stream, but may be less occluded or not occluded in the vertical view stream. For any stream composed of multiple views, the target point is not visible in some target views, but may be visible in other views", a 3D attention module is designed, which includes a view attention module and a stream attention module, which makes full use of the interdependence between feature channels in the same angular direction and the interdependence between feature channels in different angular directions (cross directions), thereby improving the depth reasoning accuracy of occluded objects.

[0144] The present invention also discloses a garbage dump 3D reconstruction system based on wide baseline light field imaging technology, comprising:

[0145] The first module is used to collect images of the garbage dump using a multi-camera array;

[0146] The second module uses deep compressed sensing technology to restore the full light field image from the acquired sparse camera array image;

[0147] The third module selects images of multiple view streams from the reconstructed full light field image as input data;

[0148] The fourth module uses the light field image sequences collected from the wide baseline light field public dataset LLF and the landfill as model training samples;

[0149] The fifth module uses the model training samples to train the deep neural network to obtain the corresponding weights and biases, and then brings the obtained weights and biases into the validation data set to calculate the prediction error of the validation data set, and save the weights and biases that minimize the prediction error of the training set;

[0150] In the sixth module, the weights and biases in the fifth module that minimize the prediction error of the training set are brought into the prediction set, and the predicted depth value is obtained by inference calculation.

[0151] The system of the present invention corresponds to the above method and also has the advantages described in the above method.

[0152] The present invention further discloses a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method for 3D reconstruction of a landfill based on wide baseline light field imaging technology as described above are performed. The present invention also discloses a computer device, including a memory and a processor, on which a computer program is stored, and when the computer program is executed by the processor, the steps of the method for 3D reconstruction of a landfill based on wide baseline light field imaging technology as described above are performed. The present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing related hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, the steps of each of the above-mentioned method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc. The memory can be used to store computer programs and / or modules. The processor implements various functions by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory. The memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices, etc.

[0153] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A method for 3D reconstruction of a landfill based on wide baseline light field imaging technology, characterized in that: Includes steps: Step S1, using a multi-camera array to collect images of a garbage dump; Step S2, using deep compressed sensing technology to restore the full light field image from the acquired sparse camera array image; Step S3, selecting images of multiple view streams from the reconstructed full light field image as input data; Step S4, using the wide baseline light field public dataset LLF and the light field image sequences collected from the landfill as model training samples; Step S5, using the model training samples to train the deep neural network to obtain corresponding weights and biases, bringing the obtained weights and biases into the verification data set, calculating the prediction error of the verification data set, and saving the weights and biases that minimize the prediction error of the training set; In the first network module, the Resnet residual network structure is used to extract and associate multi-scale deep features; in the second network module, given four mixed deep feature streams, shift interpolation, cost calculation and fusion operations are used to generate cost volumes; in the third network module, six cross-scale aggregation modules are introduced to further improve the depth prediction accuracy by fusing deep features of different scales; in the fourth network module, a 3D attention module is used to eliminate partial occlusion problems, and a 3D codec is used to regularize the output of the attention network in the disparity dimension; finally, a soft argmin operator is introduced to calculate the expected value of the weighted difference to regress continuous disparity; Step S6, bringing the weights and biases that minimize the prediction error of the training set in step S5 into the prediction set, and obtaining the predicted depth value by inference calculation.

2. The method for 3D reconstruction of a landfill based on wide baseline light field imaging technology according to claim 1, characterized in that: The step S2 comprises: Step S2.1, constructing a full light field restoration network model; Step S2.2, performing feature extraction operation; Step S2.3, performing nonlinear mapping operation; Step S2.4, perform feature aggregation operation; Step S2.5, adding a long connection.

3. The method for 3D reconstruction of a landfill based on wide baseline light field imaging technology according to claim 2 is characterized in that: In step S2.1, the full light field restoration network model includes an initial restoration model and a depth restoration model, which can be abstractly described as follows: R(y)=D(I(y)) (1) Where y is the sampling result obtained by the camera array in step S1, I is the initial recovery process, D is the depth recovery process, and R is the overall recovery process, as follows: The initial recovery process I(y) is described as: Among them, lB 2 ×n B matrix is the sampling matrix Φ B The inverse matrix of For recovered data; In the initial recovery model, a bias-free convolutional layer is used to simulate this process, which can be expressed as W int For IB 2 The size is 1×1×n B The filter has a step size of 1×1; Each column is of size 1×1×lB 2 The reconstruction vectors are deformed and linked to obtain the initial reconstructed light field data In step S2.2, the feature extraction operation is used to generate high-dimensional features from the regional receptive field. The process consists of a convolutional layer connected to an activation layer. The convolutional layer is assumed to consist of d filters of size f×f×l; this process is expressed as Where W e is a series of filters, B e is a bias term of size d×1, Act(·) is an activation operation, using the ReLU activation function; In step S2.3, the nonlinear mapping operation is used to increase the network nonlinearity and the related receptive field to ensure the reconstruction quality; the operation process consists of residual blocks, convolutional layers and activation layers, which can be expressed as: In the formula, i∈{1,2,3···,n} is the residual block, and Both contain d filters of size f×f×d. and is an offset term of size d×1, In step S2.4, feature aggregation operation is used to recover light field data from high-dimensional features. The operation process can be expressed as Where W a is a filter of size f×f×d, B a is a bias term of size l×1; In step S2.5, the light field data is initially reconstructed Output of the model with deep reconstruction Long connections are added between them to accelerate network aggregation. The final deep reconstruction process can be expressed as 4. The method for 3D reconstruction of a landfill based on wide baseline light field imaging technology according to claim 3 is characterized in that: The process of step S5 is as follows: Step S5.1, determine the error threshold ε of the neural network back propagation and set the hyperparameters; Step S5.2, each branch extracts features from the input image; Step S5.3, performing cost volume generation operation on the extracted feature elements; Step S5.4, performing adaptive cross-scale aggregation on the multi-scale fusion cost volume generated in the above steps; Step S5.5, cost regression; Step S5.6, disparity regression; Step S5.7, calculating the loss function for parameter optimization of model training; Step S5.8, return.

5. The method for 3D reconstruction of a landfill based on wide baseline light field imaging technology according to claim 4 is characterized in that: In step S5.1, set the number of initialization iterations I=0, and the maximum number of iterations is IMAX; randomly initialize the weight matrix w and bias β of each network layer; In step S5.2, the feature extraction operation is used to generate high-dimensional features from the regional receptive field. The process consists of three feature extraction convolution modules, each of which contains a strided convolution layer, a 2D convolution layer, a batch normalization layer, and an activation layer. The stride of the former convolution layer is set to 2 for downsampling input, while the latter is set to 1, using the ReLU activation function. In step S5.3, the cost volume is generated by using operations such as shift interpolation, cost calculation and fusion on the four generated feature element streams; In step S5.4, the final cost volume is obtained by adaptively combining the results of cost aggregation performed at different scales; the algorithm is approximated as in, is the resulting cost volume after cross-scale cost aggregation, is the aggregate cost within the scale k; f k is a general function used to achieve adaptive combination between cost quantities at each scale; specifically, for the cost quantity For: Where I represents the unit function, (sk) stride-2 3×3 convolution is used for 2 s-k times downsampled to achieve the same resolution, while It means that the same resolution is first bilinearly upsampled, followed by a 1×1 convolution to align the number of channels; In step S5.5, two types of 3D attention networks, view attention and flow attention, are used. The view attention module is used to exploit the interdependence between feature channels in the same angular direction, while the flow attention module is used to exploit the interdependence between feature channels in different angular directions. In step S5.6, the expected value of the weighted disparity is calculated using the differential soft argmin operator to regress the continuous disparity The final disparity map is obtained as shown in formula (8): d min ,d max Respectively represent the minimum and maximum disparity within the range; In step S5.7, a smooth L1 loss function is used in the model, which calculates the difference between the predicted disparity value and the true disparity value, as shown in formulas (9)-(10): in is the predicted disparity value, g i is the true disparity value; Step S5.8, when Loss>ε or I≤IMAX, return to step S5.

1.

6. The method for 3D reconstruction of a landfill based on wide baseline light field imaging technology according to claim 5 is characterized in that: The specific process of step S5.3 is as follows: Step S5.31, shift interpolation; Step S5.32, cost calculation; Step S5.33, fusion.

7. The method for 3D reconstruction of a landfill based on wide baseline light field imaging technology according to claim 6, characterized in that: In step S5.31, the feature feature map of the center view is used as a reference, and other feature maps along the flow direction are used as target feature maps F t , through each hypothesized disparity in the disparity range Target View F t The feature map is shifted to the reference view, and then bilinear interpolation is used to calculate the appropriate value for each pixel at the sub-pixel position: where d min ,d max and L represent the minimum disparity, maximum disparity and discrete disparity in the range respectively; the size of the feature map of each target view and reference view is (S×H 1 ×W 1 ×C 1 ), where S represents the scale level and C represents the channel; In step S5.32, after obtaining the feature map at scale l from the target view stream, the matching cost is calculated using the connection between the reference view and the target view feature map to form a 4D cost volume, as shown in formula (4) and formula (5): c i represents all feature channels of view I; In step S5.33, the computational costs between the views and streams in the previous step are fused, and the fusion module includes segmentation, splicing fusion and fusion operations; Specifically, the cross-line light field input is divided into (N-1) / 2 groups, where N is the total number of baselines. Each group has the same view baseline and contains five views, including a reference view and its four adjacent target views. For each group of light field images, all feature maps are first cascaded and fused across channel dimensions, and then all groups are summed and fused. The final fusion output consists of a horizontal part CV_0° and a vertical part CV_90°, with a size of L l ×H l ×W l ×6C.

8. A 3D reconstruction system for a landfill based on wide baseline light field imaging technology, used to execute the steps of the 3D reconstruction method for a landfill based on wide baseline light field imaging technology as claimed in any one of claims 1 to 7, characterized in that: include: The first module is used to collect images of the garbage dump using a multi-camera array; The second module uses deep compressed sensing technology to restore the full light field image from the acquired sparse camera array image; The third module selects images of multiple view streams from the reconstructed full light field image as input data; The fourth module uses the light field image sequences collected from the wide baseline light field public dataset LLF and the landfill as model training samples; The fifth module uses the model training samples to train the deep neural network to obtain the corresponding weights and biases, and then brings the obtained weights and biases into the validation data set to calculate the prediction error of the validation data set, and save the weights and biases that minimize the prediction error of the training set; In the first network module, the Resnet residual network structure is used to extract and associate multi-scale deep features; in the second network module, given four mixed deep feature streams, shift interpolation, cost calculation and fusion operations are used to generate cost volumes; in the third network module, six cross-scale aggregation modules are introduced to further improve the depth prediction accuracy by fusing deep features of different scales; in the fourth network module, a 3D attention module is used to eliminate partial occlusion problems, and a 3D codec is used to regularize the output of the attention network in the disparity dimension; finally, a soft argmin operator is introduced to calculate the expected value of the weighted difference to regress continuous disparity; In the sixth module, the weights and biases in the fifth module that minimize the prediction error of the training set are brought into the prediction set, and the predicted depth value is obtained by inference calculation.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the computer program executes the steps of the method for three-dimensional reconstruction of a landfill based on wide-baseline light field imaging technology as claimed in any one of claims 1 to 7.

10. A computer device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the computer program is executed by a processor, the computer program executes the steps of the method for three-dimensional reconstruction of a landfill based on wide-baseline light field imaging technology as claimed in any one of claims 1 to 7.

Images