Binocular stereoscopic vision flotation froth three-dimensional reconstruction method based on deep learning

Abstract

The invention discloses a binocular stereoscopic vision flotation froth three-dimensional reconstruction method based on deep learning, and belongs to the technical field of froth three-dimensional reconstruction, and the method comprises the steps: 1, carrying out the image correction through a Bouguet algorithm, carrying out the image segmentation through MDA-Net, and carrying out the preprocessing of a froth image; 2, predicting a foam disparity map of the corrected left and right foam images by using a DAEV-Stereo model; 3, calculating three-dimensional coordinates of the foam surface in a world coordinate system according to the disparity map and camera parameters through a binocular stereoscopic vision method; compared with traditional features, the three-dimensional features show less fluctuation, the stability is improved, and the method has better correlation with operation conditions. A binocular stereoscopic vision method and a DAEV-Stereo model are adopted to extract the three-dimensional features of the foam, and the method is of great significance to accurate representation of foam surface data.

Description

Technical Field

[0001] The present invention relates to the technical field of foam three-dimensional reconstruction, and specifically to a binocular stereo vision flotation foam three-dimensional reconstruction method based on deep learning. Background Art

[0002] Coal plays a vital role in the global energy sector, ensuring energy security and promoting social progress. The coal slurry flotation process relies on surface chemistry principles, utilizing the differences in the hydrophilicity of mineral surfaces to promote the efficient separation of different minerals.

[0003] Currently, most coal processing facilities in China have limited automation, digitalization, and intelligent systems. During flotation operations, workers typically rely on visual assessment of froth levels in the flotation tanks to manually adjust chemical dosages. This practice often results in poor process stability, low precision, inferior product quality, and excessive waste of flotation chemicals.

[0004] These issues stem primarily from the subjectivity and imprecision of manual intervention in the flotation process. Machine vision systems can be used to extract froth surface features, facilitate accurate identification of froth type, predict mineral grade, and quantify chemical dosage during flotation.

[0005] Previous studies have demonstrated the importance of the relationship between visual foam characteristics and process variables such as tank level and chemical ratio. Despite this, there is currently a lack of accurate and reliable methods to measure visual features related to coal slurry flotation foam. Extracting features from two-dimensional images alone, including foam size distribution, morphology, and color, is insufficient to immediately apply these features to production guidance parameters. This limitation stems from the three-dimensional nature of the foam structure. To this end, a deep learning-based binocular stereo vision-based three-dimensional reconstruction method for flotation foam is proposed. Summary of the Invention

[0006] The present invention aims to address at least one of the technical issues in the prior art: the lack of accurate and reliable methods for measuring visual features associated with coal slurry flotation froth. To this end, one objective of the present invention is to propose a method for 3D reconstruction of flotation froth using binocular stereo vision based on deep learning.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] The 3D reconstruction method of flotation foam using binocular stereo vision based on deep learning includes:

[0009] Step 1: Image correction is performed using the Bouguet algorithm, and the foam image is preprocessed using image segmentation using MDA-Net;

[0010] Step 2: Use the DAEV-Stereo model to predict the foam disparity map of the corrected left and right foam images;

[0011] Step 3: Calculate the three-dimensional coordinates of the foam surface in the world coordinate system based on the disparity map and camera parameters using binocular stereo vision.

[0012] Step 4: Convert the three-dimensional coordinate point cloud into a triangle network and use the triangular prism infinitesimal traversal method to calculate the volume and surface area of ​​the foam;

[0013] Step 5: Build a mathematical model to relate the brightness of the foam image to the distance from the light source and calculate the average three-dimensional color value of the foam.

[0014] Preferably, in step 1, the MDA-Net framework adopts a U-shaped network structure for encoding and decoding, and the specific steps of using MDA-Net for image segmentation are:

[0015] First, the foam coal image is input into the encoding network;

[0016] Subsequently, a PDA module is added to each encoder layer to identify semantic features using global context information and enhance semantic consistency through a dual-attention mechanism;

[0017] The DSC module is introduced in the jump connection between codec and decoder in the same layer. In the decoding layer, the MFFE module is used to restore the bubble contour details and image accuracy, replacing the traditional convolution.

[0018] Finally, a 1×1 convolution and a sigmoid activation function are performed to obtain a mucus foam segmentation image with the same resolution as the input foam image.

[0019] Preferably, the DAEV-Stereo model includes a multi-scale feature extractor, a combined geometric encoder, a multi-level update operator based on a convolutional neural network, and a segmentation map refinement module.

[0020] Preferably, in the feature extractor, a lightweight network is used to obtain multi-scale attention features and an MDA-Net model is used to obtain context features and segmentation map Q. The specific steps are as follows:

[0021] For the feature network, the standard left and right foam images Il(r)∈R3×H×W are input into the lightweight shared weight MobileNetV2 network, and Il(r) is subsampled to 1 / 16 resolution;

[0022] Subsequently, the up-sampling block using skip connection is used to restore the features to 1 / 2 resolution, thus in extracts multi-scale features with channel attention f(fr,i)∈RCi×Hi×Wi 1,i (i=2, 4, 8, 16, characteristic channel Ci);

[0023] fl,2 and fr,2 are used to construct the cost volume and guide cost aggregation, and fl,i (i=2, 4, 8, 16) is used to guide the 3D regularization network;

[0024] For the MDA network, the context network is used to initialize the hidden state of the ConvGRU update operator and integrated into the ConvGRU during each iteration; MDA-Net is used to generate 128-channel dual-attention multi-scale context features at image resolutions of 1 / 2, 1 / 4, and 1 / 8.

[0025] Preferably, in the combined geometric coding volume, the left feature correlation fl,2 and the volume right feature are constructed, and fr,2fl,2(fr,2) is extracted from the partition Il and divided into groups ir,Ng (Ng=8) along the channel dimension, and the correlation map is calculated group by group. The formula used is:

[0026]

[0027] Where <·,·> is the inner product, d is the parallax index, N c is the number of feature channels;

[0028] For Di×Hi×Wi and cost amount Ci (i=2, 4, 8, 16) in the cost aggregation process, the bootstrapped cost amount is recorded as:

[0029] C i =σ(f l,i )⊙C i

[0030] Where σ is the sigmoid function and ⊙ is the Hadamard product;

[0031] By applying a 3D regularized network to perform bootstrap cost volume operations, the foam surface geometry information is inferred and propagated to generate a geometric encoding volume, and the corresponding correlations between all pairs of left and right features are calculated to obtain local feature correlations.

[0032] Preferably, in the multi-level update operator, first, starting from point 0, at each iteration, the current estimated disparity is retrieved in CGEV, a set of eigenvalues ​​is obtained and connected to the ConvGRU operator;

[0033] Initialize the hidden state of the ConvGRU operator using multi-scale context features, concatenate the GEV, current disparity, and context features and input them into the ConvGRU operator to update the hidden state, and use the new hidden state to predict the updated disparity;

[0034] Apply soft-argmin to regress the initial disparity d0 from the geometric code volume CG, the formula is:

[0035]

[0036] where d is a set of predetermined disparity indices at 1 / 2 resolution, and for each iteration, a set of geometric features G is generated by linear interpolation using the current disparity {d0, ...dk} indexed from the combined geometry encoding volume f , get G f The formula is:

[0037]

[0038] Where, d k is the current disparity, r is the index radius, and p is the pooling operation.

[0039] Preferably, in the segmentation map refinement module, the DAEV-Stereo model uses a 1 / 2 resolution image and uses the MDA-Net architecture to perform superpixel prediction to generate a segmentation map Q.

[0040] Preferably, for the loss function, the smoothed L1 loss obtained by GEV regression with an initial difference of 0 is calculated as:

[0041]

[0042] Where d is the true error; L1 loss is calculated for the entire prediction variance {di}Ni＝1t , the weights grow exponentially, and the total loss is defined as:

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] The present invention first preprocesses the foam image and then proposes a foam stereo matching model - dual attention coded volume stereo. The model is trained using simulated and real-world datasets to determine the foam parallax. The distance between the foam surface and the camera is calculated using a binocular vision method based on the intrinsic and extrinsic aspects of the camera, and the foam is reconstructed in three dimensions. Its surface area and volume are calculated using triangular prism differential traversal technology. Finally, a mathematical model is established to integrate depth information and foam layer thickness to calculate color features. Compared with traditional features, stereo features show less fluctuation, improved stability, and better correlation with operating conditions. The use of binocular stereo vision methods and the DAEV-Stereo model to extract foam three-dimensional features is of great significance for the accurate characterization of foam surface data. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1This is a flow chart for establishing a simulated foam image dataset in the present invention.

[0046] Figure 2 This is a flow chart of simulating foam image data sets in the present invention.

[0047] Figure 3 This is the daev-3D network architecture diagram in the present invention.

[0048] Figure 4 This is a diagram showing the ablation experiment results in Experimental Example 1 of the present invention.

[0049] Figure 5 This is a comparison diagram of the differences between the modules in the foam stereo matching ablation experiment of the present invention; among them, (1) foam left image; (2) ground truth; (3) disparity result of the baseline model; (4) disparity result of C; (5) CG-induced disparity; (6) from ship to ship to; (7) daev-stee difference.

[0050] Figure 6 3D foam images representing five different types of typical foam states in the present invention. DETAILED DESCRIPTION

[0051] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0052] The 3D reconstruction method of flotation foam using binocular stereo vision based on deep learning includes:

[0053] Step 1: Image correction is performed using the Bouguet algorithm, and the foam image is preprocessed using image segmentation using MDA-Net;

[0054] Step 2: Use the DAEV-Stereo model to predict the foam disparity map of the corrected left and right foam images;

[0055] Step 3: Calculate the three-dimensional coordinates of the foam surface in the world coordinate system based on the disparity map and camera parameters using binocular stereo vision.

[0056] Step 4: Convert the three-dimensional coordinate point cloud into a triangle network and use the triangular prism infinitesimal traversal method to calculate the volume and surface area of ​​the foam;

[0057] Step 5: Build a mathematical model to relate the brightness of the foam image to the distance from the light source and calculate the average three-dimensional color value of the foam.

[0058] Regarding the creation and preprocessing of the dataset, in the present invention, coal samples from the Huaibei Coal Processing Plant in Anhui Province were collected for experiments, and the foam stereo dataset was obtained using a pilot plant at Taiyuan University of Technology.

[0059] The depth map was filtered to remove blurry and poorly accurate data, and the image size was appropriately cropped while ensuring that 75% of the main features of the image were present. When the model training data was similar to the processing data, the foam stereo image was modeled in 3D using Substance 3D Designer software to capture the characteristic stereo properties of the five foam types to generate a 3D image. Subsequently, an inverse calculation was applied to derive the depth image, left image, and right image, as shown in the attached figure. Figure 1 shown.

[0060] Create a simulated foam dataset from height, base color, ambient occlusion and roughness perspectives, as shown in the attached Figure 2 As shown, a simulated foam dataset was developed based on five classic foams.

[0061] In step 1, the MDA-Net framework uses a U-shaped network structure for encoding and decoding. The specific steps for using MDA-Net for image segmentation are:

[0062] First, the foam coal image is input into the encoding network;

[0063] Subsequently, a PDA module is added to each encoder layer to identify semantic features using global context information and enhance semantic consistency through a dual-attention mechanism;

[0064] The DSC module is introduced in the encoder-decoder jump connection part of the same layer. This module integrates multiple layers of deep features from local and global information to reconstruct the boundary contour of the coal slime bubble. In the decoding layer, the MFFE module is used to restore the bubble contour details and image accuracy, replacing the traditional convolution.

[0065] Finally, a 1×1 convolution and a sigmoid activation function are performed to obtain a mucus foam segmentation image with the same resolution as the input foam image.

[0066] The DAEV-Stereo model consists of a multi-scale feature extractor, a combined geometric encoder, a multi-level update operator based on a convolutional neural network, and a segmentation map refinement module.

[0067] Among them, the standard left and right coal slurry foam images are input into a multi-scale feature extraction network with shared weights to obtain multi-scale features that provide detailed foam location information.

[0068] Subsequently, the left and right features representing multi-scale data are used to create cost volumes and facilitate cost aggregation, integrating local and global information from multiple layers of deep features to reconstruct detailed foam information.

[0069] Subsequently, the MDA-Net model is used to generate multi-scale context features for initializing and updating the hidden state of ConvGRU, as shown in the attached figure. Figure 3 (b) shown.

[0070] Finally, the segmentation map Q produced by the MDA-Net model is used to guide the upsampling process.

[0071] In the feature extractor, multi-scale attention features are obtained through a lightweight network and the context features and segmentation map Q are obtained using the MDA-Net model. The specific steps are as follows:

[0072] For the feature network, the standard left and right foam images Il(r)∈R3×H×W are input into a lightweight shared-weight MobileNetV2 network, and Il(r) is subsampled to 1 / 16 resolution.

[0073] Subsequently, the up-sampling block using skip connection is used to restore the features to 1 / 2 resolution, thus in extracts multi-scale features with channel attention f(fr,i)∈RCi×Hi×Wi 1,i (i=2, 4, 8, 16, characteristic channel Ci);

[0074] fl,2 and fr,2 are used to construct the cost volume and guide cost aggregation, and fl,i (i=2, 4, 8, 16) is used to guide the 3D regularization network;

[0075] For MDA networks, the context network is used to initialize the hidden state of the ConvGRU update operator and is integrated into the ConvGRU during each iteration. Figure 3 As shown, MDA-Net is used to generate 128-channel dual-attention multi-scale context features at image resolutions of 1 / 2, 1 / 4, and 1 / 8.

[0076] In the combined geometric coding volume, the left feature correlation fl,2 and the right feature of the volume are constructed, and fr,2fl,2(fr,2) is extracted from the partition Il and divided into groups ir,Ng (Ng = 8) along the channel dimension. The correlation map is calculated group by group, and the formula used is:

[0077]

[0078] Where <·,·> is the inner product, d is the parallax index, N c is the number of feature channels.

[0079] A lightweight 3D regularization network is used for further processing, which consists of three downsampling blocks and three upsampling blocks. Each downsampling block consists of two 3×3×3 3D convolutions, and the number of channels in the three downsampling blocks is 16, 32, and 48 respectively. Each upsampling block consists of one 4×4×4 3D transposed convolution and two 3×3×3 3D convolutions to obtain the geometric code volume CG.

[0080] For Di×Hi×Wi and cost amount Ci (i=2, 4, 8, 16) in the cost aggregation process, the bootstrapped cost amount is recorded as:

[0081] C i =σ(f l,i )⊙C i

[0082] Where σ is the sigmoid function and ⊙ is the Hadamard product.

[0083] By applying a 3D regularized network to perform bootstrap cost volume operations, the foam surface geometry information is inferred and propagated to generate a geometric encoding volume, and the corresponding correlations between all pairs of left and right features are calculated to obtain local feature correlations.

[0084] Due to varying conditions during flotation, coal slurry foam can be roughly categorized into five states: dense, ideal, sparse, elliptical, and irregular. Therefore, to increase the size of the receptive field, the disparity dimension is passed through an average pooling layer p, forming a two-layer CG pyramid and a full-pair correlation CA pyramid. The CG and CA pyramids are then combined into a combined geometric encoding volume.

[0085] In the multi-level update operator, first, starting from point 0, in each iteration, the current estimated disparity is retrieved in CGEV, a set of eigenvalues ​​is obtained and connected to the ConvGRU operator;

[0086] Initialize the hidden state of the ConvGRU operator using multi-scale context features, concatenate the GEV, current disparity, and context features and input them into the ConvGRU operator to update the hidden state, and use the new hidden state to predict the updated disparity;

[0087] Apply soft-argmin to regress the initial disparity d0 from the geometric code volume CG, as follows:

[0088]

[0089] where d is a set of predetermined disparity indices at 1 / 2 resolution, and for each iteration, a set of geometric features G is generated by linear interpolation using the current disparity {d0, ...dk} indexed from the combined geometry encoding volume f Get Gf The formula is:

[0090]

[0091] Where, d k is the current disparity, r is the index radius, and p is the pooling operation.

[0092] Geometric feature G f and the predicted disparity d k Through two encoding layers, it is then connected to dk to form xk. The multi-resolution update operator makes the iterative disparity optimization process converge faster and obtain better generalization. Through the ablation experiment, the results of using three levels of iteration in the ConvGRU operator are shown in the attached figure. Figure 3 (b) shows the remaining parallax Δd. k Decoded by two convolutional layers, then the current disparity is updated:

[0093] d k+1 =d k +Δd k

[0094] In the segmentation map refinement module, the DAEV-Stereo model uses 1 / 2 resolution images and utilizes the MDA-Net architecture, as shown in the attached figure. Figure 3 As shown in (a), superpixel prediction is performed to generate a segmentation map Q. This map serves as a guide for the upsampling process and helps generate high-resolution output that accurately represents the boundaries and complex details of the foam.

[0095] For the loss function, the smoothed L1 loss obtained by GEV regression with an initial difference of 0 is calculated as:

[0096]

[0097] Where d is the true error; L1 loss is calculated for the entire prediction variance {di}Ni＝1t , the weights grow exponentially, and the total loss is defined as:

[0098]

[0099] Experimental Example 1

[0100] Experimental Purpose: Ablation experiments are conducted using a simulated foam test dataset to evaluate the effectiveness of the proposed model in estimating the quality of dispersion results.

[0101] Configuration of test parameters:

[0102] The DAEV-Stereo model was trained in a Windows 10 environment, using Python 3.7 as the programming language, PyTorch 1.11.0+cu113 as the deep learning framework, and an NVIDIA RTX4090 graphics card to enhance the computing power of the model.

[0103] The AdamW optimizer was applied to the simulated Foam Stereo dataset. The DAEV-Stereo model was trained for up to 300,000 steps with a batch size of 8. The pre-trained DAEV-Stereo model was fine-tuned for 50,000 steps using the Foam Stereo dataset. During training, images were randomly cropped to 720×560. A single-cycle learning rate schedule was used, with the learning rate set to 0.0002 for 20 update iterations. The generalized average EPE-fg was chosen as the quantitative evaluation metric, calculated as:

[0104]

[0105] in, Indicates the number of pixels in the foreground area of ​​the image, d i represents the prediction error, Represents the true error. The formula for the proportion of foreground pixels with a prediction error of D1-fg is:

[0106]

[0107] Ablation experiments evaluate the effectiveness of the ConvGRU operator, the GEV module, and the improved disparity structure using contextual features extracted by MDA-Net. Comparative analysis is conducted on a simulated foam dataset. The performance of each module is evaluated using metrics such as EPE and D1. Comprehensive evaluation criteria include model computation time and number of parameters.

[0108] The ablation test results are shown in the attached Figure 4 As shown in Figure 2, the performance of the RAFT-Stereo benchmark is improved by replacing the original backbone with MobileNetV2. After introducing the ConvGRU module that utilizes contextual features extracted by MDA-Net, the EPE is reduced to 0.83 pixels and the D1 metric is reduced to 3.25%. MDA-Net improves the performance of the ConvGRU module by incorporating contextual features using a dual-attention mechanism, thereby reducing prediction error.

[0109] Subsequently, the GEV module is combined to improve the performance, reducing EPE to 0.67 pixels and D1 to 2.84%. The GEV module plays a crucial role in providing non-local information and scene-specific knowledge, thereby significantly reducing the prediction error.

[0110] By using MobileNetV2 as the baseline architecture for the initial RAFT-Stereo model, non-local information and multi-scale full-pair correlation are combined to reduce the EPE and D1 indicators to 0.56 pixels and 2.66%, respectively. By enhancing the disparity structure segmentation map, the preservation of boundaries and details is improved, reducing the EPE to 0.50 pixels and the D1 to 2.59%.

[0111] As can be seen from the above, DAEV-Stereo has 12.6M parameters and can be used with a microcomputer, enabling its application in industrial environments. In addition, the detection time of DAEV-Stereo is 0.37s, which meets the operational requirements.

[0112] Experimental Example 2

[0113] Experimental purpose: Qualitative analysis of the foam parallax effect

[0114] By the attached Figure 5 As shown, the properties of various foam types vary significantly. A comparative analysis leverages the stereo matching performance of various modules on foam images and selects five images that demonstrate typical foam characteristics. These images depict various foam types, including ideal foam, sparse foam, irregular foam, elliptical foam, and dense foam, and are organized from bottom to top.

[0115] The baseline results for different types of foam show irregular foam boundaries, and some foam regions exceed the depth range, resulting in black spots that hinder accurate 3D reconstruction.

[0116] According to the C-disparity results, the boundaries of various bubble types show blur and discontinuity, and bubbles appear outside the depth range. In the ideal, elliptical, and dense bubble categories, the cougg disparity results show a general trend of excessive bubble depth, with many bubble instances exceeding the depth range.

[0117] The differences in g-values ​​observed for the boat-shaped transition across various foam types indicate the presence of certain foams in the shadowed region, characterized by blurred boundaries and skewed depth perception within specific foam images. Conversely, the varying results of DAEV-Stereo for different foam types indicate a reduced loss of foam boundary features. This results in clear and cohesive foam boundaries, minimizes foam depth inaccuracies, and enhances overall image clarity and density. The expected difference maps closely correspond to the true difference maps, facilitating the collection of point cloud data for 3D reconstruction.

[0118] Experimental Example 3

[0119] Experimental purpose: To verify the effect of flotation foam 3D reconstruction

[0120] After acquiring the coal disparity map using DAEV-Stereo, binocular stereo vision was used to calculate the 3D coordinates of the foam surface relative to the camera's internal reference frame, generating foam point cloud data. To mitigate the impact of noise in the 3D data, a median filter algorithm was implemented to remove abnormal interference values ​​in the foam point cloud data.

[0121] The 3D reconstruction of the foam surface is shown in the attached Figure 6 As shown, the goal of the 3D reconstruction method of the present invention is to replicate 3D flotation foam data using experimental image data. Figure 6 The foam stereo images depicted in (1)-(5) represent the different foam states observed throughout the flotation process: ideal, irregular, elliptical, dense, and sparse. The foam stereo matching dataset includes foams of different volumes and morphologies.

[0122] Therefore, the DAEV-Stereo model exhibits considerable robustness, as demonstrated by accurate 3D reconstructions that are representative of real-world scenes, validating the accuracy and versatility of DAEV-Stereo's predicted coal body disparity.

[0123] Parts not described in the present invention are the same as those in the prior art or can be implemented using the prior art. Although the embodiments of the present invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and variations may be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A binocular stereo vision flotation foam 3D reconstruction method based on deep learning, characterized by: include: Step 1: Image correction is performed using the Bouguet algorithm, and the foam image is preprocessed using image segmentation using MDA-Net; Step 2: Use the DAEV-Stereo model to predict the foam disparity map of the corrected left and right foam images; Step 3: Calculate the three-dimensional coordinates of the foam surface in the world coordinate system based on the disparity map and camera parameters using binocular stereo vision. Step 4: Convert the three-dimensional coordinate point cloud into a triangle network and use the triangular prism infinitesimal traversal method to calculate the volume and surface area of ​​the foam; Step 5: Build a mathematical model to relate the brightness of the foam image to the distance from the light source and calculate the average three-dimensional color value of the foam.

2. The method for 3D reconstruction of flotation foam based on binocular stereo vision using deep learning according to claim 1, characterized in that: In step 1, the MDA-Net framework uses a U-shaped network structure for encoding and decoding. The specific steps for using MDA-Net for image segmentation are: First, the foam coal image is input into the encoding network; Subsequently, a PDA module is added to each encoder layer to identify semantic features using global context information and enhance semantic consistency through a dual-attention mechanism; The DSC module is introduced in the jump connection between codec and decoder in the same layer. In the decoding layer, the MFFE module is used to restore the bubble contour details and image accuracy, replacing the traditional convolution. Finally, a 1×1 convolution and a sigmoid activation function are performed to obtain a mucus foam segmentation image with the same resolution as the input foam image.

3. The method for 3D reconstruction of flotation foam based on binocular stereo vision using deep learning according to claim 1, characterized in that: The DAEV-Stereo model consists of a multi-scale feature extractor, a combined geometric encoder, a multi-level update operator based on a convolutional neural network, and a segmentation map refinement module.

4. The method for 3D reconstruction of flotation foam based on binocular stereo vision using deep learning according to claim 3, characterized in that: In the feature extractor, multi-scale attention features are obtained through a lightweight network and the context features and segmentation map Q are obtained using the MDA-Net model. The specific steps are as follows: For the feature network, the standard left and right foam images Il(r)∈R3×H×W are input into the lightweight shared weight MobileNetV2 network, and Il(r) is subsampled to 1 / 16 resolution; Subsequently, the up-sampling block using skip connection is used to restore the features to 1 / 2 resolution, thus in extracts multi-scale features with channel attention f(fr,i)∈RCi×Hi×Wi}1,i (i=2, 4, 8, 16, characteristic channel Ci); fl,2 and fr,2 are used to construct the cost volume and guide cost aggregation, and fl,i (i=2, 4, 8, 16) is used to guide the 3D regularization network; For the MDA network, the context network is used to initialize the hidden state of the ConvGRU update operator and is integrated into the ConvGRU during each iteration; MDA-Net is adopted to generate 128-channel dual-attention multi-scale contextual features at image resolutions of 1 / 2, 1 / 4 and 1 / 8.

5. The method for 3D reconstruction of flotation foam based on binocular stereo vision using deep learning according to claim 3, characterized in that: In the combined geometric coding volume, the left feature correlation fl,2 and the right feature of the volume are constructed, and fr,2fl,2(fr,2) is extracted from the partition Il and divided into groups ir,Ng (Ng = 8) along the channel dimension. The correlation map is calculated group by group, and the formula used is: Where <·,·> is the inner product, d is the parallax index, N c is the number of feature channels; For Di×Hi×Wi and cost amount Ci (i=2, 4, 8, 16) in the cost aggregation process, the bootstrapped cost amount is recorded as: C i =σ(f i,i )⊙C i Where σ is the sigmoid function and ⊙ is the Hadamard product; By applying a 3D regularized network to perform bootstrap cost volume operations, the foam surface geometry information is inferred and propagated to generate a geometric encoding volume, and the corresponding correlations between all pairs of left and right features are calculated to obtain local feature correlations.

6. The method for 3D reconstruction of flotation foam based on binocular stereo vision using deep learning according to claim 3, characterized in that: In the multi-level update operator, first, starting from point 0, at each iteration, the current estimated disparity is retrieved in CGEV, a set of eigenvalues ​​is obtained and connected to the ConvGRU operator; Initialize the hidden state of the ConvGRU operator using multi-scale context features, concatenate the GEV, current disparity, and context features and input them into the ConvGRU operator to update the hidden state, and use the new hidden state to predict the updated disparity; Apply soft-argmin to regress the initial disparity d0 from the geometric code volume CG, the formula is: where d is a set of predetermined disparity indices at 1 / 2 resolution, and for each iteration, a set of geometric features G is generated by linear interpolation using the current disparity {d0, ...dk} indexed from the combined geometry encoding volume f , get G f The formula is: Where, d k is the current disparity, r is the index radius, and p is the pooling operation.

7. The method for 3D reconstruction of flotation foam based on binocular stereo vision using deep learning according to claim 3, characterized in that: In the segmentation map refinement module, the DAEV-Stereo model uses a 1 / 2 resolution image and uses the MDA-Net architecture to perform superpixel prediction to generate a segmentation map Q.

8. The method for 3D reconstruction of flotation foam based on binocular stereo vision using deep learning according to claim 1, characterized in that: For the loss function, the smoothed L1 loss obtained by GEV regression with an initial difference of 0 is calculated as: Where d is the true error; L1 loss is calculated for the entire prediction variance {d i }N i= 1t , the weights grow exponentially, and the total loss is defined as: where y = 0.9.

Images