A Fruit Surface Reconstruction Method Based on Dual Attention-Guided Stereo Matching

Abstract

This invention discloses a method for fruit surface reconstruction based on dual-attention guided stereo matching. The invention includes the following steps: First, multiple sets of depth maps, a left image of the fruit, and a right image of the fruit are acquired to construct a fruit stereo matching dataset. Next, a dual-attention guided stereo matching model is constructed, and then trained using the fruit stereo matching dataset to obtain a trained dual-attention guided stereo matching model. Finally, the image of the fruit to be tested is input into the trained dual-attention guided stereo matching model, and the reconstructed disparity map is output, thereby obtaining the depth map and completing the reconstruction of the fruit surface. This invention enables rapid and accurate acquisition of depth information from fruits with smooth surfaces and few feature points, which is beneficial for deployment and application on actual fruit sorting lines.

Description

Technical Field

[0001] This invention relates to a fruit surface reconstruction method in the field of image processing technology, and more particularly to a fruit surface reconstruction method based on dual attention-guided stereo matching. Background Technology

[0002] The external quality of fruit is a crucial indicator in its sorting process and directly impacts consumer purchasing decisions. Therefore, it is necessary to test external quality indicators such as volume and shape. Previous testing methods involved analyzing multiple images of fruit rotating on the production line. However, the variability of biological objects makes it difficult to accurately determine the true size of the fruit. Surface reconstruction of the fruit will help achieve more comprehensive and accurate external quality testing.

[0003] Due to the high real-time requirements of actual fruit sorting lines, binocular stereo vision, with its high frame rate and ease of deployment, holds promise for online surface reconstruction of fruits. Most fruits have limited surface texture features, making traditional stereo matching algorithms ineffective. Deep learning-based stereo matching methods, however, offer more robust feature extraction and matching. GwcNet (Guo,X.,Yang,K.,Yang,W.,Wang,X.,&Li,H.(2019). Group-wise correlation stereo network.In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition,pp.3273–3282.) constructs a correlation cost matrix after feature extraction from the left and right images, and then uses a 3D aggregation network to regress and obtain the final predicted disparity map. Although this network provides higher disparity accuracy and faster inference speed than traditional stereo matching algorithms, it still suffers from poor matching results for fruits with relatively smooth surfaces, such as citrus fruits.

[0004] In recent years, deep learning-based stereo matching algorithms have been continuously iterated and optimized. Improved networks such as RealtimeStereo (Chang, JR, Chang, PC, & Chen, YS (2020). Attention-aware feature aggregation for real-time stereo matching on edge devices. In: Proceedings of the Asian Conference on Computer Vision.) and ACVNet (Xu, G., Cheng, J., Guo, P., & Yang, X. (2022). Attention concatenation volume for accurate and efficient stereo matching. In: Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pp. 12981–12990.) have achieved better matching performance at the whole image scale. However, they have not optimized the matching performance for individual objects. The entire image scale is given a consistent loss function weight, which makes it difficult to effectively match fruit objects with few surface features. Some networks have also incorporated the concept of a multi-task model into the design of stereo matching networks (Dovesi, PL, Poggi, M., Andraghetti, L., Martí, M., H., Pieropan, A., & Mattoccia, S. (2020). Real-Time Semantic Stereo Matching. In: 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 10780-10787.), but this will increase the complexity of the model, and the consistent loss weights are still used for different segmentation objects, making it difficult to improve the stereo matching performance for individual objects of interest. Summary of the Invention

[0005] The technical problem this invention aims to solve is that existing stereo matching methods, which use loss functions with consistent weights, lack optimization for matching performance of individual objects, and suffer from increased model size due to the addition of multi-task learning tasks. This invention proposes a fruit surface reconstruction method based on dual attention-guided stereo matching to improve the matching of stereo matching networks for fruit images with few surface features.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] 1) Using a depth camera C depth and binocular camera C stereo Multiple sets of depth maps were acquired. depth Left image of fruit; Right image of fruit;

[0008] 2) Utilize multiple sets of depth maps I depth A stereo matching dataset of fruits is constructed using the left and right images of fruits.

[0009] 3) Construct a dual-attention guided stereo matching model. Train the dual-attention guided stereo matching model based on the fruit stereo matching dataset to obtain a trained dual-attention guided stereo matching model.

[0010] 4) Input the image of the fruit to be tested into the trained dual attention-guided stereo matching model, output the reconstructed disparity map, and then obtain the depth map, thereby completing the reconstruction of the fruit surface.

[0011] Specifically, 2) refers to:

[0012] 2.1) Compare the left image of the fruit in each group of images with the corresponding depth map I. depth After parallax processing, the corresponding parallax true values ​​are obtained respectively;

[0013] 2.2) After performing epipolar correction on the left and right images of the fruit and the disparity ground truth in each group of images, a stereo matching dataset of fruit is obtained.

[0014] In step 3), the dual attention-guided stereo matching model includes a weight-sharing feature extraction module M. f First-level attention guidance module M att1 Second Attention Guidance Module M att2 and 3D decoding network M de The input images corresponding to the left and right fruit images are respectively processed by the weight-sharing feature extraction module M. f Then, the initial left feature map and the initial right feature map are output respectively. The initial left feature map is input into the second attention guidance module M. att2 In the process, the initial left feature map and the initial right feature map are concatenated along the disparity direction to obtain the stacked cost volume Vol. con The initial left and initial right feature maps are also input together into the first attention guidance module M. att1 In the middle, the first level of attention guidance module M att1 Output and superposition cost body Vol conThe attention cost volume (Vol) obtained after performing the Hadamard product operation att Attention Cost Volume (Vol) att Input to 3D decoding network M de 3D decoding network M de Output the reconstructed disparity map, based on the second attention guidance module M. att2 The output is used to construct the total loss function of the dual attention-guided stereo matching model.

[0015] The second attention guidance module M att2 The output constructs the total loss function of the dual attention-guided stereo matching model, specifically:

[0016] First, based on the left image I of the fruit RGB_l And the fruit image on the right (I) RGB_r The segmentation dataset is constructed and obtained, and then the U-Net model M is used to analyze the segmentation dataset. seg After pre-training, a second attention guidance module M is obtained. att2 Then, based on the second level of attention guidance module M... att2 Left image of fruit I RGB_l The segmentation result is the main fruit part I in the dual attention-guided stereo matching model. fruit Loss function fruit and background area I back Loss function back We assign a first weight α and a second weight β to each of them, and then construct the total loss function of the dual attention-guided stereo matching model.

[0017] The formula for the total loss function of the dual attention-guided stereo matching model is as follows:

[0018] L all =αLoss fruit +βLoss back

[0019]

[0020] Among them, L all This is the total loss function value. For the smoothing loss calculation function, I disp_ground Let λ be the true value of the disparity. att For the first level of attention guidance module M att1 The weight of the loss, d att For attention weights, λ i M, a 3D decoding network de Output of the i-th element i The corresponding weights for the loss are: Loss = the main part of the fruit I. fruitLoss function value fruit Or background area I back Loss function value bg Output i For 3D decoding network M de The i-th output, namely Output1, Output2, Output3, and Output4, are respectively the outputs of the 3D decoding network M. de The first output, second output, third output, and fourth output.

[0021] The beneficial effects of this invention are:

[0022] This invention proposes a fruit surface reconstruction method based on dual attention-guided stereo matching. By introducing a dual attention module, the network pays more attention to the matching effect of fruit regions, improving the matching performance of fruit surface regions and achieving high-precision matching of fruit images. This provides a reliable foundation for the subsequent realization of real-time 3D reconstruction of fruits on the production line. Attached Figure Description

[0023] Figure 1 This is the overall flowchart of the present invention.

[0024] Figure 2 This is a schematic diagram of the dual attention-guided stereo matching algorithm of the present invention.

[0025] Figure 3 The 3D decoding network M of this invention de Structural diagram.

[0026] Figure 4 This is a schematic diagram of surface reconstruction and error distribution according to the present invention. Detailed Implementation

[0027] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0028] In this embodiment, navel oranges were selected as the experimental subject:

[0029] The integrated development environment used in this embodiment is PyCharm 2021.6, the programming language is Python 3.6, and the GPU is NVIDIA GeForce RTX 3060. The specific implementation environment includes, but is not limited to, the above-mentioned environment.

[0030] like Figure 1 As shown, the present invention includes the following steps:

[0031] 1) Utilizing the true value of fruit depth information to acquire the high-precision depth camera C in the system depth And binocular camera C stereoConstruct a fruit depth information truth value acquisition system. The fruit depth information truth value acquisition system mainly includes a computer, an industrial control computer, and a depth camera (C). d Binocular camera C stereo (including left camera C) l And right camera C r The camera includes a light source and a roller holder. The camera is connected to an industrial control computer via a network cable, and then to a computer to acquire data. In this embodiment, the depth camera C... depth With binocular camera C stereo The installation height is 1000mm, and the binocular camera C stereo The baseline distance b is 40mm; using depth camera C depth and binocular camera C stereo Multiple sets of depth maps were acquired. depth Left image of fruit; Right image of fruit;

[0032] 2) Utilize multiple sets of depth maps I depth A stereo matching dataset of fruits is constructed using the left and right images of fruits.

[0033] 2) Specifically:

[0034] 2.1) Compare the left image of the fruit in each group of images with the corresponding depth map I. depth After parallax processing, the corresponding parallax ground truth values ​​are obtained respectively; in specific implementation, a depth information-based backsampling method is used to process the depth map I. depth The parallax ground truth is obtained by processing the left image of the fruit, specifically:

[0035] According to depth camera C d Acquired depth map I depth It is then backsampled to the left camera C. l In the coordinate system, the true disparity value I can be obtained by converting it to a disparity value through triangulation. disp_ground The specific method is shown in the following formula:

[0036] Z d p d =K d P d

[0037] Z cl p cl =K cl P cl

[0038] P cl =R dl P d +T dl

[0039]

[0040] Where, p d Represents depth image I depth A point in pixel coordinates (converted to homogeneous coordinates), Z d p d depth, P d Indicates depth camera C d p in coordinate system d The corresponding point, p cl p d On the left camera C l The corresponding point K in the coordinate system d For depth camera C d The intrinsic parameter matrix, Z cl For p d The depth value, K cl Left camera C l The intrinsic parameter matrix, R dl For depth camera C d Coordinate system to left camera C l The rotation matrix of the coordinate system, T dl For depth camera C d Coordinate system to left camera C l Translation matrices of the coordinate system, b and f c C, respectively, are binocular cameras stereo The baseline distance and focal length.

[0041] 2.2) After epipolar correction of the left and right fruit images and disparity ground truth values ​​in each group of images, a fruit stereo matching dataset is obtained; among them, the depth camera C is calibrated using Zhang's calibration method. d With left camera C l Calibration, for left camera C l And right camera C r Calibration was performed to obtain the depth camera C. d The intrinsic parameter matrix K d Left camera C l The intrinsic parameter matrix K cl And from depth camera C d Coordinate system to left camera C l Rotation matrix R of the coordinate system dl Translation matrix T dl Binocular camera C stereo The baseline distance is b, and the focal length is f. c This allows for the correction of the left and right images of the fruit.

[0042] 2.3) Replace the left image I of each group of fruits RGB_l Fruit right image I RGB_r And the corresponding parallax truth value I disp_groundThe dataset is arranged sequentially and divided into training, validation, and test sets in a 6:2:2 ratio. In this embodiment, a total of 960 pairs of images were collected, and the dataset was divided into training, validation, and test sets in a 6:2:2 ratio.

[0043] 3) Construct a dual-attention guided stereo matching model, FruitStereo. Train the dual-attention guided stereo matching model on the fruit stereo matching dataset to obtain a trained dual-attention guided stereo matching model.

[0044] 3) In, such as Figure 2 As shown, the dual attention-guided stereo matching model includes a weight-sharing feature extraction module M. f First-level attention guidance module M att1 Second Attention Guidance Module M att2 and 3D decoding network M de ;

[0045] The input images corresponding to the left and right fruit images are respectively processed by the weight-sharing feature extraction module M. f Then, the initial left feature map and the initial right feature map are output respectively. The initial left feature map is input into the second attention guidance module M. att2 In the process, the initial left feature map and the initial right feature map are concatenated along the disparity direction to obtain the stacked cost volume Vol. con The initial left and initial right feature maps are also input together into the first attention guidance module M. att1 In the middle, the first level of attention guidance module M att1 Output and superposition cost body Vol con The attention cost volume (Vol) is obtained after performing the Hadamard product operation. att Attention Cost Volume (Vol) att Input to 3D decoding network M de 3D decoding network M de Output the reconstructed disparity map, based on the second attention guidance module M. att2 The output is used to construct the total loss function of the dual attention-guided stereo matching model.

[0046] According to the second attention guidance module M att2 The output constructs the total loss function of the dual attention-guided stereo matching model, specifically:

[0047] First, based on the left image I of the fruit RGB_l And the fruit image on the right (I) RGB_r The segmentation dataset is constructed and obtained, and then the U-Net model M is used to analyze the segmentation dataset. seg After pre-training, a second attention guidance module M is obtained.att2 Then, based on the second level of attention guidance module M... att2 Left image of fruit I RGB_l The segmentation result is the main fruit part I in the dual attention-guided stereo matching model. fruit Loss function fruit and background area I back Loss function back We assign a first weight α and a second weight β to achieve a second level of attention guidance, and then construct the total loss function of the dual-attention-guided stereo matching model, as shown in the following formula:

[0048] L all =αLoss fruit +βLoss back

[0049]

[0050]

[0051] Among them, L all This is the total loss function value. For the smoothing loss calculation function, I disp_ground Let λ be the true value of the disparity. att For the first level of attention guidance module M att1 The weight of the loss, d att For attention weights, λ i M, a 3D decoding network de Output of the i-th element i The corresponding weights for the loss are: Loss = the main part of the fruit I. fruit Loss function value fruit Or background area I back Loss function value bg Output i For 3D decoding network M de The i-th output, namely Output1, Output2, Output3, and Output4, are respectively the outputs of the 3D decoding network M. de The first output, second output, third output, and fourth output, where | represents the absolute value.

[0052] Among them, the weight-sharing feature extraction module M f : Concatenate three basic convolutional blocks in sequence conv The initial input has 3 channels, and the output has 32 channels. The convolution kernel size is 3, and data normalization and ReLU operations are performed. Subsequently, four cascaded residual modules (Blocks) are input. resFurther feature extraction is performed using the residual module Block. res It consists of data normalization, convolutional layers with a kernel size of 3, ReLU, data normalization, and another convolutional layer with a kernel size of 3, all connected in series, with skip connections added. 4 blocks. res By adding convolutional layers, the number of output channels was increased to 32, 64, 128, and 32, respectively. Finally, the output data size was 1 / 4H × 1 / 4W × 32, where H and W are the left and right images of the fruit, respectively. RGB_l The height and width. Through the weighted shared feature extraction module M... f The left image I was obtained respectively. RGB_l The initial left feature map f l And right image I RGB_r The initial right feature map f r ;

[0053] First-level attention guidance module M att1 : The initial left feature map f l and the initial right feature map f r Performing matrix dot product operation yields the correlation cost volume Vol. corr The kernel size is 1 / 4D×1 / 4H×1 / 4W×32. It is then passed through three cascaded 3D convolutional layers with a kernel size of 3, producing 64, 128, and 64 output channels respectively, and outputting attention weights d. att Used for supervised learning. The cost volume Vol is obtained by stitching together the parallax directions. con The size is 1 / 4D×1 / 4H×1 / 4W×64, and the attention weights d are... att With superposition cost body Vol con The first level of attention guidance is achieved by performing the Hadamard product operation. The relevance cost volume Vol... corr and superposition cost body Vol con The calculation method is as follows:

[0054]

[0055] Vol con (d, x, y) = Concat(f) l (x, y), f r (xd, y))

[0056] Where, N c This represents the number of channels.

[0057] 3D Decoding Network M de :like Figure 3 As shown, first use two convolutional basic blocks. convThis results in an output data size of 1 / 4D×1 / 4H×1 / 4W×1, from which the first output, Output1, is derived. Then, three stacked funnel modules, Stack1, Stack2, and Stack3, are concatenated. Stack1, Stack2, and Stack3 each consist of four concatenated convolutional layers with a kernel size of 3 and 64 output channels. These layers then pass through deconvolutional layers Deconv1 and Deconv2, which contain convolutional layers with 64 and 32 output channels respectively. These two layers are connected to the outputs of the 2nd and 4th convolutional layers in their respective stacked funnel modules. The three stacked funnel modules, Stack1, Stack2, and Stack3, then derive a second output, Output2, a third output, and a fourth output, Output4, each with a size of 1 / 4D×1 / 4H×1 / 4W×1, to better supervise the model training process.

[0058] 4) Input the images of the fruit to be tested (i.e., the left and right images of the fruit to be tested) into the trained dual attention-guided stereo matching model, output the reconstructed disparity map, and then obtain the depth map, thereby completing the reconstruction of the fruit surface.

[0059] During model training, the initial learning rate was set to 0.001, the number of iterations was 300, and the maximum disparity value was set to 192. Different combinations of weights α and β were used to select optimal model parameters. Based on the results in Table 1, the preferred parameters in this embodiment were selected as α = 1.0 and β = 0.5.

[0060] Table 1 shows the test results for different weight combinations.

[0061]

[0062] Based on this, the performance of the network proposed in this embodiment was compared with that of other stereo matching networks. As shown in Table 2, it can be seen that the proposed algorithm FruitStereo achieved the best results, with only a slightly slower inference time than GwcNet. Subsequently, the disparity values ​​were converted into depth using triangulation to obtain the final surface reconstruction result. The reconstructed point cloud is compared with the original point cloud, and the error distribution is shown in the figure. Figure 4 As shown, the RMSE of absolute depth on the test set is only 0.81 mm, which has high reconstruction accuracy, and also proves the effectiveness of the method proposed in this invention.

[0063] Table 2 compares the performance of the algorithm of this invention with other stereo matching algorithms.

[0064]

[0065] Finally, it should be noted that the above embodiments and descriptions are only used to illustrate the technical solutions of the present invention and not to limit it. Those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the disclosure of the technical solutions of the present invention, and all such modifications and substitutions should be covered within the protection scope of the claims of the present invention.

Claims

1. A method for fruit surface reconstruction based on dual attention-guided stereo matching, characterized in that, Includes the following steps: 1) Using a depth camera C depth and binocular camera C stereo Multiple sets of depth maps were acquired. depth Left image of fruit; Right image of fruit; 2) Utilize multiple sets of depth maps I depth A stereo matching dataset of fruits is constructed using the left and right images of fruits. 3) Construct a dual-attention guided stereo matching model. Train the dual-attention guided stereo matching model based on the fruit stereo matching dataset to obtain a trained dual-attention guided stereo matching model. In step 3), the dual attention-guided stereo matching model includes a weight-sharing feature extraction module M. f First-level attention guidance module M att1 Second Attention Guidance Module M att2 and 3D decoding network M de The input images corresponding to the left and right fruit images are respectively processed by the weight-sharing feature extraction module M. f Then, the initial left feature map and the initial right feature map are output respectively. The initial left feature map is input into the second attention guidance module M. att2 In the process, the initial left feature map and the initial right feature map are concatenated along the disparity direction to obtain the stacked cost volume Vol. con The initial left and initial right feature maps are also input together into the first attention guidance module M. att1 In the middle, the first level of attention guidance module M att1 Output and superposition cost body Vol con The attention cost volume (Vol) obtained after performing the Hadamard product operation att Attention Cost Volume (Vol) att Input to 3D decoding network M de 3D decoding network M de Output the reconstructed disparity map, based on the second attention guidance module M. att2 The output is used to construct the total loss function of the dual attention-guided stereo matching model; 4) Input the image of the fruit to be tested into the trained dual attention-guided stereo matching model, output the reconstructed disparity map, and then obtain the depth map, thereby completing the reconstruction of the fruit surface.

2. The fruit surface reconstruction method based on dual attention-guided stereo matching according to claim 1, characterized in that, Specifically, 2) refers to: 2.1) Compare the left image of the fruit in each group of images with the corresponding depth map I. depth After parallax processing, the corresponding parallax true values ​​are obtained respectively; 2.2) After performing epipolar correction on the left and right images of the fruit and the disparity ground truth in each group of images, a stereo matching dataset of fruit is obtained.

3. The fruit surface reconstruction method based on dual attention-guided stereo matching according to claim 1, characterized in that, The second attention guidance module M att2 The output constructs the total loss function of the dual attention-guided stereo matching model, specifically: First, based on the left image I of the fruit RGB_l And the fruit image on the right (I) RGB_r The segmentation dataset is constructed and obtained, and then the U-Net model M is used to analyze the segmentation dataset. seg After pre-training, a second attention guidance module M is obtained. att2 Then, based on the second level of attention guidance module M... att2 Left image of fruit I RGB_l The segmentation result is the main fruit part I in the dual attention-guided stereo matching model. fruit Loss function fruit and background area I back Loss function back We assign a first weight α and a second weight β to each of them, and then construct the total loss function of the dual attention-guided stereo matching model.

4. The fruit surface reconstruction method based on dual attention-guided stereo matching according to claim 3, characterized in that, The formula for the total loss function of the dual attention-guided stereo matching model is as follows: L all =αLoss fruit +βLoss back Among them, L all This is the total loss function value. For the smoothing loss calculation function, I disp_ground Let λ be the true value of the disparity. att For the first level of attention guidance module M att1 The weight of the loss, d att For attention weights, λ i M, a 3D decoding network de Output of the i-th element i The corresponding weights for the loss are: Loss = the main part of the fruit I. fruit Loss function value fruit Or background area I back Loss function value back .

Images