Binocular vision SLAM method and system based on fused GCNv2 network in orchard environment

Abstract

The invention discloses a binocular vision SLAM method and system based on a fused GCNv2 network in an orchard environment. The method comprises the following steps: using a binocular camera to shoot a stereo image pair of the orchard environment; a depth map is generated by adopting a stereo matching algorithm to serve as a label, and a GCNv2 network architecture containing an attention mechanism and multi-scale feature fusion is constructed and trained; an orchard environment image is input, key points are extracted, the corresponding relation between the left image and the right image is found through descriptor matching, and a corresponding depth value is distributed to each matched key point through a depth map predicted by a GCNv2 network; filtering feature points and combining depth information to update the map; loopback detection is carried out, geometric consistency check is carried out, and a BA global optimization algorithm is executed to correct errors in the whole track and the map; the binocular vision SLAM method provided by the invention is efficient, robust and suitable for the orchard environment so as to improve the navigation capability and the automation level of the robot in the complex orchard environment.

Description

Technical Field

[0001] The present invention relates to the field of simultaneous localization and mapping (SLAM) technology, and more specifically to a binocular vision SLAM method and system based on a fused GCNv2 network in an orchard environment. Background Art

[0002] With the increasing degree of agricultural automation, robots are being used more and more widely in orchard management. However, orchard environments usually have complex terrain and vegetation structures, which poses challenges to the autonomous navigation of robots.

[0003] Binocular vision SLAM uses two cameras to simulate the way humans observe the world with both eyes, calculates parallax to obtain depth information, and thus helps robots achieve self-positioning in unknown environments and create maps of the surrounding environment; GCNv2 is a deep neural network based on geometric consistency, designed for stereo matching, and can provide high-quality dense depth maps. Compared with traditional feature point-based methods, GCNv2 uses learned feature representations to assist or even replace manually designed feature extractors, improving the accuracy and robustness of feature matching.

[0004] Although the existing binocular vision SLAM technology and GCNv2 network have significant advantages, there are still the following main problems when applied in orchard environments:

[0005] 1) Challenge of light changes: The light conditions in the orchard vary dramatically, from strong direct sunlight to shadow areas under the shade of trees, making it difficult for traditional SLAM systems to work stably;

[0006] 2) Insufficient adaptability to dynamic environments: The orchard is a dynamically changing environment. With the change of seasons, factors such as the growth of fruit trees and the ripening of fruits will cause significant changes in appearance. Existing technologies are difficult to quickly adapt to these changes.

[0007] 3) Partial occlusion processing is difficult: Partial occlusion caused by tree trunks, branches, etc. frequently occurs, affecting the extraction and matching accuracy of feature points;

[0008] 4) Data scarcity: There is relatively little data on orchard environments, especially datasets containing rich details and variable conditions, which limits the training effect and generalization ability of the model;

[0009] 5) Real-time and computing resource requirements: Running high-precision deep learning models on embedded devices requires a lot of computing resources. Existing methods are difficult to meet real-time requirements, especially in resource-constrained agricultural robots;

[0010] 6) Complexity of path planning and obstacle recognition: Path planning and obstacle recognition in orchard environments face complex terrain and various types of obstacles (such as trees and fruits), which puts higher requirements on the accuracy of the SLAM system.

[0011] Therefore, how to provide an efficient, robust and binocular vision SLAM method suitable for orchard environments to improve the navigation ability and automation level of robots in complex orchard environments is an urgent problem that technical personnel in this field need to solve. Summary of the invention

[0012] In view of this, the present invention provides a binocular vision SLAM method and system based on the fusion of GCNv2 network in an orchard environment to solve some of the technical problems mentioned in the background technology.

[0013] In order to achieve the above object, the present invention adopts the following technical solution:

[0014] A binocular vision SLAM method based on a fused GCNv2 network in an orchard environment comprises the following steps:

[0015] S1. Collect high-quality stereo image pairs. Use a high-resolution binocular camera system to capture stereo image pairs of the orchard environment at different times and weather conditions.

[0016] S2. Use stereo matching algorithm to generate accurate depth map as label for each pair of stereo images, build GCNv2 network architecture including attention mechanism and multi-scale feature fusion, and train it;

[0017] S3. Input the binocular vision image of the orchard environment and apply the ORB feature detection algorithm to extract key points, and find the correspondence between the left and right images through descriptor matching. Use the depth map predicted by the trained GCNv2 network to assign corresponding depth values ​​to each matched key point;

[0018] S4. Filter out unstable feature points according to the depth information, and add the filtered feature points together with their depth information to the current map representation for map update;

[0019] S5. By comparing the newly observed scene with the position in the existing map, we look for possible loop opportunities and perform geometric consistency checks. Once the loop is confirmed to exist, we execute the BA global optimization algorithm to correct the errors in the entire trajectory and map.

[0020] Preferably, step S2 also includes applying one or more combined transformation operations including brightness adjustment, contrast adjustment, random cropping, rotation and flipping to the original image, and using 3D modeling tools to create a virtual orchard scene to generate a synthetic data set and increase data diversity.

[0021] Preferably, the specific content of using the stereo matching algorithm to generate an accurate depth map for each pair of stereo images is:

[0022] Preprocess the input left and right images and calculate the matching cost of the corresponding pixels of the left and right images;

[0023] Perform path cost aggregation in four horizontal and vertical directions, and select the disparity with the lowest total cost as the best matching disparity based on the aggregated cost volume;

[0024] The obtained initial disparity map is post-processed and converted into a depth map according to the disparity value and known camera parameters.

[0025] Preferably, the GCNv2 network architecture including the attention mechanism and multi-scale feature fusion is constructed, and the specific contents of the training are:

[0026] Choose a GCNv2 variant suitable for handling complex textures and lighting changes as the base architecture;

[0027] Add a spatial attention module to the network to calculate the importance weight of each position;

[0028] Design cross-layer connections to combine low-level feature maps with high-level feature maps to form richer feature representations;

[0029] The total loss function is designed by geometric consistency loss and introducing the GANs framework for adversarial learning. The Adam optimizer is used for training, and batch normalization is applied after each layer of convolution to speed up training.

[0030] Preferably, the specific method of introducing the adversarial learning framework is:

[0031] Discriminator design: A discriminator network is constructed to distinguish between the real orchard depth map and the generated orchard depth map. The discriminator adopts a convolutional neural network structure, and the last layer outputs a scalar value to represent the probability of true or false.

[0032] Generator Design: A generator network is designed by adding upsampling layers to the convolutional neural network structure to generate high-resolution orchard depth maps.

[0033] Preferably, the training process also includes performing structured pruning of the model by identifying and removing neurons or layers that contribute less to the orchard environment SLAM task.

[0034] Preferably, the training process also includes designing an incremental learning framework that allows the model to gradually update its own parameters based on newly acquired data; when the environment changes, the parameters of specific layers are fine-tuned to adapt to the changing conditions of the orchard environment.

[0035] A binocular vision SLAM system based on a fused GCNv2 network in an orchard environment, based on the binocular vision SLAM method based on a fused GCNv2 network in an orchard environment, comprising: a binocular camera system, a model building and training module, a binocular vision SLAM framework integration module, a depth information integration module and a loop detection and global optimization module;

[0036] A binocular camera system for capturing stereo image pairs of the orchard environment at different times and weather conditions;

[0037] The model building and training module is used to generate accurate depth maps as labels for each pair of stereo images using a stereo matching algorithm, build a GCNv2 network architecture that includes an attention mechanism and multi-scale feature fusion, and perform training;

[0038] The integrated module of the binocular vision SLAM framework is used to extract key points by applying the ORB feature detection algorithm to the input binocular vision image of the orchard environment, and find the correspondence between the left and right images through descriptor matching. The depth map predicted by the trained GCNv2 network is used to assign the corresponding depth value to each matched key point.

[0039] A depth information integration module is used to filter out unstable feature points based on depth information, and add the filtered feature points together with their depth information to the current map representation for map update;

[0040] The loop detection and global optimization module is used to find possible loop opportunities and perform geometric consistency checks by comparing the newly observed scene with the position in the existing map. Once the loop is confirmed to exist, the BA global optimization algorithm is executed to correct the errors in the entire trajectory and map.

[0041] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements a binocular vision SLAM method based on a fused GCNv2 network in an orchard environment.

[0042] A processing terminal comprises a memory and a processor. The memory stores a computer program that can be run on the processor. When the processor executes the computer program, a binocular vision SLAM method based on a fused GCNv2 network in an orchard environment is implemented.

[0043] It can be seen from the above technical solution that, compared with the prior art, the present invention discloses a binocular vision SLAM method and system based on the fusion GCNv2 network in an orchard environment, which has the following advantages:

[0044] Enhanced adaptability: By introducing the GCNv2 network with attention mechanism and multi-scale feature fusion, the present invention can better handle the common illumination changes and partial occlusion problems in orchard environments, thereby improving the robustness and adaptability of the system;

[0045] Higher accuracy: By using high-quality depth maps as training labels and combining them with advanced stereo matching algorithms, the present invention can achieve higher-precision depth estimation in complex environments, thereby improving the positioning and mapping accuracy of the SLAM system;

[0046] Real-time performance: Through effective feature extraction and matching strategies and efficient deep information integration solutions, the real-time performance of the system is ensured, which is suitable for dynamically changing orchard environments;

[0047] Continuous improvement capability: By designing an incremental learning framework, the model is allowed to gradually update its parameters based on newly acquired data, so that the system can operate stably for a long time and can quickly adapt to changes in the environment;

[0048] The present invention provides a binocular vision SLAM method that is efficient, robust and suitable for orchard environments, which significantly improves the robot's navigation capability and automation level in complex agricultural environments. It not only solves the problems of illumination changes and insufficient adaptability to dynamic environments in the prior art, but also improves the accuracy of path planning and obstacle recognition, which helps to provide strong technical support for future intelligent agriculture. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0050] Figure 1 A schematic diagram of a binocular vision SLAM method based on a fused GCNv2 network in an orchard environment provided by the present invention;

[0051] Figure 2 A schematic diagram of the GCNv2 network training including the attention mechanism and multi-scale feature fusion provided by the present invention;

[0052] Figure 3 A schematic diagram of a binocular vision SLAM system based on a fused GCNv2 network in an orchard environment provided by the present invention. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] The embodiment of the present invention discloses a binocular vision SLAM method based on the fusion GCNv2 network in an orchard environment. Figure 1 , including the following steps:

[0055] S1. Collect high-quality stereo image pairs. Use a high-resolution binocular camera system to capture stereo image pairs of the orchard environment at different times and weather conditions.

[0056] S2. Use stereo matching algorithm to generate accurate depth map as label for each pair of stereo images, build GCNv2 network architecture including attention mechanism and multi-scale feature fusion, and train it;

[0057] S3. Input the binocular vision image of the orchard environment and apply the ORB feature detection algorithm to extract key points, and find the correspondence between the left and right images through descriptor matching. Use the depth map predicted by the trained GCNv2 network to assign corresponding depth values ​​to each matched key point;

[0058] S4. Filter out unstable feature points according to the depth information, and add the filtered feature points together with their depth information to the current map representation for map update;

[0059] S5. By comparing the newly observed scene with the position in the existing map, we look for possible loop opportunities and perform geometric consistency checks. Once the loop is confirmed to exist, we execute the BA global optimization algorithm to correct the errors in the entire trajectory and map.

[0060] To further implement the above technical solution, step S2 further includes applying brightness adjustment to the original image (by multiplying by a factor α and adding an offset β: new =αI+β), contrast adjustment, random cropping, rotation and flipping, and the use of 3D modeling tools to create a virtual orchard scene to generate a synthetic dataset and increase data diversity.

[0061] In order to further implement the above technical solution, the specific content of using the stereo matching algorithm to generate an accurate depth map for each pair of stereo images is as follows:

[0062] Preprocess the input left and right images and calculate the matching cost of the corresponding pixels of the left and right images;

[0063] Perform path cost aggregation in four horizontal and vertical directions, and select the disparity with the lowest total cost as the best matching disparity based on the aggregated cost volume;

[0064] The obtained initial disparity map is post-processed and converted into a depth map according to the disparity value and known camera parameters.

[0065] In order to further implement the above technical solution, Figure 2 , construct the GCNv2 network architecture including attention mechanism and multi-scale feature fusion, and the specific content of training is as follows:

[0066] Choose a GCNv2 variant suitable for handling complex textures and lighting changes as the base architecture;

[0067] Add a spatial attention module to the network to calculate the importance weight of each position;

[0068] w i =f(x i ), where x i is the input feature vector, f(·) is the attention function;

[0069] f(x i )=σ(W x *x i +b x ), where W x and b x are the weight matrix and bias term respectively, and σ is the activation function;

[0070] Design cross-layer connections to combine low-level feature maps with high-level feature maps to form richer feature representations;

[0071] F out =W1F low ⊕W2F high

[0072] Among them, W1 and W2 are the weight matrices corresponding to the low-level and high-level feature maps respectively, and ⊕ represents element-wise addition or other fusion strategies;

[0073] The total loss function is designed by geometric consistency loss and introducing the GANs framework for adversarial learning. The Adam optimizer is used for training, and batch normalization is applied after each layer of convolution to speed up training.

[0074] In this embodiment, the geometric consistency constraint is added to ensure that the predicted disparity map satisfies the basic geometric relationship; for a pair of matching points (p l ,p r ), the loss term can be defined as L gc (pl ,p r )=|d(p l )-d(p r )|, where d(·) represents the depth value.

[0075] In order to further implement the above technical solution, the specific method of introducing the adversarial learning framework is as follows:

[0076] Discriminator design: A discriminator network is constructed to distinguish between the real orchard depth map and the generated orchard depth map. The discriminator adopts a convolutional neural network structure, and the last layer outputs a scalar value to represent the probability of true or false.

[0077] Generator design: The generator network is designed by adding upsampling layers to the convolutional neural network structure to generate high-resolution orchard depth maps;

[0078] L total =L adv (G,D)+λ*L content (G), where L adv is the loss against which L content is the content loss, and λ is a hyperparameter used to control the weight of the content loss and balance the two losses.

[0079] In the actual training process, the pre-trained model is used to initialize the network weights and the specific method of dynamically adjusting the learning rate is used, taking into account the scarcity of orchard data:

[0080] Transfer learning strategy: pre-train the model from a similar but richer public dataset and then fine-tune it using a small amount of orchard-specific data;

[0081] Intelligent learning rate adjustment: Adopts an adaptive learning rate adjustment algorithm, which not only considers the first-order moment estimation of the gradient, but also adds a weight decay term to prevent overfitting.

[0082] To further implement the above technical solution, the training process also includes model structured pruning by identifying and removing neurons or layers that contribute less to the orchard environment SLAM task;

[0083] Traverse all weights, set a threshold τ, and remove weights whose absolute value is less than τ. The formula is:

[0084] W pruned =W original ·(∣W original ∣>τ).

[0085] To further implement the above technical solution, the training process also includes designing an incremental learning framework that allows the model to gradually update its parameters based on newly acquired data. Specifically, each time new data is collected, it is added to the existing dataset and some layers of the model are retrained using the latest dataset, while the parameters of other layers remain unchanged.

[0086] When the environment changes, the parameters of specific layers are fine-tuned to adapt to the changing conditions of the orchard environment. Specifically, the layers that are more affected (such as shallow convolutional layers) are retrained, while the parameters of the deep feature extractors remain unchanged.

[0087] A binocular vision SLAM system based on the fusion GCNv2 network in an orchard environment. Figure 3 , based on a binocular vision SLAM method based on the fusion GCNv2 network in an orchard environment, including: a binocular camera system, a model building and training module, a binocular vision SLAM framework integration module, a depth information integration module, and a loop detection and global optimization module;

[0088] A binocular camera system for capturing stereo image pairs of the orchard environment at different times and weather conditions;

[0089] The model building and training module is used to generate accurate depth maps as labels for each pair of stereo images using a stereo matching algorithm, build a GCNv2 network architecture that includes an attention mechanism and multi-scale feature fusion, and perform training;

[0090] The integrated module of the binocular vision SLAM framework is used to extract key points by applying the ORB feature detection algorithm to the input binocular vision image of the orchard environment, and find the correspondence between the left and right images through descriptor matching. The depth map predicted by the trained GCNv2 network is used to assign the corresponding depth value to each matched key point.

[0091] A depth information integration module is used to filter out unstable feature points based on depth information, and add the filtered feature points together with their depth information to the current map representation for map update;

[0092] The loop detection and global optimization module is used to find possible loop opportunities and perform geometric consistency checks by comparing the newly observed scene with the position in the existing map. Once the loop is confirmed to exist, the BA global optimization algorithm is executed to correct the errors in the entire trajectory and map.

[0093] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements a binocular vision SLAM method based on a fused GCNv2 network in an orchard environment.

[0094] A processing terminal comprises a memory and a processor. The memory stores a computer program that can be run on the processor. When the processor executes the computer program, a binocular vision SLAM method based on a fused GCNv2 network is implemented in an orchard environment.

[0095] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0096] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A binocular vision SLAM method based on the fusion GCNv2 network in an orchard environment, characterized in that: The following steps are involved: S1. Collect high-quality stereo image pairs. Use a high-resolution binocular camera system to capture stereo image pairs of the orchard environment at different times and weather conditions. S2. Use stereo matching algorithm to generate accurate depth map as label for each pair of stereo images, build GCNv2 network architecture including attention mechanism and multi-scale feature fusion, and train it; S3. Input the binocular vision image of the orchard environment and apply the ORB feature detection algorithm to extract key points, and find the correspondence between the left and right images through descriptor matching. Use the depth map predicted by the trained GCNv2 network to assign corresponding depth values ​​to each matched key point; S4. Filter out unstable feature points according to the depth information, and add the filtered feature points together with their depth information to the current map representation for map update; S5. By comparing the newly observed scene with the position in the existing map, we look for possible loop opportunities and perform geometric consistency checks. Once the loop is confirmed to exist, we execute the BA global optimization algorithm to correct the errors in the entire trajectory and map.

2. A binocular vision SLAM method based on fusion GCNv2 network in an orchard environment according to claim 1, characterized in that: Step S2 also includes applying one or more combined transformation operations including brightness adjustment, contrast adjustment, random cropping, rotation and flipping to the original image, and using 3D modeling tools to create a virtual orchard scene to generate a synthetic data set and increase data diversity.

3. A binocular vision SLAM method based on fusion GCNv2 network in an orchard environment according to claim 1, characterized in that, The specific contents of using stereo matching algorithm to generate accurate depth map for each pair of stereo images are as follows: Preprocess the input left and right images and calculate the matching cost of the corresponding pixels of the left and right images; Perform path cost aggregation in four horizontal and vertical directions, and select the disparity with the lowest total cost as the best matching disparity based on the aggregated cost volume; The obtained initial disparity map is post-processed and converted into a depth map according to the disparity value and known camera parameters.

4. A binocular vision SLAM method based on fusion GCNv2 network in an orchard environment according to claim 1, characterized in that, The specific contents of building a GCNv2 network architecture that includes attention mechanism and multi-scale feature fusion and training are as follows: Choose a GCNv2 variant suitable for handling complex textures and lighting changes as the base architecture; Add a spatial attention module to the network to calculate the importance weight of each position; Design cross-layer connections to combine low-level feature maps with high-level feature maps to form richer feature representations; The total loss function is designed by geometric consistency loss and introducing the GANs framework for adversarial learning. The Adam optimizer is used for training, and batch normalization is applied after each layer of convolution to speed up training.

5. A binocular vision SLAM method based on fusion GCNv2 network in an orchard environment according to claim 4, characterized in that, The specific method of introducing the adversarial learning framework is: Discriminator design: A discriminator network is constructed to distinguish between the real orchard depth map and the generated orchard depth map. The discriminator adopts a convolutional neural network structure, and the last layer outputs a scalar value to represent the probability of true or false. Generator Design: A generator network is designed by adding upsampling layers to the convolutional neural network structure to generate high-resolution orchard depth maps.

6. A binocular vision SLAM method based on fusion GCNv2 network in an orchard environment according to claim 4, characterized in that, The training process also includes model structured pruning by identifying and removing neurons or layers that contribute less to the orchard environment SLAM task.

7. A binocular vision SLAM method based on fusion GCNv2 network in an orchard environment according to claim 4, characterized in that: The training process also includes designing an incremental learning framework that allows the model to gradually update its parameters based on newly acquired data; when the environment changes, the parameters of specific layers are fine-tuned to adapt to changing conditions in the orchard environment.

8. A binocular vision SLAM system based on the fusion GCNv2 network in an orchard environment, characterized in that: A binocular vision SLAM method based on a fused GCNv2 network in an orchard environment according to any one of claims 1 to 7, comprising: a binocular camera system, a model building and training module, a binocular vision SLAM framework integration module, a depth information integration module, and a loop detection and global optimization module; A binocular camera system for capturing stereo image pairs of the orchard environment at different times and weather conditions; The model building and training module is used to generate accurate depth maps as labels for each pair of stereo images using a stereo matching algorithm, build a GCNv2 network architecture that includes an attention mechanism and multi-scale feature fusion, and perform training; The integrated module of the binocular vision SLAM framework is used to extract key points by applying the ORB feature detection algorithm to the input binocular vision image of the orchard environment, and find the correspondence between the left and right images through descriptor matching. The depth map predicted by the trained GCNv2 network is used to assign the corresponding depth value to each matched key point. A depth information integration module is used to filter out unstable feature points based on depth information, and add the filtered feature points together with their depth information to the current map representation for map update; The loop detection and global optimization module is used to find possible loop opportunities and perform geometric consistency checks by comparing the newly observed scene with the position in the existing map. Once the loop is confirmed to exist, the BA global optimization algorithm is executed to correct the errors in the entire trajectory and map.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the binocular vision SLAM method based on the fusion GCNv2 network in an orchard environment as described in any one of claims 1 to 7.

10. A processing terminal, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the computer program, it implements the binocular vision SLAM method based on the fusion GCNv2 network in an orchard environment as described in any one of claims 1 to 7.

Images