A binocular vision SLAM method and system based on a fusion GCNv2 network in an orchard environment

Abstract

The application discloses a kind of orchard environment based on fusion GCNv2 network binocular vision SLAM method and system, comprising: using binocular camera to shoot the stereo image pair of orchard environment;Depth map is generated using stereo matching algorithm as label, construct and train GCNv2 network architecture including attention mechanism and multi-scale feature fusion;Input orchard environment image and extract key point, find left-right image corresponding relationship by descriptor matching, assign corresponding depth value to each matching key point using the depth map predicted by GCNv2 network;Filter feature points and update map with depth information;Loop detection and perform geometric consistency check, execute BA global optimization algorithm to correct errors in the entire trajectory and map;The application provides a kind of efficient, robust and suitable for orchard environment binocular vision SLAM method to improve the navigation ability and automation level of robot in complex orchard environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of simultaneous localization and mapping (SLAM), and more particularly to a binocular vision SLAM method and system based on a fusion GCNv2 network in an orchard environment. BACKGROUND

[0002] With the increasing automation of agriculture, robots are increasingly used in orchard management. However, the orchard environment often has complex terrain and vegetation structures, which poses challenges for the autonomous navigation of robots.

[0003] Binocular vision SLAM simulates the way humans observe the world with two eyes, calculates parallax to obtain depth information, and helps robots to realize self-localization and create a map of the surrounding environment in unknown environments. GCNv2 is a deep neural network based on geometric consistency, designed specifically for stereo matching, and can provide high-quality dense depth maps. Compared with traditional feature point-based methods, GCNv2 uses learned feature representations to assist or even replace hand-designed feature extractors, improving the accuracy and robustness of feature matching.

[0004] Although the existing binocular vision SLAM technology and GCNv2 network each have significant advantages, there are still the following main problems when applied in an orchard environment:

[0005] 1) Lighting variation challenge: The lighting conditions in the orchard change dramatically, from strong direct sunlight to shaded areas under trees, making it difficult for traditional SLAM systems to work stably;

[0006] 2) Insufficient adaptability to dynamic environment: The orchard is a dynamic environment, and as the seasons change, factors such as tree growth and fruit ripening can cause significant changes in appearance, making it difficult for existing technology to quickly adapt to these changes;

[0007] 3) Difficulty in handling partial occlusions: Partial occlusions caused by tree trunks, branches, and leaves frequently occur, affecting the extraction and matching accuracy of feature points;

[0008] 4) Data scarcity: The data in the orchard environment is relatively scarce, especially datasets containing rich details and variable conditions, limiting the training effect and generalization ability of the model;

[0009] 5) Real-time performance and computational resource requirements: Running high-precision deep learning models on embedded devices requires a large amount of computational resources, and existing methods are difficult to meet real-time requirements, especially in resource-constrained agricultural robots;

[0010] 6) Complexity of path planning and obstacle recognition: Path planning and obstacle recognition in orchard environments face complex terrain and various types of obstacles (such as trees, fruits), which pose higher requirements for the accuracy of SLAM systems.

[0011] Therefore, how to provide an efficient, robust and suitable binocular vision SLAM method for orchard environment to improve the navigation ability and automation level of robots in complex orchard environment is a problem that those skilled in the art need to solve. SUMMARY

[0012] Therefore, the present application provides a binocular vision SLAM method based on fusion GCNv2 network in orchard environment to solve some of the technical problems mentioned in the background art.

[0013] In order to achieve the above purpose, the present application adopts the following technical solutions:

[0014] A binocular vision SLAM method based on fusion GCNv2 network in orchard environment, comprising the following steps:

[0015] S1. Collect high-quality stereo image pairs, use a high-resolution binocular camera system to capture stereo image pairs of the orchard environment under different time and weather conditions;

[0016] S2. Use a stereo matching algorithm to generate accurate depth maps for each stereo image pair as labels, construct a GCNv2 network architecture containing attention mechanism and multi-scale feature fusion, and train it;

[0017] S3. Input the binocular vision image of the orchard environment and apply the ORB feature detection algorithm to extract key points, find the corresponding relationship between the left and right images through descriptor matching, and use the depth map predicted by the trained GCNv2 network to assign corresponding depth values to each matched key point;

[0018] S4. Filter out unstable feature points according to the depth information, and add the filtered feature points together with their depth information to the current map representation for map updating;

[0019] S5. Find possible loop opportunities and perform geometric consistency checks by comparing the newly observed scene with the positions in the existing map, and once the loop is confirmed to exist, execute the BA global optimization algorithm to correct errors in the entire trajectory and map.

[0020] Preferably, step S2 further comprises applying one or more combined transformation operations including brightness adjustment, contrast adjustment, random cropping, rotation and flipping to the original image, and creating a virtual orchard scene using a 3D modeling tool to generate a synthetic dataset and increase data diversity.

[0021] Preferably, the specific content of using a stereo matching algorithm to generate an accurate depth map for each pair of stereo images is:

[0022] Preprocessing the input left and right eye images, calculating the matching cost of the corresponding pixel points of the left and right eye image points;

[0023] Aggregating the path cost in four horizontal and vertical directions, and selecting the disparity with the lowest total cost as the best matching disparity based on the aggregated cost volume;

[0024] Post-processing the obtained initial disparity map, and converting the disparity map into a depth map according to the disparity value and the known camera parameters.

[0025] Preferably, the specific content of constructing a GCNv2 network architecture containing an attention mechanism and multi-scale feature fusion and training is:

[0026] Selecting a GCNv2 variant suitable for processing complex textures and lighting changes as the basic architecture;

[0027] Adding a spatial attention module to the network to calculate the importance weight of each position;

[0028] Designing cross-layer connections to combine low-level feature maps with high-level feature maps to form richer feature representations;

[0029] Designing the total loss function through geometric consistency loss and introducing GANs framework for adversarial learning, using Adam optimizer for training, and applying batch normalization after each convolution to speed up training.

[0030] Preferably, the specific method of introducing an adversarial learning framework is:

[0031] Discriminator design: build a discriminator network to distinguish between real orchard depth maps and generated orchard depth maps, the discriminator uses a convolutional neural network structure, and the last layer outputs a scalar value representing the probability of being true or false;

[0032] Generator design: design a generator network through a convolutional neural network structure to increase the upsampling layer to generate high-resolution orchard depth maps.

[0033] Preferably, the training process further includes model structural pruning by identifying and removing neurons or layers that contribute less to the orchard environment SLAM task.

[0034] Preferably, the training process further includes designing an incremental learning framework that allows the model to gradually update its parameters based on newly acquired data; when the environment changes, adapt to the changing conditions of the orchard environment by fine-tuning the parameters of specific layers.

[0035] The application discloses a binocular vision SLAM system based on a fusion GCNv2 network in an orchard environment.

[0036] The binocular camera system is used for shooting stereo image pairs of the orchard environment under different time and weather conditions.

[0037] The model construction and training module is used for generating an accurate depth map as a label for each stereo image pair by using a stereo matching algorithm, constructing a GCNv2 network architecture containing an attention mechanism and multi-scale feature fusion, and performing training.

[0038] The binocular vision SLAM framework integration module is used for extracting key points by applying an ORB feature detection algorithm to input binocular vision images of the orchard environment, finding a corresponding relationship between left and right images by descriptor matching, and assigning a corresponding depth value to each matched key point by using a depth map predicted by the trained GCNv2 network.

[0039] The depth information integration module is used for filtering out unstable feature points according to depth information, and adding the filtered feature points and their depth information to a current map representation for map updating.

[0040] The loop detection and global optimization module is used for finding a possible loop opportunity and performing geometric consistency checking by comparing a newly observed scene with a position in an existing map, and correcting errors in the entire trajectory and map by performing a BA global optimization algorithm once a loop is confirmed to exist.

[0041] A computer readable storage medium has a computer program stored thereon, and the computer program is executed by a processor to implement the binocular vision SLAM method based on the fusion GCNv2 network in the orchard environment.

[0042] A processing terminal comprises a memory and a processor, and the memory has a computer program stored therein and executable on the processor, and the processor implements the binocular vision SLAM method based on the fusion GCNv2 network in the orchard environment when executing the computer program.

[0043] Compared with the prior art, the application discloses a binocular vision SLAM method and system based on a fusion GCNv2 network in an orchard environment, and has the following advantages.

[0044] Enhanced adaptability: By introducing the GCNv2 network containing attention mechanism and multi-scale feature fusion, the invention can better handle the common illumination changes and partial occlusion problems in orchard environment, thereby improving the robustness and adaptability of the system;

[0045] Higher accuracy: Using high-quality depth maps as training labels and combining advanced stereo matching algorithms, the invention can achieve higher accuracy in depth estimation in complex environments, thereby improving the positioning and mapping accuracy of the SLAM system;

[0046] Real-time performance: Through effective feature extraction and matching strategies and efficient depth information integration schemes, the real-time performance of the system is ensured, which is suitable for dynamic changes in orchard environment;

[0047] Continuous improvement capability: By designing an incremental learning framework, the model allows itself to update its parameters gradually based on newly acquired data, enabling the system to run stably for a long time and quickly adapt to environmental changes;

[0048] The invention provides a binocular vision SLAM method that is efficient, robust and suitable for orchard environment, significantly improving the navigation ability and automation level of robots in complex agricultural environment. It not only solves the problems of insufficient adaptability to illumination changes and dynamic environment in existing technology, but also improves the accuracy of path planning and obstacle recognition, which helps to provide strong technical support for future intelligent agriculture. BRIEF DESCRIPTION OF DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description are only embodiments of the present invention, and those skilled in the art can obtain other drawings according to the provided drawings without creative labor.

[0050] Figure 1 A binocular vision SLAM method based on fusion GCNv2 network in orchard environment provided by the present invention;

[0051] Figure 2 The GCNv2 network training diagram provided by the present invention contains attention mechanism and multi-scale feature fusion;

[0052] Figure 3 A binocular vision SLAM system based on fusion GCNv2 network in orchard environment provided by the present invention. DETAILED DESCRIPTION

[0053] With reference to the accompanying drawings, the technical solutions in the embodiments of the present application will be clearly and completely described below. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of the present application.

[0054] The embodiments of the present application disclose a binocular vision SLAM method based on a fusion GCNv2 network in an orchard environment, as shown in Figure 1 , comprising the following steps:

[0055] S1. Collect high-quality stereo image pairs, use a high-resolution binocular camera system to shoot stereo image pairs of an orchard environment at different times and under different weather conditions;

[0056] S2. Use a stereo matching algorithm to generate an accurate depth map for each stereo image pair as a label, construct a GCNv2 network architecture containing an attention mechanism and multi-scale feature fusion, and perform training;

[0057] S3. Input binocular vision images of an orchard environment and apply an ORB feature detection algorithm to extract key points, find the corresponding relationship between left and right images through descriptor matching, and assign a corresponding depth value to each matched key point by using the depth map predicted by the trained GCNv2 network;

[0058] S4. Filter out unstable feature points according to the depth information, and add the filtered feature points together with their depth information to the current map representation for map updating;

[0059] S5. Find possible loop opportunities and perform geometric consistency checks by comparing the newly observed scene with the positions in the existing map, and once the existence of a loop is confirmed, execute a BA global optimization algorithm to correct errors in the entire trajectory and map.

[0060] In order to further implement the above technical solutions, step S2 further comprises applying one or more combined transformation operations including brightness adjustment (achieved by multiplying a factor a and adding an offset b: I new = aI + b), contrast adjustment, random cropping, rotation and flipping to the original images, and creating a virtual orchard scene using a 3D modeling tool to generate a synthetic data set and increase data diversity.

[0061] In order to further implement the above technical solutions, the specific content of generating an accurate depth map for each stereo image pair using a stereo matching algorithm is as follows:

[0062] The input left and right images are preprocessed, and the matching cost of the corresponding pixel points of the left and right images is calculated;

[0063] The path cost is aggregated in four horizontal and vertical directions, and based on the aggregated cost volume, the disparity with the lowest total cost is selected as the best matching disparity;

[0064] The obtained initial disparity map is post-processed, and the disparity map is converted into a depth map according to the disparity value and known camera parameters.

[0065] In order to further implement the above technical solutions, such as Figure 2 , a GCNv2 network architecture containing attention mechanism and multi-scale feature fusion is constructed, and the specific content of training is as follows:

[0066] The GCNv2 variant suitable for processing complex texture and illumination change is selected as the basic architecture;

[0067] The spatial attention module is added in the network to calculate the importance weight of each position;

[0068] w i =f(x i ), wherein x i is an input feature vector, and f(·) is an attention function;

[0069] f(x i )=σ(W x *x i +b x ), wherein W x and b x are weight matrix and bias term respectively, and σ is an activation function;

[0070] The cross-layer connection is designed to combine low-level feature maps with high-level feature maps to form richer feature representation;

[0071] F out =W1F low ⊕W2F high

[0072] Wherein, W1 and W2 are weight matrices corresponding to low-level and high-level feature maps respectively, and ⊕ represents element-wise addition or other fusion strategy;

[0073] The total loss function is designed by geometric consistency loss and adversarial learning introduced in the GANs framework, the Adam optimizer is used for training, and batch normalization is applied after each convolution to speed up the training.

[0074] In the embodiment, the geometric consistency constraint is added to ensure that the predicted disparity map meets the basic geometric relationship; for a pair of matching points (p l , p r ), the loss term can be defined as L gc (pl ,p r ) = |d(p l )-d(p r )|, where d(·) denotes the depth value.

[0075] To further implement the above technical solutions, the specific method of introducing an adversarial learning framework is:

[0076] Discriminator design: a discriminator network is constructed to distinguish between real orchard depth maps and generated orchard depth maps. The discriminator adopts a convolutional neural network structure, and the last layer outputs a scalar value representing the probability of being true or false.

[0077] Generator design: a generator network is designed by adding an up-sampling layer to the convolutional neural network structure to generate high-resolution orchard depth maps.

[0078] L total = L adv (G,D) + λ*L content (G), where L adv is the adversarial loss, L content is the content loss, and λ is a hyperparameter used to control the weight of the content loss and balance the two losses.

[0079] In the actual training process, the specific method of using a pre-trained model to initialize the network weights and dynamically adjusting the learning rate is considered in view of the scarcity of orchard data:

[0080] Transfer learning strategy: pre-train the model from a similar but more abundant public dataset, and then fine-tune it using a small amount of orchard-specific data.

[0081] Intelligent learning rate adjustment: use an adaptive learning rate adjustment algorithm that not only considers the first moment estimate of the gradient but also adds a weight decay term to prevent overfitting.

[0082] To further implement the above technical solutions, the training process also includes structurally pruning the model by identifying and removing neurons or layers that contribute less to the orchard environment SLAM task;

[0083] Iterate through all weights, set a threshold τ, and remove weights with absolute values less than τ, the formula is:

[0084] W pruned = W original ·(∣W original ∣>τ).

[0085] To further implement the above technical solutions, the training process further includes designing an incremental learning framework to allow the model to update its parameters gradually based on newly acquired data, specifically: after each new data is collected, it is added to the existing data set, and the latest data set is used to retrain part of the model layers, while keeping the parameters of other layers unchanged.

[0086] When the environment changes, the parameters of specific layers are fine-tuned to adapt to the changing conditions of the orchard environment, specifically: the layers that are more affected (such as shallow convolutional layers) are retrained, while the parameters of the deep feature extractor are kept unchanged.

[0087] An orchard environment-based binocular vision SLAM system based on a fusion GCNv2 network, as shown in Figure 3 An orchard environment-based binocular vision SLAM method based on a fusion GCNv2 network, including: a binocular camera system, a model construction and training module, a binocular vision SLAM framework integration module, a depth information integration module, and a loop detection and global optimization module.

[0088] The binocular camera system is used to capture stereo image pairs of the orchard environment under different time and weather conditions.

[0089] The model construction and training module is used to generate accurate depth maps for each stereo image pair as labels using a stereo matching algorithm, construct a GCNv2 network architecture containing attention mechanisms and multi-scale feature fusion, and perform training.

[0090] The binocular vision SLAM framework integration module is used to extract key points by applying the ORB feature detection algorithm to input binocular vision images of the orchard environment, find the corresponding relationship between left and right images through descriptor matching, and assign corresponding depth values to each matched key point using the depth map predicted by the trained GCNv2 network.

[0091] The depth information integration module is used to filter out unstable feature points based on depth information, and add the filtered feature points along with their depth information to the current map representation for map updating.

[0092] The loop detection and global optimization module is used to find possible loop opportunities and perform geometric consistency checks by comparing newly observed scenes with positions in the existing map, and once a loop is confirmed to exist, the BA global optimization algorithm is executed to correct errors in the entire trajectory and map.

[0093] A computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement an orchard environment-based binocular vision SLAM method based on a fusion GCNv2 network.

[0094] A processing terminal comprising a memory and a processor, the memory storing a computer program capable of running on the processor, and the processor implements a binocular vision SLAM method based on a fusion GCNv2 network in an orchard environment when executing the computer program.

[0095] The various embodiments are described in the specification with progressive progression, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be mutually referred to. For the device disclosed by the embodiments, since it corresponds to the method disclosed by the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part.

[0096] The above description of the disclosed embodiments enables a person skilled in the art to implement or use the present application. Various modifications to the embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A binocular vision SLAM method based on a fusion GCNv2 network in an orchard environment, characterized in that, The method comprises the following steps: S1. Collect high-quality stereo image pairs, use a high-resolution binocular camera system to take stereo image pairs of orchard environments at different times and weather conditions; S2. Use a stereo matching algorithm to generate accurate depth maps for each stereo image pair as labels, construct a GCNv2 network architecture containing attention mechanisms and multi-scale feature fusion, and train it; S3. Input the binocular vision image of the orchard environment and apply the ORB feature detection algorithm to extract key points, find the corresponding relationship between the left and right images through descriptor matching, and use the depth map predicted by the trained GCNv2 network to assign a corresponding depth value to each matched key point; S4. Filter out unstable feature points according to the depth value, and add the filtered feature points together with their depth values to the current map representation for map updating; S5. By comparing the newly observed scene with the position in the existing map, find the loop opportunity and perform geometric consistency check, and once the loop is confirmed to exist, execute the BA global optimization algorithm to correct the errors in the entire trajectory and map.

2. The binocular vision SLAM method based on the fusion GCNv2 network in an orchard environment according to claim 1, characterized in that, Step S2 further comprises applying one or more combined transformation operations including brightness adjustment, contrast adjustment, random cropping, rotation and flipping to the original image, and creating a virtual orchard scene using a 3D modeling tool to generate a synthetic data set and increase data diversity.

3. The binocular vision SLAM method based on the fusion GCNv2 network in an orchard environment according to claim 1, characterized in that, The specific content of generating accurate depth maps for each stereo image pair using a stereo matching algorithm is: Preprocess the input left and right images, calculate the matching cost of the corresponding pixel points of the left and right images; Aggregate the path cost in four horizontal and vertical directions, and select the disparity with the lowest total cost as the best matching disparity based on the aggregated cost volume; Post-process the obtained initial disparity map, and convert the disparity map to a depth map according to the disparity value and known camera parameters.

4. The binocular vision SLAM method based on the fusion GCNv2 network in an orchard environment according to claim 1, characterized in that, The specific content of constructing a GCNv2 network architecture containing attention mechanisms and multi-scale feature fusion and training it is: Select a GCNv2 variant suitable for processing complex textures and lighting changes as the basic architecture; Add a spatial attention module to the network to calculate the importance weight of each position; Design a cross-layer connection to combine low-level feature maps with high-level feature maps to form a richer feature representation; Design the total loss function through geometric consistency loss and adversarial learning by introducing a GANs framework, use the Adam optimizer for training, and apply batch normalization after each convolution to speed up training.

5. The binocular vision SLAM method based on the fusion GCNv2 network in an orchard environment according to claim 4, characterized in that, The specific method of introducing an adversarial learning framework is: Discriminator design: build a discriminator network to distinguish between real orchard depth maps and generated orchard depth maps, the discriminator uses a convolutional neural network structure, and the last layer outputs a scalar value representing the probability of being true or false; Generator design: design a generator network through a convolutional neural network structure to increase the upsampling layer to generate high-resolution orchard depth maps.

6. The binocular vision SLAM method based on the fusion GCNv2 network in an orchard environment according to claim 4, characterized in that, The training process also includes designing an incremental learning framework that allows the model to gradually update its parameters based on newly acquired data.

7. A binocular vision SLAM system based on a fusion GCNv2 network in an orchard environment, characterized in that, The binocular vision SLAM method based on the fusion GCNv2 network in an orchard environment according to any one of claims 1-6 comprises a binocular camera system, a model construction and training module, a binocular vision SLAM framework integration module, a depth information integration module, and a loop detection and global optimization module. The binocular camera system is used to shoot stereo image pairs of the orchard environment under different time and weather conditions. The model construction and training module is used to generate an accurate depth map as a label for each stereo image pair by using a stereo matching algorithm, construct a GCNv2 network architecture containing an attention mechanism and multi-scale feature fusion, and perform training. The binocular vision SLAM framework integration module is used to extract key points by applying an ORB feature detection algorithm to input binocular vision images of the orchard environment, find the corresponding relationship between left and right images through descriptor matching, and assign a corresponding depth value to each matched key point by using the depth map predicted by the trained GCNv2 network. The depth information integration module is used to filter out unstable feature points according to the depth value, and add the filtered feature points together with their depth values to the current map representation for map updating. The loop detection and global optimization module is used to find loop opportunities and perform geometric consistency checks by comparing newly observed scenes with positions in the existing map, and once a loop is confirmed to exist, perform a BA global optimization algorithm to correct errors in the entire trajectory and map.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by a processor to implement the binocular vision SLAM method based on the fusion GCNv2 network in an orchard environment according to any one of claims 1-6.

9. A processing terminal comprising a memory and a processor, the memory having stored therein a computer program to be run on the processor, characterized in that, The processor executes the computer program to implement the binocular vision SLAM method based on the fusion GCNv2 network in an orchard environment according to any one of claims 1-6.

Images