Three-dimensional scene reconstruction and navigation system based on agricultural scene

Abstract

The invention provides a three-dimensional scene reconstruction and navigation system based on an agricultural scene. The three-dimensional scene reconstruction and navigation system comprises a data acquisition module, an image processing module, a three-dimensional reconstruction module, a navigation planning module and a multi-mode sensor module. The data acquisition module acquires two-dimensional image data of an agricultural scene by using a multi-view camera, transmits the two-dimensional image data to the image processing module for preprocessing and then inputs the two-dimensional image data into the three-dimensional reconstruction module. The three-dimensional reconstruction module generates a high-precision three-dimensional scene by means of a generative adversarial network model and a denoising diffusion model. And the navigation planning module fuses the three-dimensional reconstruction data and the multi-modal sensor information to realize autonomous navigation and accurate operation path planning of the agricultural machinery. The multi-mode sensor module provides key information such as position, posture and environment parameters. The problems that traditional agricultural production is low in efficiency, large in resource waste and difficult in navigation operation in a complex environment are effectively solved, the agricultural production efficiency is remarkably improved, the labor cost is reduced, the environmental adaptability of the system is enhanced, agricultural intelligent development is promoted, and the system has important application value and innovative significance.

Description

Technical Field

[0001] The present invention relates to the field of intelligent agricultural technology, and in particular to a three-dimensional scene reconstruction and navigation system based on agricultural scenarios. Background Art

[0002] Agricultural production faces an urgent need to improve efficiency and achieve sustainable development. Traditional agricultural models rely heavily on labor, resulting in low production efficiency and significant deficiencies in crop monitoring and resource management. For example, traditional agriculture struggles to accurately monitor crop growth in real time, leading to wasted resources and delayed pest and disease control.

[0003] In agricultural operations, precise navigation and operation are crucial for improving production efficiency and resource utilization. However, the agricultural environment is highly complex and unstructured. Lighting conditions vary dramatically throughout the day, with significant differences in light intensity and angle across seasons and weather conditions, severely impacting the performance of visual sensors. During crop growth, plants often block and overlap each other, making it difficult to obtain accurate crop information. Furthermore, complex terrain, such as rolling terrain, ditches, and ponds, as well as various obstacles, including rocks, buildings, and animals, pose significant challenges to the navigation and operation of agricultural machinery.

[0004] Currently, the application of existing 3D reconstruction and navigation technologies in the agricultural sector faces numerous challenges. Traditional 3D reconstruction methods, such as passive reconstruction based on RGB cameras, struggle to accurately extract depth information in complex lighting conditions and crop occlusion, resulting in low reconstruction accuracy. Active reconstruction methods, while improving the accuracy of depth data to a certain extent, are costly and ineffective in certain complex environments. Regarding navigation technology, the Global Navigation Satellite System (GNSS) is susceptible to obstruction and signal interference, significantly reducing positioning accuracy in areas such as dense vegetation or mountainous areas. While LiDAR (LiDAR) can provide highly accurate ranging data, its high cost limits its application in large-scale agriculture. Machine vision navigation systems are relatively inexpensive, but their reliability and accuracy struggle to meet the demands of agricultural production in conditions such as low light and shadows. Furthermore, existing navigation systems often lack adaptability to dynamic changes in the agricultural environment and are unable to adjust navigation strategies in real time, resulting in low operational efficiency.

[0005] Therefore, developing a three-dimensional scene reconstruction and navigation system that can adapt to complex agricultural environments is of great practical significance for improving the level of intelligent agricultural production and realizing precision agriculture. Summary of the Invention

[0006] The present invention provides a three-dimensional scene reconstruction and navigation system based on agricultural scenarios to solve the problems of low efficiency, large resource waste, and difficulty in navigation and operation in complex agricultural environments in traditional agricultural production. It improves agricultural production efficiency, reduces labor costs, enhances system environmental adaptability, and promotes the development of intelligent agriculture.

[0007] The present invention provides a three-dimensional scene reconstruction and navigation system based on agricultural scenarios, comprising a data acquisition module, an image processing module, a three-dimensional reconstruction module, a navigation planning module and a multimodal sensor module;

[0008] The data acquisition module is connected to the image processing module, the image processing module is connected to the three-dimensional reconstruction module, the three-dimensional reconstruction module is connected to the navigation planning module, and the multimodal sensor module is connected to the navigation planning module;

[0009] The data acquisition module is composed of multiple multi-view cameras and is used to collect two-dimensional image data of agricultural scenes;

[0010] The image processing module receives the image data transmitted by the data acquisition module and performs pre-processing;

[0011] The 3D reconstruction module includes a generative adversarial network model and a denoising diffusion model. The generator of the generative adversarial network model generates a preliminary 3D scene based on multiple frames of images. The discriminator optimizes the model by comparing the generated scene with the actual data. The denoising diffusion model processes the noise in the image data.

[0012] The navigation planning module combines 3D reconstruction data with multimodal sensor information to achieve autonomous navigation and operation path planning for agricultural machinery;

[0013] The multimodal sensor module integrates multiple sensors to obtain the position, posture and environmental parameter information of agricultural machinery.

[0014] Preferably, the preprocessing operations of the image processing module include image denoising, grayscale conversion and normalization.

[0015] Preferably, the generator of the generative adversarial network model adopts a convolutional neural network combined with a transposed convolution structure, and the discriminator adopts a convolutional neural network to extract and compare features of the generated three-dimensional scene and actual data.

[0016] Preferably, the denoising diffusion model gradually adds noise to the image data in a forward process, and restores the three-dimensional structure through a denoising step in a reverse process.

[0017] Preferably, it also includes a navigation line row detection module, which is connected to the data acquisition module and the navigation planning module. The navigation line row detection module uses a Transformer-based semantic segmentation model to detect crop rows, extracts anchor points through triangle fitting, determines the initial search direction using vertical projection and image segmentation, and fits anchor point rows through the least squares method.

[0018] Preferably, the Transformer-based semantic segmentation model includes a segmentation Transformer, a shift window Transformer and a SegFormer.

[0019] Preferably, the sensors in the multimodal sensor module include a global navigation satellite system receiver, an inertial measurement unit sensor and an environmental sensor.

[0020] Preferably, a three-dimensional scene reconstruction and navigation method based on an agricultural scene comprises the following steps:

[0021] A. Collect 2D image data of agricultural scenes through the multi-view camera of the data acquisition module;

[0022] B. Transmitting the collected image data to the image processing module for preprocessing;

[0023] C. Input the pre-processed image data into the 3D reconstruction module, use the generative adversarial network model to generate a preliminary 3D scene and optimize it, and use the denoising diffusion model to deal with image noise;

[0024] D. The navigation planning module combines 3D reconstruction data with multimodal sensor information to plan the operation path of agricultural machinery and achieve autonomous navigation.

[0025] Preferably, in the navigation line row detection, a Transformer-based semantic segmentation model is used to detect crop rows, anchor points are extracted by triangle fitting, the initial search direction is determined by vertical projection and image segmentation, and the anchor point rows are fitted by the least squares method.

[0026] Beneficial effects:

[0027] (1) Through three-dimensional scene reconstruction and precise navigation, the present invention enables agricultural machinery to quickly and accurately complete tasks, reducing operating time and wasted resources. For example, in plant protection operations, agricultural machinery can accurately spray pesticides based on the actual growth conditions of crops and the distribution of pests and diseases, avoiding unnecessary duplication of work and improving the efficiency of pesticide utilization.

[0028] (2) The present invention enables autonomous navigation and operation of agricultural machinery, reducing reliance on manual operation. In the event of labor shortages, it can ensure the normal operation of agricultural production and reduce labor costs.

[0029] By real-time monitoring of crop growth status and environmental parameters, problems can be discovered in a timely manner and corresponding measures can be taken, such as reasonable fertilization, pest and disease control, etc., which helps to improve the quality and yield of agricultural products.

[0030] The above description is only an overview of the technical solutions of the embodiments of the present invention. In order to more clearly understand the technical means of the embodiments of the present invention, they can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the embodiments of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are specifically listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following is a brief introduction to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0032] Figure 1 This is a system architecture diagram of the present invention;

[0033] Figure 2 This is a principle block diagram of the multimodal sensor module of the present invention;

[0034] Figure 3 It is a flowchart of the present invention;

[0035] Explanation of the accompanying drawings: data acquisition module 1, image processing module 2, three-dimensional reconstruction module 3, navigation planning module 4, multimodal sensor module 5, navigation line detection module 6, global navigation satellite system receiver 7, inertial measurement unit sensor 8, environmental sensor 9. DETAILED DESCRIPTION

[0036] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0037] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention pertains; the terms used in the specification of the application herein are for the purpose of describing specific embodiments only and are not intended to limit the present invention; the terms "including" and "having" and any variations thereof in the specification, claims and accompanying drawings of the present invention are intended to cover non-exclusive inclusions.

[0038] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase "embodiment" in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute a separate or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0039] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense. For example, the "connection" or "connection" of a mechanical structure may refer to a physical connection. For example, the physical connection may be a fixed connection, such as a fixed connection through a fixing member, such as a fixed connection through a screw, bolt, or other fixing member; the physical connection may also be a detachable connection, such as a mutual snap connection or a snap connection; the physical connection may also be an integral connection, such as a connection formed by welding, bonding, or integral molding. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0040] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0041] See also Figure 1-Figure 3 , the present invention discloses a three-dimensional scene reconstruction and navigation system based on an agricultural scene, comprising a data acquisition module 1, an image processing module 2, a three-dimensional reconstruction module 3, a navigation planning module 4 and a multimodal sensor module 5;

[0042] The data acquisition module 1 is connected to the image processing module 2, the image processing module 2 is connected to the three-dimensional reconstruction module 3, the three-dimensional reconstruction module 3 is connected to the navigation planning module 4, and the multimodal sensor module 5 is connected to the navigation planning module 4;

[0043] The data acquisition module consists of multiple multi-view cameras, which are used to capture 2D image data of agricultural scenes. These cameras, mounted on agricultural machinery or fixed monitoring points, capture farmland environmental information from different angles, including topography, crop growth status, and obstacle distribution. The camera layout and parameter settings are carefully designed to ensure sufficient overlap in the captured images, facilitating subsequent 3D reconstruction.

[0044] The image processing module receives the image data transmitted by the data acquisition module and performs preprocessing. Preprocessing operations include image denoising, grayscale conversion, and normalization. Denoising removes noise interference from the image and improves image quality. Grayscale conversion converts color images into grayscale images, reducing data volume and improving subsequent processing efficiency. Normalization keeps the image's pixel values within a certain range, facilitating subsequent feature extraction and model training.

[0045] The 3D reconstruction module comprises a generative adversarial network (GAN) model and a denoising diffusion model. The GAN generator generates a preliminary 3D scene based on multiple image frames. The discriminator optimizes the model by comparing the generated scene with the actual data. The denoising diffusion model processes noise in the image data. The GAN generator employs a convolutional neural network combined with transposed convolutions, while the discriminator uses a convolutional neural network to extract and compare features between the generated 3D scene and the actual data. The denoising diffusion model gradually adds noise to the image data during the forward process and restores the 3D structure through a denoising step during the reverse process. In the forward process, noise is gradually added to the image data to simulate disturbances found in agricultural environments, such as low light, shadows, and dynamically changing obstacles. In the reverse process, the denoising step gradually restores the 3D structure. This model effectively handles image noise in complex environments, generating high-quality 3D reconstruction data even in poor lighting conditions or complex terrain, providing reliable support for navigation systems.

[0046] The navigation and planning module combines 3D reconstruction data with multimodal sensor information to enable autonomous navigation and path planning for agricultural machinery. This module employs advanced algorithms to plan the optimal path based on the machinery's task and current environmental information. During operation, it monitors environmental changes in real time and, when obstacles or other anomalies are detected, promptly adjusts the path to ensure the machinery completes its task safely and efficiently.

[0047] The multimodal sensor module integrates multiple sensors to acquire the position, attitude, and environmental parameter information of agricultural machinery. The sensors in the multimodal sensor module 5 include a global navigation satellite system receiver 7, an inertial measurement unit sensor 8, and an environmental sensor 9. These sensor data are integrated with the 3D reconstruction data to provide more comprehensive and accurate environmental information to the navigation planning module.

[0048] The present invention also includes a navigation line detection module 6, which is connected to the data acquisition module 1 and the navigation planning module 4. The navigation line detection module uses a Transformer-based semantic segmentation model to detect crop rows, extracts anchor points through triangle fitting, determines the initial search direction using vertical projection and image segmentation, and fits anchor point rows using the least squares method. The Transformer-based semantic segmentation model includes a segmentation Transformer, a shift window Transformer, and a SegFormer. A large number of rice seedling images are collected using an RGB color camera, and a data set is constructed using a superpixel-level annotation method using a morphological mask. After training, the model can accurately segment the stem area. After obtaining the segmentation mask, the anchor points are extracted through triangle fitting, and the bottom endpoint of the triangle is used as the cropping reference to reduce noise interference. Then, the initial search direction is determined through vertical projection and image segmentation. Starting from the initial point, a synchronous search is performed up and down, and the nearest anchor points are selected in turn until no new anchor points appear within the search range. Finally, the anchor point rows are fitted by the least squares method, providing accurate results for the detection and fitting of rice rows and a reliable path planning basis for agricultural machinery navigation.

[0049] In terms of operation path planning and efficiency optimization, a deep reinforcement learning algorithm is used to optimize the path planning of agricultural machinery, combined with 3D reconstruction data of agricultural scenes. Through model-driven reinforcement learning, the system simulates the navigation strategies of agricultural machinery in different operation scenarios and gradually optimizes path selection through a large number of trial and error processes. A dynamic model of the environment is constructed within the reinforcement learning model to predict future states and rewards, allowing agricultural machinery to consider long-term benefits during the learning process and achieve more intelligent and flexible decision-making. Furthermore, the deep reinforcement learning algorithm is combined with a deep neural network. The deep neural network extracts deep features from the 3D reconstruction data, providing precise environmental perception support for the reinforcement learning algorithm, ensuring robust and efficient path planning under complex conditions.

[0050] Working principle: A 3D scene reconstruction and navigation method based on agricultural scenarios, including the following steps:

[0051] A. Collect 2D image data of agricultural scenes through the multi-view camera of the data acquisition module;

[0052] B. Transmitting the collected image data to the image processing module for preprocessing;

[0053] C. Input the pre-processed image data into the 3D reconstruction module, use the generative adversarial network model to generate a preliminary 3D scene and optimize it, and use the denoising diffusion model to deal with image noise;

[0054] D. The navigation planning module combines 3D reconstruction data with multimodal sensor information to plan the operation path of agricultural machinery and achieve autonomous navigation.

[0055] In navigation line row detection, a Transformer-based semantic segmentation model is used to detect crop rows, anchor points are extracted through triangle fitting, vertical projection and image segmentation are used to determine the initial search direction, and the anchor point rows are fitted using the least squares method.

[0056] This invention enables autonomous navigation and operation of agricultural machinery, reducing reliance on manual operation. This ensures the smooth progress of agricultural production in the face of labor shortages, reducing labor costs. By monitoring crop growth and environmental parameters in real time, problems can be identified and appropriate measures implemented, such as appropriate fertilization and pest control, helping to improve the quality and yield of agricultural products.

[0057] In summary, through 3D scene reconstruction and precise navigation, the present invention enables agricultural machinery to quickly and accurately complete tasks, reducing operational time and wasted resources. For example, in plant protection operations, agricultural machinery can precisely spray pesticides based on the actual growth of crops and the distribution of pests and diseases, avoiding unnecessary repetitive work and improving pesticide utilization efficiency.

[0058] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions described in the above embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A three-dimensional scene reconstruction and navigation system based on agricultural scenes, characterized by: It includes a data acquisition module (1), an image processing module (2), a three-dimensional reconstruction module (3), a navigation planning module (4) and a multimodal sensor module (5); The data acquisition module (1) is connected to the image processing module (2), the image processing module (2) is connected to the three-dimensional reconstruction module (3), the three-dimensional reconstruction module (3) is connected to the navigation planning module (4), and the multimodal sensor module (5) is connected to the navigation planning module (4); The data acquisition module is composed of multiple multi-view cameras and is used to collect two-dimensional image data of agricultural scenes; The image processing module receives the image data transmitted by the data acquisition module and performs pre-processing; The 3D reconstruction module includes a generative adversarial network model and a denoising diffusion model. The generator of the generative adversarial network model generates a preliminary 3D scene based on multiple frames of images. The discriminator optimizes the model by comparing the generated scene with the actual data. The denoising diffusion model processes the noise in the image data. The navigation planning module combines 3D reconstruction data with multimodal sensor information to achieve autonomous navigation and operation path planning for agricultural machinery; The multimodal sensor module integrates multiple sensors to obtain the position, posture and environmental parameter information of agricultural machinery.

2. The three-dimensional scene reconstruction and navigation system based on agricultural scenes according to claim 1 is characterized in that: The preprocessing operations of the image processing module include image denoising, grayscale conversion and normalization.

3. The three-dimensional scene reconstruction and navigation system based on agricultural scenes according to claim 1, characterized in that: The generator of the generative adversarial network model adopts a convolutional neural network combined with a transposed convolution structure, and the discriminator uses a convolutional neural network to extract and compare features of the generated three-dimensional scene and actual data.

4. The three-dimensional scene reconstruction and navigation system based on agricultural scenes according to claim 1, characterized in that: The denoising diffusion model gradually adds noise to the image data in the forward process and restores the three-dimensional structure through the denoising step in the reverse process.

5. The three-dimensional scene reconstruction and navigation system based on agricultural scenes according to claim 1, characterized in that: The system also includes a navigation line detection module (6), which is connected to the data acquisition module (1) and the navigation planning module (4). The navigation line detection module detects crop rows using a Transformer-based semantic segmentation model, extracts anchor points through triangle fitting, determines an initial search direction using vertical projection and image segmentation, and fits anchor point rows using a least squares method.

6. The three-dimensional scene reconstruction and navigation system based on agricultural scenes according to claim 5, characterized in that: The Transformer-based semantic segmentation model includes segmentation Transformer, shift window Transformer and SegFormer.

7. The three-dimensional scene reconstruction and navigation system based on agricultural scenes according to claim 1, characterized in that: The sensors in the multimodal sensor module (5) include a global navigation satellite system receiver (7), an inertial measurement unit sensor (8) and an environmental sensor (9).

8. A three-dimensional scene reconstruction and navigation method based on an agricultural scene, characterized in that: The following steps are involved: A. Collect 2D image data of agricultural scenes through the multi-view camera of the data acquisition module; B. Transmitting the collected image data to the image processing module for preprocessing; C. Input the pre-processed image data into the 3D reconstruction module, use the generative adversarial network model to generate a preliminary 3D scene and optimize it, and use the denoising diffusion model to deal with image noise; D. The navigation planning module combines 3D reconstruction data with multimodal sensor information to plan the operation path of agricultural machinery and achieve autonomous navigation.

9. The three-dimensional scene reconstruction and navigation method based on agricultural scenes according to claim 8, characterized in that: In navigation line row detection, a Transformer-based semantic segmentation model is used to detect crop rows, anchor points are extracted through triangle fitting, vertical projection and image segmentation are used to determine the initial search direction, and the anchor point rows are fitted using the least squares method.

Images