Agricultural scene three-dimensional reconstruction method based on multi-modal sensor fusion

Abstract

The invention discloses an agricultural scene three-dimensional reconstruction method based on multi-modal sensor fusion, and belongs to the technical field of three-dimensional modeling. The method comprises the following steps: acquiring multi-modal data of a target scene based on a tracked robot, wherein the multi-modal data comprises image data acquired by a binocular camera, point cloud data acquired by a laser radar, time sequence inertial data acquired by an inertial measurement unit and position data acquired by a Beidou positioning system; extracting different modal data features in the multi-modal data, and carrying out heterogeneous fusion on the different modal data features to obtain multi-modal data fusion features; and agricultural scene three-dimensional reconstruction is carried out by using a three-dimensional dynamic construction method based on space-time-semantic joint embedding. According to the invention, the environment scene modeling precision of the agricultural robot under the interference of muddy, raised dust and vibration can be improved.

Description

Technical Field

[0001] This invention relates to the field of 3D modeling technology, and more specifically to a method for 3D reconstruction of agricultural scenes based on multimodal sensor fusion. Background Technology

[0002] In recent years, tracked robots have become core equipment in agricultural plant protection, sowing, and harvesting, requiring autonomous operation in complex environments such as mud, slopes, crop shading, and dynamic disturbances. However, existing agricultural robots generally suffer from insufficient environmental perception robustness, which severely restricts their large-scale application.

[0003] Currently, environmental perception technologies for farmland robots mostly rely on single sensors (such as RGB cameras or 2D LiDAR), which face the following limitations: ① In muddy and vegetated environments, traditional visual algorithms are prone to feature mismatches due to differences in ground reflectivity or leaf occlusion; ② In dusty / rainy / foggy environments, LiDAR point clouds generate noise due to scattering by suspended particles, resulting in significant delays in the detection of dynamic obstacles (such as personnel and moving farm machinery); ③ Vibrations caused by rugged terrain lead to pose estimation drift in SLAM (Simultaneous Localization and Mapping) systems, making it difficult to correct local deformations in 3D maps.

[0004] In existing solutions, multi-sensor fusion methods based on manual threshold rules cannot adaptively adjust weights according to the dynamic characteristics of the environment, which may lead to decision failure when sensors degrade (such as camera failure in rainy weather). In addition, traditional static scene reconstruction algorithms ignore temporal changes such as crop growth deformation and temporary obstacle movement, resulting in a mismatch between the navigation path and the actual environment, increasing the risk of the robot getting stuck in mud or colliding with crops.

[0005] Therefore, there is an urgent need for a dynamic perception and real-time map building technology for complex farmland, which can improve the environmental modeling accuracy and operational safety of agricultural robots under muddy, dusty, and vibration interference, and provide technical support for unmanned farmland management. Summary of the Invention

[0006] In view of this, the present invention provides a method for three-dimensional reconstruction of agricultural scenes based on multimodal sensor fusion, which is used to at least solve some of the technical problems in the background art.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A method for 3D reconstruction of agricultural scenes based on multimodal sensor fusion includes the following steps:

[0009] Multimodal data of the target scene collected by the tracked robot includes image data collected by binocular cameras, point cloud data collected by lidar, time-series inertial data collected by inertial measurement unit, and position data collected by Beidou positioning system;

[0010] Extract features from different modalities in multimodal data, and heterogeneously fuse these features to obtain multimodal data fusion features;

[0011] A 3D dynamic construction method based on spatiotemporal-semantic joint embedding is used for 3D reconstruction of agricultural scenes.

[0012] Furthermore, this invention extracts features of different modalities from multimodal data, specifically including:

[0013] For image data acquired by a stereo camera, a ResNet-50 backbone network with dual-path weight sharing is used for feature extraction. At the same time, a 3D Cost Volume module is constructed for depth estimation, and semantic feature maps and disparity maps are output collaboratively.

[0014] For point cloud data collected by LiDAR, a voxelized Transformer encoder is used to process the data and generate bird's-eye view features that express the geometric structure of the object.

[0015] For the time-series inertial data collected by the inertial measurement unit, potential motion features are extracted using a bidirectional gated recurrent unit network;

[0016] Location data collected by the BeiDou Navigation Satellite System is compressed into a low-dimensional vector through a fully connected layer and encoded with geographic information context to obtain fused BeiDou positioning information.

[0017] Furthermore, for the image data acquired from binocular vision data, feature extraction is performed using a dual-path weight-sharing ResNet-50 backbone network, specifically including:

[0018] The left and right images captured by the binocular camera are used to extract features using the ResNet-50 backbone network to obtain the left feature map and the right feature map.

[0019] The obtained left and right feature maps are input into the 3D Cost Volume module. Matching costs under multiple depth / disparity assumptions are applied to the left and right feature maps. The matching cost results are then subjected to 3D convolution aggregation and regression to obtain the disparity map.

[0020] The obtained left or right feature map is input into a semantic decoder that includes multiple transposed convolutional layers. By upsampling and feature fusion of feature maps at different scales, a semantic feature map is obtained.

[0021] Furthermore, heterogeneous fusion of features from different modalities is performed to obtain multimodal data fusion features, specifically including:

[0022] A cross-scale, cross-modal integration strategy is adopted to fuse the extracted semantic feature map with disparity map, bird's-eye view features, latent motion features, and BeiDou positioning information to obtain multimodal data fusion features.

[0023] Furthermore, a cross-scale, cross-modal integration strategy is employed to fuse the extracted semantic feature maps with disparity maps, bird's-eye view features, latent motion features, and BeiDou positioning information. Specifically, this includes:

[0024] A deformable convolution mechanism is introduced to project the semantic feature map onto the bird's-eye view reference frame of the LiDAR based on depth cues, and then the projected semantic feature map and the bird's-eye view features are fused at the geometric level to obtain image fusion features.

[0025] The navigation state vector is formed by splicing and fusing BeiDou positioning information with potential motion characteristics.

[0026] Image fusion features and navigation state vectors are fused at a high level using a Transformer module based on multi-head cross-attention. Modal associations are learned through adaptive weights, and multimodal data fusion features that support 3D reconstruction tasks in agricultural scenarios are output.

[0027] Furthermore, the projected semantic feature map and the bird's-eye view feature are geometrically fused to obtain image fusion features, specifically including:

[0028] The semantic feature maps are then weighted using gating to obtain dynamically weighted semantic features:

[0029]

[0030] Among them, F s Let W represent the semantic features of the input, σ represent the Sigmoid activation function, and W represent the semantic features of the input. g and b g For learnable parameters, ⊙ represents element-wise multiplication. This represents the semantic features after dynamic weighting;

[0031] The dynamically weighted semantic features and the bird's-eye view features are input into the multi-head cross-attention fusion module for fusion and stitching to obtain the image fusion features.

[0032] Furthermore, the dynamically weighted semantic features and the bird's-eye view features are input into the multi-head cross-attention fusion module for fusion and concatenation, specifically including the following formula:

[0033]

[0034] Output=Concat(head1,...,head h W 0

[0035] In the formula, F is the head-specific projection matrix. g Indicates features of a bird's-eye view, W 0 Denotes a linear transformation, d k For the key vector dimension, head i This represents the output of the i-th attention head.

[0036] Furthermore, a 3D dynamic construction method based on spatiotemporal-semantic joint embedding is used for 3D reconstruction of agricultural scenes, specifically including:

[0037] Multimodal data fusion features are encoded using a symbolic distance field implicit function to generate neural implicit joint features corresponding to three-dimensional space; the neural implicit joint features include geometric attribute component features, appearance attribute component features, semantic attribute component features, and temporal dynamic component features;

[0038] Differentiable volume rendering is used to render neural implicit joint features into a two-dimensional image, and the parameters of the neural implicit function are optimized based on the loss function between the rendering result and the real acquired data.

[0039] Based on the symbolic distance field predicted by the optimized neural implicit function, a continuous surface mesh model is extracted using the moving cube algorithm.

[0040] The semantic features queried during the extraction of the surface mesh model are input into the semantic feature decoder to assign semantic labels to each facet in the surface mesh model.

[0041] Output a dense 3D mesh model with semantic labels.

[0042] Furthermore, the multimodal data acquisition step of the target scene in this invention also includes:

[0043] Based on the physical characteristics of lidar scanning, dust scattering noise distribution is modeled, and combined with the light intensity in the images acquired by the binocular camera, a degradation perception mask is dynamically generated. Then, an adversarial generative network is used to collaboratively repair the missing areas of the point cloud and the overexposed images.

[0044] As can be seen from the above technical solution, compared with the prior art, the present invention discloses a method for three-dimensional reconstruction of agricultural scenes based on multimodal sensor fusion, which has the following beneficial effects:

[0045] This invention obtains multimodal data fusion features containing different data through multimodal data collaborative optimization and dynamic fusion, and uses a three-dimensional dynamic construction method based on spatiotemporal-semantic joint embedding to model three-dimensional scenes, which can improve the scene modeling accuracy of agricultural robots under muddy, dusty, and vibration interference. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0047] Figure 1 A schematic diagram of a multi-sensor fusion method for scene 3D reconstruction provided in this embodiment of the invention.

[0048] Figure 2 A control diagram for a tracked robot used to validate 3D environment reconstruction. Detailed Implementation

[0049] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] To address the sensor degradation problem caused by dynamic terrain changes and multi-source interference in agricultural scenarios, this invention proposes a 3D reconstruction method for agricultural scenarios based on multimodal sensor fusion. By using the different sensor characteristics of a tracked robot for modeling and data-driven optimization, high-precision dynamic analysis and map construction can be achieved for complex farmland or other scenarios.

[0051] In practical implementation, this invention can be achieved by constructing a multimodal perception and reconstruction integrated system architecture, which includes a heterogeneous sensor collaborative module and a dynamic 3D reconstruction engine. The heterogeneous sensor collaborative module integrates a solid-state LiDAR, a binocular camera, a high-precision IMU, and a BeiDou differential positioning system. The LiDAR adopts a dynamic coding scanning mode, adaptively increasing the point cloud density in key areas under dusty conditions to generate a geometric skeleton of the terrain surface. The binocular camera acquires 3D spatial information by mimicking the parallax principle of human binoculars, combining environmental perception and geometric measurement functions in the machine vision system. BeiDou and the IMU construct global motion constraints through a tightly coupled positioning model, compensating for pose drift caused by track slippage in real time, and providing a spatiotemporal alignment benchmark for multi-source data.

[0052] The dynamic three-dimensional reconstruction engine adopts a hierarchical hash voxel framework, extracts feature vectors such as terrain curvature and elevation mutations from the lidar point cloud through multi-scale sparse convolution, encodes the farmland semantic labels for the multi-spectral image through a lightweight Transformer network, and generates a semantic global scene implicit field after gated fusion of heterogeneous features. At the same time, an incremental voxel optimization strategy is introduced, a dynamic residual field is constructed using the differences in consecutive frame point clouds, local hash blocks are dynamically expanded for newly added obstacles (such as temporary mounds), voxel compression based on probability density is performed on stable background regions, combined with a semantic confidence update mechanism, and the map continuity is maintained relying on the radar geometric features during visual degradation. Finally, a real-time three-dimensional semantic map of the farmland with centimeter-level accuracy and 20 frames per second update is achieved, supporting precise agricultural operation decision-making and terrain adaptability control, and the accuracy of the three-dimensional map is verified by the path planning of a tracked robot.

[0053] As a preferred technical solution, the heterogeneous sensor collaboration module further includes a degradation-resistant processing unit. The degradation-resistant processing unit dynamically perceives the degraded area by deeply fusing the physical characteristics of the lidar and the optical characteristics of the image, and uses an advanced generative model for collaborative repair, and finally outputs enhanced and optimized sensor data to solve the collaborative problem of the missing lidar point cloud data and the overexposure / degradation of the camera image during the three-dimensional modeling process of the scene due to the influence of harsh environments such as dust, haze, and strong light.

[0054] Specifically, the degradation-resistant processing unit models the dust scattering noise distribution using the physical characteristics of the lidar scan (such as wavelength, transmission power). The dust scattering noise distribution can be physically modeled based on the Mie scattering theory to reflect the energy attenuation of the lidar in the dust:

[0055] I = I’·e -β·d +η

[0056] Where, I represents the true echo intensity of the laser pulse, I’ represents the theoretical echo intensity of the laser pulse, η represents the random scattering noise, and β is the attenuation coefficient related to the wavelength and dust concentration.

[0057] Calculate the true echo intensity and the theoretically expected echo intensity of each point to generate a point cloud confidence map. For each point in the point cloud confidence map: c i = I i / I’; Set a dynamic threshold T according to the average confidence of the scene. When c i < T, it can be considered that this point is severely attenuated or lost due to dust scattering, and finally a binary mask M1 of the point cloud missing area is obtained;

[0058] Using the calibration extrinsic parameters of the camera-LiDAR, the LiDAR point cloud P is projected onto the image plane to generate a projected point cloud map M2 aligned with the image pixels. Simultaneously, a point cloud missing mask M1 is also projected onto the image plane to obtain a point cloud missing mask M1' in the image coordinate system. A logical OR operation is performed on corresponding points in the projected point cloud map M2 and the point cloud missing mask M1' to obtain a new dynamic degradation sensing mask M1' that incorporates the image. fused .

[0059] Finally, an adversarial generative network is used to collaboratively insulate the missing areas of the point cloud and the overexposed image. This process employs a two-branch ConditionalGAN architecture, which includes a generator G and two discriminators (image discriminators D). img and point cloud discriminator D pc Furthermore, it achieves collaborative repair of missing point cloud regions and overexposed images through a cross-modal attention mechanism;

[0060] Specifically, in the image inpainting branch of generative adversarial networks, the encoder uses the original image and a dynamically degraded perceptual mask M. fused The concatenated images are used as input, and multi-scale image features are extracted by downsampling through a ResNet convolutional neural network.

[0061] The point cloud repair branch in the anti-generative network uses a 3D CNN or PointNet++ encoder to extract the spatial structural features of the point cloud.

[0062] Introducing a cross-modal attention mechanism, for the image inpainting branch, multi-scale image features are used as...

[0063] The query and point cloud features are used as the key and value for attention calculation. The decoder restores the original image resolution using the attention calculation result, resulting in the repaired image. For the point cloud inpainting branch, the point cloud features are used as the query and the image features as the key and value for attention calculation. The attention calculation result is then applied to the dynamic degradation perceptual mask M. fused The region is used to generate a new point cloud, resulting in the repaired point cloud data.

[0064] The following is combined with Figure 1 The specific steps of the present invention will be further described in detail below.

[0065] Physically Constrained Deep Reinforcement Learning Fusion of Multimodal Sensors in Complex Scenarios:

[0066] Environmental perception is fundamental for the dynamic adaptive path planning of tracked robots, especially in complex agricultural environments where they must cope with factors such as terrain changes, the presence of obstacles, and dynamic environmental changes. This invention employs multi-sensor fusion technology, combining data from multiple sensors, including LiDAR, binocular cameras, inertial measurement units (IMUs), and the BeiDou Navigation Satellite System, to achieve high-precision environmental perception. The depth information provided by LiDAR enables high-precision obstacle distance measurement for the walking device; the binocular cameras help identify different types of ground features and obstacles; the IMU system provides ground information and vehicle vibration data; and the positioning system provides positioning information, ensuring that the walking device can acquire positioning data in real time in complex environments, comprehensively providing multi-modal sensor data. By constructing a deep neural network, heterogeneous fusion, feature extraction, and joint training are performed on the sensor data based on the provided data from different dimensions, and the sensor data weights are adaptively adjusted according to different environmental conditions. Utilizing the complementary characteristics of different sensors, environmental information is monitored in real time.

[0067] In the dynamic environmental perception of agricultural robots, multi-sensor heterogeneous fusion based on deep learning constructs a data-driven global feature representation space by simulating the adaptive integration mechanism of the human cognitive system for multi-source information. Its core lies in utilizing the multi-level abstraction capabilities of neural networks to map heterogeneous signals such as LiDAR point clouds, IMU inertial data, and BeiDou positioning coordinates onto a unified high-dimensional feature space. Through cross-attention mechanisms and gating networks, it dynamically captures the complementary and contradictory relationships between sensors. The theoretical foundation of this method comes from the "soft decision" paradigm in modern information fusion theory, which involves weighted integration of different information sources through probability distributions rather than deterministic rules.

[0068] In its implementation, a bifurcated encoder architecture is first used to process the raw data from different dimensions: sparse point clouds from the LiDAR are processed using a voxelized Transformer encoder to generate bird's-eye view features representing the object's geometry; temporal inertial data from the IMU are processed through a bidirectional gated recurrent unit network to capture the robot's motion continuity; and the BeiDou position is transformed into a position embedding vector in the global coordinate system. Subsequently, a conditional gating mechanism is introduced into the multimodal fusion layer. The switching state of this gating unit is controlled by the output of the prediction branch of the current environmental context (such as light intensity and GPS signal quality)—when the dust concentration detection module reports a decrease in visibility, the network automatically reduces the weight coefficient of the LiDAR channel and instead increases the feature contribution of IMU dead reckoning and visual inertial odometry. This dynamic weight allocation is essentially a modeling of the uncertainty of sensor observations, consistent with the dynamic evaluation of information reliability in Dempster-Shafer evidence theory.

[0069] During joint training, the network achieves end-to-end optimization through a triple loss function: geometric consistency loss forces the topologically coherent 3D scene reconstructed by multiple sensors to remain consistent; temporal smoothing loss constrains the estimation results of adjacent frames to conform to kinematic laws; and adaptive entropy regularization guides the network to select the source with lower information entropy (i.e., higher determinism) when sensor information conflicts occur. This design is inspired by the optimization of latent variable distribution in variational inference, enabling the network to autonomously discover complementary relationships between sensors. For example, in the task of detecting field ridge edges, when the elevation data of the lidar is incomplete due to crop occlusion, the network automatically enhances the texture feature extraction capability of the multispectral camera in the near-infrared band, realizing boundary inference based on multi-band reflectivity differences.

[0070] Theoretically, the effectiveness of this method stems from the ability of deep neural networks to approximate high-dimensional nonlinear relationships. When observations from multiple sensors exhibit complex coupling relationships in a low-dimensional space, traditional Kalman filters based on linear assumptions struggle to accurately model these relationships, while deep networks can construct more precise implicit mappings through multi-layer nonlinear transformations. Simultaneously, the attention mechanism mimics the selective focusing characteristics of the human perception system, enabling the network to quickly adjust its feature representation strategy when faced with local anomalies (such as temporary obstacles). Mathematically, this mechanism is equivalent to imposing sparsity constraints on the sensor feature space, enhancing the model's robustness when some sensors fail.

[0071] After obtaining the multimodal fusion features, a 3D dynamic construction method based on spatiotemporal-semantic joint embedding is used to reconstruct the agricultural scene in 3D. In this step, the multimodal data fusion features are encoded through a symbolic distance field implicit function to generate neural implicit joint features corresponding to the 3D space. The neural implicit joint features include geometric attribute components, appearance attribute components, semantic attribute components, and temporal dynamic components. Differentiable volume rendering is used to render the neural implicit joint features into a 2D image, and the parameters of the neural implicit function are optimized based on the loss function between the rendering result and the real collected data. Based on the symbolic distance field predicted by the optimized neural implicit function, a continuous surface mesh model is extracted using the moving cube algorithm. The semantic features queried during the extraction of the surface mesh model are input into the semantic feature decoder to assign semantic labels to each facet in the surface mesh model. Finally, a dense 3D mesh model with semantic labels is output.

[0072] In the above steps, the implicit function of the symbolic distance field is expressed as:

[0073] f θ (x,t)→(d,c,s,v)

[0074] Where x is the spatial coordinate, t is the timestamp, d is the symbol distance value, c is the RGB color vector, s is the semantic feature vector, v is the temporal dynamic feature vector, and θ is the neural network parameter.

[0075] In the step of using differentiable volume rendering to render neural implicit joint features into a two-dimensional image, and optimizing the parameters of the neural implicit function based on the loss function between the rendered result and the actual acquired data, the differentiable volume rendering process is achieved by integrating the sampling points on the camera ray:

[0076]

[0077] in: For cumulative transmittance, α i For symbolic distance d i The calculated ray termination probability, c i For color values. The loss function between the rendered result and the actual acquired data includes multiple supervised components:

[0078] Υ=λ rgb Υ rgb +λ depth Υ depth +λ sem Υ sem +λ sdf Υ sdf

[0079] Among them Υ rgb For color reconstruction loss, Υ depth For depth consistency loss, Υ sem For semantic segmentation loss, Υ sdf For the symbolic distance field regularization loss

[0080] The 3D scene reconstruction process also includes modeling the motion trajectory of dynamic targets in the agricultural scene using the aforementioned time dynamic component features, encoding the time series using a spatiotemporal Transformer, and predicting the future motion trajectory of dynamic targets such as agricultural machinery and personnel; the specific dynamic trajectory prediction is achieved by querying an implicit function for future moments.

[0081] τ future =Φ({f θ (x,t future ) x∈R})

[0082] Where Φ is the trajectory decoder and R is the target region, the dynamic target in the agricultural scene is identified through semantic features.

[0083] For the dynamically changing regions of agricultural scenarios, an incremental optimization strategy is adopted, which only updates the local implicit neural function in the changing parts to achieve real-time evolution of the scenario. The incremental optimization strategy is achieved through local parameter adjustments.

[0084] θ new =θ old -η▽ θ Υ local

[0085] Incremental optimization is limited to network parameters corresponding to local areas in agricultural scenarios that perceive changes, in order to ensure efficient adaptation to dynamic changes in the agricultural environment.

[0086] To verify the environmental perception and dynamic adaptive path tracking control scheme based on multi-sensor fusion and deep learning, this invention will conduct a series of experiments on a physical prototype of a tracked robot, such as... Figure 2 As shown, a path tracking control system based on deep learning and multi-sensor fusion is designed. This control system continuously acquires current position information, velocity, acceleration, and path deviation, and adjusts the path tracking strategy in real time. The path tracking control system mainly consists of three modules:

[0087] The path deviation calculation module calculates the path deviation based on real-time position, velocity, and a predetermined path, and adjusts the position and attitude according to sensor data. By fusing high-precision GNSS positioning data, angular velocity signals from the IMU inertial measurement unit, and kinematic observations from the laser odometry, a two-dimensional plane state estimator is constructed. An extended Kalman filter algorithm is used to optimally estimate the robot's real-time pose. The actual position is projected onto the reference path in the coordinate system to generate lateral and heading angle deviations. Subsequently, a model predictive control framework is introduced, and feedforward-feedback composite compensation is generated through rolling time-domain optimization. This includes not only proportional-integral adjustment terms but also velocity-related predictive compensation factors derived from the dynamic model, enabling proactive adjustment of nonlinear disturbances in rugged terrain.

[0088] The deep learning inference module, combining deep learning algorithms, makes intelligent decisions based on current environmental perception data, obstacles, terrain, etc., and optimizes path correction schemes. The environmental perception and decision-making module constructs a multimodal deep neural network architecture. The front end uses a sparse convolutional neural network to process LiDAR point cloud data, extracting high-dimensional obstacle spatial distribution features. Simultaneously, an improved semantic segmentation network is used to analyze road surface types and dynamic targets in visual images, generating pixel-level semantic masks. At the decision layer, reinforcement learning and generative adversarial networks are innovatively integrated. A proximal policy optimization algorithm is used to train the action policy function, and the reward function combines collision risk indicators and energy consumption costs to make multi-objective trade-offs. At the same time, the implicit representation of the neural network is used to generate a terrain-adaptive path deformation field, so that the corrected path achieves the best balance between obstacle avoidance requirements and motion smoothness.

[0089] The execution control module, based on path planning and deviation adjustment results, controls key parameters such as voltage, current, and speed of the tracked drive system and steering system in real time to ensure the tracked robot accurately tracks the planned path. At the decision-making level, global path planning uses the A* algorithm, comprehensively considering terrain elevation, obstacle distribution, and passage costs to generate the optimal reachable path point sequence. The actuators then perform high-dynamic control based on the planned path and real-time perceived environmental deviations (such as lateral position error and heading angle deviation). The high-dynamic control stage establishes a dynamic model of the permanent magnet synchronous motor. Addressing the strong nonlinear characteristics of the tracked drive system, a dual-closed-loop sliding mode variable structure controller is designed. In the current loop, a boundary layer approximation method is used to suppress high-frequency chattering, and in the speed loop, fuzzy logic is combined to dynamically adjust the switching gain coefficient, effectively reducing motor torque ripple. The steering servo system constructs an LQR optimal regulator based on Lyapunov stability theory, mapping the state vector, including path tracking error, to the steering angle control quantity, and using FPGA hardware acceleration to achieve high-frequency refresh of multiple PWM signals. Ultimately, a complete time-domain closed loop is formed, consisting of "perception (environmental information) - decision-making (A* global planning + real-time deviation adjustment) - execution (high-precision motor and steering control)," achieving dynamic control accuracy with small lateral tracking error and small maximum path deviation under complex terrain.

[0090] The first test involves environmental perception and path planning. Real-time data collected by sensors verifies the system's perception accuracy and path planning effectiveness in complex terrain and dynamic environments. The second test is path tracking control. Multi-scenario path tracking experiments are conducted in an agricultural environment to verify the response speed, accuracy, and stability of the path tracking control system under actual working conditions. The third test is system integration and optimization verification. Through the integration of a computer and embedded hardware system, the execution efficiency and reliability of the entire control system are tested, ensuring efficient coordination between data processing, path planning, and execution control. The tests will comprehensively evaluate the performance of the multi-tracked machinery in a real-world environment, focusing on its adaptability, stability, and safety in dynamically changing environments.

[0091] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0092] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for three-dimensional reconstruction of an agricultural scene based on multi-modal sensor fusion, characterized in that, The method comprises the following steps: Based on the multi-modal data collected by the tracked robot in the target scene, including image data collected by a binocular camera, point cloud data collected by a laser radar, time series inertia data collected by an inertial measurement unit, and position data collected by a Beidou positioning system; Extract different modal data features from the multi-modal data, and perform heterogeneous fusion on the different modal data features to obtain multi-modal data fusion features; Use a three-dimensional dynamic construction method based on spatio-temporal-semantic joint embedding to perform three-dimensional reconstruction of the agricultural scene.

2. The method of claim 1, wherein, Extracting different modal data features from multi-modal data, specifically including: For image data collected by a binocular camera, use a dual-path weight-shared ResNet-50 backbone network for feature extraction, and simultaneously construct a 3D Cost Volume module for depth estimation, and cooperatively output semantic feature maps and disparity maps; For point cloud data collected by a laser radar, use a voxelized Transformer encoder for data processing to generate an aerial view feature expressing the geometric structure of the object; For time series inertia data collected by an inertial measurement unit, use a bidirectional gated recurrent unit network to extract latent motion features; For position data collected by a Beidou positioning system, compress it into a low-dimensional vector through a fully connected layer, and encode the geographical information context to obtain the fusion of Beidou positioning information.

3. The method of claim 2, wherein, For image data collected by binocular vision data, use a dual-path weight-shared ResNet-50 backbone network for feature extraction, specifically including: Use the ResNet-50 backbone network to extract features from the left and right images collected by the binocular camera to obtain left and right feature maps; Input the obtained left and right feature maps into the 3D Cost Volume module to match the left and right feature maps under multiple depth / disparity hypotheses, and aggregate and regress the matching cost results through 3D convolution to obtain a disparity map; Input the obtained left or right feature map into a semantic decoder including multiple transpose convolution layers to obtain a semantic feature map through upsampling and feature fusion of feature maps of different scales.

4. The method of claim 2, wherein, Perform heterogeneous fusion on different modal data features to obtain multi-modal data fusion features, specifically including: Use a cross-scale and cross-modal integration strategy to fuse the extracted semantic feature maps and disparity maps, aerial view features, latent motion features, and fusion of Beidou positioning information to obtain multi-modal data fusion features.

5. The method of claim 4, wherein, Use a cross-scale and cross-modal integration strategy to fuse the extracted semantic feature maps and disparity maps, aerial view features, latent motion features, and fusion of Beidou positioning information, specifically including: Introduce a deformable convolution mechanism to project the semantic feature maps to the laser radar's aerial view reference frame according to the depth clues, and fuse the projected semantic feature maps with the aerial view features at the geometric level to obtain image fusion features; Fuse the Beidou positioning information and the latent motion features to form a navigation state vector; The image fusion feature and the navigation state vector are fused at a high level by a Transformer module based on multi-head cross attention, the modal correlation is learned by adaptive weight, and a multi-modal data fusion feature supporting the three-dimensional reconstruction of the agricultural scene is output.

6. The method of claim 5, wherein, The projected semantic feature map and the aerial view feature are fused at a geometric level to obtain an image fusion feature, which specifically includes: The semantic feature map is weighted by a gate to obtain a dynamically weighted semantic feature: where F s represents the input semantic feature, σ represents a Sigmoid activation function, W g and b g are learnable parameters, and represents element-wise multiplication. represents the dynamically weighted semantic feature. The dynamically weighted semantic feature and the aerial view feature are input into a multi-head cross attention fusion module for fusion and splicing to obtain an image fusion feature.

7. The method of claim 6, wherein, The dynamically weighted semantic feature and the aerial view feature are input into a multi-head cross attention fusion module for fusion and splicing, which specifically includes the following formula: Output = Concat(head1,...,head h )W 0 wherein, is a head-specific projection matrix, F g denotes an aerial view feature, W 0 denotes a linear transformation, d k is a key vector dimension, head i denotes the output of the i-th attention head.

8. The method of claim 1, wherein, A three-dimensional dynamic construction method based on spatio-temporal-semantic joint embedding is used for three-dimensional reconstruction of the agricultural scene, which specifically includes: The multi-modal data fusion feature is encoded by a signed distance field implicit function to generate a neural implicit joint feature corresponding to the three-dimensional space, which includes geometric attribute component features, appearance attribute component features, semantic attribute component features, and temporal dynamic component features; The neural implicit joint feature is rendered into a two-dimensional image by a differentiable volume rendering, and the parameters of the neural implicit function are optimized based on the loss function between the rendering result and the real acquisition data; Based on the predicted signed distance field of the optimized neural implicit function, a continuous surface mesh model is extracted by a marching cubes algorithm; The semantic features queried in the process of extracting the surface mesh model are input into a semantic feature decoder to assign semantic labels to each face sheet in the surface mesh model. A dense three-dimensional mesh model with semantic labels is output.

9. The method for 3D reconstruction of agricultural scene based on multi-modal sensor fusion according to any one of claims 1-8, characterized in that, In the multi-modal data acquisition step of the target scene, it also includes: According to the modeling of the scattering noise distribution of the laser radar scanning physical characteristics, combined with the light intensity in the image collected by the binocular camera, a degradation perception mask is dynamically generated, and the missing area of the point cloud and the overexposed image are collaboratively repaired by a generative adversarial network.

Images