Multi-mode SLAM implementation method for intelligent garden supervision, software system and hardware device

Abstract

The invention discloses a multi-mode SLAM implementation method for intelligent garden supervision, a software system and a hardware device, and belongs to the technical field of intelligent garden supervision, synchronous positioning and map construction. According to the technical scheme, the multi-mode SLAM implementation method for intelligent garden supervision comprises the following steps: S1, sensor calibration: acquiring a camera internal reference matrix and distortion parameters by adopting a checkerboard calibration method; compensating motion distortion through a constant-speed model, cutting sub-frames, and calculating a transformation matrix from the radar to the IMU in combination with the output of a pure laser odometer and IMU data; generating a transformation matrix from the radar to the camera through feature extraction, matching and external parameter optimization; through multi-modal sensor fusion and space-time synchronization technologies and in combination with collaborative optimization of the laser-vision-inertial odometer, high-precision map construction with geometric structure and color information is realized in a complex garden scene, and the method has the advantages of improving positioning precision and map information integrity in a complex garden environment.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of intelligent garden supervision and simultaneous localization and mapping, and particularly relates to a multi-modal SLAM implementation method, software system and hardware device for intelligent garden supervision. BACKGROUND

[0002] With the advancement of smart garden construction, the demand for fine supervision of garden environment is increasingly urgent, and real-time positioning and three-dimensional map construction are the core basis for realizing intelligent inspection, vegetation monitoring, security warning and other functions. As a key technology for environment perception of autonomous mobile devices, simultaneous localization and mapping (SLAM) technology can construct a map and determine its own position in real time through sensor data in an unknown environment, and has become a research hotspot in the field of intelligent garden supervision.

[0003] Existing SLAM technology faces many challenges in the garden supervision scene. The visual SLAM system is prone to lose feature points under complex lighting conditions in the garden (such as strong light irradiation, tree shade blocking) or low-texture areas (such as lawns, water areas), resulting in positioning drift and map breakage. Although laser SLAM is not affected by light, the generated point cloud map lacks color and texture information, making it difficult to meet the fine supervision needs of garden vegetation species identification, pest detection and other functions. Although the inertial measurement unit (IMU) can provide high-frequency motion data, when used alone, the cumulative error problem is serious, and it cannot achieve stable navigation over long distances.

[0004] Current multi-sensor fusion schemes still have obvious deficiencies in garden scenes. In terms of sensor calibration, traditional methods rely on static scenes or manual operation, making it difficult to adapt to device vibration and temperature changes in the garden environment, resulting in external parameter drift problems. In terms of time synchronization, the sampling frequencies of different sensors (such as lidar, camera, IMU) differ greatly, and when the unmanned vehicle moves quickly, it will produce serious motion distortion. In terms of map construction, existing fusion schemes mostly focus on positioning accuracy optimization, and the generated map lacks the semantic information required by the garden scene, and cannot directly support vegetation health assessment and other advanced applications. In addition, the garden environment contains narrow passages blocked by trees, open lawns, water areas and other sub-scenes, and existing single fusion strategies cannot maintain stable performance in all scenes. SUMMARY

[0005] The application provides a multi-modal SLAM implementation method for intelligent garden supervision, and its software system and hardware device, to solve at least one of the above technical problems.

[0006] The technical scheme adopted by the application is: A multi-modal SLAM implementation method for intelligent garden supervision, characterized by comprising the following steps: S1, sensor calibration: acquire camera intrinsic matrix and distortion parameters by using checkerboard calibration method; compensate motion distortion by constant speed model and split sub-frames, calculate radar to IMU transformation matrix by combining pure laser odometry output and IMU data; generate radar to camera transformation matrix by feature extraction, matching and extrinsic parameter optimization; S2, timestamp unification: time initialization for IMU message, adjust timestamp to system reference when processing radar data, unify data timestamp of radar, camera and IMU; S3, fusion of SLAM and map construction: based on the intrinsic, distortion parameters and transformation matrix, construct map geometry by laser-inertial odometry, add RGB color information to the map by combining visual-inertial odometry, realize radar-vision-inertial fusion SLAM.

[0007] Further, the application further proposes that the camera calibration in step S1 includes: using a 7x10 specification checkerboard, shooting multiple images at a distance of 1.5-3 meters, performing sub-pixel level fine processing on the checkerboard corner points, and selecting a low deviation image group for re-calibration by three-dimensional point re-projection deviation comparison to obtain the final parameters.

[0008] Further, the application further proposes that the radar-IMU transformation matrix calculation in step S1 includes: rotating and translating the device in x, y and z directions, selecting an environment containing planar and edge features, and starting to be stationary for more than 30 seconds; inputting pure laser odometry output and IMU data into an initialization module, aligning by time sliding, optimizing time difference and extrinsic parameters, estimating device bias and gravity vector, and calculating the transformation matrix by tight coupling laser-inertial odometry, and the angular velocity unit is unified to .

[0009] Further, the application further proposes that the radar-camera transformation matrix generation in step S1 includes: in the feature extraction stage, converting point cloud to voxel grid (1 meter outdoors, 0.5 meter indoors), fitting planes and extracting intersecting edges by RANSAC; in the feature matching stage, mapping edge points to images by Euclidean transformation and projection correction, and matching by K-d tree nearest neighbor search; in the extrinsic parameter optimization stage, constructing constraint equations, and solving by weighted least squares after linearization.

[0010] Further, the application further proposes that the timestamp unification in step S2 includes: recording the ROS time and time offset of radar start, and ignoring the initial 100 frames of data; adjusting the timestamp to the system time reference when publishing the radar point cloud message, and realizing time synchronization of three sensor data.

[0011] Further, the application further proposes that the fusion SLAM in step S3 comprises: laser-inertial odometry registers radar scans to a global map and minimizes point-to-plane residuals; visual-inertial odometry minimizes PnP re-projection error and frame-to-map photometric error, and adds RGB color information to the geometric map.

[0012] A multi-modal SLAM software system for intelligent garden supervision, comprising: A sensor calibration module for performing the calibration operations described in steps S1 of claim 1 and claims 2-4; A timestamp synchronization module for performing the time synchronization operations described in step S2 of claim 1 and claim 5; A fusion and map construction module for performing the fusion and map construction operations described in step S3 of claim 1 and claim 6; The system is based on the Flask framework and adopts a non-separated front-end and back-end architecture.

[0013] Further, the application further proposes that the sensor calibration module comprises a camera calibration submodule, a radar-IMU calibration submodule, and a radar-camera calibration submodule, which respectively perform the operations described in claims 2, 3, and 4.

[0014] A multi-modal SLAM hardware device for intelligent garden supervision, comprising a sensor unit: an RGB camera, a solid-state laser radar, and an IMU; a computing unit: a memory, a processor, and a display, the memory stores the software system described in claims 7-8, and the processor executes the system to implement the method described in claims 1-6; A mobile device: a tracked unmanned trolley, for carrying the sensor unit and the computing unit to move and survey.

[0015] Further, the application further proposes that the computing unit is a computer, which is connected to the RGB camera, the solid-state laser radar, the IMU, and the tracked unmanned trolley through multiple interfaces to realize data transmission and control.

[0016] As the above technical solutions are adopted, the application has the following beneficial effects: 1. The present scheme eliminates data mismatch by unifying the time reference; existing maps are mostly single-modal data, and the present scheme generates a composite map through multi-modal fusion, enhancing practicality; existing fusion strategies are unstable in performance in diversified scenarios, and the present scheme improves scene adaptability through adaptive calibration and synchronization mechanisms.

[0017] The application solves the problems of insufficient sensor calibration accuracy, large time synchronization error, lack of semantic information in the map, and poor scene adaptability, realizes high-precision positioning and map construction of multi-modal SLAM in a complex garden environment, and generates a composite map that can support fine supervision needs such as vegetation health assessment and pest identification.

[0018] 2. The present scheme ensures accurate corner recognition under complex garden lighting conditions by increasing the size of the checkerboard and limiting the shooting distance range. Existing methods usually directly use all collected images for calibration without considering abnormal data introduced by factors such as leaf shaking and light changes in dynamic environments. The present scheme effectively excludes interference data through a re-projection error screening mechanism, improving the reliability of calibration parameters.

[0019] The present application can stably obtain high-precision camera parameters in a garden environment with vegetation obstruction and uneven lighting, solving the problem of parameter drift caused by environmental interference in traditional calibration methods, and providing accurate visual data basis for subsequent multi-modal SLAM fusion.

[0020] 3. The present scheme uses natural planes and edge features in the garden scene to complete calibration, adapting to the actual working environment. Existing technologies often ignore IMU zero offset compensation, while the present scheme accurately estimates device bias through a static initialization stage, improving the robustness of calibration in dynamic environments. The present application effectively solves the problem of positioning drift caused by multi-sensor space-time calibration error, can automatically complete high-precision external parameter calibration in a natural garden environment, and eliminates external parameter drift caused by device vibration and temperature changes, providing an accurate space-time reference for subsequent multi-modal data fusion. At the same time, this scheme avoids dependence on artificial calibration scenes and can directly complete the calibration process on the job site, improving the deployment efficiency of intelligent garden supervision systems.

[0021] 4. The present scheme can automatically extract stable structural features in complex natural scenes through voxel grid adaptive downsampling and RANSAC plane fitting, such as effectively distinguishing between tree trunk planes and randomly distributed leaf point clouds. At the same time, through projection correction and weighted optimization strategies, the interference of sensor view angle differences and dynamic objects on external parameter calibration can be eliminated, such as continuously correcting the external parameter matrix during the movement of an unmanned vehicle, avoiding external parameter drift caused by device vibration.

[0022] The present application solves the problem of insufficient radar and camera external parameter calibration accuracy in a dynamic garden environment, realizes efficient matching and robust optimization of cross-modal features, and provides an accurate spatial alignment basis for subsequent multi-sensor fusion SLAM, significantly reducing the fusion error of geometric maps and visual semantic information, such as the accurate correspondence of the position of a tree trunk in a point cloud map and its color information in an image, thereby supporting fine supervision functions such as vegetation morphology analysis and pest area labeling.

[0023] 5. The scheme effectively reduces the timing noise in the sensor cold start phase by actively discarding initialization data and combining dynamic time offset compensation. At the same time, through the continuous alignment of the global time reference, the problem of time-space mismatch caused by the difference in sampling frequency of multiple sensors is solved. The application can eliminate the time asynchronous error in the multi-sensor data fusion process, especially in the high-speed moving scene of unmanned vehicles, significantly reducing the motion distortion caused by timestamp deviation, providing accurate and synchronized multi-modal data input for subsequent laser-vision-inertial fusion algorithms, and thus improving the accuracy and stability of three-dimensional map construction and positioning.

[0024] 6. The scheme realizes joint optimization of geometric precision and color consistency by simultaneously constraining point-to-plane geometric residual, re-projection geometric residual and photometric residual, solving the problem of incomplete map information caused by single optimization target in traditional methods in complex garden environment. The application can accurately maintain the spatial position of geometric structures such as tree trunks and shrubs in dense vegetation areas, while eliminating visual positioning drift in leaf texture repetitive areas through color consistency constraints, generating map data containing accurate three-dimensional coordinates and real color information, providing a directly usable multi-modal environment model for subsequent vegetation species identification and canopy health assessment.

[0025] 7. The scheme reduces cross-platform communication overhead through the non-separation design of front-end and back-end of Flask framework. The existing sensor calibration module usually runs independently and lacks automation process, while the scheme integrates the calibration operation into a unified module, supporting online optimization of parameters in dynamic environment. The existing fusion algorithm mostly uses single residual optimization strategy, while the scheme realizes the simultaneous improvement of map geometric precision and semantic richness through joint optimization of geometric residual and photometric error. The application effectively solves the technical problems of insufficient calibration accuracy of multiple sensors, large time synchronization error and limited map practicality in garden scenes, realizes high-precision spatio-temporal alignment of multi-modal data fusion, and generates a map that can directly support vegetation color identification, morphology analysis and other fine regulation needs, while simplifying the system architecture to improve deployment efficiency and running stability.

[0026] 8. The scheme implements special optimization for optical imaging distortion, inertial data drift and cross-modal feature mismatch through three sub-modules designed separately, eliminating the influence of multi-source error on subsequent SLAM process in the calibration stage.

[0027] The application solves the problem of sensor calibration accuracy decline caused by device vibration and dynamic occlusion in garden environment, and realizes stable calibration of multi-modal sensors under complex lighting and vegetation interference conditions. The design of separate calibration module makes the parameter optimization process of each sensor independent, reducing the frequency of system restart caused by calibration failure, and improving the deployment efficiency of intelligent inspection equipment in diversified garden sub-scenes. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 A multi-modal SLAM implementation method flow chart for intelligent garden supervision is provided in the present application. Figure 2 A multi-modal SLAM software algorithm framework for intelligent garden supervision is provided in the present application. Figure 3 A multi-modal SLAM hardware device structure schematic diagram for intelligent garden supervision is provided in the present application.

[0029] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the principles of the application. In the drawings:

[0030] In the drawings: 101, computer; 102, RGB camera; 103, solid-state laser radar; 104, unmanned vehicle. DETAILED DESCRIPTION

[0031] In order to more clearly illustrate the overall concept of the present application, the following will be described in detail with reference to the accompanying drawings.

[0032] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. However, it will be apparent to one skilled in the art that the present application can be practiced without the specific details and, therefore, the scope of the present application is not limited to the specific embodiments disclosed.

[0033] In the present application, unless otherwise explicitly specified and limited, the first feature is "on" or "under" the second feature, which can be direct contact between the first and second features, or indirect contact between the first and second features through an intermediate medium. In the description of the present application, the description of the terms "embodiment", "example", "one embodiment", "exemplary" or "specific example" means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present application, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples.

[0034] Those skilled in the art can understand that, in the prior art, smart garden supervision needs real-time positioning and three-dimensional map construction to support intelligent inspection, vegetation monitoring and other functions, but the existing SLAM technology has significant limitations. Visual SLAM is prone to feature loss in complex lighting or low-texture areas, laser SLAM lacks semantic information, and the use of IMU alone has cumulative errors. Although multi-modal fusion solutions have developed, problems such as insufficient sensor calibration accuracy, large time synchronization error, limited map practicality, and poor scene adaptability still restrict their application in garden environments. For example, device vibration and temperature changes in the garden environment can cause sensor extrinsic parameter drift, and existing calibration methods rely on static scenes, making it difficult to adapt to dynamic changes; the differences in sampling frequencies of lidar, camera and IMU cause timestamp inconsistencies, leading to data mismatch; the generated map lacks semantic information, making it impossible to meet the needs of vegetation monitoring; and the diversity of sub-scenes makes single fusion strategies unstable in performance.

[0035] To solve the above problems, the inventors have conducted in-depth research on the problems of insufficient sensor calibration accuracy, large time synchronization error, limited map practicality, and poor scene adaptability in the prior art. First, considering that sensor extrinsic parameters are prone to drift in dynamic garden environments, existing static calibration methods cannot meet the needs, and a dynamic calibration method needs to be developed to improve calibration accuracy. Second, to address the data mismatch problem caused by the different timestamps of multiple sensors, a synchronization mechanism with a unified time reference needs to be designed. In addition, the lack of semantic information in existing maps prompts the inventors to explore multi-modal data fusion methods to enhance the practicality of the map. Finally, to cope with the challenges of diverse garden scenes, an adaptive fusion strategy needs to be constructed. By analyzing the characteristics of sensors and the needs of garden environments, the inventors propose a technical solution that combines dynamic calibration, time synchronization optimization, and multi-modal fusion, gradually forming a systematic method to solve the above problems.

[0036] Reference Figures 1 to 3 A multi-modal SLAM implementation method for intelligent garden supervision, characterized by the following steps: S1, sensor calibration: using a checkerboard calibration method to obtain the camera intrinsic matrix and distortion parameters; compensating for motion distortion through a constant speed model and dividing sub-frames, combining the pure laser odometry output with IMU data to calculate the transformation matrix from radar to IMU; generating the transformation matrix from radar to camera through feature extraction, matching and extrinsic parameter optimization; S2, timestamp unification: time initialization of IMU messages, adjusting the timestamp to the system reference when processing radar data, and unifying the data timestamps of radar, camera and IMU; S3, fusion SLAM and map construction: based on intrinsic, distortion parameters and transformation matrix, constructing the geometric structure of the map through laser-inertial odometry, adding RGB color information to the map through visual-inertial odometry, and realizing radar-vision-inertial fusion SLAM.

[0037] The sensor calibration refers to the process of establishing a unified coordinate system through multi-sensor parameter calibration, and can be specifically implemented by using a checkerboard calibration method combined with motion distortion compensation, and the calibration parameters are dynamically optimized to adapt to environmental changes to solve the problem of external parameter drift. The time stamp unification refers to aligning the time base of multi-source sensor data, and can be specifically implemented by initializing the IMU message time and adjusting the radar data timestamp to eliminate the space-time mismatch caused by the difference in sampling frequency. The fusion of SLAM and map construction refers to integrating multi-modal sensor data to generate a composite map, and can be specifically implemented by using laser-inertial odometry to construct geometric structures, and adding color information to visual-inertial odometry to realize the complementarity of geometric and semantic information.

[0038] Specifically, in the sensor calibration stage, the camera parameters are dynamically obtained by the checkerboard calibration method, and the motion distortion is compensated by the constant speed model to ensure the calibration accuracy in dynamic environment. Then, the spatial transformation relationship between radar and IMU is calculated by using laser odometry and IMU data, and the external parameter matrix of radar and camera is optimized by feature matching. In the time stamp unification stage, the radar data timestamp is adjusted to the system time axis by initializing the IMU time base, so as to realize the accurate synchronization of multi-sensor data. In the fusion SLAM stage, the geometric map generated by laser-inertial odometry and the color information extracted by visual-inertial odometry are fused to form a composite map with geometric accuracy and semantic features, which meets the needs of garden supervision for vegetation identification and environmental analysis.

[0039] Compared with the prior art, the existing calibration method relies on static scenes and cannot adapt to dynamic garden environment, while the present scheme improves the calibration accuracy by motion distortion compensation and dynamic external parameter optimization; the existing time synchronization method cannot handle the frequency difference of multiple sensors, and the present scheme eliminates data mismatch by unifying the time base; the existing map is mostly single modal data, and the present scheme generates a composite map by multi-modal fusion to enhance the practicality; the existing fusion strategy has unstable performance in diversified scenes, and the present scheme improves the scene adaptability by adaptive calibration and synchronization mechanism.

[0040] Through the above technical solutions, the present application solves the problems of insufficient sensor calibration accuracy, large time synchronization error, lack of semantic information in the map, and poor scene adaptability, and realizes high-precision positioning and map construction of multi-modal SLAM in complex garden environment. The generated composite map can support fine supervision needs such as vegetation health evaluation and pest identification.

[0041] The present application further proposes a camera calibration method, which includes using a 7x10 checkerboard, taking multiple images at a distance of 1.5-3 meters, performing sub-pixel level fine processing on the corner points of the checkerboard, comparing the deviation of three-dimensional point re-projection, and selecting a low-deviation image group for re-calibration to obtain the final parameters.

[0042] The 7x10 specification chessboard is a plane calibration board with 7 rows and 10 columns of black and white alternating squares, which can be specifically made by printing or laser etching. The size of the chessboard is increased to expand the calibration coverage range to meet the long-distance calibration needs of the garden scene. The sub-pixel level refinement refers to improving the accuracy of the initial detected corner coordinates to below the pixel level, which can be specifically implemented by using an iterative optimization algorithm based on gray gradient to eliminate the influence of image noise and optical distortion on corner positioning. The three-dimensional point re-projection deviation refers to mapping the three-dimensional space coordinates of the calibration board to the two-dimensional image plane through the camera projection model, calculating the position difference between the projected point and the actual detected corner, and specifically using the Euclidean distance as the deviation measurement index to filter out the effective calibration data less affected by environmental interference.

[0043] Specifically, in the garden scene, a 7x10 chessboard is fixed on a flat wall or ground, and the operator shoots multiple images at different orientations at a distance of 1.5 to 3 meters. During shooting, the chessboard is kept complete in the shot and has no obvious reflection or obstruction. The automatic corner detection algorithm is used to identify the intersection points of the chessboard and perform sub-pixel optimization. Then, a camera projection model is established to project the three-dimensional world coordinates of each corner to the two-dimensional image plane, and the re-projection error of each corner is calculated. When the average error of a single image exceeds the preset threshold, the image is automatically excluded and the data is re-collected. Finally, only the image group with the smallest error is retained for parameter optimization, and the stable intrinsic matrix and distortion coefficient are obtained through multiple iterations.

[0044] Compared with the prior art, the traditional chessboard calibration usually uses a 5x7 or 6x9 specification chessboard, which leads to a decrease in calibration accuracy due to insufficient corner resolution when shooting at a long distance. The present scheme increases the size of the chessboard and limits the shooting distance range to ensure accurate identification of the corners under complex lighting conditions in the garden. The existing method usually directly uses all collected images for calibration without considering abnormal data introduced by factors such as leaf shaking and light changes in dynamic environments. The present scheme effectively excludes interference data through the re-projection error screening mechanism to improve the reliability of the calibration parameters.

[0045] Through the above technical solutions, the present application can stably obtain high-precision camera parameters in a garden environment with vegetation obstruction and uneven lighting, solve the parameter drift problem caused by environmental interference in traditional calibration methods, and provide accurate visual data basis for subsequent multi-modal SLAM fusion.

[0046] The application further proposes a technical solution for radar-IMU transformation matrix calculation, including device rotation and translation operations in x, y, and z directions, selection of an environment containing planar and edge features, static state maintenance for more than 30 seconds after starting, pure laser odometry output and IMU data input initialization module, time sliding alignment, time difference and external parameter optimization processing, device bias and gravity vector estimation, tight coupling laser-inertial odometry calculation of transformation matrix, and angular velocity unit unification to rad / s.

[0047] The time sliding alignment refers to time axis matching of sensor data with different sampling frequencies. Specifically, a sliding window algorithm can be used to interpolate and align lidar and IMU data, solving the problem of time asynchronization caused by sampling rate difference. The tight coupling laser-inertial odometry refers to joint state estimation of lidar data and IMU data. Specifically, an extended Kalman filter can be used to synchronize and optimize sensor observation values and motion model prediction values, eliminating single sensor error accumulation. The device bias estimation refers to calculation of IMU zero bias error. Specifically, sensor data in a static state can be collected during a static initialization phase, and a noise model can be established for compensation. The gravity vector estimation refers to determination of the angle between the device coordinate system and the gravity direction. Specifically, the gravity projection component can be calculated from the IMU accelerometer data in a static state.

[0048] Specifically, after selecting a garden scene containing planar and edge features, the device is kept in a static state for more than 30 seconds to collect stable initial data. The point cloud features of the pure laser odometry output and the angular velocity and acceleration data of the IMU are input into the initialization module, the time axis of the two data is matched through the time sliding window, and the micro time difference caused by the asynchronous hardware clock is eliminated. Then, an optimization equation containing the external parameter matrix, device bias and gravity vector is constructed, and a nonlinear least squares method is used for joint optimization. In the tight coupling framework, the planar feature residual observed by the lidar and the IMU pre-integration result are simultaneously included in the state estimation, and finally the accurate radar-IMU spatial transformation matrix is output. The angular velocity unit is unified to the international standard radian system, avoiding calculation errors caused by unit system differences.

[0049] Compared with the prior art, the existing laser-inertial calibration mostly uses a loose coupling method to process sensor data separately, without considering the coupling effects of time asynchronization and motion distortion. The present solution realizes data space-time unification through time sliding alignment and tight coupling optimization. Traditional methods rely on artificial calibration boards or specific experimental environments, while the present solution uses natural planar and edge features in a garden scene to complete calibration, adapting to actual working environments. Existing technologies often ignore IMU zero bias compensation, while the present solution accurately estimates device bias through a static initialization phase, improving the robustness of calibration in dynamic environments.

[0050] By the technical solution, the positioning drift problem caused by the multi-sensor space-time calibration error is effectively solved, high-precision external parameter calibration can be automatically completed in the garden natural environment, the external parameter drift caused by equipment vibration and temperature change is eliminated, and an accurate space-time reference is provided for subsequent multi-modal data fusion. Meanwhile, the scheme avoids the dependence on manual calibration scene, and the calibration process can be directly completed on the work site, thereby improving the deployment efficiency of the intelligent garden supervision system.

[0051] The application further proposes that the radar-camera transformation matrix generation in step S1 includes: a feature extraction stage, which converts the point cloud into a voxel grid, and sets 1 meter outdoors and 0.5 meters indoors, fits a plane and extracts an intersection edge by RANSAC; a feature matching stage, which maps the edge points to the image by Euclidean transformation and projection correction, and matches through K-d tree nearest neighbor search; and an external parameter optimization stage, which constructs a constraint equation, and solves by weighted least squares after linearization.

[0052] The voxel grid refers to dividing the three-dimensional point cloud data into regular cubic units, and specifically, different grid sizes can be set to realize adaptive downsampling of outdoor and indoor scenes, for example, 1-meter grid is used outdoors to reduce redundant point cloud in sparse vegetation areas, and 0.5-meter grid is used indoors to retain structural details of narrow passages, thereby balancing the calculation efficiency and feature retention requirements. The RANSAC plane fitting refers to detecting the plane structure in the point cloud by the random sample consensus algorithm, and specifically, an iterative screening of inliers and fitting of the optimal plane can be used to achieve this, for example, candidate plane regions are extracted from the voxel grid, the main plane direction is determined by calculating the point cloud normal vector distribution, thereby eliminating the interference of non-structural noise such as leaves and shrubs in the garden environment. The Euclidean transformation and projection correction refer to converting the edge points in the radar coordinate system to the camera pixel coordinate system, and specifically, joint operation of the rigid transformation matrix and the camera intrinsic matrix can be used to achieve this, for example, after projecting the three-dimensional edge points to the two-dimensional image plane, the geometric deformation caused by the angle difference is corrected by affine transformation, ensuring the spatial consistency of cross-modal features. The K-d tree nearest neighbor search refers to quickly finding matching feature points in high-dimensional space, and specifically, a binary tree structure can be used to divide the image feature space to achieve this, for example, a search area is established around the projected edge points, and the best matching pair is selected by distance threshold, thereby improving the real-time performance of cross-sensor feature matching. The weighted least squares solution refers to optimizing the external parameter matrix by assigning different matching pairs with different weight coefficients, and specifically, the residual equation can be constructed and the parameters can be iteratively updated to achieve this, for example, high-confidence matching pairs are assigned with larger weights, and the influence of the optimization process caused by the mis-matching pairs due to dynamic objects or occlusions is suppressed.

[0053] Specifically, in the feature extraction stage, the point cloud data is first divided into voxel grids to reduce the data density, for example, a 1-meter grid size is used in outdoor scenes to filter the noise of distant point clouds, and a 0.5-meter grid is used in indoor scenes to retain fine structures such as walls and flower stands. Then, the RANSAC algorithm is used to extract planar regions from each voxel, and the intersection of adjacent planes is calculated as an edge feature, for example, in a garden scene, the intersection of the tree trunk surface plane and the ground plane is identified as a stable edge. In the feature matching stage, the radar edge points are converted to the camera coordinate system through Euclidean transformation, and the projection correction is used to eliminate the geometric deviation caused by the difference in sensor viewing angle, for example, the tree trunk edge points are projected onto the corresponding tree trunk contour area in the image. Then, the K-d tree is used to search for the nearest neighbors of the image edge features, and the correspondence between the cross-modal features is quickly established. In the extrinsic parameter optimization stage, a spatial constraint equation is constructed based on the matched features, and the rotation and translation parameters of the radar to the camera are iteratively optimized by the weighted least squares algorithm, for example, higher weights are given to the matching pairs of static edges such as tree trunks and fences, while lower weights are given to the matching pairs of wind-swaying branches, thereby improving the robustness of the extrinsic parameter calibration result.

[0054] Compared with the prior art, the traditional radar-camera calibration method relies on manual calibration boards or feature extraction in specific static scenes, for example, a chessboard calibration board needs to be arranged in the laboratory for offline calibration, which is difficult to adapt to the dynamic interference of vegetation shading and light changes in the garden environment. The present scheme can automatically extract stable structural features in complex natural scenes through adaptive downsampling of voxel grids and RANSAC plane fitting, for example, effectively distinguishing between tree trunk planes and randomly distributed leaf point clouds. At the same time, through projection correction and weighted optimization strategies, the interference of sensor viewing angle differences and dynamic objects on extrinsic parameter calibration can be eliminated, for example, the extrinsic parameter matrix is continuously corrected during the movement of the unmanned vehicle, avoiding the problem of extrinsic parameter drift caused by device vibration.

[0055] Through the above technical solutions, the present application solves the problem of insufficient accuracy of radar and camera extrinsic parameter calibration in a garden dynamic environment, realizes efficient matching and robust optimization of cross-modal features, provides an accurate spatial alignment basis for subsequent multi-sensor fusion SLAM, and significantly reduces the fusion error of geometric maps and visual semantic information, for example, the position of a tree trunk in the point cloud map can be accurately matched with its color information in the image, thereby supporting fine supervision functions such as vegetation morphology analysis and pest and disease area labeling.

[0056] The present application further proposes a timestamp unification including recording the ROS time when the radar starts and the time offset, ignoring the initial 100 frames of data; adjusting the timestamp to the system time reference when publishing the radar point cloud message, realizing time synchronization of the three sensor data.

[0057] ROS time is the system time reference corresponding to the radar start time, which can be recorded by the built-in timestamp function of the ROS operating system to establish a global unified time coordinate system. The time offset is the difference between the internal clock of the radar device and the system time reference, which can be determined by calculating the difference between the ROS time at the radar start time and the timestamp of the first data packet generated by the radar, and is used for subsequent time correction of the data. The initial 100 frames of data are the first 100 frames of point cloud data collected after the radar starts, which can be selectively discarded by setting a counter threshold to avoid timing errors caused by unstable sensor output during device initialization.

[0058] Specifically, the ROS system time is recorded as a global time reference at the radar start time, and the first data packet timestamp generated by the radar itself is obtained at the same time. The time offset is obtained by calculating the difference between the two, and the offset is continuously applied to dynamically compensate the radar point cloud timestamp in subsequent data processing. During data acquisition, the first 100 frames of data are skipped by setting a counter to eliminate abnormal time fluctuations during sensor initialization. When publishing the point cloud message, the compensated timestamp is aligned with the camera exposure time and the IMU sampling time to the ROS time reference to ensure the consistency of multi-source data in the time and space dimensions.

[0059] Compared with the prior art, the existing method usually only uses a fixed offset for timestamp compensation, does not consider the timing jitter problem during sensor startup, and does not establish a continuous time offset tracking mechanism in a dynamic environment. The present scheme actively discards the initialization data and combines dynamic time offset compensation to effectively reduce the timing noise during sensor cold start. At the same time, through the continuous alignment of the global time reference, the problem of time and space mismatch caused by the difference in sampling frequency of multiple sensors is solved.

[0060] Through the above technical scheme, the present application can eliminate the time asynchronous error in the multi-sensor data fusion process, especially in the high-speed moving scene of unmanned vehicles, significantly reducing the motion distortion caused by timestamp deviation, providing accurate and synchronized multi-modal data input for subsequent laser-vision-inertial fusion algorithms, and further improving the accuracy and stability of three-dimensional map construction and positioning.

[0061] The present application further proposes that the fusion SLAM in step S3 includes laser-inertial odometry to register radar scans to the global map and minimize point-to-plane residuals; visual-inertial odometry minimizes PnP reprojection error and frame-to-map photometric error to add RGB color information to the geometric map.

[0062] The point-to-plane residual refers to calculating the vertical distance between the radar scanning point and the nearest neighbor plane in the global map as part of the optimization objective function, which can be implemented using the plane matching strategy in the iterative closest point algorithm, and is used to optimize the pose estimation of the lidar and reduce the geometric structure registration error. The PnP reprojection error refers to calculating the position difference between the three-dimensional map point projected onto the current image plane and the detected two-dimensional feature point, which can be implemented using the bundle adjustment based on nonlinear optimization, and is used to optimize the camera pose and improve the accuracy of visual positioning. The frame-to-map photometric error refers to comparing the brightness difference between the current image frame and the stored key frame in the map at the corresponding pixel of the same three-dimensional point, which can be implemented using the photometric consistency constraint in the direct method visual odometry, and is used to optimize the color information of the three-dimensional point in the map and enhance the color consistency.

[0063] Specifically, the laser-inertial odometry registers each frame of radar scan data with the plane features in the map when constructing the global map, optimizes the current radar pose by minimizing the point-to-plane residual, and fuses the high-frequency motion data of the IMU to suppress the cumulative error. The visual-inertial odometry first calculates the current camera pose and the projection relationship of the three-dimensional point in the map by the PnP algorithm during runtime, minimizes the feature point reprojection error, and optimizes the RGB color value of the three-dimensional point in the map by pixel-by-pixel comparison of the photometric difference between the current frame and the key frame. The outputs of the two odometry are fused in a tightly coupled manner, so that the geometric map retains high-precision structure information and superimposes color information with spatial consistency to form a semantic map suitable for garden vegetation analysis. Figure Three

[0064] Compared with the prior art, the existing fusion SLAM scheme usually only uses a single residual term for optimization, such as using only geometric residual or photometric residual, resulting in a lack of effective fusion of multi-modal information in the map. The present scheme simultaneously constrains the point-to-plane geometric residual, the reprojection geometric residual, and the photometric residual to achieve joint optimization of geometric accuracy and color consistency, and solves the problem of incomplete map information caused by single optimization target in traditional methods in complex garden environments.

[0065] Through the above technical solutions, the present application can accurately maintain the spatial positions of geometric structures such as tree trunks and shrubs in dense vegetation areas, and eliminate visual positioning drift through color consistency constraints in leaf texture repetitive areas, generating map data containing accurate three-dimensional coordinates and real color information, which provides a directly usable multi-modal environment model for subsequent vegetation species identification and canopy health assessment.

[0066] ​The application further provides a multi-modal SLAM software system for intelligent garden supervision, comprising a sensor calibration module, a timestamp synchronization module, and a fusion and map construction module, wherein the sensor calibration module is configured to perform operations of acquiring a camera intrinsic matrix and distortion parameters through a checkerboard calibration method, compensating motion distortion through a constant speed model, and dividing sub-frames, and calculating a transformation matrix of a radar to an IMU in combination with pure laser odometry output and IMU data, and operations of generating a transformation matrix of a radar to a camera through feature extraction, matching, and external parameter optimization; the timestamp synchronization module is configured to perform operations of time initialization on IMU messages, and adjusting timestamps to a system reference when processing radar data; and the fusion and map construction module is configured to perform operations of constructing a map geometry through a laser-inertial odometry, and adding RGB color information to the map in combination with a visual-inertial odometry. The system is based on a Flask framework and adopts a front-end and back-end non-separated architecture.

[0067] The sensor calibration module refers to accurate calibration of spatial relationships of multiple sensors through a multi-step calibration process, which can be implemented by, for example, checkerboard image acquisition, point cloud feature extraction and matching, and external parameter optimization algorithm, and is configured to solve the problem of fusion accuracy decline caused by sensor external parameter drift in a garden environment. The timestamp synchronization module refers to time sequence alignment through adjustment of time references of sensor data, which can be implemented by, for example, ROS time offset compensation and initial frame filtering mechanism, and is configured to eliminate the spatio-temporal mismatch caused by differences in sampling frequencies of multiple sensors. The fusion and map construction module refers to generating a map with both geometric and semantic information through fusion of laser, visual, and inertial data, which can be implemented by, for example, point-to-plane residual optimization and reprojection error minimization algorithm, and is configured to improve the practicability of the map in scenarios such as vegetation identification and security warning. The Flask framework refers to a lightweight Web application development framework, which can be implemented by, for example, template rendering and route management functions to realize a front-end and back-end non-separated architecture, and is configured to reduce system complexity and improve data processing efficiency.

[0068] Specifically, the sensor calibration module sequentially performs camera intrinsic calibration, radar-IMU external parameter calibration, and radar-camera external parameter calibration during runtime, for example, through sub-pixel level corner detection of checkerboard images, or fitting of point cloud plane features through RANSAC algorithm. The timestamp synchronization module performs offset compensation on radar start time during data preprocessing, for example, ignoring the initial 100 frames of data to eliminate the influence of device start-up jitter. The fusion and map construction module synchronously processes laser scanning data and visual images during the SLAM process, for example, after registering laser point clouds to a global map, matching image edge features through K-d tree nearest neighbor search to add color information. The system integrates data processing logic and user interface in a single service based on the Flask framework, for example, directly rendering map data to the front-end page through a template engine.

[0069] Compared with the prior art, the existing multi-modal SLAM system usually adopts a front-end and back-end separation architecture, needs to maintain an API interface and a data transmission protocol additionally, and the scheme reduces the cross-platform communication overhead through the front-end and back-end separation design of the Flask framework. The sensor calibration module in the prior art usually runs independently and lacks an automatic process, and the scheme integrates the calibration operation into a unified module and supports online optimization of parameters in a dynamic environment. The existing fusion algorithm usually adopts a single residual optimization strategy, and the scheme realizes synchronous improvement of the geometric precision and semantic richness of a map by jointly optimizing geometric residuals and photometric errors.

[0070] Through the technical scheme, the technical problems of insufficient multi-sensor calibration precision, large time synchronization error and limited map practicability in a garden scene are effectively solved, multi-modal data fusion with high-precision space-time alignment is realized, and the generated map can directly support fine supervision needs such as vegetation color recognition and morphology analysis. Meanwhile, the deployment efficiency and running stability are improved through simplifying the system architecture.

[0071] The scheme further proposes that the sensor calibration module comprises a camera calibration sub-module, a radar-IMU calibration sub-module and a radar-camera calibration sub-module, which respectively perform camera parameter calibration based on a checkerboard, radar-IMU transformation matrix calculation based on static initialization and tight coupling optimization, and radar-camera extrinsic parameter generation operation based on a voxel grid and feature matching.

[0072] The camera calibration sub-module is a device calibration unit through checkerboard image acquisition and parameter optimization, and can be implemented by a process of multi-distance image acquisition, sub-pixel corner point detection and three-dimensional re-projection deviation screening, and the robustness of the calibration parameters under complex lighting conditions is improved by dynamically screening low-deviation image groups. The radar-IMU calibration sub-module is a space alignment unit that fuses laser odometry and inertial data, and can be implemented by a combination method of acquiring planar feature data in a static initialization stage, aligning sensor data streams in a time sliding window and tightly coupling and optimizing extrinsic parameters and bias parameters, and the influence of motion noise on transformation matrix estimation is reduced by using stable observation data in the device static stage. The radar-camera calibration sub-module is a coordinate conversion unit for cross-modal feature matching and optimization, and can be implemented by an iterative optimization process of voxel down-sampling point cloud data, RANSAC plane segmentation to extract geometric features and image edge projection matching, and the cross-sensor data fusion precision is improved by establishing the correspondence between geometric features and visual features.

[0073] Specifically, when the camera calibration sub-module is deployed in a garden scene, an operator can arrange a checkerboard calibration board in a vegetation-covered area, and cover multi-view observation data in the range of 1.5 meters to 3 meters by adjusting the shooting distance. After the system automatically performs corner sub-pixel optimization, the image distortion samples caused by branch and leaf shaking are excluded by using a re-projection error threshold screening mechanism, and the final calibration parameters are ensured to adapt to the dynamic environment. When the radar-IMU calibration sub-module is started, the device remains stationary in the feature area containing the trunk plane and the path edge, and the system constructs an initial observation model by continuously collecting static data for more than 30 seconds, and then synchronously corrects the time offset and spatial transformation parameters in the tight coupling optimization process, eliminating the coordinate drift caused by device vibration. When the radar-camera calibration sub-module is working, the point cloud data is down-sampled by an outdoor 1-meter voxel grid, the trunk plane and branch edge features are extracted, the three-dimensional edge points are mapped to the image plane through projection transformation, the K-d tree is used to accelerate the matching process and construct a weighted least squares constraint equation, and finally a spatial transformation matrix that adapts to the vegetation form is generated.

[0074] Compared with the prior art, the existing calibration method usually uses a single calibration process to handle multi-sensor parameters, and does not design independent optimization modules for different sensor characteristics, which is prone to calibration failure in the garden dynamic environment due to branch and leaf shaking and device vibration interference. The scheme solves the problem of sensor calibration precision decline caused by device vibration and dynamic occlusion in the garden environment, and realizes stable calibration of multi-modal sensors under complex lighting and vegetation interference conditions. The design of the separate calibration module makes the sensor parameter optimization process independent of each other, reduces the frequency of system restart caused by calibration failure, and improves the deployment efficiency of intelligent inspection equipment in diversified garden sub-scenes.

[0075] Through the above technical solutions, the application solves the problem of sensor calibration precision decline caused by device vibration and dynamic occlusion in the garden environment, and realizes stable calibration of multi-modal sensors under complex lighting and vegetation interference conditions. The design of the separate calibration module makes the sensor parameter optimization process independent of each other, reduces the frequency of system restart caused by calibration failure, and improves the deployment efficiency of intelligent inspection equipment in diversified garden sub-scenes.

[0076] The application further proposes a multi-modal SLAM hardware device for intelligent garden supervision, which includes a sensor unit, a computing unit and a mobile device. The sensor unit includes an RGB camera 102, a solid-state laser radar 103 and an IMU; the computing unit includes a memory, a processor and a display, the memory stores a multi-modal SLAM software system, and the processor executes the system to realize a multi-modal SLAM method; the mobile device adopts a tracked unmanned vehicle 104, which is used to carry the sensor unit and the computing unit for movement surveying.

[0077] Wherein, the RGB camera 102 refers to an imaging device with red, green and blue three-channel color perception capability, which can be specifically implemented by a global shutter CMOS sensor, and is used to capture environmental texture and color information. The solid-state laser radar 103 refers to a laser ranging device using solid-state scanning technology, which can be specifically implemented by MEMS micro-mirror or optical phased array technology, and is used to obtain high-precision three-dimensional point cloud data. The IMU refers to an inertial measurement unit containing a three-axis accelerometer and a three-axis gyroscope, which can be specifically implemented by a MEMS sensor, and is used to collect device angular velocity and linear acceleration. The computing unit refers to an embedded computer with data storage and processing capability, which can be specifically implemented by an industrial-grade industrial computer, and realizes sensor data fusion and map construction through multi-thread parallel processing. The tracked unmanned vehicle 104 refers to a mobile platform driven by rubber tracks, which can be specifically implemented by a low-center-of-gravity wide-track structure, and is used to move stably and carry sensors and computing devices in complex garden terrain.

[0078] Specifically, the RGB camera 102, the solid-state laser radar 103 and the IMU in the sensor unit are fixed on the top of the mobile device through rigid connection, ensuring stable spatial positional relationship. The computing unit is connected with the camera through a USB 3.0 interface, communicates with the laser radar through an Ethernet interface, and receives IMU data through a serial port. During the travel of the tracked unmanned vehicle 104, the sensor unit continuously collects environmental data, and the computing unit synchronously executes sensor calibration, time synchronization and fusion SLAM algorithm to generate a semantic map containing geometric structure and color information. When encountering tree roots or sloping terrain, the tracked chassis keeps stable movement through differential steering and high-torque output, avoiding data distortion caused by sensor vibration.

[0079] In some specific embodiments, the solid-state laser radar 103 can be selected to have a horizontal field of view angle of 120 degrees, and is installed at a height of 1.2 meters from the ground, and is aligned with the camera optical axis in parallel. The computing unit can be equipped with a cooling fan and a waterproof shell to adapt to temperature and humidity changes in the garden environment. The drive motor of the tracked unmanned vehicle 104 can be configured with an encoder to feedback the travel speed in real time for motion distortion compensation.

[0080] Compared with the prior art, the traditional garden surveying and mapping equipment mostly uses a wheeled robot to carry a single-line laser radar, which is prone to slipping or data loss in densely vegetated areas. The present scheme enhances the terrain adaptability through the tracked mobile device, and expands the horizontal scanning range by combining the solid-state laser radar 103, effectively capturing the three-dimensional structure of tree trunks and low shrubs. Compared with the split sensor arrangement scheme, the present device reduces the external parameter calibration error and environmental interference through rigid connection and unified power supply design.

[0081] By the technical solution, the application solves the problem of sensor data distortion caused by poor equipment mobility in complex garden terrain, realizes high-precision semantic map construction through synchronous acquisition of multi-modal sensors and real-time processing of embedded computing units.

[0082] The application further proposes a multi-modal SLAM hardware device for intelligent garden monitoring, wherein the computing unit is a computer 101, which is connected with an RGB camera 102, a solid-state laser radar 103, an IMU and a tracked unmanned vehicle 104 through multiple interfaces to realize data transmission and control.

[0083] The computing unit refers to an electronic device with data processing capability, which can be implemented by an industrial control computer or an embedded computer, and is used for running a multi-modal SLAM algorithm and coordinating sensor data streams. The multiple interfaces refer to data transmission channels with different physical specifications, which can be implemented by communication interfaces such as USB 3.0, Ethernet, CAN bus, etc. For example, the USB interface is used to connect the RGB camera 102, the Ethernet interface is used to connect the solid-state laser radar 103, and the CAN bus is used to connect the control unit of the tracked unmanned vehicle 104, so as to realize matching of data transmission rates of each device through differentiated interface design. The data transmission and control refer to information interaction and instruction transmission between devices, which can be realized by establishing a topic subscription mechanism through a ROS communication framework. For example, the computer 101 subscribes to the point cloud topic of the laser radar and the inertial measurement topic of the IMU to realize time alignment and fusion processing of multi-source data.

[0084] Specifically, the computing unit, as the core of the hardware device, establishes point-to-point connection with each sensor through physical interfaces. In the running process, the visual data collected by the RGB camera 102 is transmitted to the computing unit through a high-speed interface, the point cloud data of the solid-state laser radar 103 is uploaded in real time through a special communication interface, and the inertial measurement data of the IMU is transmitted synchronously through a serial interface. The time synchronization module built in the computing unit calibrates the hardware-level time stamp of the data transmitted through each interface, eliminating the transmission delay caused by the difference in interface types. For the motion control of the tracked unmanned vehicle 104, the computing unit sends speed instructions through the control interface and receives the pose feedback data of the vehicle at the same time, forming a closed-loop control. This multi-interface independent transmission architecture can avoid the bandwidth bottleneck caused by bus competition and ensure the parallel processing of high-frequency IMU data and low-frequency laser radar data.

[0085] Compared with the prior art, the conventional multi-modal SLAM device often uses a single interface to concentrate the transmission of multi-sensor data, which is prone to cause data congestion and time synchronization error. The scheme allocates a dedicated interface for each sensor, realizes data channel isolation at the hardware level, and effectively reduces the mutual interference of multi-modal data transmission. At the same time, according to the vibration characteristics of the tracked mobile platform in the complex terrain of the garden, the industrial interface connection mode with strong anti-interference ability is adopted, which improves the data transmission reliability compared with the ordinary civil interface.

[0086] Through the above technical scheme, the present application solves the timing consistency problem of real-time transmission of multi-sensor data in the intelligent garden supervision scene, and guarantees the time accurate alignment of the laser radar point cloud, the visual image and the inertial measurement data. The multi-interface independent transmission mechanism effectively avoids the data packet loss phenomenon, provides complete and synchronous input data for the subsequent fusion SLAM algorithm, and thus improves the precision of three-dimensional map construction and the stability of system operation.

[0087] The places not mentioned in the present application can be realized by adopting or referring to the existing technology.

[0088] Each embodiment in the specification is described in a progressive manner, and the same or similar parts between each embodiment can be referred to each other, and each embodiment mainly describes the difference from other embodiments.

[0089] The above is only an embodiment of the present application and is not used to limit the present application. For those skilled in the art, the present application can have various changes and variations. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application shall be included in the scope of the claims of the present application.

Claims

1. A multi-modal SLAM implementation method for smart garden supervision, characterized in that, The method comprises the following steps: S1, sensor calibration: acquire camera intrinsic matrix and distortion parameters by using checkerboard calibration method; compensate motion distortion by constant speed model and split subframes, combine pure laser odometer output and IMU data to calculate radar to IMU transformation matrix; generate radar to camera transformation matrix by feature extraction, matching and external parameter optimization; S2, timestamp unification: time initialization for IMU message, adjust timestamp to system reference when processing radar data, unify data timestamp of radar, camera and IMU; S3, fusion SLAM and map construction: based on the intrinsic, distortion parameters and transformation matrix, construct map geometry by laser-inertial odometer, add RGB color information to the map by visual-inertial odometer, realize radar-vision-inertial fusion SLAM.

2. The multi-modal SLAM implementation method for smart garden monitoring according to claim 1, wherein, Camera calibration in step S1 includes: using 7*10 specification checkerboard, shooting multiple images at a distance of 1.5-3 meters, performing sub-pixel level fine processing on the checkerboard corner points, comparing the deviation by three-dimensional point re-projection, screening low deviation image group for re-calibration to obtain final parameters.

3. The multi-modal SLAM implementation method for smart garden monitoring according to claim 1, wherein, The radar-IMU transformation matrix calculation in step S1 includes: device rotation and translation in x, y, z directions, selection of an environment containing a plane and edge features, and starting to be stationary for more than 30 seconds; inputting the pure laser odometer output and the IMU data into an initialization module, performing time sliding alignment, time difference and external parameter optimization, estimating the device bias and the gravity vector, and calculating the transformation matrix through a tightly coupled laser-inertial odometer, and the angular velocity unit is unified to .

4. The multi-modal SLAM implementation method for smart garden monitoring according to claim 1, wherein, Radar-camera transformation matrix generation in step S1 includes: in the feature extraction stage, converting point cloud to voxel grid (1 meter outdoors, 0.5 meter indoors), fitting plane by RANSAC and extracting intersecting edges; in the feature matching stage, mapping the edge points to the image by Euclidean transformation and projection correction, and matching by K-d tree nearest neighbor search; in the external parameter optimization stage, constructing constraint equations, and solving by weighted least squares after linearization.

5. The multi-modal SLAM implementation method for smart garden monitoring according to claim 1, wherein, Timestamp unification in step S2 includes: recording the ROS time and time offset of radar start, ignoring the initial 100 frames of data; adjusting the timestamp to the system time reference when publishing radar point cloud message, realizing time synchronization of three sensor data.

6. The multi-modal SLAM implementation method for smart garden monitoring according to claim 1, wherein, The method is characterized in that, Fusion SLAM in step S3 includes: laser-inertial odometer registers radar scanning to the global map and minimizes point-to-plane residual error; visual-inertial odometer minimizes PnP re-projection error and frame-to-map photometric error, and adds RGB color information to the geometric map.

7. A multi-modal SLAM software system for smart garden supervision, characterized in that, The method comprises: A sensor calibration module for performing the calibration operations described in steps S1 of claim 1 and claims 2-4; A timestamp synchronization module for performing the time synchronization operations described in step S2 of claim 1 and claim 5; A fusion and map construction module for performing the fusion and map construction operations described in step S3 of claim 1 and claim 6; The system is based on Flask framework and adopts a non-separated front-end and back-end architecture.

8. The smart garden oriented surveillance multi-modal SLAM software system according to claim 7, wherein, The sensor calibration module includes a camera calibration sub-module, a radar-IMU calibration sub-module and a radar-camera calibration sub-module, which correspondingly perform the operations described in claims 2, 3 and 4. 9.A multi-modal SLAM hardware device for smart garden supervision, characterized in that, A sensor unit: RGB camera (102), solid-state laser radar (103), IMU; a computing unit: memory, processor, display, the memory stores the software system described in claims 7-8, and the processor executes the system to realize the method described in claims 1-6; Mobile device: a tracked unmanned vehicle (104) for carrying a sensor unit and a computing unit for mobile mapping.

10. The smart garden oriented surveillance multi-modal SLAM hardware apparatus according to claim 9, wherein, The computing unit is a computer (101) connected with an RGB camera (102), a solid-state laser radar (103), an IMU and the tracked unmanned vehicle (104) through multiple interfaces for data transmission and control.

Images