Method and apparatus for monitoring growth of field crops
By collecting multi-source sensor data through inspection robots and performing timestamp interpolation calibration and Kalman filtering, a full-domain 3D map is generated, which solves the data fragmentation problem in the monitoring of open-field vegetable growth and realizes real-time accurate monitoring and refined management of open-field crop growth parameters.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING RES CENT FOR INFORMATION TECH & AGRI
- Filing Date
- 2025-09-23
- Publication Date
- 2026-06-02
AI Technical Summary
In the monitoring of open-field vegetable growth, there are problems such as insufficient data coverage from a single sensor, limited information dimensions, and spatiotemporal fragmentation of data from multiple sensors, making it difficult to achieve real-time and accurate monitoring.
By collecting multi-source sensor data through inspection robots, and using timestamp interpolation calibration and extended Kalman filtering algorithms for time and space alignment, a full-domain 3D map is generated. Semantic segmentation is then performed to identify the core location and growth parameters of the material.
It enables real-time and accurate monitoring of growth parameters of open-field crops, adapts to the complexity of the open-field environment, provides comprehensive and accurate growth monitoring data support, and supports refined management.
Smart Images
Figure CN121415086B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of precision agriculture technology, and in particular to a method and device for monitoring the growth of open-field crops. Background Technology
[0002] Monitoring the growth of open-field vegetables is a core component of precision agriculture, enabling refined management. Its core requirement is the real-time and accurate acquisition of crop physiological status, growth environment parameters, and spatial distribution information to provide data support for agricultural decision-making. However, the open-field environment is characterized by its vast area, complex terrain, densely interwoven crops, variable lighting, and the susceptibility of mobile monitoring equipment to motion distortion. This leads to problems such as insufficient data coverage and limited information dimensions with single sensors, and spatiotemporal data fragmentation and insufficient feature fusion with multi-source sensors, making it difficult to meet the core requirement of real-time and accurate monitoring of open-field vegetable growth parameters. Summary of the Invention
[0003] This invention provides a method and apparatus for monitoring the growth of open-field crops, which solves the problem that the growth parameters of open-field vegetables cannot be monitored in real time and accurately.
[0004] This invention provides a method for monitoring the growth of open-field crops, comprising: collecting multi-source sensor data of open-field crops using an inspection robot, wherein the multi-source sensor data includes depth image data, position data, and inertial measurement unit (IMU) data; performing time synchronization processing on the multi-source sensor data using a timestamp interpolation calibration method; performing spatial alignment processing on the depth image data and the position data using the IMU coordinate system as a reference; and obtaining a spatiotemporal alignment state vector by correcting the dynamic errors of the multi-source sensors based on an extended Kalman filter algorithm; performing densification processing on the spatiotemporal alignment state vector to generate a global 3D map covering the entire open-field crop area and including the three-dimensional morphology and spatial position of the open-field crops; and performing semantic segmentation on the global 3D map to identify the centroid position of each crop and determine the growth parameters of each crop.
[0005] This invention also provides an open-field crop growth monitoring device, comprising the following modules: an acquisition module and a processing module; the acquisition module is used to collect multi-source sensor data of open-field crops through an inspection robot, the multi-source sensor data including depth image data, position data, and inertial measurement unit (IMU) data; the processing module is used to perform time synchronization processing on the multi-source sensor data using a timestamp interpolation calibration method, perform spatial alignment processing on the depth image data and the position data based on the IMU coordinate system, and obtain a spatiotemporal alignment state vector by correcting the dynamic error of the multi-source sensors based on the extended Kalman filter algorithm; the spatiotemporal alignment state vector is densified to generate a global 3D map covering the entire open-field crop area and including the three-dimensional morphology and spatial position of the open-field crops; the global 3D map is semantically segmented to identify the centroid position of each crop and determine the growth parameters of each crop.
[0006] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the open-field crop growth monitoring method as described above.
[0007] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the open-field crop growth monitoring method as described above.
[0008] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the open-field crop growth monitoring method as described above.
[0009] The open-field crop growth monitoring method and device provided by this invention utilizes multi-source sensor data, consisting of depth image data, location data, and IMU data collected by an inspection robot. Combined with timestamp interpolation calibration and spatial alignment processing based on the IMU coordinate system, this effectively solves the problem of spatiotemporal data fragmentation from multiple sensors, achieving precise matching of data from different sources in both time and space dimensions. By correcting the dynamic errors of the multi-source sensors using an extended Kalman filter algorithm and obtaining a spatiotemporal alignment state vector, it can specifically improve data deviations caused by motion distortion of mobile monitoring equipment, enhancing data reliability. By densifying the spatiotemporal alignment state vector to generate a full-domain 3D map, it overcomes the limitations of insufficient coverage and single-dimensional information from a single sensor, fully presenting the 3D morphology and spatial distribution of open-field crops, adapting to the characteristics of wide open-field environments with complex terrain and densely interwoven crops. Semantic segmentation of the full-domain 3D map and extraction of the centroid location and growth parameters of individual crops enable refined perception of crop growth status, meeting the core requirement of real-time and accurate monitoring of open-field vegetable growth parameters. In this way, the advantages of multi-source data can be efficiently integrated to provide comprehensive and accurate growth monitoring data support for the refined management of open-field crops. Attached Figure Description
[0010] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0011] Figure 1 This is a flowchart illustrating the open-field crop growth monitoring method provided by the present invention;
[0012] Figure 2 This is a schematic diagram of the structure of the open-field crop growth monitoring device provided by the present invention;
[0013] Figure 3 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation
[0014] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0015] It should be noted that in the embodiments of this application, the words "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the words "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.
[0016] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.
[0017] To facilitate a clear description of the technical solutions of the embodiments of this application, the terms "first" and "second" are used in the embodiments of this application to distinguish the same or similar items with essentially the same function and effect. Those skilled in the art can understand that the terms "first" and "second" are not intended to limit the quantity or execution order.
[0018] This application describes some exemplary embodiments for illustrative purposes. It should be understood that this application may be implemented in other ways not specifically shown in the accompanying drawings.
[0019] like Figure 1 As shown, this application provides a method for monitoring the growth of open-field crops, which can be applied to an open-field crop growth monitoring device. The method may include steps S101-S104:
[0020] S101, the open-field crop growth monitoring device collects multi-source sensor data of open-field crops through an inspection robot.
[0021] The aforementioned multi-source sensor data includes depth image data, position data, and inertial measurement unit (IMU) data.
[0022] Optionally, the inspection robot is equipped with a depth camera, an IMU sensor, and a Global Positioning System (GPS) module; before collecting multi-source sensor data of open-field crops through the inspection robot, the open-field crop growth monitoring device can perform internal parameter calibration and external parameter calibration of the depth camera, the IMU sensor, and the GPS module.
[0023] Specifically, the inspection robot is responsible for collecting environmental data on open-field crops in field settings. This inspection robot uses a tracked chassis and is equipped with a depth camera with a built-in IMU sensor and a GPS module. The optical axis of the depth camera is parallel to the robot's direction of travel, the IMU coordinate system is strictly aligned with the robot's coordinate system, and the GPS module's antenna is placed on the unobstructed area of the robot's top to ensure simultaneous acquisition of data from multiple sensors.
[0024] Furthermore, before collecting multi-source sensor data of open-field crops via the inspection robot, the open-field crop growth monitoring device can first perform intrinsic parameter calibration on the depth camera, the IMU sensor, and the GPS module. Specifically, this includes: calibrating the depth camera's intrinsic parameters to determine its focal length, principal point position, and distortion coefficient parameters; collecting the IMU sensor's zero bias and acceleration scale factor in a static state; and configuring the positioning mode, data update frequency, and other parameters for the high-precision GPS module's receiver. Then, extrinsic parameter calibration is performed on the depth camera, the IMU sensor, and the GPS module. Specifically, camera-IMU extrinsic parameter calibration can utilize the depth camera to translate and rotate along yaw, pitch, and roll angles, recording video including the calibration board, and using the Kalibr algorithm to calculate the camera-IMU transformation matrix. GPS-IMU extrinsic parameter calibration can align the position, velocity, or attitude information of both in static and dynamic modes to solve for the rotation matrix R and translation vector T from the GPS coordinate system to the IMU coordinate system.
[0025] It is important to note that intrinsic parameter calibration clarifies the "basic measurement benchmark" for each sensor, preventing data distortion caused by deviations in their own parameters and providing a precise "single data source" for subsequent data fusion. Extrinsic parameter calibration determines the "position and angle correlation" between different sensors in physical space, transforming multi-source data originally in their respective coordinate systems to a unified benchmark and resolving the "spatiotemporal fragmentation" problem of multi-source data. In this way, sensor-specific errors and data barriers between different sensors can be eliminated, ensuring that depth image data, position data, and IMU data possess "accuracy" and "interoperability," preventing misjudgments of crop growth parameters due to data deviations during subsequent analysis.
[0026] After the hardware deployment and calibration are completed, the open field crop growth monitoring device can control the inspection robot to move along a preset path in the open field crop planting area. The depth camera collects depth image data of the crop canopy in real time, the GPS module collects the position data of the inspection robot in real time, and the IMU sensor collects the posture and motion status data of the inspection robot in real time (such as whether the robot tilts due to the bumps of the field ridges, whether the movement speed is stable, to ensure that the influence of equipment motion interference on the data can be eliminated in subsequent analysis). The three work together to form multi-source sensor data.
[0027] S102. The open-field crop growth monitoring device uses a timestamp interpolation calibration method to perform time synchronization processing on the multi-source sensor data. Using the IMU coordinate system as a reference, it performs spatial alignment processing on the depth image data and the position data. Based on the extended Kalman filter algorithm, it corrects the dynamic error of the multi-source sensors to obtain the spatiotemporal alignment state vector.
[0028] Optionally, the sampling frequency of IMU sensors is typically tens to hundreds of times per second, while the sampling frequency of GPS modules and depth cameras is mostly 1-10 times per second. To address the data time asynchrony problem caused by the difference in sampling frequencies among depth cameras, IMU sensors, and GPS modules, the open-field crop growth monitoring device can adopt a timestamp interpolation calibration method to achieve data time synchronization according to the following logic: First, preprocessing is used to clarify the timestamp distribution of data from various sensors and to analyze the data acquisition patterns on the time axis; then, using the timestamps of high-frequency IMU data as anchor points, linear interpolation is performed on low-frequency GPS location data and depth image data to generate low-frequency data estimates corresponding to each IMU timestamp, allowing the low-frequency data to adapt to high-frequency time nodes and completing the initial data synchronization; finally, a sliding window is used to dynamically optimize the interpolation results, correcting any errors that may occur during the interpolation process, ultimately ensuring the accuracy of multi-source sensor synchronized data.
[0029] Optionally, the depth camera, IMU sensor, and GPS module collect data based on their own coordinate systems. To address the data spatial fragmentation caused by the inconsistency of coordinate systems among the three types of sensors, the open-field crop growth monitoring device can use the IMU coordinate system as a reference and employ a coordinate system-unified calibration method to achieve data spatial alignment according to the following logic: First, through sensor extrinsic parameter calibration, obtain the spatial correlation parameters between the depth camera and IMU, and between GPS and IMU (including the camera-IMU transformation matrix, the GPS-IMU rotation matrix R, and the translation vector T), clarifying the transformation relationship between different coordinate systems; then, using the IMU coordinate system as a unified reference, perform coordinate transformation on the depth image data and GPS location data using the aforementioned calibration parameters, transforming the crop 3D points in the camera coordinate system and the field location points in the GPS coordinate system to the universal coordinates in the IMU coordinate system, allowing multi-source data to adapt to the same spatial reference and completing the initial spatial alignment of the data; finally, through coordinate accuracy verification, compare the coordinate deviations of multi-source data at the same physical location after transformation, correct calibration parameter errors, and ensure the consistency and accuracy of spatially aligned data from multiple sensors.
[0030] For example, depth image data acquired by an RGB-D depth camera The transformation formula is as follows:
[0031] ;
[0032] in, This is the rotation matrix for transforming the depth camera coordinate system to the IMU coordinate system. This is the translation vector for the transformation from the depth camera coordinate system to the IMU coordinate system.
[0033] Optionally, the open-field crop growth monitoring device obtains a spatiotemporally aligned state vector by correcting the dynamic errors of multi-source sensors based on the extended Kalman filter algorithm, including: constructing a state vector, which describes the motion state of the inspection robot at different times and sensor deviations; constructing a state transition equation based on the extended Kalman filter algorithm using the optimal state estimate of the previous time step and IMU data to predict the current device state; and when the position data arrives, fusing the global position and the local pose of the depth camera as observations to correct the predicted state and obtain the spatiotemporally aligned state vector.
[0034] Specifically, even after time synchronization and spatial alignment, multi-source sensor data still suffers from sensor dynamic errors, such as drift caused by long-term operation of IMU sensors, positioning fluctuations caused by GPS module signal blockage, and local deviations of depth cameras due to lighting conditions. To address the problem of these errors compounding and affecting data reliability, open-field crop growth monitoring devices can employ a filtering and fusion optimization method to achieve tightly coupled spatiotemporal processing according to the following logic:
[0035] First, by constructing a state vector, the core information of the inspection robot, such as its 3D position, movement speed, quaternion pose, and IMU bias, is integrated to identify the multi-source data association parameters that need to be optimized.
[0036] ;
[0037] in, Indicates three-dimensional position, Indicates movement speed. Represents quaternion pose, This represents the IMU bias correction value.
[0038] Then, based on the extended Kalman filter algorithm and combined with the state transition equation, the current device state is predicted using IMU high-frequency data:
[0039] Based on the optimal state estimate of the previous time step and IMU measurements The state at the current moment is predicted by the state transition function, using the state transition equation. The state at the current moment To make predictions, construct the IMU nonlinear state transition equations:
[0040] ;
[0041] in, This refers to noise in the process.
[0042] Only when GPS position data arrives, the GPS global position and the depth camera local pose are fused as observations. Low-frequency GPS observation data are fused into the state estimation through Kalman gain to suppress IMU drift, random noise from the depth camera and GPS, so as to correct the predicted state.
[0043] Finally, a high-precision spatiotemporally aligned state vector with error correction is output to ensure deep coupling and consistency of multi-source data in the spatiotemporal dimension.
[0044] It should be noted that by using time synchronization, spatial alignment, and dynamic error correction based on extended Kalman filtering, the time asynchrony problem caused by the difference in sampling frequency between IMU, GPS, and depth camera can be solved, the data space fragmentation problem caused by the inconsistency of coordinate system can be eliminated, and dynamic errors such as IMU drift, GPS signal fluctuation, and depth camera illumination deviation can be corrected, thereby improving the accuracy and reliability of multi-source data fusion.
[0045] S103. The open-field crop growth monitoring device performs densification processing on the spatiotemporal alignment state vector to generate a full-domain three-dimensional map covering the entire open-field crop area and including the three-dimensional morphology and spatial location of the open-field crops.
[0046] Optionally, the open-field crop growth monitoring device densifies the spatiotemporal alignment state vector to generate a global 3D map covering the entire open-field crop area and including the 3D morphology and spatial location of the open-field crops. This includes: parsing the spatiotemporal alignment state vector, extracting the global pose sequence and IMU deviation correction value of the inspection robot during the monitoring process, and using the IMU deviation correction value to compensate for errors in the original IMU data; associating the compensated IMU data with the depth image data acquired by the depth camera, and based on a simultaneous localization and mapping (SLAM) algorithm, using the global pose sequence as a constraint. Scale calibration is performed to eliminate deviations in the scale dimension of depth image data; graph optimization algorithms are used to jointly optimize the global pose sequence and depth image feature points to correct pose drift caused by device movement; triangulation is used to densify discrete depth image data, filling in areas lacking depth information to form continuous 3D point cloud data; after the inspection robot completes full-domain path traversal, loop closure detection is triggered, and the currently collected 3D point cloud data is matched and fused with historical mapping data to update the overlapping areas of the panoramic map and generate the full-domain 3D map.
[0047] Specifically, open-field crop growth monitoring devices can align state vectors in time and space. Extracting the 3D position in global coordinates Movement speed Quaternion posture Stored as a high-frequency pose sequence in timestamp order. Quaternion pose The data is converted into a rotation matrix to facilitate subsequent 3D data coordinate transformation. Then, IMU bias correction values are extracted from the spatiotemporal alignment state vector to correct the original IMU measurement data in real time.
[0048] Next, the RGB image acquired by the depth camera is aligned with the depth image to obtain the raw 3D data. The timestamped 3D data pixels are matched with the timestamps of the pose sequence, and the global pose at the corresponding time point is obtained through linear interpolation. Points in the camera coordinate system are converted to points in the global coordinate system through an indirect IMU coordinate system. Noise points are removed using voxel mesh filtering and statistical filtering.
[0049] At the front-end odometry, the corrected IMU data is used as the pre-integration input to the inertial measurement unit (IMU) and fed into a SLAM algorithm, such as ORB-SLAM3, to calculate the inertial pre-integration between adjacent frames, which is then combined with the global pose sequence. Scale calibration is performed on the inertial pre-integration results; in the back-end graph optimization constraints, the global pose sequence is... As a global constraint node for graph optimization, it is jointly optimized with the local pose obtained by matching visual features.
[0050] The 3D data is densified using the principle of triangulation. For missing depth information, interpolation or prediction algorithms are used to fill in the gaps, generating continuous 3D point cloud data. Finally, after the inspection robot completes the full-domain path traversal, loop closure detection is triggered, and the currently collected 3D point cloud data is matched and fused with historical mapping data to update the overlapping areas of the panoramic map and generate the full-domain 3D map.
[0051] S104. The open-field crop growth monitoring device performs semantic segmentation on the full-area three-dimensional map, identifies the centroid location of each crop, and determines the growth parameters of each crop.
[0052] Specifically, the open-field crop growth monitoring device can first preprocess the dense 3D image data corresponding to the global 3D map, converting it into a format suitable for deep learning model input. Then, the labeled data includes categories such as cabbage heads, weeds, and soil. A suitable deep learning model, such as PointNet++, is selected to perform semantic segmentation on the dense 3D image data. The model is trained to obtain semantic labels for each spatial point, thereby segmenting individual crops from the background. The centroid of each crop is calculated as its position in the global coordinate system. Next, growth parameters are extracted from the semantically segmented individual 3D data using spatial geometry methods. For example, by calculating the bounding box of an individual cabbage plant, the plant height and head diameter can be obtained. Based on a triangulated mesh, the surface area of the leaves is calculated using Heron's formula.
[0053] Optionally, after determining the growth parameters of each crop, the open-field crop growth monitoring device can also generate a planting plan based on the growth parameters and remotely control the inspection robot to execute the planting plan.
[0054] Specifically, users can view crop growth parameters in real time. The open-field crop growth monitoring device can generate planting plans based on real-time crop growth parameters and remotely control the inspection robot to execute the planting plans through the network module.
[0055] For example, if it is found that the cabbage plants in a certain area are generally shorter than average, it may be due to insufficient water and fertilizer. The open-field crop growth monitoring device will generate a targeted water and fertilizer supplementation plan and control the inspection robot through the network module to perform the corresponding water and fertilizer application operations.
[0056] It should be noted that, through the above processing, on the one hand, each crop can be accurately identified and its location in the field can be determined, which facilitates subsequent precise management; on the other hand, key growth parameters of the crop can be extracted, providing data support for the formulation of planting plans, and realizing intelligent and precise monitoring and management of open-field crop growth, thereby improving planting efficiency and crop yield and quality.
[0057] In this embodiment, multi-source sensor data, consisting of depth image data, location data, and IMU data collected by an inspection robot, combined with timestamp interpolation calibration and spatial alignment processing based on the IMU coordinate system, effectively solves the problem of spatiotemporal data fragmentation from multiple sensors, achieving precise matching of data from different sources in both time and space dimensions. By correcting the dynamic errors of the multi-source sensors using an extended Kalman filter algorithm and obtaining a spatiotemporal alignment state vector, data deviations caused by motion distortion of mobile monitoring equipment can be specifically improved, enhancing data reliability. By densifying the spatiotemporal alignment state vector to generate a full-domain 3D map, the limitations of insufficient coverage and single-dimensional information from a single sensor can be overcome, fully presenting the 3D morphology and spatial distribution of open-field crops, adapting to the characteristics of wide open-field environments with complex terrain and densely interwoven crops. Semantic segmentation of the full-domain 3D map and extraction of the centroid location and growth parameters of individual crops enable refined perception of crop growth status, meeting the core requirement of real-time and accurate monitoring of open-field vegetable growth parameters. Thus, the advantages of multi-source data can be efficiently integrated, providing comprehensive and accurate growth monitoring data support for the refined management of open-field crops.
[0058] The foregoing mainly describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the above functions, it includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, in conjunction with the units and algorithm steps of the various examples described in the embodiments disclosed herein, the embodiments of this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0059] It should be noted that the device in the embodiments of this application includes a virtual device and a physical device. The virtual device can be an open-field crop growth monitoring device, and the physical device can include electronic devices, computer storage media, and computer program products.
[0060] The open-field crop growth monitoring method provided in this application can be implemented by an open-field crop growth monitoring device or a control module for open-field crop growth monitoring within that device. This application uses an open-field crop growth monitoring device to implement the open-field crop growth monitoring method as an example to illustrate the open-field crop growth monitoring device provided in this application.
[0061] It should be noted that the embodiments of this application can divide the open-field crop growth monitoring device into functional modules according to the above method examples. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing module. The integrated modules can be implemented in hardware or as software functional modules. Optionally, the module division in the embodiments of this application is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.
[0062] like Figure 2 As shown in the figure, this application provides an open-field crop growth monitoring device 200. The open-field crop growth monitoring device 200 includes: an acquisition module 201 and a processing module 202;
[0063] The acquisition module 201 is used to collect multi-source sensor data of open-field crops through an inspection robot. The multi-source sensor data includes depth image data, position data, and inertial measurement unit (IMU) data. The processing module 202 is used to perform time synchronization processing on the multi-source sensor data using a timestamp interpolation calibration method, and to perform spatial alignment processing on the depth image data and the position data based on the IMU coordinate system. The dynamic error of the multi-source sensors is corrected based on the extended Kalman filter algorithm to obtain a spatiotemporal alignment state vector. The spatiotemporal alignment state vector is then densified to generate a global 3D map covering the entire open-field crop area and including the three-dimensional morphology and spatial position of the open-field crops. The global 3D map is then semantically segmented to identify the centroid position of each crop and determine the growth parameters of each crop.
[0064] Optionally, the inspection robot is equipped with a depth camera, an IMU sensor, and a GPS module; the processing module 202 is used to perform internal parameter calibration and external parameter calibration on the depth camera, the IMU sensor, and the GPS module before the inspection robot collects multi-source sensor data of open-field crops.
[0065] Optionally, the processing module 202 is used to construct a state vector, which describes the motion state and sensor bias of the inspection robot at different times; based on the extended Kalman filter algorithm, a state transition equation is constructed using the optimal state estimate of the previous time step and IMU data to predict the current device state; when the position data arrives, the global position and the local pose of the depth camera are fused as observations to correct the predicted state and obtain the spatiotemporally aligned state vector.
[0066] Optionally, the processing module 202 is used to parse the spatiotemporal alignment state vector, extract the global pose sequence and IMU deviation correction value of the inspection robot during the monitoring process, and use the IMU deviation correction value to compensate for errors in the original IMU data; associate the compensated IMU data with the depth image data acquired by the depth camera, and perform scale calibration based on the global pose sequence as a constraint using a simultaneous localization and mapping (SLAM) algorithm to eliminate the deviation of the depth image data in the scale dimension; jointly optimize the global pose sequence and depth image feature points using a graph optimization algorithm to correct pose drift caused by device movement; use triangulation to densify the discrete depth image data, fill in the missing depth information areas, and form continuous three-dimensional point cloud data; after the inspection robot completes the full-domain path traversal, trigger loop closure detection, perform feature matching and fusion of the currently acquired three-dimensional point cloud data and historical mapping data, update the overlapping area of the panoramic map, and generate the full-domain three-dimensional map.
[0067] Optionally, the processing module 202 is used to generate a planting plan based on the growth parameters and remotely control the inspection robot to execute the planting plan.
[0068] In this embodiment, multi-source sensor data, consisting of depth image data, location data, and IMU data collected by an inspection robot, combined with timestamp interpolation calibration and spatial alignment processing based on the IMU coordinate system, effectively solves the problem of spatiotemporal data fragmentation from multiple sensors, achieving precise matching of data from different sources in both time and space dimensions. By correcting the dynamic errors of the multi-source sensors using an extended Kalman filter algorithm and obtaining a spatiotemporal alignment state vector, data deviations caused by motion distortion of mobile monitoring equipment can be specifically improved, enhancing data reliability. By densifying the spatiotemporal alignment state vector to generate a full-domain 3D map, the limitations of insufficient coverage and single-dimensional information from a single sensor can be overcome, fully presenting the 3D morphology and spatial distribution of open-field crops, adapting to the characteristics of wide open-field environments with complex terrain and densely interwoven crops. Semantic segmentation of the full-domain 3D map and extraction of the centroid location and growth parameters of individual crops enable refined perception of crop growth status, meeting the core requirement of real-time and accurate monitoring of open-field vegetable growth parameters. Thus, the advantages of multi-source data can be efficiently integrated, providing comprehensive and accurate growth monitoring data support for the refined management of open-field crops.
[0069] Figure 3 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 3As shown, the electronic device may include: a processor 310, a communications interface 320, a memory 330, and a communications bus 340, wherein the processor 310, the communications interface 320, and the memory 330 communicate with each other through the communications bus 340. The processor 310 can call logic instructions in the memory 330 to execute a method for monitoring the growth of open-field crops. This method includes: collecting multi-source sensor data of open-field crops using an inspection robot, the multi-source sensor data including depth image data, position data, and inertial measurement unit (IMU) data; performing time synchronization processing on the multi-source sensor data using a timestamp interpolation calibration method; spatially aligning the depth image data and the position data using the IMU coordinate system as a reference; correcting the dynamic errors of the multi-source sensors using an extended Kalman filter algorithm to obtain a spatiotemporal alignment state vector; densifying the spatiotemporal alignment state vector to generate a global 3D map covering the entire open-field crop area and including the three-dimensional morphology and spatial location of the open-field crops; performing semantic segmentation on the global 3D map to identify the centroid position of each crop and determine the growth parameters of each crop.
[0070] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0071] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the open-field crop growth monitoring method provided by the above methods. The method includes: collecting multi-source sensor data of open-field crops through an inspection robot, the multi-source sensor data including depth image data, position data, and inertial measurement unit (IMU) data; performing time synchronization processing on the multi-source sensor data using a timestamp interpolation calibration method; performing spatial alignment processing on the depth image data and the position data based on the IMU coordinate system; and obtaining a spatiotemporal alignment state vector by correcting the dynamic errors of the multi-source sensors based on an extended Kalman filter algorithm; performing densification processing on the spatiotemporal alignment state vector to generate a global three-dimensional map covering the entire open-field crop area and including the three-dimensional morphology and spatial position of the open-field crops; performing semantic segmentation on the global three-dimensional map to identify the centroid position of each crop and determine the growth parameters of each crop.
[0072] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the open-field crop growth monitoring method provided by the above methods. The method includes: collecting multi-source sensor data of open-field crops using an inspection robot, the multi-source sensor data including depth image data, position data, and inertial measurement unit (IMU) data; performing time synchronization processing on the multi-source sensor data using a timestamp interpolation calibration method; performing spatial alignment processing on the depth image data and the position data using the IMU coordinate system as a reference; correcting the dynamic errors of the multi-source sensors based on an extended Kalman filter algorithm to obtain a spatiotemporal alignment state vector; performing densification processing on the spatiotemporal alignment state vector to generate a global three-dimensional map covering the entire open-field crop area and including the three-dimensional morphology and spatial position of the open-field crops; performing semantic segmentation on the global three-dimensional map to identify the centroid position of each crop and determine the growth parameters of each crop.
[0073] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0074] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0075] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for monitoring the growth of open-field crops, characterized in that, include: The inspection robot collects multi-source sensor data of open-field crops, including depth image data, position data, and inertial measurement unit (IMU) data. The time-synchronization processing of the multi-source sensor data is performed using the timestamp interpolation calibration method. The spatial alignment processing of the depth image data and the position data is performed with the IMU coordinate system as the reference. The spatiotemporal alignment state vector is obtained by correcting the dynamic error of the multi-source sensors based on the extended Kalman filter algorithm. The spatiotemporal alignment state vector is densified to generate a full-domain 3D map covering the entire open field crop area and including the 3D morphology and spatial location of the open field crops. Semantic segmentation is performed on the global 3D map to identify the centroid location of each crop and determine the growth parameters of each crop. The method for correcting the dynamic errors of multi-source sensors using the extended Kalman filter algorithm to obtain the spatiotemporal aligned state vector includes: Construct a state vector, which is used to describe the motion state of the inspection robot and sensor deviations at different times; Based on the extended Kalman filter algorithm, a state transition equation is constructed using the optimal state estimate from the previous time step and IMU data to predict the current device state. When the location data arrives, the global position and the local pose of the depth camera are fused as observations to correct the predicted state, thus obtaining the spatiotemporal aligned state vector.
2. The method for monitoring the growth of open-field crops according to claim 1, characterized in that, The inspection robot is equipped with a depth camera, an IMU sensor, and a GPS module. Before collecting multi-source sensor data of open-field crops using the inspection robot, the method further includes: The depth camera, the IMU sensor, and the GPS module are subjected to internal parameter calibration and external parameter calibration.
3. The method for monitoring the growth of open-field crops according to claim 1, characterized in that, The process of densifying the spatiotemporal aligned state vector to generate a global 3D map covering the entire open-field crop area and including the 3D morphology and spatial location of the open-field crops includes: The spatiotemporal alignment state vector is analyzed to extract the global pose sequence and IMU deviation correction value of the inspection robot during the monitoring process. The IMU deviation correction value is then used to compensate for errors in the original IMU data. The compensated IMU data is associated with the depth image data acquired by the depth camera. Based on the simultaneous localization and mapping algorithm, scale calibration is performed with the global pose sequence as a constraint to eliminate the deviation of the depth image data in the scale dimension. The global pose sequence and depth image feature points are jointly optimized using a graph optimization algorithm to correct pose drift caused by device movement and vibration. Triangulation is used to densify discrete depth image data, filling in areas lacking depth information and forming continuous 3D point cloud data. After the inspection robot completes the full-domain path traversal, it triggers loop closure detection, performs feature matching and fusion of the currently collected 3D point cloud data and historical mapping data, updates the overlapping area of the panoramic map, and generates the full-domain 3D map.
4. The method for monitoring the growth of open-field crops according to claim 1, characterized in that, After determining the growth parameters of each crop, the method further includes: generating a planting plan based on the growth parameters, and remotely controlling the inspection robot to execute the planting plan.
5. A device for monitoring the growth of open-field crops, characterized in that, include: Acquisition module and processing module; The acquisition module is used to collect multi-source sensor data of open-field crops through the inspection robot. The multi-source sensor data includes depth image data, position data, and inertial measurement unit (IMU) data. The processing module is used to perform time synchronization processing on the multi-source sensor data using a timestamp interpolation calibration method, to perform spatial alignment processing on the depth image data and the position data using the IMU coordinate system as a reference, and to obtain a spatiotemporal alignment state vector by correcting the dynamic error of the multi-source sensors based on the extended Kalman filter algorithm. The spatiotemporal aligned state vector is densified to generate a global 3D map covering the entire open field crop area and including the 3D morphology and spatial location of the open field crops; the global 3D map is semantically segmented to identify the centroid position of each crop and determine the growth parameters of each crop. The processing module is specifically used for: Construct a state vector, which is used to describe the motion state of the inspection robot and sensor deviations at different times; Based on the extended Kalman filter algorithm, a state transition equation is constructed using the optimal state estimate from the previous time step and IMU data to predict the current device state. When the location data arrives, the global position and the local pose of the depth camera are fused as observations to correct the predicted state, thus obtaining the spatiotemporal aligned state vector.
6. The open-field crop growth monitoring device according to claim 5, characterized in that, The inspection robot is equipped with a depth camera, an IMU sensor, and a GPS module. The processing module is used to perform internal parameter calibration and external parameter calibration on the depth camera, the IMU sensor and the GPS module before the inspection robot collects multi-source sensor data of open field crops.
7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the open-field crop growth monitoring method as described in any one of claims 1 to 4.
8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the open-field crop growth monitoring method as described in any one of claims 1 to 4.
9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the open-field crop growth monitoring method as described in any one of claims 1 to 4.