Automatic inspection method and equipment for robot in greenhouse, medium and product
By using robot automatic inspection methods in the greenhouse, combining instant positioning and map construction and fusion positioning algorithms, the problem of inefficient crop detection in greenhouse environments is solved, high-precision automatic navigation and data acquisition are achieved, and artificial errors are reduced. cost.
Patent Information
- Application Number
- CN202411111553.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-14
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-08-14
AI Technical Summary
In greenhouse environments, the phenotype and pest detection of agricultural crops are inefficient, labor intensity and inaccurate data collection. Traditional manual inspection methods consume a lot of manpower and material resources and are prone to artificial errors, making it difficult to meet the growth of crops. Accurately evaluate requirements.
A method of automatic inspection of greenhouse robots is adopted, and the real-time positioning and map construction algorithms are used to construct high-precision maps. The fusion positioning algorithm is used to realize centimeter-level high-precision positioning, combined with finite state machines and crop labels, and automatic robot navigation and crop data are realized. fully automatic collection. The method includes the robot equipped with an RGB camera, a multi-spectral camera, a wheel encoder, an inertial measurement unit and a lidar, using an improved tightly coupled laser SLAM algorithm and a Kalman filter, embedded a repositioning module and an obstacle avoidance control algorithm to achieve automatic patrol Inspection tasks.
It improves navigation accuracy and crop data collection efficiency, reduces labor costs and errors, realizes automatic navigation of robots in greenhouses and fully automatic collection of crop data, reduces manpower and material investment, and improves data accuracy.
Smart Images

Figure CN119022927A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of robot inspection technology, and in particular to a method, equipment, medium and product for automatic robot inspection in a greenhouse. Background Art
[0002] Statistics show that in 2016, the total area of protected horticulture in China was 4.765 million hectares, and the types of facilities included small plastic greenhouses, large and medium plastic greenhouses, solar greenhouses, heated and multi-span greenhouses, with plastic greenhouses accounting for 44%. Greenhouses not only provide a controlled growth environment for agricultural production, so that crops are no longer subject to the constraints of seasons and climate, but also provide ideal conditions for creating high-quality and high-yield crops. As the global population continues to grow and the demand for excellent quality agricultural products continues to increase, the huge potential of greenhouse agriculture has been widely recognized. This modern agricultural method not only ensures the stability of food supply, but also plays a key role in improving the quality of agricultural products, reducing the use of pesticides and fertilizers, and increasing the sustainability of agricultural production. Therefore, greenhouse agriculture has significant strategic value in meeting the growing global demand for food and promoting technological innovation in the agricultural field.
[0003] At present, the overall mechanization level of China's facility agriculture has only reached 32%, while in contrast, the mechanization level of field grain crops can reach 70-80%. Against the backdrop of global agricultural labor shortages, the introduction of agricultural robots into greenhouse agriculture has become one of the important ways to solve the labor shortage problem. In addition, compared with the complex and changeable field environment, although the greenhouse environment is relatively simple, there are still many challenges that need to be solved due to high temperature and humidity, high plant density, and even the possible presence of harmful gases. Therefore, robots have broad application prospects in inspection tasks in greenhouses, and are expected to contribute to improving the efficiency of agricultural production, reducing costs, and providing a safer working environment.
[0004] Crop phenotyping and pest detection in agriculture faces problems such as low efficiency, high labor intensity, and inaccurate data collection. Traditional manual inspection methods not only consume a lot of manpower and material resources, but are also prone to human errors, resulting in inaccurate data. With the development of agricultural technology, the demand for accurate assessment of crop growth conditions is increasing, so an efficient, accurate, and automated solution is needed to replace traditional methods. The introduction of automation and intelligent technology can greatly improve the efficiency and accuracy of data collection, reduce labor costs and errors, and thus provide reliable support for crop breeding, precision management, and pest control.
[0005] In greenhouse scenarios, due to factors such as limited signals and changing environments, positioning and global path planning face more severe challenges. Positioning is a key step in automatic navigation and an important part of the robot's autonomous operation. In field scenarios, due to less signal obstruction, the Global Positioning System (GPS) can provide accurate absolute positioning for the robot. In recent years, Simultaneous Localization And Mapping (SLAM) technology has played an increasingly important role in greenhouse mapping and positioning. With the development of sensor technology and three-dimensional reconstruction technology, the accuracy, robustness and scope of use of SLAM have been greatly improved. SLAM technology can cope with greenhouse scenarios with high autonomy requirements and no GPS signals, providing agricultural robots with solutions for accurate mapping and reliable positioning. Summary of the invention
[0006] The purpose of this application is to provide a method, equipment, medium and product for automatic inspection of robots in a greenhouse, which can realize automatic navigation of robots in the greenhouse and fully automatic collection of crop data, thereby improving the accuracy of navigation and the efficiency of crop data collection.
[0007] To achieve the above objectives, this application provides the following solutions:
[0008] In a first aspect, the present application provides a method for automatic inspection of a robot in a greenhouse, the method for automatic inspection of a robot in a greenhouse is applied to an automatic inspection system for a robot in a greenhouse, the automatic inspection system for a robot in a greenhouse comprising: a robot and a plurality of crop tags; the crop tags are respectively located next to crops of different varieties in a greenhouse scene, and record the variety information of the crops; the robot is equipped with an RGB camera, a multispectral camera, a wheel encoder, an inertial measurement unit and a laser radar, the RGB camera and the multispectral camera are used to collect crop data, and the wheel encoder, the inertial measurement unit and the laser radar are used to determine the global position point of the robot;
[0009] The automatic inspection method of the greenhouse robot comprises:
[0010] Control the robot to walk in the greenhouse scene, and use the real-time positioning and map building algorithm to build a map of the greenhouse scene to obtain a global map;
[0011] Control the robot to walk along the inspection path in the greenhouse scene, and use the fusion positioning algorithm to update and record the global posture points of the robot during the walking process to obtain the teaching path;
[0012] A finite state machine is established according to the inspection task of the robot; the finite state machine is used to define the state and state transition rules of the robot;
[0013] The robot automatically performs inspection tasks based on the finite state machine according to the global map and the taught path.
[0014] Optionally, the instant positioning and mapping algorithm is an improved tightly coupled laser SLAM algorithm; the improved tightly coupled laser SLAM algorithm is based on the FAST-LIO2 architecture, adds ground constraints in the residual calculation process, and adds stable triangle descriptor loop detection.
[0015] Optionally, the fusion positioning algorithm includes local positioning and global positioning; the local positioning uses a Kalman filter to fuse the data of the inertial measurement unit and the wheel encoder, and provides a coordinate transformation relationship from the odometer coordinate system to the robot coordinate system; the global positioning uses the front end of the improved FAST-LIO2 architecture to fuse the data of the inertial measurement unit and the lidar, and provides a coordinate transformation relationship from the map coordinate system to the odometer coordinate system; the fusion positioning algorithm determines the coordinate transformation relationship from the robot coordinate system to the map coordinate system according to the local positioning and the global positioning, and realizes the update and recording of the global posture points of the robot during walking.
[0016] Optionally, a repositioning module is embedded in the fusion positioning algorithm; the repositioning module uses an iterative nearest point cloud registration algorithm to search for the best pose within a range of 1 meter and 30 degrees of a given pose point; the given pose point includes an initial pose point and a current pose point at each set time interval, and each pose point includes position and orientation information; the 1 meter and 30 degree range is a 1×1 square meter range of a given position and a 30 degree range of a given orientation.
[0017] Optionally, the robot's states include: idle state, record state, run state, pause state, photo state, goto state, follow state and wait state; and the robot's state transition rules include:
[0018] When the robot is idle, it is in idle state;
[0019] In the idle state: when the command to start recording is given, the robot enters the record state and records the teaching path; when the command to execute the teaching path is given, the robot enters the run state and performs the inspection task;
[0020] In the record state, the robot obtains the current position and pose points and writes a series of position and pose points into a file; when receiving the command to end recording, the robot returns to the idle state;
[0021] In the run state: when a pause task instruction is given, the robot enters the pause state; when a task end instruction is given or the task is successfully executed, the robot enters the idle state; when a crop tag is detected, the robot enters the photo state; the goto state is also nested in the run state; when the robot is in the run state, it enters the goto state by default, and executes the global path planning algorithm and obstacle avoidance control algorithm for point-to-point path planning;
[0022] In the pause state: when given an instruction to continue execution, the robot returns to the run state and continues to perform the inspection task; when given an instruction to end the task, the robot enters the idle state;
[0023] In the photo state: the robot uses the RGB camera and multispectral camera to collect crop data, and returns to the run state at set intervals to continue the inspection task;
[0024] In the goto state: when the robot reaches the starting point of the teaching path, it enters the follow state and automatically follows the teaching path;
[0025] In the follow state: when the robot encounters an obstacle, it enters the wait state;
[0026] In the wait state: when the obstacle is removed within the waiting time, it enters the follow state and continues to track the taught path; when the obstacle is not removed within the waiting time, it enters the goto state, searches for subsequent points without obstacles, and navigates to the next point without obstacles.
[0027] Optionally, the global path planning algorithm adopts a teaching path approach; the obstacle avoidance control algorithm adopts a time elastic band control algorithm.
[0028] Optionally, the crop tag is an April tag QR code tag.
[0029] In a second aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the automatic inspection method of the greenhouse robot.
[0030] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the automatic inspection method of the greenhouse robot.
[0031] In a fourth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the automatic inspection method of the greenhouse robot.
[0032] According to the specific embodiments provided in this application, this application discloses the following technical effects:
[0033] The present application provides a method, device, medium and product for automatic inspection of robots in a greenhouse, which realizes high-precision mapping of greenhouse scenes through real-time positioning and map construction algorithms, realizes centimeter-level high-precision positioning in the greenhouse through fusion positioning algorithms, realizes automatic navigation of robots in the greenhouse by building a global map, recording teaching paths and establishing a finite state machine, and realizes fully automatic collection of crop data by using crop tags to record crop variety information. Compared with the traditional manual inspection method, the present application uses robots to automatically perform inspection tasks, which does not require a lot of manpower and material resources, and is not prone to human errors and inaccurate data. Therefore, the method, device, medium and product for automatic inspection of robots in a greenhouse provided by the present application can realize automatic navigation of robots in the greenhouse and fully automatic collection of crop data, and improve the accuracy of navigation and the efficiency of crop data collection. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0035] Figure 1 A flow chart of the automatic inspection method of a robot in a greenhouse provided in this application;
[0036] Figure 2 A schematic diagram of the finite state machine provided for this application;
[0037] Figure 3 Crop tag detection flow chart provided for this application;
[0038] Figure 4 A schematic diagram of the static layer of the costmap provided for this application;
[0039] Figure 5 A complete schematic diagram of the costmap provided for this application. DETAILED DESCRIPTION
[0040] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0041] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.
[0042] In an exemplary embodiment, the present application provides a method for automatic robot inspection in a greenhouse, which is applied to an automatic robot inspection system in a greenhouse. The system includes: a robot and a plurality of crop tags; the crop tags are respectively located next to different varieties of crops in the greenhouse scene, and record the variety information of the crops; the robot is equipped with an RGB camera, a multi-spectral camera, a wheel encoder, an inertial measurement unit and a laser radar, the RGB camera and the multi-spectral camera are used to collect crop data, and the wheel encoder, the inertial measurement unit and the laser radar are used to determine the global position of the robot. Figure 1 As shown, the method includes the following steps 1 to 4. Wherein:
[0043] Step 1: Control the robot to walk in the greenhouse scene, and use the real-time positioning and map construction algorithm to build a map of the greenhouse scene to obtain a global map.
[0044] Preferably, the instant positioning and mapping algorithm is an improved tightly coupled laser SLAM algorithm; the improved tightly coupled laser SLAM algorithm is based on the FAST-LIO2 (Fast Direct LiDAR-inertial Odometry) architecture, adds ground constraints in the residual calculation process, and adds a stable triangle descriptor (Stable Triangle Descriptor, STD) loop detection.
[0045] High-precision mapping of greenhouse scenes is achieved through an improved tightly coupled laser SLAM algorithm. The FAST-LIO2 architecture adopted specifically includes: 1. Use solid-state laser radar to obtain laser scanning points, and use the inertial measurement unit to obtain acceleration and angular velocity information. 2. The acceleration and angular velocity information of the inertial measurement unit are forward propagated and integrated to obtain the initial pose points. 3. The laser scanning points are back-propagated to remove motion distortion caused by acquisition at different times. 4. Calculate the residual of the laser odometer and use iterative Kalman filtering to estimate the pose transformation until convergence. 5. Construct a map based on the pose estimated by the odometer in the previous step, and update the global map maintained by the incremental k-dimensional tree (ikd-tree) data structure. The optimization of the above process is as follows:
[0046] Optimization 1: The IEEE1588v2.0PTP standard time synchronization of the master-slave (master: IPC clock device, slave: solid-state LiDAR sensor device) network measurement and control system is ensured to have the correct timestamps for the inertial measurement unit data and LiDAR scan data. This avoids map overlap and odometer flickering during the mapping process.
[0047] Optimization 2: Add ground constraints to the backend of the original SLAM framework (i.e., FAST-L1O2 architecture). The following introduces ground constraints from three aspects: representation, calculation, and optimization of ground residual terms.
[0048] Representation of the ground residual term:
[0049] Initialize the global ground plane equation in the world coordinate system as:
[0050] A w x+B w y+C w z+D w =0
[0051] The parameter is denoted as θ w :(A w , B w , C w , D w ). Among them, for the parameter A of the initialized global ground plane w =0, B w =0, C w =1,D w =0.
[0052] Assuming that the laser sub-image acquired in the current frame is the k-th frame sub-image, the ground equation of the current frame radar point cloud (k-th frame) is:
[0053] A k x+Bk y+C k z+D k =0
[0054] The parameter is denoted as θ k :(A k , B k , C k , D k ).
[0055] Unification of plane coordinate system: The global ground equation in the initialized world coordinate system is converted to the current k-th frame radar coordinate system, and the parameter is expressed as θ′ w :(A′ w , B′ w , C′ w , D′ w ), parameter conversion follows the following formula:
[0056]
[0057] Among them, T kw represents the pose transformation from the kth frame radar coordinate system to the world coordinate system, T wk It represents the pose transformation from the world coordinate system to the k-th frame radar coordinate system, that is, the pose of the current lidar, and the pose transformation is known.
[0058] Calculation of ground residual term:
[0059] Let θ′ w The first three dimensions of the parameters A′ w , B′ w , C′ w (normal vector) as the basis to construct a three-dimensional rotation space. k The first three dimensions of A k , B k , C k (normal vector) is converted into this constructed rotation space to obtain vector n: (A kn , B kn , C kn ), then the residual error in rotation can be expressed as:
[0060]
[0061] The residual error in translation can be expressed as the distance d = D' of the plane intercept w -D k .
[0062] Therefore, the final plane residual term of the current k-th frame is:
[0063] E k =(α, β, d)
[0064] Optimization of ground residual terms:
[0065] After obtaining the ground residual term, this residual term is put into the iterative error state Kalman filter to optimize the error and improve the accuracy of pose estimation and mapping.
[0066] Prediction of the filter's error state:
[0067]
[0068] in, represents the prior error state estimate at time k, F ek-1 represents the state transfer matrix at time k-1, represents the posterior error state estimate at time k-1, B ek- 1 represents the noise transfer matrix at time k-1, W ek represents the measurement noise at time k.
[0069] The covariance prediction corresponding to the error state quantity is:
[0070]
[0071] in, represents the prior estimation error covariance at time k, Represents the posterior estimation error covariance at time k-1.
[0072] Kalman gain calculation:
[0073]
[0074] Among them, K ek represents the Kalman gain at time k, G ek Represents the transformation matrix from error state variables to observations, connecting states and observations. ek Represents the error measurement noise covariance at time k.
[0075] Update of the error state, computing the posterior estimate:
[0076]
[0077] in, represents the posterior estimation error covariance at time k, represents the posterior error state estimate at time k, Y ek Represents the error measurement (observation).
[0078] Optimization 3: Added STD loop detection in SLAM, which detects stable triangle descriptors to avoid ghosting problems in the global map after scanning the same position multiple times. The start and end points are optimization issues for loop conditions and z-axis drift problems.
[0079] The implementation steps of STD loop detection are as follows:
[0080] 1. Get point cloud keyframes. Keyframes are accumulated from scanning points over a period of time.
[0081] 2. Plane detection: Use the region growing algorithm to distinguish between plane voxels and boundary voxels.
[0082] 3. Key point selection. Project the boundary voxels onto the plane and create an image that overlaps the plane for each plane. Each pixel represents the maximum distance from a point in the boundary voxel to the plane. For a point, select the point with the maximum pixel value in its 5×5 neighborhood as the key point.
[0083] 4. Construct a kd number structure for all key points, search for 20 nearest key points in the neighborhood of each key point, and form a triangular descriptor.
[0084] 5. All triangle descriptors are stored in a hash table, with the key value being the dot product of the side length of each descriptor and the normal projection vector (a total of six). The key value of each descriptor in the current frame is calculated and matched in the hash table. If the descriptor key value is the same as the previous key frame, vote for the corresponding key frame. The top 10 key frames will be used as loop candidates.
[0085] 6. Geometric verification. The relative transformation T = (R, t) between two keyframes is calculated using singular value decomposition (SVD) based on the correspondence between the descriptor vertices between the keyframes. Assume that there are triangle a and triangle b descriptors on the two keyframes, and the key values of the two triangles are consistent. Then, the SVD decomposition yields:
[0086]
[0087] [U, S, V] = SVD(J)
[0088] R=VUT , t=-R*c a +c b
[0089] in, are the three vertices of triangle a and triangle b, c a 、c bis the center point of triangles a and b. After decomposition, the rotation matrix R and translation vector t between the two key frames can be obtained. In addition, the RANSAC algorithm is used to find the transformation that maximizes the correct matching of descriptors. Based on the best change, the number of planes that can be matched is calculated and compared with the previously set hyperparameters. If the number of matched planes is greater than the hyperparameters, the key frame is considered to have a loop with the current frame through geometric verification.
[0090] Step 2: Control the robot to walk along the inspection path in the greenhouse scene, and use the fusion positioning algorithm to update and record the global pose points of the robot during the walking process to obtain the teaching path.
[0091] Preferably, the fusion positioning algorithm includes local positioning and global positioning; the local positioning uses a Kalman filter to fuse the data of the inertial measurement unit and the wheel encoder to provide a coordinate transformation relationship from the odometer coordinate system to the robot coordinate system; the global positioning uses the front end of the improved FAST-L1O2 architecture to fuse the data of the inertial measurement unit and the laser radar to provide a coordinate transformation relationship from the map coordinate system to the odometer coordinate system; the fusion positioning algorithm determines the coordinate transformation relationship from the robot coordinate system to the map coordinate system based on the local positioning and the global positioning, and realizes the update and recording of the global posture points of the robot during walking. In addition, a repositioning module is also embedded in the fusion positioning algorithm to determine the starting posture and eliminate the accumulated error.
[0092] Among them, the repositioning module uses the iterative closest point (ICP) point cloud registration algorithm to search for the best pose within the range of 1 meter and 30 degrees of a given pose point; the given pose point includes the initial pose point and the current pose point at each set interval, and each pose point includes position and orientation information; the 1 meter and 30 degree range is a 1×1 square meter range of a given position and a 30 degree range of a given orientation.
[0093] The centimeter-level high-precision positioning in the greenhouse is achieved by using a fusion positioning algorithm. Local positioning uses a Kalman filter to fuse the data of the inertial measurement unit and the wheel odometer, providing high-frequency (100HZ) low-precision positioning information and the transformation relationship from the odom to the base_link coordinate system. Global positioning uses the front end of the improved FAST-LIO2 architecture to provide low-frequency (10HZ) high-precision positioning information and the coordinate transformation relationship from map to odom. Among them, map: map coordinate system, a fixed global coordinate system; odom: odometer coordinate system, a virtual coordinate system based on the transformation between the coordinates obtained by actual calculation and the real coordinates; base_link: robot coordinate system, representing the robot body, is the center of the robot. The problem of low accuracy and low robustness of a single positioning method is solved by a multi-sensor fusion positioning method. The specific fusion positioning is to achieve fusion positioning by fusing the data of multiple sensors such as wheel encoders, inertial measurement units and lidar. First, local positioning integrates the data of the inertial measurement unit and the wheel encoder, and puts the orientation, acceleration, angular velocity of the inertial measurement unit and the position, orientation, linear velocity, and angular velocity information of the wheel encoder into the extended Kalman filter. After filtering, the local pose information will be output. This local pose information includes position and orientation information. This pose information provides the coordinate transformation relationship from odom to base_link coordinate system, and the frequency is 100HZ. Global positioning integrates the data of the inertial measurement unit and the lidar. The specific fusion method adopts the front-end odometer part of the improved FAST-LIO2 architecture. The algorithm outputs the pose transformation relationship and provides the coordinate transformation relationship from map to odom, and the frequency is 10HZ. Finally, through the above two coordinate transformation relationships, the coordinate transformation relationship of map->odom->base_link is provided, and the relationship between map and base_link is the coordinate relationship between the robot body and the global map, so that the pose of the robot in the global map can be obtained, and accurate positioning can be achieved. The relocalization module specifically uses the ICP point cloud registration algorithm to search for the best pose within 1 meter and 30 degrees of the specified initial pose. Helps the robot find the correct initial position again after restarting or going offline. Relocalization is given once after the global map is built and before the robot starts patrolling. Specifically, it gives the robot an approximate position, including the current position and orientation information on the global map. Relocalization searches for the best position nearby based on the position and orientation information given by the user. Specifically, it searches for the best position within a 1×1 square meter range of the user-given position, and searches for the best orientation within a 30-degree range of the user-given orientation, and finally obtains the best position based on the ICP point cloud registration algorithm.In addition to before the inspection begins, the robot will reposition itself every 10 seconds during the positioning process. During this repositioning process, the robot will automatically obtain its position in the global map and use the repositioning module to match the best position within 1×1 square meters and 30 degrees.
[0094] Step 3: Establish a finite state machine based on the robot's inspection task.
[0095] Finite state machines can clearly represent the various states of a system and the transition relationships between states by defining clear states and state transition rules. This structured representation method makes the design, analysis, and implementation of algorithms more intuitive and controllable. In addition, finite state machines also facilitate the verification and debugging of system behavior, ensuring the correctness and stability of the algorithm under various conditions.
[0096] like Figure 2As shown, the robot's states include: idle state, record state, run state, pause state, photo state, goto state, follow state and wait state. The robot's state transition rules include: when the robot is idle, it is in idle state; in idle state: when given an instruction to start recording, the robot enters record state and records the teaching path; when given an instruction to execute the teaching path, the robot enters run state and performs inspection tasks; in record state, the robot obtains the current posture point and writes a series of posture points to a file; when receiving an instruction to end recording, the robot returns to idle state; in run state: when given an instruction to pause the task, the robot (stops running) enters pause state; when given an instruction to end the task or the task is successfully executed, the robot enters idle state; when a crop tag is detected, the robot enters photo state (starts taking pictures); in run state, goto state is also nested; when the robot is in run state, it enters goto state by default at the same time to execute global path planning algorithm and obstacle avoidance control The robot uses the control algorithm to plan the point-to-point path; in the pause state: when the instruction to continue execution is given, the robot returns to the run state and continues to perform the inspection task; when the instruction to end the task is given, the robot enters the idle state; in the photo state: the robot uses the RGB camera and the multispectral camera to collect crop data, and returns to the run state at a set interval to continue the inspection task; in the goto state: when the robot reaches the starting point of the teaching path, it enters the follow state and automatically tracks the teaching path (global path); in the follow state: when the robot encounters an obstacle, it enters the wait state; in the wait state: when the obstacle is removed during the waiting time, it enters the follow state and continues to track the teaching path; when the obstacle is not removed during the waiting time, it enters the goto state, searches for subsequent points without obstacles on the global path, and navigates to the next point without obstacles.
[0097] Preferably, the crop tag is an April tag QR code tag, which is a visual reference library (similar to a QR code, but with a simpler structure, less information transmitted but strong real-time performance). By placing an April tag QR code tag near the crop for which information needs to be collected, accurate positioning of the target plant and automatic data collection can be achieved. During the operation, the robot will use a camera to continuously shoot and detect the April tag. The main steps include: 1. Extracting image edges based on the gradient in the image. 2. Quadrilateral detection. First, detect the straight line edges in the image, and determine the quadrilateral based on the adjacent edges of the straight line. 3. Encoding and decoding. Generate a dot matrix inside the quadrilateral to calculate the color block value, and use the local binary mode for classification to obtain the code. Match with the code in the library to achieve decoding. The robot is equipped with an RGB camera, which continuously detects the QR code during the automatic navigation process. When the QR code is detected, the crop variety information is recorded. After waiting to stop, the RGB image and multispectral image of the current target plant are taken. This information can be used for subsequent phenotypic or pest and disease detection. The main functions of the QR code are as follows: 1. Help the robot locate the target plant to be photographed, achieve accurate data collection, and avoid the acquisition of a large amount of useless data. 2. Improve the quality of data acquisition. If a QR code is detected, the robot will stop running and wait for the acquisition of RGB and multi-spectral data. This can avoid problems such as image blur and ghosting caused by robot movement. 3. Record variety information. Breeders usually face hundreds or even thousands of varieties, so it is very important to accurately record different varieties. The QR code can provide variety information to help breeders manage. Crop label detection process is as follows Figure 3 As shown, it includes: 1. Start detection. When the robot is in the process of inspection, it will continuously detect the QR code. 2. If the QR code is not detected, it will return to the first step, otherwise it will go to the next step. 3. Get the ID. The ID information of the QR code records all crop varieties, helping breeders to identify and manage different crop varieties. 4. Find out whether the list exists. During an inspection, the varieties that have collected data will be put into the list. If there are duplicate varieties, they will be repeated with the list records. At this time, the QR code will be ignored and the detection will continue to prevent duplicate data acquisition. If there is no such variety in the list, it means it is a new variety, then go to the next step. 5. Wait for two seconds and take a photo. Waiting for two seconds is to allow the robot to stop and ensure the quality of data acquisition. RGB and multispectral data can be obtained by taking pictures.
[0098] Preferably, the global path planning algorithm adopts the teaching path method. The operator only needs to remotely control the robot to drive the required path before use. The specific path is determined by the inspection requirements. During the recording process, the robot will record the current global posture, mainly recording the conversion relationship between the map and base_link coordinate systems. This transformation is used as the global posture of the robot during driving.
[0099] Preferably, the obstacle avoidance control algorithm adopts a time elastic band (TEB) control algorithm. The TEB algorithm is implemented based on costmap, and Costmap is composed of three layers, namely a static layer, an obstacle layer, and an expansion layer. Among them, the static layer is obtained based on the global map, and is specifically composed of a two-dimensional grid map obtained by statistical filtering and through-filtering of the global map, that is, the most basic layer, which represents the distribution of static obstacles, including crop ridges, fixed instruments, etc. The obstacle layer is obtained in real time by the robot during the inspection process. The distance of the dynamic obstacles is obtained through the laser scanning points and superimposed on the static layer, which can better reflect the real-time obstacles in the inspection process. The expansion layer is superimposed on the static layer and the obstacle layer. Because the robot needs to retain a certain margin when avoiding obstacles, the range of the obstacles must be expanded to ensure safer avoidance of obstacles. In this embodiment, the static map refers to Figure 4 , black represents the static obstacles detected previously in the figure, and the complete costmap cost map can be found in Figure 5 The color represents the cost. If the color is red, it means the cost is higher, blue means the cost is lower, and white means the cost is 0 and you can pass.
[0100] The implementation principles and steps of the TEB algorithm are as follows:
[0101] 1. Path initialization
[0102] First, we need to determine the path sequence consisting of a series of pose points and time information. Assuming there are n pose points on the path, the path sequence consists of the following two parts:
[0103] X={x i} i=0,1,…,n
[0104] τ={ΔT i} i=0,1,…,n-1
[0105] Among them, X represents a series of pose points, x i is the i-th pose point on the path. τ represents the time series set on the path. ΔT i It represents the time required for the robot to move from the i-th pose point on the path to the i+1-th pose point.
[0106] Based on the above representation, the path can be represented as a series of pose points and time series:
[0107] B:=(X,τ)
[0108] Among them, B is the complete representation of the path. At this point, the initialization of the path is completed.
[0109] 2. Construction of cost function
[0110] Assuming there are k optimization objectives in total, the total cost function can be expressed as:
[0111]
[0112] Among them, f j (B) represents the jth optimization objective function, γ j represents the coefficient weight of the jth objective function, which represents the importance of this optimization goal. f(B) represents the total cost function.
[0113] The optimization objectives mainly include the following:
[0114] (1) Path following and obstacle avoidance constraints
[0115] The planned path needs to be consistent with the global path as much as possible. The further it deviates from the global path, the higher the cost. The planned path needs to be far away from obstacles. The closer to obstacles, the higher the cost. The objective function is constructed with these two goals.
[0116] (2) Speed and acceleration constraints
[0117] Robots have different mechanical parameters or different requirements of users for specific application scenarios, and they all have requirements for speed and acceleration. According to specific requirements, upper and lower limits can be imposed on the speed and acceleration of the robot to ensure that the movement of the robot is restricted during the inspection process.
[0118] (3) Kinematic constraints
[0119] Since the robot uses a two-wheel differential kinematic model, it can only move or rotate along the current direction, so the movement form will also impose constraints on the algorithm.
[0120] (4) Time constraints
[0121] The robot is required to move faster and increase its running speed. Therefore, the time spent on the path should be minimized to make ΔT i As small as possible.
[0122] 3. Minimization of the total cost function
[0123] Using the gradient descent method to minimize the cost function is to balance multiple optimization objectives to minimize the cost.
[0124] B * =arg B minf(B)
[0125] Among them, B * Represents the minimum value of the total cost function.
[0126] 4. Calculate the specific control amount and update the motion state
[0127] The parameters with the smallest total cost function calculated above are extracted and converted into specific control quantities (linear velocity and angular velocity), so as to control the robot system and realize real-time update of the motion state.
[0128] Step 4: The robot automatically performs inspection tasks based on the finite state machine according to the global map and the taught path.
[0129] Furthermore, the present application also includes: voice interaction module development, integrated intelligent voice control system. Use iFlytek's six-microphone ring microphone array and offline voice recognition library to realize the robot's voice wake-up, one-key inspection and inspection progress report. Through the design of these three functions, users can easily operate the device through natural language instructions without tedious manual input or specific technical training, thereby significantly improving the robot's human-computer interaction performance. The six-microphone ring microphone array can collect the user's voice information and play the role of audio preprocessing and noise reduction. Because it is a ring array, it can also distinguish the location of the sound source. The role of the offline voice recognition library is to recognize the voice information collected by the microphone ring array in an offline state, and convert the user's audio into a string for subsequent command recognition, so that the robot can make corresponding actions according to the user's words. Implementation of voice wake-up: When the microphone is powered on, it will continuously perform audio detection. When the collected audio matches the keyword set by the user, it will be awakened, and the prompt voice will be automatically played. Implementation of one-key inspection: When the microphone is awakened, it will continue to recognize the voice. When the voice of starting the inspection is received, the subsequent program will automatically issue instructions to the robot to perform the inspection according to the pre-set global path. At this time, no complicated operations are required, and the robot automatically enters the inspection state. Implementation of inspection progress report: In the flowchart of detecting QR codes, a list is used, and the id information of the QR code varieties that have been identified will be put into the list. When the robot detects the inspection progress report, it will check the number of list libraries in the navigation system, and then broadcast the total number, so that the user can know how many varieties of data have been collected so far.
[0130] In summary, this application adopts the following scheme:
[0131] 1. The remote control robot walks through the greenhouse scene that needs to be automatically created. In this process, it is not necessary to walk through all the scenes, but only to reconstruct the entire scene. The improved FAST-LIO2 tightly coupled laser SLAM algorithm is used. In this process, a high-precision map of the greenhouse (global map) will be gradually established. In addition, statistical filtering and through-filtering will be used to compress the global map into two dimensions, and finally a two-dimensional grid map will be produced. This provides a basis for subsequent positioning, recording of teaching paths, and obstacle avoidance control.
[0132] 2. Positioning and recording the teaching path. After the establishment of the complete map, it is necessary to reposition first, that is, to give the initial positioning. Then the remote control robot takes a required path. During this process, the robot will continuously record its global posture points and write them into the file. This path is the teaching path, which will be used as the global path in the inspection process. Before recording the teaching path, it is necessary to open the fusion positioning module and give the initial posture of the robot. When the robot is already in the correct position, the remote control robot can record the corresponding path. According to the different inspection needs in the scene, several different inspection paths can be recorded, and the corresponding paths can be saved after recording. For different inspection tasks in the scene, such as varieties that have been eliminated in the inspection process before, there is no need to pay attention to the growth of these varieties in the subsequent inspections, and the corresponding crop ridges may not be visited later. In this case, the path will change. Using another path can improve efficiency and prevent the inspection of eliminated varieties.
[0133] 3. Perform inspection tasks. Use voice control to start inspection, pause inspection, and report inspection progress. Help operators control the robot and grasp the robot's inspection progress in real time. The final collected RGB images and multispectral images can be saved locally or uploaded to the cloud for subsequent phenotyping or pest and disease detection.
[0134] Compared with the prior art, this application has the following advantages:
[0135] 1. Precise navigation allows the robot to pass smoothly and change rows between narrow ridges. Thanks to high-precision mapping and positioning, the difference between the actual path the robot walks and the taught path is at the centimeter level. Even without a high-precision GPS signal, the robot can achieve high-precision navigation in the greenhouse.
[0136] 2. Multi-scenario greenhouse inspection operations. Compared with navigation line extraction based on deep learning, this navigation method does not require the collection of a large number of data sets, and only requires simple exploration in the early stage. Compared with beacons, guide rails, ribbons, and magnetic strips, there is no need to make additional arrangements for multiple scenarios, which reduces equipment and maintenance costs. Therefore, this navigation method can be applied in greenhouse scenarios with different planting methods without the need for additional arrangements.
[0137] 3. High-quality and efficient data collection. The April tag is used to locate the target plant, so the target plant image can be obtained efficiently, rather than collecting information on all plants in the greenhouse on a large scale, which is more targeted. In addition, since the QR code detection is embedded in the automatic navigation system in the form of a state machine, when the QR code is detected, it will give navigation feedback, and the information collected after stopping makes the image quality higher, without blurring or ghosting.
[0138] 4. Good human-machine interaction performance. The integrated intelligent voice system reduces the difficulty of robot operation and the threshold of use, and improves the control efficiency of the staff. Even if you are not a navigation-related professional, you can easily complete the control of the robot and help realize the automatic inspection of crops in the greenhouse. The function of reporting the inspection progress can also help the staff to check the inspection situation, which is conducive to human-machine interaction.
[0139] In an exemplary embodiment, the present application further provides a computer device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.
[0140] In an exemplary embodiment, the present application further provides a computer-readable storage medium storing a computer program, which implements the steps in the above-mentioned method embodiments when executed by a processor.
[0141] In an exemplary embodiment, the present application further provides a computer program product, including a computer program, which implements the steps in the above-mentioned method embodiments when executed by a processor.
[0142] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant legal provisions.
[0143] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).
[0144] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.
[0145] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0146] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.
Claims
1. A greenhouse robot automatic inspection method, characterized in that: The greenhouse robot automatic inspection method is applied to a greenhouse robot automatic inspection system, which includes: a robot and a plurality of crop tags; the crop tags are respectively located next to crops of different varieties in a greenhouse scene and record the variety information of the crops; the robot is equipped with an RGB camera, a multi-spectral camera, a wheel encoder, an inertial measurement unit and a laser radar, the RGB camera and the multi-spectral camera are used to collect crop data, and the wheel encoder, the inertial measurement unit and the laser radar are used to determine the global position point of the robot; The automatic inspection method of the greenhouse robot comprises: Control the robot to walk in the greenhouse scene, and use the real-time positioning and map building algorithm to build a map of the greenhouse scene to obtain a global map; Control the robot to walk along the inspection path in the greenhouse scene, and use the fusion positioning algorithm to update and record the global posture points of the robot during the walking process to obtain the teaching path; A finite state machine is established according to the inspection task of the robot; the finite state machine is used to define the state and state transition rules of the robot; The robot automatically performs inspection tasks based on the finite state machine according to the global map and the taught path.
2. The automatic inspection method of a greenhouse robot according to claim 1, characterized in that: The instant positioning and mapping algorithm is an improved tightly coupled laser SLAM algorithm; the improved tightly coupled laser SLAM algorithm is based on the FAST-LIO2 architecture, adds ground constraints in the residual calculation process, and adds stable triangle descriptor loop detection.
3. The automatic inspection method of a greenhouse robot according to claim 2, characterized in that: The fusion positioning algorithm includes local positioning and global positioning; the local positioning uses a Kalman filter to fuse the data of the inertial measurement unit and the wheel encoder to provide a coordinate transformation relationship from the odometer coordinate system to the robot coordinate system; the global positioning uses the front end of the improved FAST-LIO2 architecture to fuse the data of the inertial measurement unit and the lidar to provide a coordinate transformation relationship from the map coordinate system to the odometer coordinate system; The fusion positioning algorithm determines the coordinate transformation relationship from the robot coordinate system to the map coordinate system according to the local positioning and the global positioning, thereby updating and recording the global posture points of the robot during walking.
4. The automatic inspection method of a greenhouse robot according to claim 3, characterized in that: A repositioning module is embedded in the fusion positioning algorithm; the repositioning module uses an iterative nearest point cloud registration algorithm to search for the best posture within a range of 1 meter and 30 degrees of a given posture point; the given posture point includes an initial posture point and a current posture point at each set interval, and each posture point includes position and orientation information; the 1 meter and 30 degree range is a 1×1 square meter range of a given position and a 30 degree range of a given orientation.
5. The automatic inspection method of a greenhouse robot according to claim 1, characterized in that: The robot's states include: idle state, record state, run state, pause state, photo state, goto state, follow state and wait state; the robot's state transition rules include: When the robot is idle, it is in idle state; In the idle state: when the command to start recording is given, the robot enters the record state and records the teaching path; when the command to execute the teaching path is given, the robot enters the run state and performs the inspection task; In the record state, the robot obtains the current position and pose points and writes a series of position and pose points into a file; when receiving the command to end recording, the robot returns to the idle state; In the run state: when a pause task instruction is given, the robot enters the pause state; when a task end instruction is given or the task is successfully executed, the robot enters the idle state; when a crop tag is detected, the robot enters the photo state; the goto state is also nested in the run state; when the robot is in the run state, it enters the goto state by default, and executes the global path planning algorithm and obstacle avoidance control algorithm for point-to-point path planning; In the pause state: when given an instruction to continue execution, the robot returns to the run state and continues to perform the inspection task; when given an instruction to end the task, the robot enters the idle state; In the photo state: the robot uses the RGB camera and multispectral camera to collect crop data, and returns to the run state at set intervals to continue the inspection task; In the goto state: when the robot reaches the starting point of the teaching path, it enters the follow state and automatically follows the teaching path; In the follow state: when the robot encounters an obstacle, it enters the wait state; In the wait state: when the obstacle is removed within the waiting time, it enters the follow state and continues to track the taught path; when the obstacle is not removed within the waiting time, it enters the goto state, searches for subsequent points without obstacles, and navigates to the next point without obstacles.
6. The automatic inspection method of a greenhouse robot according to claim 5, characterized in that: The global path planning algorithm adopts a teaching path method; the obstacle avoidance control algorithm adopts a time elastic band control algorithm.
7. The automatic inspection method of a greenhouse robot according to claim 1, characterized in that: The crop tag is an April tag QR code tag.
8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the automatic inspection method for a greenhouse robot according to any one of claims 1 to 7.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the automatic inspection method of a robot in a greenhouse according to any one of claims 1 to 7 is implemented.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the automatic inspection method of a robot in a greenhouse according to any one of claims 1 to 7 is implemented.
Citation Information
Patent Citations
Athletic field patrol intelligent robot based on Internet of things
CN106493730A
Greenhouse management system based on mobile robot
CN111679666A
Agricultural robot positioning and navigation method and system based on visual inspection
CN115235470A
Laser SLAM mapping method for greenhouse inspection robot
CN115294287A
Positioning method and mapping method for orchard mobile robot and electronic equipment
CN116660916A
Cited By
Body unmanned aerial vehicle water conservancy inspection method and system based on state machine
CN120980190A
