A greenhouse robot automatic inspection method, device, medium and product

Abstract

The application discloses a greenhouse robot automatic inspection method and device, medium and product, and relates to the technical field of robot inspection. The method comprises the following steps: controlling a robot to walk in a greenhouse scene once, and adopting a real-time positioning and map construction algorithm to construct a map of the greenhouse scene, so as to obtain a global map; controlling the robot to walk along an inspection path in the greenhouse scene, and adopting a fusion positioning algorithm to update and record global pose points in the walking process of the robot, so as to obtain a teaching path; establishing a finite state machine according to an inspection task of the robot; the finite state machine is used for defining the state of the robot and a state transition rule; and the robot automatically executes the inspection task based on the finite state machine according to the global map and the teaching path. The greenhouse robot automatic inspection method, device, medium and product provided by the application can realize automatic navigation of the robot in the greenhouse and fully automatic acquisition of crop data, and improve the navigation precision and the efficiency of crop data acquisition.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot inspection, in particular to a greenhouse robot automatic inspection method, device, medium and product. BACKGROUND

[0002] Statistical data shows that the total area of facility horticulture in China in 2016 was 476.5 million mu, and the facility types included plastic small sheds, plastic large and medium-sized sheds, sunlight greenhouses, heated greenhouses, and continuous-span greenhouses, with plastic greenhouses accounting for 44%. Greenhouses not only provide a controlled growing environment for agricultural production, allowing crops to be free from the constraints of seasons and weather, but also provide ideal conditions for creating high-quality and high-yield crops. With the continuous growth of the global population and the increasing demand for superior quality agricultural products, the great potential of greenhouse agriculture has been widely recognized. This modern agricultural approach not only ensures the stability of food supply, but also plays a key role in improving the quality of agricultural products, reducing the use of pesticides and fertilizers, and increasing the sustainability of agricultural production. Therefore, greenhouse agriculture has significant strategic value in meeting the growing global demand for food and driving technological innovation in the agricultural sector.

[0003] Currently, the overall mechanization level of facility agriculture in China is only 32%, while the mechanization level of field grain crops can reach 70-80%. In the context of global agricultural labor shortages, the introduction of agricultural robots into greenhouse agriculture is one of the important ways to solve the problem of labor shortage. In addition, compared to the complex and variable field environment, the greenhouse environment, although relatively simple, still faces many challenges that need to be addressed due to high temperature and humidity, high plant density, and even the presence of harmful gases. Therefore, robots have broad application prospects in the inspection tasks in greenhouses and are expected to contribute to improving the efficiency of agricultural production, reducing costs, and providing a safer working environment.

[0004] Crop phenotyping and pest detection in agriculture face problems such as low efficiency, high labor intensity, and inaccurate data collection. Traditional manual investigation methods not only consume a lot of manpower and resources, but also are prone to human errors, resulting in inaccurate data. With the development of agricultural technology, there is an increasing demand for accurate assessment of crop growth conditions, so a high-efficiency, accurate, and automated solution is needed to replace traditional methods. The introduction of automation and intelligent technology can greatly improve the efficiency and accuracy of data collection, reduce labor costs and errors, and thus provide reliable support for crop breeding, precision management, and pest control.

[0005] In the greenhouse scene, due to the limited signal, the changeable environment and other factors, positioning and global path planning face more severe challenges. Positioning is a key step of automatic navigation and an important part of realizing autonomous operation of the robot. In the field scene, due to less signal shielding, the robot can be provided with accurate absolute positioning by relying on the global positioning system (GPS). In recent years, the simultaneous localization and mapping (SLAM) technology occupies an increasingly important position in the mapping and positioning of the greenhouse. With the development of sensor technology and three-dimensional reconstruction technology, the accuracy, robustness and use range of SLAM have been greatly improved. The SLAM technology can meet the high autonomy demand and the greenhouse scene without GPS signal, and provides an accurate mapping and reliable positioning solution for the agricultural robot. SUMMARY

[0006] The purpose of the present application is to provide a greenhouse robot automatic inspection method, device, medium and product, which can realize automatic navigation of the greenhouse robot and full-automatic acquisition of crop data, and improve the accuracy of navigation and the efficiency of crop data acquisition.

[0007] To achieve the above purpose, the present application provides the following solutions.

[0008] In a first aspect, the present application provides a greenhouse robot automatic inspection method, which is applied to a greenhouse robot automatic inspection system, the greenhouse robot automatic inspection system comprising a robot and a plurality of crop labels; the crop labels are respectively located beside crops of different varieties in a greenhouse scene, and record the variety information of the crops; the robot is loaded with an RGB camera, a multispectral camera, a wheel encoder, an inertial measurement unit and a laser radar; the RGB camera and the multispectral camera are used to acquire crop data, and the wheel encoder, the inertial measurement unit and the laser radar are used to determine the global pose point of the robot.

[0009] The greenhouse robot automatic inspection method comprises:

[0010] controlling the robot to walk through the greenhouse scene, and using a simultaneous localization and mapping algorithm to map the greenhouse scene and obtain a global map;

[0011] controlling the robot to walk along an inspection path in the greenhouse scene, and using a fusion positioning algorithm to update and record the global pose point of the robot during walking, and obtaining a teaching path;

[0012] establishing a finite state machine according to the inspection task of the robot; the finite state machine is used to define the state and state transition rule of the robot;

[0013] The robot automatically performs the inspection task according to the global map and the teaching path based on a finite state machine.

[0014] Optionally, the instant positioning and map building algorithm is an improved tight coupling laser SLAM algorithm; the improved tight coupling laser SLAM algorithm is based on a FAST-LIO2 architecture, adds ground constraints in a residual calculation process, and adds a stable triangle descriptor loop detection.

[0015] Optionally, the fusion positioning algorithm includes local positioning and global positioning; the local positioning uses a Kalman filter to fuse data of an inertial measurement unit and a wheel encoder, and provides a coordinate transformation relationship from a mileage coordinate system to a robot coordinate system; the global positioning uses a front end of an improved FAST-LIO2 architecture to fuse data of the inertial measurement unit and a laser radar, and provides a coordinate transformation relationship from a map coordinate system to the mileage coordinate system; the fusion positioning algorithm determines a coordinate transformation relationship from the robot coordinate system to the map coordinate system according to the local positioning and the global positioning, and realizes updating and recording of global pose points in a robot walking process.

[0016] Optionally, a repositioning module is embedded in the fusion positioning algorithm; the repositioning module uses an iterative closest point point cloud registration algorithm to search for an optimal pose within a 1-meter and 30-degree range of a given pose point; the given pose point includes an initial pose point and a current pose point at a set time interval, and each pose point includes position and orientation information; the 1-meter and 30-degree range is a 1x1 square meter range of a given position and a 30-degree range of a given orientation.

[0017] Optionally, the states of the robot include an idle state, a record state, a run state, a pause state, a photo state, a goto state, a follow state, and a wait state; and the state transition rules of the robot include:

[0018] When the robot is idle, the idle state is entered;

[0019] In the idle state: when a start recording instruction is given, the robot enters the record state and records the teaching path; when an execute teaching path instruction is given, the robot enters the run state and performs the inspection task;

[0020] In the record state, the robot obtains a current pose point and writes a series of pose points into a file; when an end recording instruction is received, the robot returns to the idle state;

[0021] In the run state: when a pause task instruction is given, the robot enters the pause state; when an end task instruction is given or the task is executed successfully, the robot enters the idle state; when a crop label is detected, the robot enters the photo state; the run state is also nested in the goto state; when the robot is in the run state, it defaults to entering the goto state at the same time, executing the global path planning algorithm and obstacle avoidance control algorithm for point-to-point path planning;

[0022] In the pause state: when a continue execution instruction is given, the robot returns to the run state and continues to execute the inspection task; when an end task instruction is given, the robot enters the idle state;

[0023] In the photo state: the robot uses the RGB camera and the multispectral camera to collect crop data, and returns to the run state at a set interval to continue the inspection task;

[0024] In the goto state: when the robot reaches the starting point of the teaching path, it enters the follow state and automatically tracks the teaching path;

[0025] In the follow state: when the robot encounters an obstacle, it enters the wait state;

[0026] In the wait state: when the obstacle is removed within the waiting time, the robot enters the follow state and continues to track the teaching path; when the obstacle is not removed within the waiting time, the robot enters the goto state, searches for the next point without obstacles, and navigates to the next point without obstacles.

[0027] Optionally, the global path planning algorithm adopts a teaching path mode; and the obstacle avoidance control algorithm adopts a time elastic band control algorithm.

[0028] Optionally, the crop label is an April tag two-dimensional code label.

[0029] In a second aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the greenhouse robot automatic inspection method.

[0030] In a third aspect, the present application provides a computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the greenhouse robot automatic inspection method.

[0031] In a fourth aspect, the present application provides a computer program product comprising a computer program, wherein the computer program is executed by a processor to implement the greenhouse robot automatic inspection method.

[0032] According to specific embodiments provided in the present application, the present application discloses the following technical effects:

[0033] The present application provides a greenhouse robot automatic inspection method, device, medium and product. High-precision mapping of a greenhouse scene is realized through real-time positioning and map construction algorithms. Centimeter-level high-precision positioning in the greenhouse is realized through a fusion positioning algorithm. Automatic navigation of the greenhouse robot is realized through constructing a global map, recording a teaching path, and establishing a finite state machine. Full-automatic acquisition of crop data is realized by recording crop variety information through crop labels. Compared with the traditional manual inspection method, the present application uses a robot to automatically perform an inspection task, which does not need to consume a large amount of manpower and material resources, and is not prone to human errors, resulting in inaccurate data. Therefore, the greenhouse robot automatic inspection method, device, medium and product provided by the present application can realize automatic navigation of the greenhouse robot and full-automatic acquisition of crop data, and improve the navigation accuracy and the efficiency of crop data acquisition. BRIEF DESCRIPTION OF DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0035] Figure 1 A greenhouse robot automatic inspection method flowchart is provided for the present application.

[0036] Figure 2 A finite state machine schematic diagram is provided for the present application.

[0037] Figure 3 A crop label detection flowchart is provided for the present application.

[0038] Figure 4 A static layer schematic diagram of a costmap cost map is provided for the present application.

[0039] Figure 5 A complete schematic diagram of a costmap cost map is provided for the present application. DETAILED DESCRIPTION

[0040] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all the other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0041] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0042] In one exemplary embodiment, the present application provides a greenhouse robot automatic inspection method, which is applied to a greenhouse robot automatic inspection system. The system comprises a robot and a plurality of crop labels; the crop labels are respectively located beside different varieties of crops in the greenhouse scene, and record the variety information of the crops; the robot is equipped with an RGB camera, a multispectral camera, a wheel encoder, an inertial measurement unit and a laser radar, the RGB camera and the multispectral camera are used to collect crop data, and the wheel encoder, the inertial measurement unit and the laser radar are used to determine the global pose point of the robot. As shown in the figure, the method comprises the following steps 1 to 4. Among them: Figure 1

[0043] Step 1: control the robot to walk through the greenhouse scene once, and use the real-time positioning and map building algorithm to build a map of the greenhouse scene, and obtain a global map.

[0044] Preferably, the real-time positioning and map building algorithm is an improved tight coupling laser SLAM algorithm; the improved tight coupling laser SLAM algorithm is based on the FAST-LIO2 (Fast Direct LiDAR-inertial Odometry) architecture, adds ground constraints in the residual calculation process, and adds a stable triangle descriptor (STD) loop detection.

[0045] ​High-precision mapping of greenhouse scenes is achieved by improving the tight coupling laser SLAM algorithm, which adopts the FAST-LIO2 architecture, specifically including: 1. Obtain laser scanning points by using a solid-state laser radar, and obtain acceleration and angular velocity information by using an inertial measurement unit. 2. The acceleration and angular velocity information of the inertial measurement unit is forward propagated, and the initial pose point is integrated. 3. Laser scanning points are backward propagated to remove motion distortion caused by acquisition at different times. 4. Calculate the residual of the laser odometry, estimate the pose transformation using iterative Kalman filtering until convergence. 5. According to the pose estimated by the odometry in the last step, construct a map and update the global map maintained by the incremental k-dimensional tree (ikd-tree) data structure. The optimization of the above process is as follows:

[0046] Optimization 1: Through the Precision Timing Protocol (PTP) network measurement and control system of the master-slave end (master: clock device of industrial computer, slave: solid-state laser radar sensor device), the IEEE1588v2.0 PTP standard time synchronization is ensured, and the timestamps of the inertial measurement unit data and the laser radar scanning data are correct. Avoid the problems of map overlap and odometry flicker in the mapping process.

[0047] Optimization 2: Ground constraints are added to the original SLAM framework (i.e. FAST-L1O2 architecture) backend. The ground constraint is introduced from three aspects of ground residual term representation, calculation and optimization.

[0048] Representation of ground residual term:

[0049] Initialize the global ground plane equation in the world coordinate system as:

[0050] A w x+B w y+C w z+D w =0

[0051] The parameters are represented as θ w : (A w , B w , C w , D w ). Among them, for the initialized global ground plane parameters A w = 0, B w = 0, C w = 1, D w = 0.

[0052] Assuming that the laser subgraph obtained in the current frame is the kth subgraph, the ground equation of the current frame radar point cloud (kth frame) is:

[0053] A k x+Bk y + C k z + D k = 0

[0054] Parameters are represented as θ k : (A k , B k , C k , D k ).

[0055] The unity of the plane coordinate system: convert the global ground equation under the initialized world coordinate system to the current k-th frame radar coordinate system, and the parameters are represented as θ' w : (A' w , B' w , C' w , D' w ), and the parameter conversion follows the following formula:

[0056]

[0057] where T kw represents the pose transformation from the k-th frame radar coordinate system to the world coordinate system, and T wk represents the pose transformation from the world coordinate system to the k-th frame radar coordinate system, i.e. the pose of the current laser radar, which is known.

[0058] Calculation of ground residual term:

[0059] The first three dimensions A' w , B' w , C' w (normal vector) of the θ' w parameter are used as the base to construct a three-dimensional rotation space. The first three dimensions A k , B k , C k (normal vector) of θ k are converted into this constructed rotation space to obtain the vector n: (A kn , B kn , C kn ), and then the rotation aspect residual can be represented as:

[0060]

[0061] The translation aspect residual can be represented as the distance of the plane intercept d = D' w - D k .

[0062] Therefore, the final plane residual term of the current k-th frame is:

[0063] E k = (α, β, d)

[0064] Optimization of the ground residual term:

[0065] Put the residual term into the iterative error state Kalman filter after getting the ground residual term, optimize the error, and improve the pose estimation and mapping accuracy.

[0066] Prediction of the error state quantity of the filter:

[0067]

[0068] where, represents the prior error state estimate at time k, F ek-1 represents the state transition matrix at time k-1, represents the posterior error state estimate at time k-1, B ek- 1 represents the noise transition matrix at time k-1, W ek represents the measurement noise at time k.

[0069] Covariance prediction corresponding to the error state quantity:

[0070]

[0071] where, represents the prior error covariance at time k, G represents the posterior error covariance at time k-1.

[0072] Kalman gain calculation:

[0073]

[0074] where, K ek represents the Kalman gain at time k, G ek represents the conversion matrix of the error state variable to the observation, which connects the relationship between the state and the observation. R ek represents the error measurement noise covariance at time k.

[0075] Error state update, calculate the posterior estimate:

[0076]

[0077] where, represents the posterior error covariance at time k, Y represents the posterior error state estimate at time k, Y ek represents the error measurement value (observation value).

[0078] Optimization 3: Add STD loop detection in SLAM, avoid ghosting problem in global map after multiple scans of the same location by detecting stable triangle descriptors, start and end points are optimization problems of loop case, z-axis drift problem.

[0079] The implementation steps of STD loop detection are as follows:

[0080] 1. Get the point cloud key frame. The key frame is accumulated by a period of scanning points.

[0081] 2. Plane detection. Use region growing algorithm to distinguish plane voxels and boundary voxels.

[0082] 3. Key point selection. Project the boundary voxels onto the plane, and create an image overlapping the plane for each plane. Each pixel represents the maximum distance from the point in the boundary voxel to the plane. For a point, select the point with the maximum pixel value in its 5x5 neighborhood as the key point.

[0083] 4. Construct a k-d number structure with all key points, search for 20 nearest key points in the neighborhood of each key point, and form a triangle descriptor.

[0084] 5. Store all triangle descriptors in the form of hash table, the key value is the dot product of the edge length and the normal vector projection vector of each descriptor (a total of six). Calculate the key value of each descriptor of the current frame and match it in the hash table, if the key value of the descriptor is the same as that of the previous key frame, give the corresponding key frame a vote. The top 10 key frames with votes will be loop candidates.

[0085] 6. Geometric verification. Calculate the relative transformation T=(R, t) between two key frames by Singular Value Decomposition (SVD) using the correspondence of descriptor vertices between key frames. Assume that there are triangle a and triangle b descriptors on two key frames respectively, and the key values of the two triangles are the same. Then, by SVD decomposition, we get:

[0086]

[0087] [U, S, V] = SVD(J)

[0088] R = VUT ， t = -R*c a +c b

[0089] where, are the three vertices of triangle a and triangle b, c a , c bThe centers of the triangles a and b are calculated. Finally, the rotation matrix R and the translation vector t between two key frames can be obtained after decomposition. In addition, the RANSAC algorithm is used to find the maximum correct matching descriptor transformation. Based on the best change, the number of matched planes is calculated and compared with the previously set hyperparameters. If the number of matched planes is more than the hyperparameters, it is considered that there is a loop between the key frame and the current frame through geometric verification.

[0090] Step 2: Control the robot to walk along the inspection path in the greenhouse scene, and update and record the global pose points of the robot during walking by using the fusion positioning algorithm to obtain the teaching path.

[0091] Preferably, the fusion positioning algorithm comprises local positioning and global positioning; the local positioning uses a Kalman filter to fuse the data of an inertial measurement unit and a wheel encoder, and provides a coordinate transformation relationship from a odometer coordinate system to a robot coordinate system; the global positioning uses a front end of an improved FAST-L102 architecture to fuse the data of the inertial measurement unit and a lidar, and provides a coordinate transformation relationship from a map coordinate system to the odometer coordinate system; the fusion positioning algorithm determines a coordinate transformation relationship from the robot coordinate system to the map coordinate system according to the local positioning and the global positioning, and realizes updating and recording of global pose points of the robot during walking. In addition, a repositioning module is embedded in the fusion positioning algorithm, which is used to determine a starting pose and eliminate cumulative errors.

[0092] The repositioning module uses an Iterative Closest Point (ICP) point cloud registration algorithm to search for the best pose within a 1-meter and 30-degree range of a given pose point; the given pose point includes an initial pose point and a current pose point every interval of a set time, and each pose point includes position and orientation information; the 1-meter and 30-degree range is a 1x1 square meter range of a given position and a 30-degree range of a given orientation.

[0093] The fusion positioning algorithm is used to achieve high-precision positioning in the greenhouse. The local positioning uses a Kalman filter to fuse the data of the inertial measurement unit and the wheel odometer, providing high-frequency (100HZ) low-precision positioning information and the transformation relationship between the odom and base_link coordinate systems. The global positioning uses the improved FAST-LIO2 architecture front-end, providing low-frequency (10HZ) high-precision positioning information and the coordinate transformation relationship between the map and odom. Among them, map: map coordinate system, fixed global coordinate system; odom: odometer coordinate system, virtual coordinate system obtained by transforming the actual calculated coordinates and the true coordinates; base_link: robot coordinate system, representing the robot body, which is the center of the robot. The multi-sensor fusion positioning method solves the problem of low precision and low robustness of single positioning method. The specific fusion positioning is to fuse the data of wheel encoder, inertial measurement unit and laser radar multiple sensors to achieve fusion positioning. First, the local positioning fuses the data of the inertial measurement unit and the wheel encoder, and puts the orientation, acceleration, angular velocity of the inertial measurement unit and the position, orientation, linear velocity, angular velocity of the wheel encoder into the extended Kalman filter. After filtering, the local pose information is output, which includes position and orientation information. This pose information provides the coordinate transformation relationship between odom and base_link coordinate systems, with a frequency of 100HZ. The global positioning fuses the data of the inertial measurement unit and the laser radar. The specific fusion method uses the improved FAST-LIO2 architecture front-end odometer part. This algorithm outputs the pose transformation relationship, provides the coordinate transformation relationship between map and odom, and provides a frequency of 10HZ. Finally, through the above two coordinate transformation relationships, the coordinate transformation relationship of map->odom->base_link is provided, and the relationship between map and base_link is the coordinate relationship between the robot body and the global map. In this way, the robot's pose in the global map can be obtained, and accurate positioning can be achieved. The repositioning module specifically uses the ICP point cloud registration algorithm to search for the best pose within 1 meter and 30 degrees of the specified initial pose. It helps the robot to find the correct initial position after restarting or offline. The repositioning is given once before the robot starts the inspection after building the global map. Specifically, the approximate pose of the robot is given, including the current position and orientation information in the global map. The repositioning searches for the best position based on the position and orientation information given by the user. Specifically, it searches for the best position within a 1x1 square meter range of the user's given position, and searches for the best orientation within a 30-degree range of the user's given orientation. Finally, the best pose is obtained based on the ICP point cloud registration algorithm.In addition to the start of the inspection before, in the robot positioning process every 10 s will also be repositioned, this time in the process of repositioning, the robot will automatically get their own in the global map pose and use the repositioning module in 1x1 square meter and 30 degrees range to match the best pose.

[0094] Step 3: Establish a finite state machine according to the robot's inspection task.

[0095] Finite state machine can clearly represent the state of the system and the transition relationship between states by defining clear states and state transition rules. This structured representation method makes the design, analysis and implementation of the algorithm more intuitive and controllable. In addition, finite state machine also facilitates the verification and debugging of system behavior, ensuring the correctness and stability of the algorithm under various conditions.

[0096] For example, Figure 2As shown, the state of the robot includes: idle state, record state, run state, pause state, photo state, goto state, follow state and wait state. The state transition rule of the robot includes: when the robot is idle, the idle state; in the idle state: when the instruction of starting recording is given, the robot enters the record state to record the teaching path; when the instruction of executing the teaching path is given, the robot enters the run state to execute the inspection task; in the record state, the robot obtains the current pose point and writes a series of pose points into a file; when the instruction of ending recording is received, the robot returns to the idle state; in the run state: when the instruction of pausing the task is given, the robot (stops running) enters the pause state; when the instruction of ending the task or the task is executed successfully, the robot enters the idle state; when the crop label is detected, the robot enters the photo state (starts taking pictures); the run state is also nested in the goto state; when the robot is in the run state, it defaults to enter the goto state at the same time to execute the global path planning algorithm and the obstacle avoidance control algorithm for point-to-point path planning; in the pause state: when the instruction of continuing execution is given, the robot returns to the run state to continue executing the inspection task; when the instruction of ending the task is given, the robot enters the idle state; in the photo state: the robot collects crop data using the RGB camera and the multispectral camera, and returns to the run state at a set interval to continue executing the inspection task; in the goto state: when the robot reaches the starting point of the teaching path, it enters the follow state to automatically track the teaching path (global path); in the follow state: when the robot encounters an obstacle, it enters the wait state; in the wait state: when the obstacle is removed within the waiting time, it enters the follow state to continue tracking the teaching path; when the obstacle is not removed within the waiting time, it enters the goto state to search for a point on the global path that has no subsequent obstacles and navigate to the next point with no obstacles.

[0097] Preferably, the crop label is an April tag two-dimensional code label, which is a visual fiducial library (similar to two-dimensional code, but the structure is simpler, less information is transmitted, but real-time is strong). By placing an April tag two-dimensional code label near the crop that needs to collect information, the precise positioning of the target plant and the automatic collection of data are achieved. The robot will use the camera to take pictures constantly during the operation, and the April tag detection is carried out, the main steps of which include: 1. According to the gradient in the image, the image edge is extracted. 2. Quadrilateral detection. First, the straight edge in the image is detected, and the quadrilateral is judged according to the adjacent edge of the straight line. 3. Encoding and decoding. The dot matrix is generated inside the quadrilateral to calculate the color block value, and the local binary pattern is used for classification to obtain the code. Decoding is realized by matching the code in the library. The RGB camera is carried on the robot, and the two-dimensional code is detected constantly during automatic navigation. When the two-dimensional code is detected, the variety information of the crop is recorded, and after waiting for stable stop, the RGB image and multispectral image of the current target plant are taken. These information can be used for subsequent phenotype or pest detection. The main functions of the two-dimensional code are three: 1. Help the robot locate the target plant that needs to be taken, realize accurate data collection, and avoid a large amount of useless data acquisition. 2. Improve data acquisition quality. If the two-dimensional code is detected, the robot will stop running, and wait for the acquisition of RGB and multispectral data. Image blur, ghosting and other problems caused by robot movement can be avoided. 3. Record the variety information. Breeders usually face hundreds or even thousands of varieties, so it is very important to accurately record different varieties. The two-dimensional code can provide variety information to help breeders manage. The crop label detection process is shown in Figure 3 , including: 1. Start detection. The robot will constantly detect the two-dimensional code during the inspection process. 2. If no two-dimensional code is detected, go back to step 1, otherwise go to the next step. 3. Get id. The id information of the two-dimensional code records all the crop varieties, helping breeders identify and manage different crop varieties. 4. Check if the list exists. In one inspection process, the varieties that have collected data will be put into the list, and if there are repeated varieties, they will be recorded in the list. At this time, this two-dimensional code will be ignored, and the detection will continue to prevent repeated data acquisition. If there is no list, it means it is a new variety, then go to the next step. 5. Wait two seconds and take a picture. Wait two seconds to make the robot stable and ensure data acquisition quality. RGB and multispectral data can be obtained by taking a picture.

[0098] Preferably, the global path planning algorithm adopts the method of teaching path. The operator only needs to remotely control the robot to run the path that needs to be run before use, and the specific path is determined by the inspection demand. During the recording process, the robot will record the current global pose, mainly recording the conversion relationship between the map and the base_link coordinate system. This conversion serves as the global pose of the robot during movement.

[0099] Preferably, the obstacle avoidance control algorithm adopts the Time Elastic Band (TEB) control algorithm. The TEB algorithm is based on costmap implementation. The costmap is composed of three layers, namely the static layer, the obstacle layer and the inflation layer. Among them, the static layer is obtained based on the global map, and is specifically composed of a two-dimensional grid map obtained by statistical filtering and straight-through filtering of the global map, that is, the most basic layer, representing the distribution of static obstacles, including crop ridges, fixed instruments, etc. The obstacle layer is obtained in real time during the inspection of the robot, and the distance of the dynamic obstacle is obtained through laser scanning points and superimposed on the static layer, which can better reflect the real-time obstacles in the inspection process. The inflation layer is superimposed on the static layer and the obstacle layer. Because the robot needs to leave a certain margin when avoiding obstacles, the range of the obstacle needs to be inflated to ensure that the obstacle can be avoided more safely. In this embodiment, the static map is shown in Figure 4 , and the black color represents the static obstacle detected in the previous map. The complete costmap cost map is shown in Figure 5 , and the color represents the cost. If the color is reddish, the cost is higher, the blue color represents the lower cost, and the white color represents the cost of 0 which can be passed.

[0100] The implementation principle and steps of the TEB algorithm are as follows:

[0101] 1. Path initialization

[0102] First, the path sequence composed of a series of pose points and time information needs to be determined. Assuming that there are n pose points on the path, the path sequence includes the following two parts:

[0103] X={x i} i＝0,1,…,n

[0104] τ={ΔT i} i＝0,1,…,n-1

[0105] Among them, X represents a series of pose point sets, x i is the i-th pose point on the path. τ represents the time sequence set on the path. ΔT i represents the time required for the robot to move from the i-th pose point on the path to the i+1-th pose point.

[0106] Based on the above representation, the path can be represented as a series of pose points and time sequence set:

[0107] B:=(X,τ)

[0108] Where B is the complete representation of the path, and thus the initialization of the path is completed.

[0109] 2. Construction of the cost function

[0110] Assuming that there are k optimization objectives in total, the total cost function can be represented as:

[0111]

[0112] Where f j (B) represents the jth optimization objective function, γ j represents the coefficient weight of the jth objective function, representing the importance of this optimization objective. f(B) represents the total cost function.

[0113] The optimization objectives mainly include the following:

[0114] (1) Following the path and obstacle avoidance constraints

[0115] The planned path needs to be consistent with the global path as much as possible, and the farther it deviates from the global path, the higher the cost. The planned path needs to be far away from obstacles, and the closer it is to obstacles, the higher the cost. The objective function is constructed through these two objectives.

[0116] (2) Velocity and acceleration constraints

[0117] Due to different mechanical parameters or different requirements of users for specific application scenarios, robots have requirements for speed and acceleration. According to specific requirements, upper and lower limits can be imposed on the speed and acceleration of robot motion to ensure that the motion of the robot is limited during the inspection process.

[0118] (3) Kinematic constraints

[0119] Since the robot adopts a two-wheel differential kinematic model, it can only move along the current orientation or rotate, so there will also be constraints on the algorithm in terms of motion form.

[0120] (4) Time constraints

[0121] The motion speed of the robot is required to be fast to improve the running speed. Therefore, the time spent on the path should be minimized, so that ΔT i is as small as possible.

[0122] 3. Minimization of the total cost function

[0123] The gradient descent method is used to minimize the cost function, which is to balance the minimization of the cost of achieving multiple optimization objectives.

[0124] B * = arg B minf(B)

[0125] where B * represents the minimum value of the total cost function.

[0126] 4. Calculate the specific control amount, update the motion state

[0127] The parameters calculated above to minimize the total cost function are extracted and converted into specific control amounts (linear velocity and angular velocity), thereby controlling the robot system to achieve real-time update of the motion state.

[0128] Step 4: The robot automatically performs the inspection task based on the finite state machine according to the global map and the teaching path.

[0129] Further, the present application also includes: voice interaction module development, integrated intelligent voice control system. Using the six-mic ring microphone array and offline speech recognition library of the University of Science and Technology of China to realize the voice wake-up, one-key inspection and inspection progress report of the robot. Through the design of these three functions, users can easily operate the device through natural language instructions without the need for tedious manual input or specific technical training, thereby significantly improving the human-computer interaction performance of the robot. The six-mic ring microphone array can collect voice information of the user and play the role of audio preprocessing and noise reduction. Since it is a ring array, it can also distinguish the position of the sound source. The role of the offline speech recognition library is to recognize the voice information collected by the microphone ring array in the offline state, convert the audio spoken by the user into a string for subsequent instruction recognition, and facilitate the robot to make corresponding actions according to the user's speech. The implementation of voice wake-up: after the microphone is powered on, it will continuously perform audio detection, and when the collected audio matches the user's set keyword, it will be woken up, at which time a prompt voice will be automatically played. The implementation of one-key inspection: after the microphone is woken up, it will continue to recognize the voice, and when the start inspection voice is received, the subsequent program will automatically issue instructions to the robot to execute the inspection according to the pre-set global path. At this time, there is no need for tedious operation, and the robot automatically enters the inspection state. The implementation of inspection progress reporting: list is used in the flowchart of detecting two-dimensional codes, which will put the recognized two-dimensional code variety id information into the list. When the robot detects the inspection progress reporting, it will check the number of list libraries in the navigation system, and then report the total number, so that the user can know how many varieties of data have been collected.

[0130] In summary, the present application adopts the following scheme:

[0131] 1. Remote control robot walks through the greenhouse scene that needs to be automatically rebuilt. In this process, it is not necessary to walk through the whole scene, but only to rebuild the whole scene. The improved FAST-LIO2 tightly coupled laser SLAM algorithm is used. In this process, the greenhouse high-precision map (global map) is gradually established. In addition, statistical filtering and straight-through filtering are used to compress the global map into two dimensions, and finally a two-dimensional grid map is completed. It provides a basis for subsequent positioning, recording of teaching path and obstacle avoidance control.

[0132] 2. Positioning and recording teaching path. After establishing the global map, repositioning is needed, that is, the initial positioning is given. Then the remote control robot walks a desired path, during which the robot continuously records its global pose points and writes them into a file. This path is the teaching path, which will be used as the global path during the inspection process. Before recording the teaching path, the fusion positioning module needs to be turned on and the initial pose of the robot needs to be given. When the robot is in the correct position, the remote control robot can record the corresponding path. Different inspection paths can be recorded for different inspection needs in the scene, and the corresponding paths can be saved after recording. For different inspection tasks in the scene, such as the varieties that have been eliminated in the previous inspection process, the growth of these varieties does not need to be concerned in the subsequent inspection, and the robot may not go to the corresponding crop ridge in the future. In this case, the path will change, and using another path can improve efficiency and prevent the robot from inspecting the eliminated varieties.

[0133] 3. Perform the inspection task. Use voice control to start, pause and report the inspection progress of the robot. Help the operator control the robot and keep track of the robot's inspection progress in real time. The final RGB image and multispectral image can be saved locally or uploaded to the cloud for subsequent phenotype or pest detection.

[0134] Compared with the prior art, the application has the following advantages:

[0135] 1. Accurate navigation. The robot can smoothly pass through and change lanes between narrow ridges. Thanks to high-precision mapping and positioning, the difference between the actual path of the robot and the teaching path is centimeter-level. Even without high-precision GPS signals, the robot can achieve high-precision navigation in the greenhouse.

[0136] 2. Multi-scene greenhouse inspection operation. Compared with deep learning-based navigation line extraction, this navigation method does not need to collect a large amount of data set, but only needs to do simple exploration in the early stage. Compared with beacons, guide rails, color bands and magnetic strips, it does not need to be additionally arranged for multiple scenes, reducing equipment and maintenance costs. Therefore, this navigation method can be applied in greenhouse scenes with different planting methods without additional arrangement.

[0137] 3. High data collection quality and efficiency. The target plant is located by using April tag, so that the target plant image can be efficiently obtained, instead of collecting information of all plants in the greenhouse, which is more targeted. In addition, since the two-dimensional code detection is embedded in the automatic navigation system in the form of a state machine, the navigation is fed back when the two-dimensional code is detected, and the information is collected after stopping, so that the image quality is higher, and the phenomena of blur and ghosting do not occur.

[0138] 4. Good human-computer interaction performance. The integrated intelligent voice system reduces the operation difficulty and use threshold of the robot, and improves the control efficiency of the staff. Even if it is not a navigation-related professional, the control of the robot can be easily completed, helping to realize the automatic inspection of crops in the greenhouse. The function of reporting the inspection progress can also help the staff to check the inspection situation, which is conducive to human-computer interaction.

[0139] In an exemplary embodiment, the present application also provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0140] In an exemplary embodiment, the present application also provides a computer readable storage medium storing a computer program, wherein the computer program is executed by a processor to implement the steps in the above method embodiments.

[0141] In an exemplary embodiment, the present application also provides a computer program product comprising a computer program, wherein the computer program is executed by a processor to implement the steps in the above method embodiments.

[0142] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant legal regulations.

[0143] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0144] The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a blockchain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0145] The technical features of the above embodiments can be combined arbitrarily. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combinations of the technical features do not exist contradictory, they should be considered as the scope of the present application.

[0146] The principles and implementation manners of the present application are described herein by using specific examples, and the above examples are only used to help understand the method of the present application and its core idea; meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manners and application ranges will have changes. In conclusion, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A method for automatically inspecting a greenhouse by a robot, characterized in that, The greenhouse robot automatic inspection method is applied to a greenhouse robot automatic inspection system, the greenhouse robot automatic inspection system comprising: a robot and a plurality of crop labels; the crop labels are respectively located beside crops of different varieties in a greenhouse scene, and record the variety information of the crops; the robot is provided with an RGB camera, a multispectral camera, a wheel encoder, an inertial measurement unit and a laser radar, the RGB camera and the multispectral camera are used to collect crop data, and the wheel encoder, the inertial measurement unit and the laser radar are used to determine the global pose point of the robot; The greenhouse robot automatic inspection method comprises: controlling the robot to walk through the greenhouse scene, and using a simultaneous localization and mapping algorithm to map the greenhouse scene to obtain a global map; controlling the robot to walk along the inspection path in the greenhouse scene, and using a fusion positioning algorithm to update and record the global pose point of the robot during walking to obtain a teaching path; establishing a finite state machine according to the inspection task of the robot; the finite state machine is used to define the state and state transition rule of the robot; the robot automatically executes the inspection task based on the global map and the teaching path according to the finite state machine; the simultaneous localization and mapping algorithm is an improved tightly coupled laser SLAM algorithm; the improved tightly coupled laser SLAM algorithm is based on the FAST-LIO2 architecture, adds ground constraints in the residual calculation process, and adds a stable triangle descriptor loop detection; the fusion positioning algorithm comprises local positioning and global positioning; the local positioning uses a Kalman filter to fuse the data of the inertial measurement unit and the wheel encoder, and provides a coordinate transformation relationship from the odometer coordinate system to the robot coordinate system; the global positioning uses the front end of the improved FAST-LIO2 architecture to fuse the data of the inertial measurement unit and the laser radar, and provides a coordinate transformation relationship from the map coordinate system to the odometer coordinate system; the fusion positioning algorithm determines the coordinate transformation relationship from the robot coordinate system to the map coordinate system according to the local positioning and the global positioning, and realizes the update and recording of the global pose point of the robot during walking; a repositioning module is embedded in the fusion positioning algorithm; the repositioning module uses an iterative closest point point cloud registration algorithm to search for the best pose within 1 meter and 30 degrees of a given pose point; the given pose point includes an initial pose point and a current pose point every interval set time, and each pose point includes position and orientation information; the 1 meter and 30 degree range is a 1x1 square meter range of the given position and a 30 degree range of the given orientation.

2. The method of claim 1, wherein, The states of the robot include: idle state, record state, run state, pause state, photo state, goto state, follow state and wait state; the state transition rule of the robot comprises: when the robot is idle, it is in idle state; In the idle state: when a start recording instruction is given, the robot enters the record state to record the teaching path; when an instruction to execute the teaching path is given, the robot enters the run state to execute the inspection task; In the record state, the robot acquires the current pose point and writes a series of pose points into a file; when an end recording instruction is received, the robot returns to the idle state; In the run state: when a task pausing instruction is given, the robot enters the pause state; when an end task instruction or the task is successfully executed, the robot enters the idle state; when a crop label is detected, the robot enters the photo state; the run state is also nested in the goto state; when the robot is in the run state, it enters the goto state by default, executes the global path planning algorithm and the obstacle avoidance control algorithm to plan a point-to-point path; In the pause state: when a continue execution instruction is given, the robot returns to the run state to continue the inspection task; when an end task instruction is given, the robot enters the idle state; In the photo state: the robot collects crop data using an RGB camera and a multispectral camera, and returns to the run state at a set interval to continue the inspection task; In the goto state: when the robot reaches the starting point of the teaching path, it enters the follow state to automatically track the teaching path; In the follow state: when the robot encounters an obstacle, it enters the wait state; In the wait state: when the obstacle is removed within the waiting time, the robot enters the follow state to continue tracking the teaching path; When the obstacle is not removed within the waiting time, the robot enters the goto state to search for a subsequent point without obstacles and navigate to the next point without obstacles.

3. The method of claim 2, wherein, The global path planning algorithm adopts a teaching path mode; the obstacle avoidance control algorithm adopts a time elastic band control algorithm.

4. The method of claim 1, wherein, The crop label is an April tag two-dimensional code label.

5. A computer device comprising: A memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that the processor executes the computer program to implement the greenhouse robot automatic inspection method of any one of claims 1-4.

6. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the greenhouse robot automatic inspection method of any one of claims 1-4.

7. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the greenhouse robot automatic inspection method of any one of claims 1-4.

Images