A method and device for monitoring tomato plant phenotypes based on a mobile robot

Abstract

The application provides a tomato plant phenotype monitoring method and device based on a movable robot. The monitoring device comprises a movable robot body, a monitoring module and a control module carried on the movable robot body. The movable robot body is used for movement. The control module is used for controlling the movable robot body to move according to a monitoring task sent by a base station. The monitoring module is used for implementing a monitoring item, judging whether a tomato plant has a pest condition, whether the growth of the tomato plant is good, and whether the tomato plant is pollinated. In application, the control module controls the movable robot body to move, and the monitoring module completes monitoring. For an abnormal plant, abnormal plant information is uploaded to the base station. The application can systematically, massively and efficiently detect crop phenotype information, can realize detection of crop pests, growth conditions and flowering conditions, reduces artificial cost, and provides efficiency.

Description

Technical Field

[0001] This invention relates to the field of robot monitoring technology, specifically to a method and device for monitoring the phenotypic characteristics of tomato plants based on a mobile robot. Background Technology

[0002] The growth, pest and disease status, and flowering of tomato plants are crucial indicators in tomato cultivation, as they foreshadow whether the tomatoes will ultimately produce high-quality fruit. Current technology primarily relies on human experience for assessment; however, this is extremely costly due to its labor-intensive nature. Other technologies include ground-based platforms mounted on fixed tracks, which offer high measurement accuracy but cannot be moved between locations, limiting their application. Unmanned aerial vehicles (UAVs) are also available, allowing for site changes, but they have limited payload capacity and short flight times.

[0003] Agricultural machinery platforms, such as tractors, have been widely used in field phenotyping studies due to their relatively low cost, ease of sensor system design, and ability to carry large payloads. However, their large size and wide tires make them unsuitable for planting patterns with small row spacing, their speed is difficult to control precisely, and most are fuel-powered, resulting in relatively large vibrations during operation, all of which affect measurement accuracy.

[0004] In recent years, hyperspectral imaging technology has been widely used in agricultural production, mainly for the rapid and accurate detection of crop phenotypic information, including the extraction of growth information, monitoring crop growth, and monitoring crop diseases and pests. However, because spectral imaging technology for crop indicator detection is often used in remote sensing monitoring, its relatively low spatial resolution limits its application in areas requiring high spatial resolution. Therefore, combining it with mobile robots for close-range monitoring of plants can better determine plant phenotypic characteristics.

[0005] In summary, current methods for monitoring tomato plant phenotypic information still suffer from high labor costs, low measurement accuracy, and insufficient monitoring capabilities. Summary of the Invention

[0006] To overcome the problems in the prior art, this invention provides a method that can systematically, extensively, and efficiently detect crop phenotypic information, enabling the detection of crop diseases and pests, growth status, and flowering conditions, and improving the quantity and quality of crop phenotypic information, which is an unsolved problem.

[0007] This invention first provides a tomato plant phenotypic monitoring device based on a mobile robot, which includes:

[0008] The mobile robot body, and the monitoring module and control module mounted on the mobile robot body;

[0009] The control module is used to control the movement or stopping of the mobile robot body according to the monitoring task sent by the base station;

[0010] The monitoring module includes:

[0011] A camera used to capture images of abnormal tomato plants;

[0012] Distance sensor module is used to detect the distance between the mobile robot body and the tomato plant;

[0013] A six-degree-of-freedom robotic arm, one end of which is connected to the mobile robot body, and the other end is equipped with a spectral imager, which is used to move the spectral imager.

[0014] A spectral imager is used to acquire hyperspectral images of tomato plants and transmit them to a signal processing module.

[0015] The signal processing module is used to process hyperspectral images to determine whether tomato plants have pests or diseases, whether they are growing well, and whether they have been pollinated.

[0016] The signal processing module contains a pre-trained pest and disease identification model, a plant growth model, and a flower pollination judgment model.

[0017] Among them, the pest and disease identification model is used to determine whether tomato plants are affected by pests and diseases, the plant growth model is used to determine whether tomato plants are growing well, and the flower pollination judgment model is used to identify flowers and determine whether the flowers are pollinated.

[0018] As a preferred embodiment of the present invention, the mobile robot body is equipped with a wheel drive motor, a battery module, and a signal transceiver module; the mobile robot is provided with several wheels for movement, the wheel drive motor is used to drive the wheels to roll, and the battery module is used to supply power to the monitoring module, the signal transceiver module, and the wheel drive motor; the signal transceiver module communicates with the base station wirelessly to receive control commands and upload data; the signal transceiver module and the monitoring module are connected via a signal line.

[0019] The present invention also provides a method for monitoring the phenotypic characteristics of tomato plants based on the aforementioned device, comprising the following steps:

[0020] 1) The mobile robot moves to the initial monitoring position and adjusts the components of the monitoring module to their initial state; after the reset is complete, the signal transceiver module on the mobile robot begins to receive monitoring tasks from the base station; the monitoring tasks include the monitoring route and monitoring items.

[0021] 2) After acquiring the monitoring task, the control module drives the mobile robot body to move according to the monitoring route. The tomato plants on the same row are planted at equal intervals, and the mobile robot body moves along a rectangle parallel to the row. When the monitoring of one row is completed, it moves to the next row along the set route. The control module controls the wheel drive motor to drive the wheels to roll a set distance to achieve movement between plants.

[0022] 3) When the mobile robot moves to the position corresponding to the tomato plant, the distance sensor module detects the distance between the mobile robot and the tomato plant. Based on the detected distance, the six-degree-of-freedom robotic arm drives the spectral imager to acquire a hyperspectral image of the plant.

[0023] The signal processing module calls the corresponding model according to the monitoring items corresponding to the monitoring task; the signal processing module first removes irrelevant background images and interference images, and extracts the target image; then it obtains the characteristic bands and hyperspectral images of different height parts of the tomato plant, inputs them into the corresponding model for prediction, and determines the severity of pests and diseases, plant growth, and flowering and pollination status.

[0024] 4) After monitoring one plant, the mobile robot moves to the monitoring position corresponding to the next plant and repeats step 3) until the monitoring task is completed. For plants with abnormalities, since the initial monitoring position is fixed and the travel route and wheel travel distance are known, the position corresponding to each plant is also determined. The camera takes pictures of the plant and transmits the plant images and the corresponding positions of the plants to the base station through the signal transceiver module.

[0025] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0026] (1) The present invention adopts mobile crop characterization, which overcomes the limitations of the plant phenotypic site in the prior art, thereby giving fuller play to the advantages of the robot's reusability and site transformation capabilities.

[0027] (2) The detection method and device proposed in this invention can detect crop phenotypic information over a large area with high efficiency, and can realize comprehensive detection of crop diseases and pests, plant growth, flowering and pollination, thereby improving crop productivity and production quality.

[0028] (3) The present invention provides a mobile robot that can pollinate in a timely manner and provide cultivation plans according to the on-site conditions, thus overcoming the comprehensive management problems in the prior art and achieving timeliness of plant growth status. Attached Figure Description

[0029] Figure 1 Side view of the mobile inspection robot;

[0030] Figure 2This is a schematic diagram of the mobile inspection robot structure;

[0031] Figure 3 This is a flowchart of the testing process;

[0032] Figure 4 This is a schematic diagram of the hyperspectral imager's detection output.

[0033] In the diagram, 1. Wheel drive motor, 2. Wheel, 3. Battery module, 4. Signal receiver, 5. Signal transmitter, 6. Camera, 7. Distance sensor module, 8. Six-DOF robotic arm, 9. Spectral imager. Detailed Implementation

[0034] The present invention will be further described and illustrated below with reference to specific embodiments. The embodiments described are merely examples of the content of this disclosure and do not limit the scope of the invention. The technical features of each embodiment in the present invention can be combined accordingly, provided that there is no mutual conflict.

[0035] like Figure 1 and 2 The diagram shows the structure of the tomato plant phenotypic monitoring device based on a mobile robot according to the present invention. The monitoring device of the present invention includes a mobile robot body, a monitoring module, and a control module. The present invention is based on a mobile robot body and utilizes unmanned driving technology to realize the robot's automatic movement; it uses high-definition image recognition technology to realize the real-time acquisition of plant characterization data (information on leaf and fruit diseases, pests, growth, nutritional status, etc.); and finally, it is combined with a power supply system to complete the driving of the entire mobile robot body.

[0036] Among them, such as Figure 1 As shown, the mobile robot body is equipped with a wheel drive motor 1, a battery module 3, and a signal transceiver module; the mobile robot is equipped with several wheels 2 for movement, the wheel drive motor 1 is used to drive the wheels 2 to roll, and the battery module 3 is used to supply power to the monitoring module, the signal transceiver module, and the wheel drive motor 1; the signal transceiver module communicates with the base station wirelessly to receive control commands and upload data; the signal transceiver module and the monitoring module are connected through a signal line.

[0037] The signal transceiver module includes a signal receiver 4 and a signal transmitter 5; the signal receiver and signal transmitter can communicate wirelessly with the base station via wireless communication methods such as Bluetooth, Wi-Fi, 4G / 5G.

[0038] The control module of this invention is used to control the movement or stopping of a mobile robot body according to the monitoring tasks sent by the base station. The monitoring tasks include a monitoring route and monitoring items, which include one or more of the following: monitoring pest and disease conditions, monitoring growth, and detecting pollination. Generally, during the pollination period, all three monitoring tasks will be selected simultaneously. When the plant is not flowering, monitoring pest and disease conditions and monitoring growth can be selected.

[0039] The monitoring module includes:

[0040] Camera 6 is used to capture images of abnormal tomato plants;

[0041] Distance sensor module 7 is used to detect the distance between the mobile robot body and the tomato plant;

[0042] A six-degree-of-freedom robotic arm 8 has one end connected to a movable robot body and the other end equipped with a spectral imager for moving the imager. This invention's six-degree-of-freedom robotic arm 8 is a five-axis robotic arm with a vertical lifting axis capable of lifting heights from 900mm+0 to 1800mm. It can rotate horizontally from -180° to +180°; and its vertical rotation axis can pitch from -75° to +75°.

[0043] The spectral imager 9 uses a hyperspectral imager to acquire hyperspectral images of tomato plants and transmit them to the signal processing module.

[0044] The signal processing module is used to process hyperspectral images to determine whether tomato plants are affected by pests and diseases, whether they are growing well, and whether they have been pollinated. The signal processing module contains a trained pest and disease recognition model, a plant growth model, and a flower pollination judgment model. The pest and disease recognition model is used to determine whether tomato plants are affected by pests and diseases, the plant growth model is used to determine whether tomato plants are growing well, and the flower pollination judgment model is used to identify flowers and determine whether they have been pollinated.

[0045] like Figure 3As shown, the hyperspectral imager primarily detects plant growth, pest and disease conditions, and pollination status. Plant growth includes the size of roots, stems, and leaves, fruit maturity, and flowering status. Excessive root and stem growth is considered etiolation and should be controlled with growth regulators. Poor growth requires more fertilizer and water. During the fruiting period, water and nutrient requirements are high; for higher quality, water and fertilizer can be appropriately controlled, but yield will be lower. However, excessive fertilizer and water will result in larger fruits, affecting taste. Therefore, hyperspectral imaging can identify fruit size and color, allowing for targeted treatment and solutions. Pest and disease conditions include leaf, root, and fruit pests and diseases, which can be addressed with appropriate chemical control. Pollination status includes the color of the flower head; a blackened head indicates pollination has been ongoing for some time. The flower head also indicates pollination success; after pollination, the flower head will fall off first when fruiting begins. Brown marks left by bumblebees during pollination on the style also indicate successful pollination.

[0046] like Figure 2 As shown, the tail end of the six-degree-of-freedom robotic arm 8 is mounted on a vertical lifting frame via a rotating device, and the vertical lifting frame is mounted on the mobile robot body. The rotating axis of the rotating device is the Z-axis, which can drive the six-degree-of-freedom robotic arm to rotate around the Z-axis. The vertical lifting frame can drive the six-degree-of-freedom robotic arm 8 to move in the height direction. The six-degree-of-freedom robotic arm consists of several connecting arms and rotating joints located on the connecting arms. The spectral imager 9 is located at the end of the six-degree-of-freedom robotic arm. Through the coordinated movement of the six-degree-of-freedom robotic arm 8, the rotating device, and the vertical lifting frame, the spectral imager 9 can move and change orientation in three-dimensional space to obtain hyperspectral data at the required position and viewing angle.

[0047] The signal processing module of this invention is pre-trained using a labeled hyperspectral image dataset of tomato plants to develop a pest and disease identification model, a plant growth model, and a flower pollination determination model. The training dataset is manually labeled, including both normal tomato plants and a predetermined number of abnormal plants. Abnormal plants include pest and disease-affected plants, plants with excessive growth, and unpollinated plants. For pest and disease-affected plants, specific pest and disease categories are also manually labeled.

[0048] During monitoring, the corresponding model is called according to the monitoring items corresponding to the monitoring task; the signal processing module first removes irrelevant background images and interference images, and extracts the target image; then it acquires the characteristic bands and hyperspectral images of different height parts of the tomato plant, inputs them into the corresponding model for prediction, and determines the severity of pests and diseases, plant growth, and flowering and pollination status.

[0049] When using the device of the present invention to monitor the phenotypic characteristics of tomato plants, the following steps can be taken:

[0050] 1) The mobile robot moves to the initial monitoring position and adjusts the components of the monitoring module to their initial state; after the reset is complete, the signal transceiver module on the mobile robot begins to receive monitoring tasks from the base station; the monitoring tasks include the monitoring route and monitoring items.

[0051] 2) After acquiring the monitoring task, the control module drives the mobile robot body to move according to the monitoring route. The tomato plants on the same row are planted at equal intervals, and the mobile robot body moves along a rectangle parallel to the row. When the monitoring of one row is completed, it moves to the next row along the set route. The control module controls the wheel drive motor to drive the wheels to roll a set distance to achieve movement between plants.

[0052] 3) When the mobile robot moves to the position corresponding to the tomato plant, the distance sensor module detects the distance between the mobile robot and the tomato plant. Based on the detected distance, the six-degree-of-freedom robotic arm drives the spectral imager to acquire a hyperspectral image of the plant.

[0053] The signal processing module calls the corresponding model according to the monitoring items corresponding to the monitoring task; the signal processing module first removes irrelevant background images and interference images, and extracts the target image; then it obtains the characteristic bands and hyperspectral images of different height parts of the tomato plant, inputs them into the corresponding model for prediction, and determines the severity of pests and diseases, plant growth, and flowering and pollination status.

[0054] 4) After monitoring one plant, the mobile robot moves to the monitoring position corresponding to the next plant and repeats step 3) until the monitoring task is completed. For plants with abnormalities, since the initial monitoring position is fixed and the travel route and wheel travel distance are known, the position corresponding to each plant is also determined. The camera takes pictures of the plant and transmits the plant images and the corresponding positions of the plants to the base station through the signal transceiver module.

[0055] like Figure 4 As shown, the monitoring items during the pollination period include monitoring the status of pests and diseases, monitoring growth, and detecting pollination. At this time, the signal processing module first calls the pest and disease identification model to identify pests and diseases. If the identification result is that pests and diseases exist, the plant is directly marked as an abnormal plant, and the subsequent plant growth model and flower pollination judgment model are not called. Only when the identification result is that there are no pests and diseases will the plant growth model be called.

[0056] The plant growth model determines whether the plant is growing well. If the plant is growing poorly, it is directly marked as an abnormal plant and the subsequent flower pollination judgment model is not called. If the plant is growing well, the flower pollination judgment model is called.

[0057] The flower pollination determination model first checks whether flowers are present. If no flowers are detected, the plant is marked as an abnormal plant. If flowers are present, it checks whether the flowers are pollinated. If pollinated, the monitoring is completed and the model moves to the next plant. If not pollinated, the model sends the coordinates of the unpolluted plant and an early warning message, and then moves to the next plant.

[0058] In a specific application, based on an existing database, the monitoring device can further provide treatment suggestions (pest and disease control programs, fertilization programs, etc.) based on the abnormal conditions of abnormal plants. During tomato cultivation, tomatoes grow vigorously during the flowering period. After entering the initial flowering stage, bumblebees are introduced for pollination. If the stamens have turned black, it indicates successful pollination; if brown marks (bumblebee proboscis marks) are left on the style, it can be judged as normal flower visiting by the bee colony. If poor growth or excessive vegetative growth is detected in the tomato plants, the system indicates the specific plant location and the need for growth regulator control. For example, for excessive vegetative growth, a 1000-fold dilution of chlormequat chloride can be sprayed on the leaves, 5-7 days / time, for two consecutive times, followed by 10-15 days / time. If pests or diseases are found, the system determines and controls them according to the specific type. For example, if the pest or disease is whiteflies on the leaves, the monitoring system indicates the plant coordinates and suggests spraying with a 2500-fold dilution of 25% imidacloprid wettable powder.

[0059] Therefore, this invention can systematically, extensively, and efficiently detect crop phenotypic information, and can detect crop diseases, pests, growth status, and flowering conditions.

[0060] The above-described embodiments are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. Those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A phenotypic monitoring method for tomato plants based on a mobile robot phenotypic monitoring device, characterized in that, The tomato plant phenotypic monitoring device includes: The mobile robot body, and the monitoring module and control module mounted on the mobile robot body; The control module is used to control the movement or stopping of the mobile robot body according to the monitoring task sent by the base station; the monitoring module includes: A camera used to capture images of abnormal tomato plants; Distance sensor module is used to detect the distance between the mobile robot body and the tomato plant; A six-degree-of-freedom robotic arm, one end of which is connected to the mobile robot body, and the other end is equipped with a spectral imager, which is used to move the spectral imager. A spectral imager is used to acquire hyperspectral images of tomato plants and transmit them to a signal processing module. The signal processing module is used to process hyperspectral images to determine whether tomato plants have pests or diseases, whether they are growing well, and whether they have been pollinated. The signal processing module contains a pre-trained pest and disease identification model, a plant growth model, and a flower pollination judgment model. Among them, the pest and disease identification model is used to determine whether tomato plants are affected by pests and diseases, the plant growth model is used to determine whether tomato plants are growing well, and the flower pollination judgment model is used to identify flowers and determine whether the flowers are pollinated. The phenotypic monitoring method includes the following steps: 1) The mobile robot moves to the initial monitoring position and adjusts the components of the monitoring module to their initial state; after the reset is complete, the signal transceiver module on the mobile robot begins to receive monitoring tasks from the base station; the monitoring tasks include the monitoring route and monitoring items. 2) After acquiring the monitoring task, the control module drives the mobile robot body to move according to the monitoring route. The tomato plants on the same row are planted at equal intervals, and the mobile robot body moves along a rectangle parallel to the row. When the monitoring of one row is completed, it moves to the next row along the set route. The control module controls the wheel drive motor to drive the wheels to roll a set distance to achieve movement between plants. 3) When the mobile robot moves to the position corresponding to the tomato plant, the distance sensor module detects the distance between the mobile robot and the tomato plant. Based on the detected distance, the six-degree-of-freedom robotic arm drives the spectral imager to acquire a hyperspectral image of the plant. The signal processing module calls the corresponding model according to the monitoring items corresponding to the monitoring task; the signal processing module first removes irrelevant background images and interference images, and extracts the target image; then it obtains the characteristic bands and hyperspectral images of different height parts of the tomato plant, inputs them into the corresponding model for prediction, and determines the severity of pests and diseases, plant growth, and flowering and pollination status. In step 3), when the monitoring items simultaneously include monitoring pest and disease status, monitoring growth, and detecting pollination, the signal processing module first calls the pest and disease identification model to identify pests and diseases. If the identification result indicates the presence of pests and diseases, the plant is directly marked as an abnormal plant, and the subsequent plant growth model and flower pollination judgment model are not called. Only when the identification result indicates the absence of pests and diseases is the plant growth model called. The plant growth model determines whether the plant is growing well. If the plant is growing poorly, it is directly marked as an abnormal plant and the subsequent flower pollination judgment model is not called. If the plant is growing well, the flower pollination judgment model is called. The flower pollination determination model first checks whether flowers are present. If no flowers are detected, the plant is marked as an abnormal plant. If flowers are present, it checks whether the flowers are pollinated. If pollinated, the monitoring is completed and the model moves to the next plant. If not pollinated, the model sends the coordinates of the unpollinated plant and an early warning signal before moving to the next plant. 4) After monitoring one plant, the mobile robot moves to the monitoring position corresponding to the next plant and repeats step 3) until the monitoring task is completed. For plants with abnormalities, since the initial monitoring position is fixed and the travel route and wheel travel distance are known, the position corresponding to each plant is also determined. The camera takes pictures of the plant and transmits the plant images and the corresponding positions of the plants to the base station through the signal transceiver module.

2. The phenotypic monitoring method for tomato plant phenotypic monitoring devices based on mobile robots according to claim 1, characterized in that, The mobile robot body is equipped with a wheel drive motor, a battery module, and a signal transceiver module. The mobile robot has several wheels for movement, the wheel drive motor drives the wheels to roll, and the battery module supplies power to the monitoring module, the signal transceiver module, and the wheel drive motor. The signal transceiver module communicates with the base station wirelessly to receive control commands and upload data. The signal transceiver module and the monitoring module are connected via a signal line.

3. The phenotypic monitoring method for tomato plant phenotypic monitoring devices based on mobile robots according to claim 2, characterized in that, The signal transceiver module includes a signal receiver and a signal transmitter; the signal receiver and signal transmitter communicate wirelessly with the base station via Bluetooth, Wi-Fi, or 4G / 5G communication methods.

4. The phenotypic monitoring method for tomato plant phenotypic monitoring devices based on mobile robots according to claim 1, characterized in that, The control module receives monitoring tasks sent by the base station. The monitoring tasks include monitoring routes and monitoring items. The monitoring items include one or more of the following: monitoring pest and disease conditions, monitoring growth, and detecting pollination.

5. The phenotypic monitoring method for tomato plant phenotypic monitoring devices based on mobile robots according to claim 1, characterized in that, The tail end of the six-degree-of-freedom robotic arm is mounted on a vertical lifting frame via a rotating device, which is installed on the mobile robot body. The rotating device's rotation axis is the Z-axis, enabling the six-degree-of-freedom robotic arm to rotate around the Z-axis. The vertical lifting frame enables the six-degree-of-freedom robotic arm to move in the height direction. The six-degree-of-freedom robotic arm consists of several connecting arms and rotating joints located on the connecting arms. A spectral imager is located at the end of the six-degree-of-freedom robotic arm. Through the coordinated movement of the six-degree-of-freedom robotic arm, the rotating device, and the vertical lifting frame, the spectral imager can move and change orientation in three-dimensional space to obtain hyperspectral data at the desired position and viewing angle.

6. The phenotypic monitoring method for tomato plant phenotypic monitoring devices based on mobile robots according to claim 1, characterized in that, The signal processing module is pre-trained using an annotated hyperspectral image dataset of tomato plants to identify pests and diseases, plant growth, and flower pollination. During monitoring, the corresponding model is called according to the monitoring items corresponding to the monitoring task; the signal processing module first removes irrelevant background images and interference images, and extracts the target image; then it acquires the characteristic bands and hyperspectral images of different height parts of the tomato plant, inputs them into the corresponding model for prediction, and determines the severity of pests and diseases, plant growth, and flowering and pollination status.

Images