An agricultural robot for a gantry cultivation scene and an image acquisition method thereof

Abstract

The application discloses an agricultural robot and an image acquisition method thereof for a gantry cultivation scene, relates to the technical field of crop monitoring, and upgrades a camera carried by the agricultural robot to an RGBD camera and additionally adds a camera lifting mechanism in the hardware structure, uses a gantry horizontal line which is generally present and fixed in form in the gantry cultivation scene as a positioning reference in the software control part, uses visual algorithm to identify actual pixel height and actual depth information of the gantry in combination with information provided by the RGBD image, and determines target pixel height in combination with prior knowledge and characteristic parameters, and then adjusts the camera lifting mechanism to make the camera field of view cover the range of crop plants, guarantees the integrity of the crop plants in the acquired crop image, can effectively improve the effectiveness of the acquired crop image, and greatly improves the quality of agricultural big data and the downstream application value.

Description

Technical Field

[0001] This application relates to the field of agricultural monitoring technology, and in particular to an agricultural robot and its image acquisition method for rack cultivation scenarios. Background Technology

[0002] Substrate bags, cultivation troughs, hydroponics, and other rack cultivation methods are the core of modern facility agriculture, urban agriculture, and vertical farms, aiming to produce crops more efficiently and in a more controllable manner, and have gradually been widely used.

[0003] In trellis cultivation, data collection is also necessary during crop growth for monitoring purposes, including subsequent phenotypic analysis, growth monitoring, and yield prediction. With the development of intelligent agriculture, agricultural robots have gradually replaced manual operation. The camera height of the agricultural robot is adjusted according to the theoretical height of the planting trellis, and then the robot is controlled to move along the trellis to capture crop images. Subsequent image processing techniques are then used to obtain relevant crop growth data. However, due to factors such as land subsidence, structural aging, or pipeline maintenance, the actual height of the planting trellis is not always consistent with the theoretical height, often fluctuating in practice. This leads to problems such as cropping and missed shots in the crop images or videos collected by the agricultural robot, preventing the acquisition of complete crop images and affecting the quality and completeness of the data required for subsequent phenotypic analysis, growth monitoring, and yield prediction. Summary of the Invention

[0004] This application addresses the aforementioned problems and technical needs by proposing an agricultural robot and its image acquisition method for rack cultivation scenarios. The technical solution of this application is as follows:

[0005] An image acquisition method for agricultural robots in rack cultivation scenarios, wherein the agricultural robot is equipped with an RGBD camera and the RGBD camera is fixed on a camera lifting mechanism, and the image acquisition method for agricultural robots includes:

[0006] Determine the characteristic parameters of the crops planted on the support frame, query the prior knowledge base to determine the relative positional relationship between the plants and the support frame corresponding to the crop characteristic parameters, and determine the target pixel height y of the support frame relative to the upper boundary of the image based on the relative positional relationship between the plants and the support frame. target ;

[0007] At the current acquisition location, an RGBD image of the crop on the trellis is acquired using an RGBD camera. Based on line detection technology, the actual pixel height y of the trellis relative to the upper boundary of the RGBD image is determined. anchor And the actual depth information d of the platform relative to the RGBD camera;

[0008] When the actual pixel height y of the platform anchor and its target pixel height ytarget When the pixel deviation reaches the deviation threshold, the camera lifting mechanism is controlled to lift the RGBD camera up and down, based on the actual depth information d of the stage, until the actual pixel height y of the stage is reached. anchor and its target pixel height y target When the pixel deviation is less than the deviation threshold, the RGBD image of the crop on the pedestal at the current acquisition position is reacquired by the RGBD camera.

[0009] A further technical solution is that the relative positional relationship between the plant and the support includes the maximum vertical height H between the upper end of the crop canopy and the centerline of the support. up And the maximum vertical height H between the lower end of the drooping fruit cluster and the center line of the trellis. down .

[0010] A further technical solution involves determining the target pixel height y of the plant support frame based on the relative positional relationship between the plant and the support frame. target include:

[0011] The target pixel height y of the frame is determined with the goal of ensuring that the upper end of the crop canopy reaches the upper boundary of the image and the lower end of the drooping fruit clusters reaches the lower boundary of the image. target for:

[0012]

[0013] Where H is the total pixel height of the image captured by the RGBD camera.

[0014] A further technical solution involves using the actual depth information d of the test platform to control the camera lifting mechanism, thereby raising and lowering the RGBD camera to achieve the actual pixel height y of the test platform. anchor Reaching target pixel height y target The location includes:

[0015] Calculate the lifting control parameters based on the actual depth information d of the platform. Among them, f y It is the vertical focal length in the intrinsic parameter matrix of the RGBD camera;

[0016] When ΔY>0, the camera lifting mechanism is controlled to raise the RGBD camera by a distance |ΔY| from the target; when ΔY<0, the camera lifting mechanism is controlled to lower the RGBD camera by a distance |ΔY| from the target.

[0017] A further technical solution involves controlling the camera lifting mechanism to move the RGBD camera up and down, including:

[0018] After controlling the camera lifting mechanism to raise and lower the RGBD camera by a target distance |ΔY|, the RGBD image of the crop on the platform is re-acquired through the RGBD camera for closed-loop feedback control until the actual pixel height y of the platform is reached.anchor and its target pixel height y target The pixel deviation is less than the deviation threshold.

[0019] A further technical solution involves determining the actual pixel height y of the platform relative to the upper boundary of the RGBD image based on line detection technology. anchor And the actual depth information d of the platform relative to the RGBD camera includes:

[0020] The gantry baseline in the RGBD image is identified based on line detection technology, and the pixel height and depth values ​​of the pixels within the gantry baseline are determined.

[0021] The actual pixel height y of the test bench is obtained by calculating the average pixel height of the pixels within the test bench baseline. anchor ;

[0022] The actual depth information d of the platform relative to the RGBD camera is obtained by calculating the average depth value of the pixels within the platform baseline.

[0023] A further technical solution involves identifying the gantry baseline in an RGBD image based on line detection technology, including:

[0024] After image preprocessing of the RGBD image, line detection is performed to obtain several candidate line segments. The RANSAC technique combined with the least squares method is used to fit the line to obtain the benchmark line of the test bench.

[0025] The further technical solution is that the crop's characteristic parameters include crop variety, planting area parameters, current growth cycle, and agronomic management parameters.

[0026] A further technical solution is that the agricultural robot image acquisition method also includes:

[0027] When it is determined that the current growth cycle of the crop belongs to the early growth stage, the height y of the center pixel of the image is directly used. center The target pixel height y of the test bench target Otherwise, proceed with the step of querying the prior knowledge base to determine the relative positional relationship between the plant and the support frame corresponding to the crop's characteristic parameters.

[0028] An agricultural robot for rack cultivation scenarios includes a robot body, a camera lifting mechanism fixed to the robot body, an RGBD camera fixed to the camera lifting mechanism, and a robot controller. The robot controller executes the agricultural robot image acquisition method of the first aspect and controls the camera lifting mechanism and the RGBD camera while controlling the movement of the robot body.

[0029] The beneficial technical effects of this application are:

[0030] This application discloses an agricultural robot and its image acquisition method for trellis cultivation scenarios. In terms of hardware, the camera mounted on the agricultural robot is upgraded to an RGBD camera, and a camera lifting mechanism is added. In the software control section, the horizontal line of the trellis, which is common and has a fixed shape in trellis cultivation scenarios, is used as a positioning reference. Combined with information provided by the RGBD image, a visual algorithm is used to identify the actual pixel height and actual depth information of the trellis. The target pixel height is determined by combining prior knowledge and characteristic parameters. Then, the camera lifting mechanism is adjusted so that the camera's field of view covers the crop area, ensuring the integrity of the crop in the acquired crop image. This effectively improves the effectiveness of the acquired crop images, increasing the proportion of effective data frames by 30%-60%, greatly enhancing the quality and downstream application value of agricultural big data.

[0031] This method introduces real-time visual feedback for closed-loop control, which can effectively avoid problems such as adjustment lag or overshoot. While ensuring adjustment accuracy, it does not require the introduction of additional high-cost infrastructure such as physical guide rails or beacons. Therefore, it has good operational flexibility and can adapt well to the changing and complex agricultural production environment. It can seamlessly adapt to various platform planting modes such as substrate bags, cultivation troughs, and hydroponics. It is highly versatile and robust, and is not affected by crop growth or foliage shading. Attached Figure Description

[0032] Figure 1 This is a flowchart of an agricultural robot image acquisition method according to an embodiment of this application.

[0033] Figure 2 This is a schematic diagram of the upper plant canopy and the drooping fruit clusters of the crop on the trellis in a trellis cultivation scenario.

[0034] Figure 3 This is a schematic diagram of an RGBD image captured by an RGBD camera in an example.

[0035] Figure 4 This is a flowchart of an agricultural robot image acquisition method according to another embodiment of this application. Detailed Implementation

[0036] The specific embodiments of this application will be further described below with reference to the accompanying drawings.

[0037] To avoid the problem of low image quality when existing agricultural robots collect images of crops in trellis cultivation scenarios at a fixed height, this application optimizes the structure of traditional agricultural robots and provides an agricultural robot for trellis cultivation scenarios. This agricultural robot, like traditional agricultural robots, includes a robot body and a robot controller. The robot controller is connected to and controls the movement of the robot body to perform various functions of traditional agricultural robots except for image acquisition. The robot body can adopt the structure of existing agricultural robots except for the camera, and this application does not limit it in this regard.

[0038] The agricultural robot of this application upgrades and optimizes the image acquisition system based on the existing main structure of agricultural robots, including adding a camera lifting mechanism and upgrading the ordinary RGB camera to an RGBD camera. The camera lifting mechanism is fixed vertically to the robot body, the RGBD camera is fixed to the camera lifting mechanism, and the robot controller connects to and controls the camera lifting mechanism and the RGBD camera. When the agricultural robot disclosed in this application acquires images of crops in a platform cultivation scenario, the robot controller controls the RGBD camera to acquire RGBD images containing RGB information and depth information, and controls the camera lifting mechanism based on the acquired RGBD images to adjust the height of the RGBD camera relative to the ground, thereby adjusting the field of view coverage of the RGBD camera to ensure that the acquired RGBD images can completely cover the crops on the platform to ensure image quality.

[0039] This application also discloses an image acquisition method for agricultural robots in rack cultivation scenarios. In the agricultural robot disclosed in this application, the robot controller, while controlling the movement of the robot body, executes this image acquisition method to control the camera lifting mechanism and the RGBD camera to achieve the aforementioned adaptive and precise framing goal. This agricultural robot image acquisition method includes the following steps; please refer to... Figure 1 The flowchart shown:

[0040] Step 110: Determine the characteristic parameters of the crops planted on the platform, and query the prior knowledge base to determine the relative positional relationship between the plants and the platform corresponding to the characteristic parameters of the crops.

[0041] Crop characteristic parameters are input externally. One approach is to input them directly on-site through the agricultural robot's human-machine interface, in which case the robot controller obtains the externally input crop characteristic parameters via the robot's human-machine interface. Another approach is to have the agricultural robot have network connectivity, in which case the robot controller remotely obtains the externally input crop characteristic parameters from a server.

[0042] Crop characteristic parameters include various parameters affecting crop plant morphology, such as crop variety, planting location, current growth cycle, and agronomic management parameters. Planting location parameters include the latitude, longitude, and altitude of the crop's location. Agronomic management parameters include trellis cultivation mode, planting environment parameters, and management parameters. Trellis cultivation mode can be substrate bag mode, cultivation trough mode, or hydroponics. Planting environment parameters include soil condition parameters and climate condition parameters. Management parameters include planting density, fertilizer application rate, and irrigation rate.

[0043] Crops exhibit certain growth patterns. Based on these patterns, the plant morphology can be determined for each set of characteristic parameters. The relative positional relationship between the crop plant and its support frame can then be recorded. Different crop characteristic parameters often result in different relative positional relationships between the plant and its support frame. This mapping relationship between the crop's characteristic parameters and the relative positional relationship between the plant and its support frame is recorded, and a priori knowledge base is constructed based on this.

[0044] If the crop is grown on a trellis, and the crop plant includes the upper canopy above the trellis and the drooping fruit clusters below the trellis, then please combine... Figure 2 In one embodiment, the relative positional relationship between the plant and the support includes the maximum vertical height H between the upper end of the crop canopy and the centerline of the support. up And the maximum vertical height H between the lower end of the drooping fruit cluster and the center line of the trellis. down .

[0045] Step 120: Determine the target pixel height y of the plant support relative to the upper boundary of the image based on the queried relative positional relationship of the plant support. target .

[0046] Based on the information contained in the relative positional relationship of the plant trellis, after retrieving the relative positional relationship of the plant trellis, the target pixel height y of the trellis can be determined with the goal of ensuring that the upper end of the upper part of the crop canopy reaches the upper boundary of the image and the lower end of the drooping fruit cluster reaches the lower boundary of the image. target for:

[0047]

[0048] In this application, the top left corner of the image is taken as the origin of the coordinate system, and downward is the direction in which the pixel height increases. H is the total pixel height of the image captured by the RGBD camera.

[0049] Step 130: At the current acquisition location, acquire an RGBD image of the crop on the pedestal using an RGBD camera. The acquired RGBD image includes the RGB image and the depth value of each pixel. Each pixel in the RGB image has its own pixel coordinates and RGB information. For example, in one instance, the acquired RGBD image might look like this: Figure 3As shown.

[0050] Step 140: Determine the actual pixel height y of the gantry relative to the upper boundary of the RGBD image based on line detection technology. anchor And the actual depth information d of the platform relative to the RGBD camera.

[0051] Regardless of the specific rack cultivation method used, the rack is the most ubiquitous and fixed load-bearing support horizontal line in rack cultivation scenarios. Based on this characteristic of rack cultivation scenarios, this application uses the rack as a positioning reference. Since the rack has a horizontal structure, the reference line is first identified in the acquired RGBD image using line detection technology. When identifying the reference line, the RGBD image is first preprocessed, including distortion correction and image filtering. Then, line detection is performed to obtain several candidate line segments. Finally, RANSAC technology combined with the least squares method is used for line fitting to obtain the rack reference line. For example, in... Figure 3 The red line in the middle represents the baseline of the test bench obtained through fitting.

[0052] Determine the pixel height and depth values ​​of the pixels falling within the rig's baseline. Then, calculate the average pixel height of the pixels within the rig's baseline to obtain the actual pixel height y of the rig. anchor The actual depth information d of the platform relative to the RGBD camera is obtained by calculating the average depth values ​​of the pixels within the platform's baseline.

[0053] Step 150: Detect the actual pixel height y of the testing platform. anchor and its target pixel height y target Whether the pixel deviation reaches the deviation threshold, which can be customized. When the actual pixel height y of the stage... anchor and its target pixel height y target If the pixel deviation does not reach the deviation threshold, the currently acquired RGBD image is directly used as the crop image at the current acquisition position, and then it can continue to move to the next acquisition position to continue image acquisition.

[0054] Step 160, when the actual pixel height y of the stage anchor and its target pixel height y target When the pixel deviation reaches the deviation threshold, the camera lifting mechanism is controlled to lift the RGBD camera up and down, based on the actual depth information d of the stage, until the actual pixel height y of the stage is reached. anchor and its target pixel height y target When the pixel deviation is less than the deviation threshold, the RGBD image of the crop on the pedestal at the current acquisition position is reacquired by the RGBD camera.

[0055] When controlling the camera lifting mechanism to move the RGBD camera up and down, it is necessary to determine the lifting direction and distance. The lifting control parameters can be calculated by using imaging principles combined with the actual depth information d of the platform. Among them, f y It is the vertical focal length in the intrinsic parameter matrix of the RGBD camera.

[0056] When ΔY>0, it indicates that the actual position of the support frame is too close to the upper boundary of the image, which will cause the upper canopy of the crop to not be fully included in the RGBD image, such as... Figure 3 As shown in the example, at this time, the camera lifting mechanism is controlled to raise the RGBD camera to a target distance of |ΔY|.

[0057] When ΔY<0, it means that the actual position of the platform is too close to the lower boundary of the image, which will cause the drooping fruit clusters of the crop to not be fully contained in the RGBD image. In this case, it is necessary to control the camera lifting mechanism to drive the RGBD camera to lower the target distance |ΔY|.

[0058] One approach is to assume that the actual pixel height y of the platform is determined by controlling the camera lifting mechanism to raise or lower the RGBD camera by a target distance |ΔY|. anchor and its target pixel height y target If the pixel deviation is less than the deviation threshold, then the RGBD image of the crop on the platform at the current acquisition position is re-acquired by the RGBD camera and used as the crop image acquired at the current acquisition position.

[0059] However, to avoid adjustment lag or overshoot, another approach introduces a visual closed-loop control strategy. That is, after controlling the camera lifting mechanism to raise or lower the RGBD camera by a target distance |ΔY|, the RGBD image of the crop on the platform is re-acquired through the RGBD camera, and the actual pixel height y of the platform is re-determined using step 140. anchor Then perform closed-loop feedback control, please combine Figure 4 The flowchart shown continues until the actual pixel height y of the stage is finally determined. anchor and its target pixel height y target When the pixel deviation is less than the deviation threshold, the RGBD image at this point is used as the crop image at the current acquisition location. This method introduces closed-loop feedback control, which can ensure more accurate adjustment and avoid lag or overshoot caused by external influences or interference during the adjustment process.

[0060] In practical applications, multiple racks in the same cultivation scenario often have crops with identical characteristic parameters. For example, multiple racks in the same greenhouse may grow the same crop variety and be managed using the same agronomic management parameters. Therefore, these crops will have the same crop variety, planting location parameters, current growth cycle, and agronomic management parameters, resulting in identical characteristic parameters. Then, steps 110 and 120 are used to determine the target pixel height y. target Then, the robot controller can control the agricultural robot to move along the platform to different collection positions, and at each collection position, directly utilize the target pixel height y target Following steps 130 to 160, adaptive framing adjustment is used to acquire crop images at each acquisition location, without needing to determine the target pixel height y separately at each acquisition location. target .

[0061] Furthermore, considering that crops are often small in their early growth stages, the corresponding H for crops... up and H down If both are relatively small, adjusting the platform to the center of the image will generally ensure the integrity of the image of the upper plant canopy and the drooping fruit clusters. In another embodiment, after determining the characteristic parameters of the crop planted on the platform, it is first determined whether the current growth cycle of the crop belongs to the early growth stage. The time period covered by the early growth stage can be customized, such as setting the seed stage and seedling stage as the early growth stage. When the current growth cycle of the crop belongs to the early growth stage, the volume of the upper plant canopy and the drooping fruit clusters are relatively small. At this time, the image center pixel height y is directly used. center The target pixel height y of the test bench target Otherwise, the relative position of the plant and the support corresponding to the crop's characteristic parameters is determined by querying the prior knowledge base.

[0062] Since agricultural robots typically move to different acquisition positions on a platform to continuously acquire data, the horizontal offset between the RGBD camera and the platform has little impact on the image acquisition results. It is usually the height offset that affects image integrity. Therefore, this application primarily adjusts the vertical height of the RGBD camera, rather than adjusting the horizontal orientation. This allows for faster response times and ensures real-time adjustment while maintaining improved image acquisition quality. Furthermore, the same principle applies to video acquisition in practical applications; individual image frames in the video stream can be processed using the same method.

[0063] The above descriptions are merely preferred embodiments of this application, and this application is not limited to the above embodiments. It is understood that other improvements and variations that can be directly derived or conceived by those skilled in the art without departing from the spirit and concept of this application should be considered to be included within the protection scope of this application.

Claims

1. An image acquisition method for agricultural robots in platform cultivation scenarios, characterized in that, The agricultural robot is equipped with an RGBD camera, which is fixed on a camera lifting mechanism. The image acquisition method of the agricultural robot includes: The characteristic parameters of the crops planted on the support frame are determined. A prior knowledge base is consulted to determine the relative positional relationship between the plants and the support frame corresponding to the crop's characteristic parameters. Based on the relative positional relationship between the plants and the support frame, the target pixel height of the support frame relative to the upper boundary of the image is determined. The prior knowledge base is used to record the mapping relationship between crop characteristic parameters and the relative positional relationship between the plant and the trellis. The relative positional relationship between the plant and the trellis records the relative positional relationship between the crop and the trellis. The crop includes the upper canopy above the trellis and the drooping fruit clusters below the trellis. The relative positional relationship between the plant and the trellis includes the maximum vertical height between the upper end of the upper canopy and the centerline of the trellis. And the maximum vertical height between the lower end of the drooping fruit cluster and the center line of the trellis. The target pixel height of the test bench is determined with the goal of ensuring that the upper end of the crop canopy reaches the upper boundary of the image and the lower end of the drooping fruit clusters reaches the lower boundary of the image. , The total pixel height of the image captured by the RGBD camera; At the current acquisition location, an RGBD image of the crop on the pedestal is acquired using an RGBD camera. Based on line detection technology, the actual pixel height of the pedestal relative to the upper boundary of the RGBD image is determined. And the actual depth information of the platform relative to the RGBD camera. ; When the actual pixel height of the platform and its target pixel height When the pixel height deviation reaches the deviation threshold, the actual depth information of the test bench is considered. The camera lifting mechanism is controlled to raise and lower the RGBD camera until it reaches the actual pixel height of the platform. and its target pixel height When the pixel height deviation is less than the deviation threshold, the RGBD image of the crop on the pedestal at the current acquisition position is reacquired by the RGBD camera.

2. The agricultural robot image acquisition method according to claim 1, characterized in that, Combined with the actual depth information of the test bench The camera lifting mechanism is controlled to raise and lower the RGBD camera until it reaches the actual pixel height of the platform. and its target pixel height When the pixel height deviation is less than the deviation threshold, the process of re-acquiring the RGBD image of the crop on the pedestal at the current acquisition location using the RGBD camera includes: Based on the actual depth information of the test bench Calculate lifting control parameters ,in, It is the vertical pixel focal length in the intrinsic parameter matrix of the RGBD camera; when At the same time, the camera lifting mechanism is controlled to raise the RGBD camera to the target distance. ;when At the same time, the camera lifting mechanism is controlled to drive the RGBD camera to lower the target distance. .

3. The agricultural robot image acquisition method according to claim 2, characterized in that, Controlling the camera lifting mechanism to drive the RGBD camera to lift and lower includes: The camera lifting mechanism is controlled to move the RGBD camera up and down by a target distance. Then, RGBD images of the crops on the pedestal are re-acquired using an RGBD camera for closed-loop feedback control until the actual pixel height of the pedestal is reached. and its target pixel height The pixel height deviation is less than the deviation threshold.

4. The agricultural robot image acquisition method according to claim 1, characterized in that, The actual pixel height of the gantry relative to the upper boundary of the RGBD image is determined based on line detection technology. And the actual depth information of the platform relative to the RGBD camera. include: The gantry baseline in the RGBD image is identified based on line detection technology, and the pixel height and depth values ​​of the pixels within the gantry baseline are determined. The actual pixel height of the test bench is obtained by calculating the average pixel height of the pixels within the test bench baseline. ; The actual depth information of the rig relative to the RGBD camera is obtained by calculating the average depth value of the pixels within the rig's baseline. .

5. The agricultural robot image acquisition method according to claim 4, characterized in that, Identifying the gantry baseline in the RGBD image based on line detection technology includes: After image preprocessing of the RGBD image, line detection is performed to obtain several candidate line segments. The RANSAC technique combined with the least squares method is used to fit the line to obtain the rig baseline.

6. The agricultural robot image acquisition method according to claim 1, characterized in that, Crop characteristic parameters include crop variety, planting location parameters, current growth cycle, and agronomic management parameters.

7. The agricultural robot image acquisition method according to claim 6, characterized in that, The agricultural robot image acquisition method also includes: When it is determined that the current growth cycle of the crop belongs to the early growth stage, the height of the center pixel of the image is used directly. Target pixel height of the test bench Otherwise, proceed with the step of querying the prior knowledge base to determine the relative positional relationship between the plant and the support frame corresponding to the crop's characteristic parameters.

8. An agricultural robot for rack cultivation scenarios, characterized in that, The agricultural robot includes a robot body, a robot controller, a camera lifting mechanism fixed on the robot body, and an RGBD camera fixed on the camera lifting mechanism. During the process of controlling the movement of the robot body, the robot controller executes the agricultural robot image acquisition method as described in any one of claims 1-7 and controls the camera lifting mechanism and the RGBD camera.

Images