A system and method for determining bonsai grasping posture and parking robot

Through the potted plant identification system based on RGB-D camera and OpenCV, combined with intelligent maintenance greenhouse boxes, the independent identification and precise handling of potted plants are achieved, and the problems of high cost and difficulty of potted plants are solved, and the full-weather accurate radiation and mechanical pipeline of potted plants are achieved. Water regularly.

CN115657531BActive Publication Date: 2024-11-26LANZHOU UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202211273414.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-10-18
Publication Date
2024-11-26
Estimated Expiration
2042-10-18

AI Technical Summary

Technical Problem

The existing potted plant handling robots cannot independently identify and locate flower pots, and lack integration with small indoor intelligent maintenance greenhouse boxes, resulting in high cost and difficulty in potted plant management. The existing technology has poor identification accuracy under natural light conditions, which cannot be achieved. Accurate irradiation around the clock and regular watering of mechanical pipes.

Method used

A potted plant recognition system based on RGB-D camera and OpenCV function library was designed, combining the Raspberry Pi 4B microprocessor and Realsense D435i RGB-D camera to fit circles through color segmentation, edge detection and least squares method to achieve flowering The precise identification and positioning of the basin is combined with the indoor small intelligent maintenance greenhouse box to achieve accurate all-weather illumination and regular watering of mechanical pipes.

Benefits of technology

It realizes the independent identification and precise handling of potted plants, reduces management difficulties and costs, is highly robust, and can accurately identify the center and radius of potted plants under natural light conditions, and supports all-weather pot maintenance and management.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115657531B_ABST
    Figure CN115657531B_ABST
Patent Text Reader

Abstract

The present invention discloses a system and method for determining a bonsai grasping posture and parking a robot, wherein a system for determining a bonsai grasping posture and parking a robot comprises an information Internet of Things and intelligent sensing node and a motion control node; the information Internet of Things and intelligent sensing node comprises a Raspberry Pi 4B microprocessor and an RGB‑D camera; the motion control node is a bottom PCB motion control board, which is used for the robot to complete the control of the chassis and the mechanical arm under the drive of a motor, and determine the bonsai grasping posture. The potted plant handling robot is combined with an indoor small intelligent maintenance greenhouse box terminal to form an indoor small potted plant intelligent maintenance and reconstruction deployment system, which realizes functions such as all-weather precise irradiation of potted plants and timed watering of mechanical pipelines, can maintain and manage indoor small potted plants, and redeploy the potted plants to their original positions after maintenance.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to robot technology, and in particular to a system and method for determining a bonsai grabbing posture and parking a robot based on a mixed-color camera. Background Art

[0002] At present, potted plants are evolved from traditional Chinese garden art. As for the cultivation of ornamental plants, it originated from ancient garden landscaping, which imitated natural landscapes to create gardens. Potted plants have the functions of beautifying the environment and refreshing the air, and have been widely used in office buildings, school offices, homes, student dormitories and other occasions. Generally speaking, due to the characteristics of potted plants such as small size, small number and small coverage area, if special personnel are hired to carry, care for and manage potted plants, it will increase management costs and human resources to a certain extent.

[0003] Based on the above problems, the research team combined the previous research foundation and the technical advantages of the team members, combined with mainstream technologies such as OpenCV, and designed a method and system for using a mixed-color camera to determine the position of potted plants and park the robot. The system includes an indoor potted plant handling robot and an indoor small intelligent maintenance greenhouse terminal, which can maintain and manage indoor small potted plants and redeploy the potted plants to their original locations after maintenance, which can reduce the difficulty and cost of potted plant management to a certain extent.

[0004] Among them, the proposed indoor potted plant handling robot can autonomously realize the recognition and positioning of potted plants, and carry the potted plants according to the obtained location information. During the handling process, the robot will first realize the synchronous positioning and mapping of the current indoor environment, and plan the corresponding path for handling. In terms of potted plant robots, related research mainly focuses on mechanical watering or timed watering with pipes. In the development of robots towards intelligence, designing an intelligent robot for accurate identification and handling of potted plants has important practical significance and application value. The proposed indoor desktop greenhouse terminal has a lighting module and a watering module, which can realize all-weather precise irradiation of potted plants and mechanical pipe timed watering.

[0005] There is no dedicated potted plant handling robot on the market. Potted plant handling robots can be seen as a combination of manipulators and mobile robots. At present, most mobile robots in practical applications require preset routes. Relying on electromagnetic, magnetic tape, and visual QR code for navigation, but they each have obvious defects. Electromagnetic and magnetic tape require site construction, and later product line updates require re-construction, which takes a long time and requires a large investment; QR codes are not so troublesome, but they need to be cleaned and maintained regularly, and QR codes must be re-laid when changing product lines. Autonomous positioning and navigation is the development trend of handling robots. At present, autonomous positioning and navigation is becoming more and more mature. There are mainly laser SLAM and visual SLAM, but both have their own shortcomings. Laser SLAM has good stability, high reliability, long measurement distance and high accuracy (generally 3-5cm, and auxiliary positioning can reach ±1cm), but it will fail on black surfaces, glass, mirrors, and single long-distance environments. The most important thing is that it is expensive; visual SLAM has a wide measurement range and complete information acquisition, but it processes a large amount of data, has poor real-time performance, and is greatly affected by the environment (such as light); in actual applications, they will be coordinated and supplemented by ultrasonic, IMU (inertial measurement unit), odometer and other sensors, but most companies only use ultrasonic waves as decorations and do not use them because they will interfere with lidar.

[0006] There is a strong demand for handling robots in factories, and many companies are considering using handling robots to replace manpower. However, there are two core problems. First, the cost is high, mainly in the lidar. The radar cost of a handling robot may be 20,000 to 30,000 yuan, while the company's price for the whole handling robot is around 50,000 yuan. Second, there is a technical problem. The autonomous navigation algorithm is not yet perfect and will fail in some environments. Third, there are related supporting equipment. It is impossible for an enterprise to only purchase a complete handling robot. It also needs supporting automatic charging, shelves, containers, and scheduling systems (to connect to their WMS warehouse management system). However, most companies do not have the ability to provide a full set of products and services. Only when these two core problems are solved will autonomous positioning and navigation handling robots be popularized in factories.

[0007] At present, the common mobile robot products are:

[0008] Turtlebot is an entry-level mobile robot. It provides basic hardware architecture and mobile platform, as well as an open source operating system based on ROS. Programmers only need to use the various interfaces provided by the platform to implement the required functions (robot positioning, SLAM mapping, navigation, following, etc.).

[0009] Turtlebot2 is equipped with a Microsoft Kinect camera as standard. After installing a lidar, it can realize visual SLAM and laser SLAM based on ROS. At the same time, a charging station can be freely selected.

[0010] Clearpath Robotics, Canada's Clearpath company's products are mainly aimed at universities and research institutes. It provides various outdoor mobile platforms equipped with different sensors to achieve autonomous navigation, mobile reconnaissance, mobile grasping and other functions.

[0011] Ridgeback is mainly used for indoor mobile platforms, 960x793x296mm, maximum load 100Kg, maximum speed 1.1m / s, and can achieve autonomous movement, obstacle avoidance, and path planning based on lidar.

[0012] KUKA, YouBot Mecanum wheeled omnidirectional mobile platform, equipped with a 5-axis robotic arm, is a classic product for researching mobile grasping.

[0013] Kuaikan Technology did not use the more technologically advanced fully autonomous navigation technology, but instead used the more reliable method of visual QR code + inertial navigation.

[0014] Many scholars have conducted research on automatic assembly lines, palletizing and handling, and container handling robots, but have rarely involved other aspects. For example, Zhang Yawan et al. have developed a set of sorting robots with STM32H7 as the core controller, which uses OpenMV cameras to collect, process and analyze images, and can achieve efficient and intelligent sorting; Li Maosen designed a visual recognition system for intelligent airport baggage handling based on a robotic arm. The system uses machine vision and a robotic arm, uses industrial cameras to process and analyze luggage data, and grabs, transports, and stacks luggage on the airport slide, changing the traditional manual handling work method and realizing intelligent loading and unloading of passenger luggage. Xu Guoshu et al. proposed an industrial robot based on RGBD visual positioning to realize automatic lithium battery loading technology, using an industrial robot hand-eye automatic calibration method based on scene features and a lithium battery recognition detection based on point cloud images, and a lithium battery posture estimation method based on RGB-D color images and depth images, etc., which can realize automatic lithium battery loading. Xiao Shaotong used RGB-D camera and point cloud segmentation algorithm based on concave-convex features to realize point cloud target recognition, and verified the effectiveness of the algorithm through experiments. Qi Rong Zhang et al. used OpenCV functions to perform a series of image processing on cherry images in natural environments, and then drew the center and circular outline of the cherry, achieving the purpose of machine picking. Wenbin Zhang et al. proposed a research plan of "color recognition and grasping robot based on OpenCV", using visual recognition and positioning technology and manipulator device to sort and transport targets with different color features. Nong Yuanfeng used OpenCV to design a urine test strip recognition system with high execution efficiency, which provided basic conditions for Internet medical care. Xu Xiaoyan et al. designed a color recognition and sorting design system based on single-chip microcomputer, and used TCS3200 color sensor to identify the collected color information. Zhao Wei et al. introduced an intelligent color recognition robot based on VEX platform and implemented its design using ROBOTC language. It can realize autonomous walking through wheeled chassis, realize color recognition through color sensor and can grasp objects with mechanical arm. Zhou Xin, Liu Shuodi and others proposed a deep neural network model based on multi-color space information to realize vehicle color recognition in natural traffic scenes, which can achieve high recognition accuracy and low computational complexity. Wei Qi proposed an object recognition method based on the fusion of color and shape features by integrating the shape recognition and color recognition of objects in two-dimensional images, which effectively improved the object recognition rate. Sakshi et al. proposed a handwritten mathematical expression segmentation and contour detection algorithm based on OpenCV, which has a high accuracy rate on the acquired data set; Harish Adusumalli et al. proposed a mask detection algorithm based on OpenCV. Meng Wang, Wenzhi Wu, Bai Jingshu and others have made many attempts in the recognition strategy based on OpenCV.

[0015] In the existing potted plant maintenance robots and systems, only watering or lighting are required for the potted plants. The potted plants are in a fixed position and need to be manually moved, which still incurs labor costs, and the flower pots cannot be identified and located. The existing RGB-D or OpenCV-based visual recognition and positioning robots do not combine the object contour and depth to obtain the object's precise coordinates and object information.

[0016] The existing handling robots have not been combined with indoor small-scale intelligent maintenance greenhouse box terminals to form a small potted plant intelligent maintenance and reconstruction deployment system. QR codes are not used when positioning and adjusting the robot's posture. Summary of the invention

[0017] The main purpose of the present invention is to provide a system and method for determining the bonsai grasping posture and parking the robot, combining the potted plant handling robot with an indoor small-scale intelligent maintenance greenhouse box terminal to form an indoor small-scale potted plant intelligent maintenance and reconstruction deployment system, realizing all-weather precise irradiation of potted plants and timed watering of mechanical pipelines, etc., which can maintain and manage indoor small-scale potted plants and redeploy the potted plants to their original positions after maintenance.

[0018] According to one aspect of the present invention, there is provided a system for determining a bonsai grasping posture and parking a robot, comprising an information IoT and intelligent sensing node and a motion control node;

[0019] The information IoT and intelligent sensing node includes a Raspberry Pi 4B microprocessor and an RGB-D camera;

[0020] The motion control node is the bottom PCB motion control board, which is used for the robot to complete the control of the chassis and the robotic arm under the drive of the motor and determine the bonsai grasping posture.

[0021] According to another aspect of the present invention, there is provided a method for determining a bonsai grasping posture and parking a robot, comprising:

[0022] The information IoT and intelligent perception nodes process the color images taken by the RGB-D camera through the OpenCV series functions and convert the obtained flower pot centroid into the X coordinate system of the camera coordinate system. C ,Y C and Z C The coordinates, and the radius R of the flower pot are sent to the motion control node in the form of low eight bits and high eight bits respectively;

[0023] After receiving the command from the host computer, the motion control node controls the robot to move near the flower pot and adjust the opening and closing of the mechanical claw; the single chip controls the mechanical claw to grab the flower pot by controlling the servo;

[0024] The flower pots are recognized by color. Each frame of the RGB image acquired by the depth camera is segmented by color contrast. The segmented images are binarized and then dilated and eroded. Smaller areas are then removed to extract the largest connected domain.

[0025] The canny operator is used to detect image edges and extract the edge of the flower pot. Finally, the edge points of the flower pot and their depth information are combined, the flower pot is divided into multiple slices according to height, and each height slice is processed separately. The horizontal coordinate of the edge point is retained, and the vertical coordinate is replaced with the depth information of the edge point to obtain a quasi-circle of the edge of the flower pot from the above perspective. The circle is fitted to the quasi-circle using the least squares method to obtain the centroid of the circle, that is, the centroid of the flower pot, so as to facilitate the subsequent grasping and carrying of the flower pot and accurately position the flower pot.

[0026] Furthermore, the accurate positioning of the flower pot includes the following algorithm:

[0027] Highly robust flower pot block extraction;

[0028] Key parameter fitting algorithm for flower pot grabbing and handling;

[0029] Calibration calculation of external parameters for potted plant handling robot;

[0030] Gripper posture calculation of omnidirectional mobile robot for potted plant handling;

[0031] Parking posture calculation of omnidirectional mobile robot.

[0032] Furthermore, the highly robust flower pot block extraction includes:

[0033] Use RGB three-channel color contrast for color;

[0034] Image binarization operation: convert the three-channel RGB image into a grayscale image through the cvtColor function in the OpenCV function library; in the obtained grayscale image, determine the binarization threshold through the Otsu method to obtain a binary image;

[0035] The image is processed using the dilation and erosion techniques. Dilation is to convolve some images with the kernel to find the local maximum value, while erosion is the reverse operation of dilation, which calculates the minimum value of the pixels in the kernel area.

[0036] Extract the largest connected domain, calibrate the connected domains in the image, output the number of pixels in each connected domain, and paint the smaller areas in black to match the background color.

[0037] Furthermore, the key parameter fitting algorithm for flower pot grabbing and handling includes:

[0038] Edge detection, using the Canny algorithm to detect the edge of the flower pot;

[0039] Based on the least squares circle fitting, the edge points in the slice with the most edge points are fitted with a circle using the least squares method.

[0040] Furthermore, the external parameter calibration calculation of the potted plant handling robot includes:

[0041] The intersection of the center axis of the chassis of the potted plant handling robot and the ground is taken as the origin O of the coordinate system of the potted plant handling robot. r , place the potted plant in the coordinate system of the potted plant handling robot, and take any three points on the surface of the potted plant. The coordinates are obtained by measurement, which are A(X r1 ,Y r1 ,Z r1 ), B(X r2 ,Y r2 ,Z r2 ), C(X r3 ,Y r3 ,Z r3 ); Read the pixel coordinates of the above three points A, B, and C in the mixed color camera pixel coordinate system, which are [u1, v1] respectively. T , [u2,v2] T , [u3,v3] T ;

[0042] Assume that the coordinates of points A, B, and C in the camera coordinate system are [X1, Y1, Z1] T , [X2,Y2,Z2] T , [X3,Y3,Z3] T ; The coordinates of points A', B', and C' on the physical imaging plane are [X'1, Y'1, Z'1] respectively. T ,[X'2,Y'2,Z'2] T ,[X'3,Y'3,Z'3] T ,According to the triangle similarity relationship, we have: Where f is the focal length;

[0043] If the imaging plane is symmetrical to the front of the camera, the above formula can be simplified to: Arranged:

[0044] There is a difference between the pixel coordinate system and the imaging plane, a scaling and a translation of the origin. Suppose the pixel coordinate system is scaled by α times on the u axis and β times on the v axis. At the same time, the origin is translated by [c x ,c y ] T , then the relationship between the coordinates of point A' and the pixel coordinates is:

[0045]

[0046] From the above formula, αf is combined into f x , merge βf into f y ,have to:

[0047]

[0048] The above formula can be written in matrix form as follows:

[0049] but:

[0050]

[0051] Since Z1 is the depth information directly read by the camera, the RGB-D mixed color camera is obtained by calibration

[0052] The head internal parameter matrix is

[0053] It is fixed after the camera leaves the factory and will not change with use.

[0054] [u1,v1] T Knowing this, we can find the coordinates of point A in the camera coordinate system, and the same goes for points B and C.

[0055] The transformation relationship between the two coordinate systems is:

[0056]

[0057] Furthermore, the posture calculation of the hand gripper of the potted plant transporting omnidirectional mobile robot includes:

[0058] After the omnidirectional mobile robot approaches the flower pot, it switches between four gripper postures in sequence: initial posture, aiming posture, embracing posture, and clamping posture to smoothly pick up the flower pot.

[0059] In the initial posture, the robot places the gripper in front, that is, the gripper x coordinate x cs = 0, the closest position to the body is the gripper y coordinate y cs = l min , the height is the highest point, that is, the hand height h cs =h max , gripper angle θ cs =0°;

[0060] The aiming and embracing posture methods of the gripper need to first use the aforementioned method to calculate the coordinate position x of the flower pot relative to the center of the robot. hp ,h hp , height h hp , radius r hp ;

[0061] The alignment posture of the robot gripper is calculated based on the following given geometric constraints. The distance b between the two fingertips of the gripper in the alignment posture is dz It should be slightly larger than the diameter of the flower pot, which is set here as the fingertip flower pot radius r hp twice plus the fingertip gap b in the initial state of the gripper cs ;

[0062] The line connecting the two fingertips of the gripper in the aiming posture b dz Tangent to the border of the flower pot;

[0063] Lifting height of gripper h dz Equal to the height of repotting h hp ;

[0064] The calculation formula is as follows:

[0065]

[0066] In the above formula, sz That is, the length of the fingers of the gripper, and then the gripper alignment posture adopted is derived, including the gripper opening angle θ dz , x coordinate x dz , y coordinate y dz , gripper height h dz The estimation formula is as follows:

[0067]

[0068] The hugging posture of the robot gripper is calculated based on the following given geometric constraints:

[0069] The environmental posture of the robot gripper is calculated based on the following given geometric constraints,

[0070] The environmental posture of the gripper is equivalent to the robot arm extending and pushing a distance d in the aiming posture hb , d hz The radius of the flower pot must be exceeded to ensure that the center of the flower pot is within the gripper's embrace area. Set d hb The value is the radius of the flower pot r hp Add the given forward margin d qj

[0071] The distance between the two fingertips and the height of the gripper in the hugging posture should be the same as in the aiming posture.

[0072] The calculation formula is as follows:

[0073]

[0074] Derive the adopted gripper alignment posture, including the gripper opening angle θ hb , x coordinate x hb , y coordinate y hb , gripper height hhb The estimation formula is as follows

[0075]

[0076] When the robot arm's gripper enters the clamping state from the environmental posture, the position of the gripper does not change, but only contracts. If a sudden increase in the gripper current is detected, it is determined that the flower pot has been clamped. At this time, the gripper angle position is maintained, and then the gripper position is adjusted to be consistent with the initial state, that is, the picking process is completed. The robot can then approach the potted plant maintenance box, and under the guidance of the parking QR code, adjust the robot to the specified parking position relative to the maintenance box, and then use the given gripper coordinates to place the flower pot on a fixed workstation in the greenhouse box so that the potted plant maintenance box can spray and maintain the potted plants in the flower pot.

[0077] Furthermore, the parking posture calculation of the omnidirectional mobile robot includes:

[0078] Guided by the parking QR code, the robot adjusts its designated parking position relative to the maintenance box. The robot captures the QR code image directly below the vacant station of the bonsai maintenance box, captures the pixel coordinates of the four positioning points A in the upper left corner, B in the upper right corner, C in the lower right corner, and D in the lower right corner of the QR code, and uses the pixel coordinate values ​​to express the parking adjustment of the posture:

[0079] The robot's orientation is adjusted by calculating the ratio λ of the pixel distance AD ​​between the upper left corner and the lower left corner and the pixel distance BC between the upper right corner and the lower right corner; if the ratio λ is greater than 1, it means that the robot is tilted to the right, that is, its left side is close to the greenhouse box and its left side is far from the greenhouse box. At this time, the robot should be rotated counterclockwise in situ, that is, slowly to the left; conversely, if the ratio λ is less than 1, the robot should be rotated slowly clockwise in situ; by rotating clockwise and counterclockwise, the value is made close to 1, so that the robot's orientation is basically consistent with the workstation entrance;

[0080] When the orientation of the robot is basically consistent with the entrance of the workstation, the average value of the horizontal coordinates of the pixels at the four corner points is calculated. If the average value is less than the central horizontal coordinate of the camera's field of view, it means that the workstation is on the left side of the robot, and the robot should slowly move to the left. If the average value is greater than the central horizontal coordinate of the camera's field of view, it means that the workstation is on the right side of the robot, and the robot should slowly move to the right. If the average value is close to the central horizontal coordinate of the camera's field of view, it means that the workstation is directly in front of the robot.

[0081] When the robot is facing the same direction as the workstation entrance and the workstation is directly in front of the robot, move vertically toward the workstation and use the following formula to calculate the area S of the quadrilateral enclosed by the four corner points of ABCD: ABCD :

[0082] When SABCD Will continue to increase when S ABCD When it increases to a certain given value, it means that the robot is close to the greenhouse box at a given distance. At this moment, the direction of the robot is basically consistent with the entrance of the workstation, and it is parked at a given distance in front of an empty workstation in the greenhouse box. The flower pot can be placed on the workstation in front in a fixed posture through the manipulator for spraying and maintenance.

[0083] Advantages of the present invention:

[0084] The present invention innovatively integrates mainstream technologies such as ROS robot system, RGB-D camera, image processing method of OpenCV4 function library, and designs a flower pot parameter extraction algorithm for indoor potted plant handling robot system based on RGB-D camera, which can autonomously realize the recognition and positioning of potted plants, and carry potted plants according to the key position information obtained. Compared with the traditional manual handling method, the present invention reduces the difficulty and cost of potted plant management to a certain extent.

[0085] Under natural lighting conditions and incandescent lamp irradiation conditions, the present invention can realize accurate recognition and positioning of the potted plants to be transported, obtain accurate information on the centroid and radius of the potted plants, and has the characteristics of strong robustness, etc., providing technical support for applications in object recognition and positioning.

[0086] The present invention combines a potted plant handling robot with an indoor small-scale intelligent maintenance greenhouse box terminal to form an indoor small-scale potted plant intelligent maintenance and reconstruction deployment system, which realizes functions such as all-weather precise irradiation of potted plants and timed watering through mechanical pipelines. It can maintain and manage indoor small-scale potted plants and redeploy the potted plants to their original positions after maintenance, providing a technical reference for the design and application of intelligent handling robots and intelligent potted plant maintenance and reconstruction systems.

[0087] In addition to the above-described purposes, features and advantages, the present invention has other purposes, features and advantages. The present invention will be further described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] The drawings constituting a part of this application are used to provide a further understanding of the present invention. The illustrative embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0089] Figure 1 It is a diagram of the robot coordinate system and the mixed color camera coordinate system of the present invention;

[0090] Figure 2 These are four posture diagrams of the hand gripper of the potted plant handling robot of the present invention;

[0091] Figure 3 It is a schematic diagram of the mobile robot parking and the parking QR code of the present invention;

[0092] Figure 4 This is a structural diagram of the electric control system of the small-sized intelligent greenhouse box of the present invention;

[0093] Figure 5 is a flow chart of flower pot identification and processing of the present invention;

[0094] Figure 6 is a schematic diagram of the potted plant transporting robot of the present invention;

[0095] Figure 7 It is a system illustration diagram of the present invention.

[0096] Reference numerals:

[0097] 1 is a PCB motion control board, 2 is a Raspberry Pi controller, 3 is a potted plant, 4 is a depth camera, 5 is a Mecanum wheel, 6 is an indoor small-scale smart potted plant maintenance greenhouse, 7 is a cylindrical bottom potted plant, 8 is an RGB-D camera, 9 is a parking guidance QR code and 10 is an omnidirectional mobile potted plant transporting machine. DETAILED DESCRIPTION

[0098] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0099] The potted plant handling robot system of the present invention is based on standardization and modularization. According to different functions, the system is divided into the following two parts: information Internet of Things and intelligent perception node, and motion control node. The information Internet of Things and intelligent perception node is composed of a Raspberry Pi 4B microprocessor and a Realsense D435i RGB-D camera. The motion control node is the bottom PCB, and the robot completes the control of the chassis and the mechanical arm under the drive of the motor.

[0100] The information IoT and intelligent perception nodes process the color images taken by the RGB-D camera through a series of OpenCV functions, and then convert the obtained flower pot centroid into the X coordinate system of the camera coordinate system. C ,Y C and Z C The coordinates and the radius R of the flower pot (in mm) are sent to the motion control node in the form of low eight bits and high eight bits respectively. After receiving the command from the host computer, the motion control node controls the robot to move near the flower pot and adjust the opening and closing of the manipulator. The single chip controls the manipulator to grab the flower pot by controlling the servo.

[0101] The flower pot recognition can be realized by using OpenCV functions. This technology uses Windows 10 as the development platform, Visual studio 2019 and OpenCV 4.5.5 open source library as development tools to design and program the flower pot recognition system. The flower pot recognition process is as follows: Figure 5 shown.

[0102] First, the flower pot is identified by color. Each frame of the RGB image obtained by the depth camera is segmented by color contrast. After the segmented image is binarized, the expansion and corrosion operation is performed, and then the smaller area is eliminated to extract the largest connected domain. The canny operator is then used to detect the edge of the image and extract the edge of the flower pot. Finally, the edge points of the flower pot and its depth information are combined, and the flower pot is divided into multiple "slices" according to the height, and each height "slice" is processed separately. The horizontal coordinate of the edge point is retained, and the vertical coordinate is replaced with the depth information of the edge point to obtain the circle-like edge of the flower pot from the above perspective. The circle is fitted to the circle-like circle by the least squares method to obtain the centroid of the circle, that is, the centroid of the flower pot, so as to facilitate the subsequent grasping and handling of the flower pot to accurately locate the flower pot.

[0103] 1. Highly robust flowerpot block extraction strategy

[0104] 1. Color segmentation

[0105] Since the color of the potted flowerpot is brown, which is quite different from other colors in the environment, using the color features of the flowerpot for identification is an effective way. When the robot works under natural lighting conditions, light and other color-like objects (such as when the experimenter's hand or other objects of similar color are near the flowerpot) have a greater impact on image color segmentation. In order to eliminate or reduce the impact of lighting, color segmentation is performed through the color contrast of the three channels R, G, and B.

[0106] The steps of color segmentation using RGB three-channel color contrast are as follows: First, traverse the RGB image and find the minimum component of the three channels R, G, and B in each pixel. Since the transparent plastic on the surface of the flower pot has reflective properties, the color values ​​of the three channels are larger than the actual values. Therefore, the color value of each channel is subtracted by 0.9 times the minimum component (this value is a test constant) to reduce the reflection of the plastic and highlight the color characteristics. After that, multiply the color value of each color channel of each pixel by 255 and divide it by the maximum component color value to reduce the difference in color segmentation results caused by lighting differences, so that its color characteristics can be effectively expressed under various lighting conditions, effectively enhancing the robustness of the algorithm to lighting. Since the color of the flower pot is brown, the R channel must be the largest component among its three color channels. Set two different Mats, namely Delete and Remain (to save the deleted part and the retained flower pot part respectively). After setting and testing different constant values ​​for many times, the following potted plant retention method is finally obtained: if the red color of the pixel in the image is the maximum component (i.e. R=255) and the blue channel component is greater than 8 and less than 100 and the green channel component is greater than or equal to the blue channel component, and the green channel component is less than the blue channel component plus 90. If this condition is met, the pixel point is drawn in Remain; if this condition is not met, the pixel coordinates are drawn in Delete and the pixel point is blackened in Remain.

[0107] 2. Binarization

[0108] The image binarization operation greatly reduces the amount of data in the image, so that the contour of the target in the image can be detected. It is widely used in image processing and is often used for target selection. The three-channel RGB image is converted into a grayscale image through the cvtColor function in the OpenCV function library. In the obtained grayscale image, the binarization threshold is determined by the Otsu method to obtain a binary image. While binarizing the image, the depth of pixels with a depth greater than 800 mm is set to zero to reduce the influence of the depth error point on the result of the least squares fitting circle.

[0109] 3. Expansion corrosion

[0110] After the binarization operation, the image obtained has a certain amount of noise, so it needs to be morphologically processed. Using the dilation and erosion technique to process the image can significantly eliminate or reduce the noise, so that the image can achieve the maximum degree of connectivity. Dilation refers to convolving some images with the kernel to find the local maximum value, and erosion is the inverse operation of dilation, which is to calculate the minimum value of the pixels in the kernel area. In OpenCV, the "cvDilate()" function is used to implement the dilation operation, and the "cvErode()" function is used to implement the erosion operation.

[0111] Generally speaking, dilation expands the area and fills the holes, while erosion shrinks the area and eliminates protrusions. Dilation is usually used to find connected branches, that is, large blocks of pixels with similar colors and intensities that are separated from each other, while erosion is generally used to remove "speckle" noise in the image. It can corrode the spots and ensure that larger areas in the image still exist. Therefore, the method of first corroding, then dilating, and then corroding in the present invention can effectively remove the edge noise and other noise effects of the detected object, and can achieve the purpose of restoring the edge contour of the flower pot.

[0112] 4. Extract the largest connected domain

[0113] After the image expansion and corrosion operation, in addition to the flower pot, there are other small areas with similar colors to the flower pot in the image. Therefore, by calibrating the connected domains in the image, outputting the number of pixels in each connected domain, the smaller areas are painted black with the same color as the background, so that only the flower pot is retained in the picture. In the present invention, the connected area is detected mainly by using the connectedComponentsWithStats() function of OpenCV4.

[0114] (II) Key parameter fitting algorithm for flower pot grabbing and handling

[0115] 1. Edge Detection

[0116] The present invention uses the Canny algorithm to detect the edge of the flower pot. The Canny edge detection method is currently the most widely used edge detection technology. Its main feature is to try to assemble individual candidate pixels into contours. The contour can be generated by applying hysteresis thresholds to these pixels. The upper and lower thresholds used in this experiment are 3:2.

[0117] 2. Circle fitting based on least squares method

[0118] After using the Canny algorithm to detect edges and find contours, for ease of processing, only the Y coordinates in the camera coordinate system are retained. C (i.e. the Y-axis coordinate of the pixel in the camera coordinate system, in mm) The edge point information with coordinates between -100mm and 100mm. The Y axis is divided into 19 height slices with a thickness of 20mm and an overlap of 10mm (i.e. [-100, -80) as the first slice, [-90, 70) as the second slice, and so on, and the last slice is [80, 100]. Combined with the camera intrinsic matrix, the obtained X-axis of the edge point is C Coordinates (i.e. the X-axis coordinates of the pixel point in the camera coordinate system, in mm) and Z C (i.e. the depth information of the pixel point, in mm) according to its height information in the camera coordinate system (in Y CThe units are mm) and the edge points in each slice are counted. The edge points in the slice with the most edge points are fitted with the least squares method to obtain the centroid and radius information of the flower pot, which is convenient for the robot gripper to grasp and carry the flower pot.

[0119] (III) Explanation of external parameter calibration calculation of potted plant handling robot

[0120] The relationship between the potted plant handling robot coordinate system and the mixed color camera coordinate system is as follows: Figure 1 shown.

[0121] The intersection of the center axis of the chassis of the potted plant handling robot and the ground is taken as the origin O of the coordinate system of the potted plant handling robot. r , place the potted plant in the coordinate system of the potted plant handling robot, and take any three points on the surface of the potted plant. The coordinates are obtained by measurement, which are A(X r1 ,Y r1 ,Z r1 ), B(X r2 ,Y r2 ,Z r2 ), C(X r3 ,Y r3 ,Z r3 ). Read the pixel coordinates of the above three points A, B, and C in the mixed color camera pixel coordinate system, which are [u1, v1] respectively. T , [u2,v2] T , [u3,v3] T . Assume that the coordinates of points A, B, and C in the camera coordinate system are [X1, Y1, Z1] T , [X2,Y2,Z2] T , [X3,Y3,Z3] T The coordinates of points A', B', and C' on the physical imaging plane are [X'1, Y'1, Z'1] respectively. T ,[X'2,Y'2,Z'2] T ,[X'3,Y'3,Z'3] T ,According to the triangle similarity relationship, we have: Where f is the focal length. To simplify the model, the imaging plane is symmetrical to the front of the camera, and the above formula is simplified to: Arranged:

[0122] There is a difference between the pixel coordinate system and the imaging plane, a scaling and a translation of the origin. Suppose the pixel coordinate system is scaled by α times on the u axis and β times on the v axis. At the same time, the origin is translated by [c x ,c y ] T , then the relationship between the coordinates of point A' and the pixel coordinates is:

[0123]

[0124] From the above formula, αf is combined into f x , merge βf into f y ,have to:

[0125]

[0126] The above formula can be written in matrix form as follows:

[0127] but:

[0128]

[0129] Since Z1 is the depth information directly read by the camera, the RGB-D mixed color camera internal parameter matrix obtained by calibration is: It is fixed after the camera leaves the factory and will not change with use. [u1,v1] T Knowing this, we can find the coordinates of point A in the camera coordinate system, and the same goes for points B and C. The transformation relationship between the two coordinate systems is (taking the average):

[0130]

[0131] (IV) Calculation of the gripper posture of the omnidirectional mobile robot for potted plant handling

[0132] The omnidirectional mobile robot approaches the flower pot and passes by Figure 2 The four gripper postures shown, namely, initial posture, aiming posture, embracing posture and clamping posture, are switched in sequence to complete the smooth picking up of the flower pot;

[0133] In the initial posture, the robot places the gripper in front, that is, the gripper x coordinate x cs = 0, the closest position to the body is the gripper y coordinate y cs = l min , the height is the highest point, that is, the hand height h cs =h max , gripper angle θ cs =0°.

[0134] The aiming and embracing posture methods of the gripper need to first use the aforementioned method to calculate the coordinate position x of the flower pot relative to the center of the robot. hp ,h hp , height h hp , radius r hp .

[0135] The alignment pose of the robot gripper is calculated based on the following given geometric constraints,

[0136] 1) The distance b between the two fingertips of the gripper in the aiming posturedz It should be slightly larger than the diameter of the flower pot, which is set here as the fingertip flower pot radius r hp twice plus the fingertip gap b in the initial state of the gripper cs .

[0137] 2) The line b connecting the two fingertips of the gripper in the aiming posture dz Tangent to the border of the flower pot

[0138] 3) Lifting height of the gripper h dz Equal to the height of repotting h hp .

[0139] The calculation formula is as follows:

[0140]

[0141] In the above formula, sz That is, the length of the fingers of the gripper, and then the gripper alignment posture adopted is derived, including the gripper opening angle θ dz , x coordinate x dz , y coordinate y dz , gripper height h dz The estimation formula is as follows

[0142]

[0143] The hugging posture of the robot gripper is calculated based on the following given geometric constraints:

[0144] The environmental posture of the robot gripper is calculated based on the following given geometric constraints,

[0145] 1) The environmental posture of the gripper is equivalent to the robot arm extending and pushing a distance d in the aiming posture hb , d hz The radius of the flower pot must be exceeded to ensure that the center of the flower pot is within the gripper's embrace area. Set d hb The value is the radius of the flower pot r hp Add the given forward margin d qj

[0146] 2) The distance between the two fingertips and the height of the gripper in the embracing posture should be the same as in the aiming posture

[0147] The calculation formula is as follows:

[0148]

[0149] Then the adopted gripper alignment posture is derived, including the gripper opening angle θ hb , x coordinate x hb , y coordinate y hb , gripper height h hb The estimation formula is as follows

[0150]

[0151] When the robot arm's gripper enters the clamping state from the environmental posture, the position of the gripper does not change, but only contracts. If a sudden increase in the gripper current is detected, it is determined that the flower pot has been clamped. At this time, the gripper angle position is maintained, and then the gripper position is adjusted to be consistent with the initial state, that is, the picking process is completed. The robot can then approach the potted plant maintenance box, and under the guidance of the parking QR code, adjust the robot to the specified parking position relative to the maintenance box, and then use the given gripper coordinates to place the flower pot on a fixed workstation in the greenhouse box so that the potted plant maintenance box can spray and maintain the potted plants in the flower pot.

[0152] (V) Calculation of the parking posture of the omnidirectional mobile robot

[0153] As described in the above method, under the guidance of the parking QR code, the robot adjusts the designated parking posture relative to the maintenance box. The robot uses the QR code image captured directly below the vacant station of the bonsai maintenance box to capture the pixel coordinates of the four positioning points A in the upper left corner, B in the upper right corner, C in the lower right corner, and D in the QR code, and uses the pixel coordinate value to express the parking adjustment of the posture:

[0154] The robot's orientation is adjusted by calculating the ratio λ of the pixel distance AD ​​between the upper left corner and the lower left corner and the pixel distance BC between the upper right corner and the lower right corner; if the ratio λ is greater than 1, it means that the robot is tilted to the right, that is, its left side is close to the greenhouse box and its left side is far from the greenhouse box. At this time, the robot should be rotated counterclockwise in situ, that is, slowly to the left; conversely, if the ratio λ is less than 1, the robot should be rotated slowly clockwise in situ; by rotating clockwise and counterclockwise, the value is made close to 1, so that the robot's orientation is basically consistent with the workstation entrance;

[0155] When the orientation of the robot is basically consistent with the entrance of the workstation, the average horizontal coordinates of the pixels at the four corner points are calculated. If the average value is smaller than the central horizontal coordinate of the camera's field of view, it means that the workstation is on the left side of the robot, and the robot should slowly move to the left. If the average value is larger than the central horizontal coordinate of the camera's field of view, it means that the workstation is on the right side of the robot, and the robot should slowly move to the right. If the average value is close to the central horizontal coordinate of the camera's field of view, it means that the workstation is directly in front of the robot.

[0156] When the robot is facing the same direction as the workstation entrance and the workstation is directly in front of the robot, move vertically toward the workstation and use the following formula to calculate the area S of the quadrilateral enclosed by the four corner points of ABCD: ABCD :

[0157]

[0158] When S ABCD Will continue to increase when SABCD When the robot is close to a certain value, it means that the robot is close to the greenhouse box at a given distance. At this time, the robot's direction is basically consistent with the station entrance, and it is parked in front of the vacant station in the greenhouse box at a given distance. The robot arm can place the flower pot in the station in front of it in a fixed posture to receive spray maintenance.

[0159] (VI) Indoor small intelligent maintenance greenhouse

[0160] (1) Electronic control system structure

[0161] The electrical control system structure diagram of the indoor small intelligent maintenance greenhouse designed in this project is as follows: Figure 4 shown.

[0162] The electronic control system uses ESP32 as the core controller. The peripheral circuit is composed of DHT11 temperature and humidity sensor, VOC air quality sensor, and relay. When the system is working, the sensor collects environmental information in the greenhouse module, and sends the collected relevant data to the main control ESP32 chip for processing through UART and single bus. The processed information is sent to the host computer for display through Socket. At the same time, the host computer can also send instructions downward through Socket to remotely control lighting and pump irrigation.

[0163] (2) ESP32

[0164] The indoor small intelligent greenhouse uses ESP32 as the main control chip and the core of the entire control system. ESP32 is a high-performance, low-cost, ultra-low power, and highly integrated embedded IoT chip, which includes a 32-bit dual-core processor with a normal operating frequency of 80MHz and a maximum of 240MHz, and a computing power of up to 600MIPS; the chip has a built-in 448KB read-only memory and 520KB static random access memory; it supports up to 16MB off-chip SPI flash memory and 8MB off-chip SPI static random access memory.

[0165] The chip can operate in a temperature range of -40℃ to +125℃, which meets the requirements of greenhouse modules working in extreme climates. In addition to the rich interfaces, the chip also integrates a wireless communication module, of which the WiFi module supports 802.11-related wireless protocols, can operate in multiple carrier frequency bands, and has a transmission speed of up to 150Mbps; the integrated Bluetooth module supports the full standard of Bluetooth v4.2, including traditional Bluetooth and low-power Bluetooth. In this project, ESP32 connects to the specified WiFi network in STATION mode, establishes a Socket connection with the host computer in the same network, and realizes wireless communication. The chip interfaces used in the peripheral circuits include: DHT11 temperature and humidity sensor, I / O interface used for relay control, and UART interface used by VOC sensor, etc.

[0166] When performing flower pot recognition, in addition to using the traditional OpenCV method, you can also choose target detection algorithms such as YOLO V5 for flower pot recognition operations;

[0167] The color of the flowerpot may be a color different from the brown mentioned in the present invention. Accordingly, when performing the color segmentation operation, the settings of different color channel components in the algorithm may also be different.

[0168] When performing flower pot edge detection operations, other edge detection algorithms may also be selected;

[0169] When fitting the centroid and radius of the flower pot, in addition to the least squares method, other methods such as Hough circular transformation can also be used to fit the circle.

[0170] In the present invention

[0171] (1) Solving the centroid and radius of the potted plant

[0172] After performing a highly robust flower pot block extraction operation on the image captured by the RGB-D camera, the Canny algorithm is used for edge detection and contour search. Then, innovative slicing is performed, with the height divided into 19 height slices with a thickness of 20 mm and an overlap of 10 mm. The edge points in the slice with the most edge points are fitted with a circle using the least squares method to obtain the centroid and radius information of the potted plant, which is convenient for the subsequent robot gripper to grasp and carry the potted plant.

[0173] (2) Enhanced algorithm robustness

[0174] When the robot works under natural lighting conditions, if HSV color segmentation recognition is used, light and other color-like objects (such as when the hands of the experimenter or other objects of similar color are near the flower pot) have a greater impact on the color segmentation of the image. In order to eliminate or reduce the impact of lighting, color segmentation is performed through the color contrast of the three channels R, G, and B. Traverse the RGB image and find the minimum component in the three channels R, G, and B. Since the transparent plastic on the surface of the potted plant has reflective properties, the color values ​​of the three channels are larger than the actual values. Therefore, the color value of each channel is subtracted by 0.9 times the minimum component to reduce the reflection of the plastic and highlight the color characteristics. After that, the color value of each channel is multiplied by 255 and then divided by the maximum component color value to reduce the difference in color segmentation results caused by lighting differences, so that its color characteristics can be effectively expressed under various lighting conditions, effectively enhancing the robustness of the algorithm to lighting.

[0175] (3) Color segmentation using RGB three-channel color contrast

[0176] Since the color of the flowerpot is brown, the R channel is the largest component among its three color channels. Set two different Mats, Delete and Remain (to save the deleted part and the retained flowerpot part respectively). After setting and testing different constant values ​​for many times, the following method of retaining the pot is finally obtained: if the red color of the pixel in the image is the largest component (i.e. R=255) and the blue channel component is greater than 8 and less than 100 and the green channel component is greater than or equal to the blue channel component, and the green channel component is less than the blue channel component plus 90. If this condition is met, draw this pixel in Remain; if this condition is not met, draw this pixel coordinate in Delete, and paint this pixel black in Remain.

[0177] (4) Indoor potted plant handling robots combined with indoor small-scale intelligent maintenance greenhouse boxes form an indoor small-scale potted plant intelligent maintenance and reconstruction deployment system

[0178] The present invention combines the popular ROS system in recent years, the target recognition and processing algorithm of the OpenCV4 function library and the ORB-SLAM2 system to design an indoor potted plant handling robot that can accurately recognize and process potted plants. For the identified flower pots, the robot can carry them and simultaneously realize synchronous positioning and mapping.

[0179] In addition, the present invention changes the conventional irrigation method, abandoning the irrigation method of using a mechanical arm to water all potted plants within a certain range, and instead moves the potted plants to a small indoor intelligent maintenance greenhouse box, using the watering module in the greenhouse box terminal to achieve precise irrigation of the potted plants. And introduces a lighting module to simulate natural light to ensure the necessary photosynthesis of the potted plants.

[0180] For the indoor potted plant handling robot: the potted plant target recognition and processing method based on the OpenCV4 function library is used to perform binarization and other processing on the flower pots, and accurate and fast flower pot recognition is designed for the indoor potted plant handling robot. After extracting the depth information, the centroid of the flower pot is obtained to facilitate the grasping and carrying of the flower pots; combined with ORB-SLAM2, the robot's synchronous positioning and mapping system is designed and implemented; four Mecanum wheels are used to achieve reliable and flexible omnidirectional movement of the robot.

[0181] For indoor desktop greenhouse terminals: ESP32 is used to connect electromagnetic relays to control water pumps and light strips. In addition, ESP32 is used to create a hotspot to provide network connection for the Raspberry Pi and ensure that ESP32 and the Raspberry Pi communicate through sockets.

[0182] The purpose of the present invention is to use the RGB-D mixed color camera carried by a potted plant handling robot to identify flower pots and extract key parameters of the flower pots, and then guide the robotic claw to grasp and carry the flower pots. A flower pot parameter extraction algorithm based on RGB-D and OpenCV is provided, so that the potted plant handling robot can achieve satisfactory potted plant recognition, grasping and handling effects after obtaining the centroid and radius parameters of the flower pot.

[0183] The purpose of the present invention is to combine a potted plant handling robot with an indoor small-scale intelligent maintenance greenhouse box terminal to form an indoor small-scale potted plant intelligent maintenance and reconstruction deployment system, which can realize functions such as all-weather precise irradiation of potted plants and timed watering through mechanical pipelines, and can maintain and manage indoor small-scale potted plants, and redeploy the potted plants to their original positions after maintenance.

[0184] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for determining a bonsai grasping posture and parking a robot, characterized in that: include: The information IoT and intelligent perception nodes process the color images taken by the RGB-D camera through the OpenCV series functions and convert the obtained flower pot centroid into the X coordinate system of the camera coordinate system. C ,Y C and Z C The coordinates, and the radius R of the flower pot are sent to the motion control node in the form of low eight bits and high eight bits respectively; After receiving the command from the host computer, the motion control node controls the robot to move near the flower pot and adjust the opening and closing of the mechanical claw; the single chip controls the mechanical claw to grab the flower pot by controlling the servo; The flower pots are recognized by color. Each frame of the RGB image acquired by the depth camera is segmented by color contrast. The segmented images are binarized and then dilated and eroded. Smaller areas are then removed to extract the largest connected domain. Use the Canny operator to detect the edge of the image and extract the edge of the flower pot. Finally, combine the edge points of the flower pot and their depth information, divide the flower pot into multiple slices according to the height, and process each height slice separately. Keep the horizontal coordinate of the edge point, replace the vertical coordinate with the depth information of the edge point, and obtain the circle-like shape of the edge of the flower pot from the above perspective. Fit the circle to the circle-like shape by the least squares method to obtain the centroid of the circle, that is, the centroid of the flower pot, so as to facilitate the subsequent grasping and handling of the flower pot and accurately locate the flower pot. The precise positioning of the flower pots includes the following algorithms: Highly robust flower pot block extraction; Key parameter fitting algorithm for flower pot grabbing and handling; Calibration calculation of external parameters for potted plant handling robot; Gripper posture calculation of omnidirectional mobile robot for potted plant handling; Parking posture calculation of omnidirectional mobile robot.

2. The method for determining the bonsai grasping posture and parking the robot according to claim 1, characterized in that: The potted plant handling robot external parameter calibration calculation includes: The intersection of the center axis of the chassis of the potted plant handling robot and the ground is taken as the origin O of the coordinate system of the potted plant handling robot. r , place the potted plant in the coordinate system of the potted plant handling robot, and take any three points on the surface of the potted plant. The coordinates are obtained by measurement, which are A(X r1 ,Y r1 ,Z r1 ), B(X r2 ,Y r2 ,Z r2 ), C(X r3 ,Y r3 ,Z r3 ); Read the pixel coordinates of the above three points A, B, and C in the mixed color camera pixel coordinate system, which are [u1, v1] respectively. T , [u2,v2] T , [u3,v3] T ; Assume that the coordinates of points A, B, and C in the camera coordinate system are [X1, Y1, Z1] T , [X2,Y2,Z2] T , [X3,Y3,Z3] T ; The coordinates of points A', B', and C' on the physical imaging plane are [X'1, Y'1, Z'1] respectively. T ,[X'2,Y'2,Z'2] T ,[X'3,Y'3,Z'3] T ,According to the triangle similarity relationship, we have: Where f is the focal length; If the imaging plane is symmetrical to the front of the camera, the above formula can be simplified to: Arranged: There is a difference between the pixel coordinate system and the imaging plane, a scaling and a translation of the origin. Suppose the pixel coordinate system is scaled by α times on the u axis and β times on the v axis. At the same time, the origin is translated by [c x ,c y ] T , then the relationship between the coordinates of point A' and the pixel coordinates is: From the above formula, αf is combined into f x , merge βf into f y ,have to: The above formula can be written in matrix form as follows: but: Since Z1 is the depth information directly read by the camera, the RGB-D mixed color camera internal parameter matrix obtained by calibration is: It is fixed after the camera leaves the factory and will not change with use. [u1,v1] T Knowing this, we can find the coordinates of point A in the camera coordinate system, and the same goes for points B and C. The transformation relationship between the two coordinate systems is:

3. The method for determining the bonsai grasping posture and parking the robot according to claim 1, characterized in that: The hand gripper posture calculation of the potted plant transporting omnidirectional mobile robot includes: After the omnidirectional mobile robot approaches the flower pot, it switches between four gripper postures in sequence: initial posture, aiming posture, embracing posture, and clamping posture to smoothly pick up the flower pot. In the initial posture, the robot places the gripper in front, that is, the gripper x coordinate x cs = 0, the closest position to the body is the gripper y coordinate y cs = l min , the height is the highest point, that is, the hand height h cs =h max , gripper angle θ cs =0°; The aiming and embracing posture methods of the gripper need to first use the aforementioned method to calculate the coordinate position x of the flower pot relative to the center of the robot. hp ,h hp , height h hp , radius r hp ; The alignment posture of the robot gripper is calculated based on the following given geometric constraints. The distance b between the two fingertips of the gripper in the alignment posture is dz It should be slightly larger than the diameter of the flower pot, which is set here as the radius of the fingertip flower pot r hp twice plus the fingertip gap b in the initial state of the gripper cs ; The line connecting the two fingertips of the gripper in the aiming posture b dz Tangent to the border of the flower pot; Lifting height of gripper h dz Equal to the height of repotting h hp ; The calculation formula is as follows: In the above formula, sz That is, the length of the fingers of the gripper, and then the gripper alignment posture adopted is derived, including the gripper opening angle θ dz , x coordinate x dz , y coordinate y dz , gripper height h dz The estimation formula is as follows: The hugging posture of the robot gripper is calculated based on the following given geometric constraints: The environmental posture of the robot gripper is calculated based on the following given geometric constraints, The environmental posture of the gripper is equivalent to the robot arm extending and pushing a distance d in the aiming posture hb , d hz The radius of the flower pot must be exceeded to ensure that the center of the flower pot is within the gripper's embrace area. Set d hb The value is the radius of the flower pot r hp Add the given forward margin d qj The distance between the two fingertips and the height of the gripper in the hugging posture should be the same as in the aiming posture. The calculation formula is as follows: Derive the adopted gripper alignment posture, including the gripper opening angle θ hb , x coordinate x hb , y coordinate y hb , gripper height h hb The estimation formula is as follows When the robot arm's gripper enters the clamping state from the environmental posture, the position of the gripper does not change, but only contracts. If a sudden increase in the gripper current is detected, it is determined that the flower pot has been clamped. At this time, the gripper angle position is maintained, and then the gripper position is adjusted to be consistent with the initial state, that is, the picking process is completed. The robot can then approach the potted plant maintenance box, and under the guidance of the parking QR code, adjust the robot to the specified parking position relative to the maintenance box, and then use the given gripper coordinates to place the flower pot on a fixed workstation in the greenhouse box so that the potted plant maintenance box can spray and maintain the potted plants in the flower pot.

4. The method for determining the bonsai grasping posture and parking the robot according to claim 1, characterized in that: The highly robust flower pot block extraction includes: Use RGB three-channel color contrast for color; Image binarization operation: convert the three-channel RGB image into a grayscale image through the cvtColor function in the OpenCV function library; in the obtained grayscale image, determine the binarization threshold through the Otsu method to obtain a binary image; The image is processed using the dilation and erosion techniques. Dilation is to convolve some images with the kernel to find the local maximum value, while erosion is the reverse operation of dilation, which calculates the minimum value of the pixels in the kernel area. Extract the largest connected domain, calibrate the connected domains in the image, output the number of pixels in each connected domain, and paint the smaller areas in black to match the background color.

5. The method for determining the bonsai grasping posture and parking the robot according to claim 3, characterized in that: The key parameter fitting algorithm for flower pot grabbing and handling includes: Edge detection, using the Canny algorithm to detect the edge of the flower pot; Based on the least squares circle fitting, the edge points in the slice with the most edge points are fitted with a circle using the least squares method.

6. The method for determining the bonsai grasping posture and parking the robot according to claim 1, characterized in that: The parking posture calculation of the omnidirectional mobile robot includes: Guided by the parking QR code, the robot adjusts its designated parking position relative to the maintenance box. The robot captures the QR code image directly below the vacant station of the bonsai maintenance box, captures the pixel coordinates of the four positioning points A in the upper left corner, B in the upper right corner, C in the lower right corner, and D in the lower right corner of the QR code, and uses the pixel coordinate values ​​to express the parking adjustment of the posture: The robot's orientation is adjusted by calculating the ratio λ of the pixel distance AD ​​between the upper left corner and the lower left corner and the pixel distance BC between the upper right corner and the lower right corner; if the ratio λ is greater than 1, it means that the robot's orientation is tilted to the right, that is, its left side is close to the greenhouse box, and its left side is far from the greenhouse box. At this time, the robot should be rotated counterclockwise in situ, that is, slowly to the left; conversely, if the ratio λ is less than 1, the robot should be rotated slowly clockwise in situ; by rotating clockwise and counterclockwise, the value is made close to 1, so that the robot's orientation is consistent with the workstation entrance; When the robot is facing the same direction as the workstation entrance, the average value of the horizontal coordinates of the four corner pixels is calculated. If the average value is less than the central horizontal coordinate of the camera's field of view, it means that the workstation is on the left side of the robot, and the robot should slowly move to the left. If the average value is greater than the central horizontal coordinate of the camera's field of view, it means that the workstation is on the right side of the robot, and the robot should slowly move to the right. If the average value is close to the central horizontal coordinate of the camera's field of view, it means that the workstation is directly in front of the robot. When the robot is facing the same direction as the workstation entrance and the workstation is directly in front of the robot, move vertically toward the workstation and use the following formula to calculate the area S of the quadrilateral enclosed by the four corner points of ABCD: ABCD : When S ABCD Will continue to increase when S ABCD When it increases to a certain given value, it means that the robot is close to the greenhouse box at a given distance. At this moment, the robot's direction is consistent with the entrance of the workstation, and it is parked at a given distance in front of an empty workstation in the greenhouse box. The robot can place the flower pot on the workstation in front in a fixed posture through the manipulator for spraying and maintenance.

7. A system using the method of determining the bonsai grasping posture and parking the robot according to any one of claims 1 to 6, characterized in that: Including information Internet of Things, intelligent sensing nodes and motion control nodes; The information IoT and intelligent sensing node includes a Raspberry Pi 4B microprocessor and an RGB-D camera; The motion control node is the bottom PCB motion control board, which is used for the robot to complete the control of the chassis and the robotic arm under the drive of the motor and determine the bonsai grasping posture.