Intelligent picking robot and monitoring system

Abstract

The invention provides an intelligent picking robot and a monitoring system. Field environment monitoring data are generated through an environment monitoring module; identifying one or more target picking objects based on a pre-trained picking object identification model through a picking object identification module according to the field environment monitoring data and the acquired image data of each to-be-picked object; generating a three-dimensional map of a to-be-picked area and an optimal picking path through a picking navigation module; a picking execution control module is used for controlling the walking structure to arrive at each target picking position in sequence, and controlling the mechanical arm structure to execute a picking task; therefore, the intelligent picking robot can adapt to picking tasks of fruits and vegetables of different types and different maturity degrees, accurate recognition, positioning and picking of a target picking object can be achieved in a complex agricultural environment, and the fruit and vegetable picking efficiency and quality are improved.

Description

Technical Field

[0001] The present application relates to the technical field of agricultural robots, and in particular to an intelligent harvesting robot and a monitoring system. Background Art

[0002] With the growth of the global population and the increasing demand for high-quality agricultural products, the agricultural sector is gradually moving towards modernization and intelligence. Traditional fruit and vegetable picking mainly relies on manual operations. Although humans can judge the maturity of fruits and pick them based on vision and touch, this method has many shortcomings.

[0003] First, manual picking is costly, especially in areas with a labor shortage. Picking has very high seasonal and time requirements, which increases the difficulty and cost of picking. Secondly, the efficiency of manual picking is difficult to meet the needs of large-scale agricultural production. With the expansion of planting scale, manual picking cannot complete the picking of large-scale crops in a timely and effective manner. In addition, during the manual picking process, fruit damage caused by human factors often occurs, affecting the commercial value of the fruit.

[0004] In order to solve the above problems, in recent years, with the development of modern agriculture, intelligent agricultural robot technology has emerged and has gradually been applied in the planting, management, and harvesting of crops. Especially in the field of fruit and vegetable picking, intelligent picking robots can accurately locate the fruit by carrying sensors, and realize automatic fruit picking through actuators such as robotic arms.

[0005] However, existing intelligent picking robots still face many challenges in practical applications.

[0006] On the one hand, the maturity judgment standards of different fruits and vegetables are different, and the types of fruits and vegetables are diverse, with large differences in shapes and sizes. The existing intelligent picking robots have poor flexibility and adaptability, and only support the picking of single, specific types of fruits and vegetables. They cannot adapt to the picking of multiple fruits and vegetables, and are prone to damage the fruit, affecting the quality and commercial value of the fruit. On the other hand, fruit and vegetable picking generally takes place in complex field environments. The existing intelligent picking robots have poor ability to quickly identify and locate fruits, and are unable to quickly identify and locate ripe fruits and vegetables that need to be picked. They are also unable to accurately avoid obstacles and accurately plan picking paths, resulting in low picking efficiency. Summary of the invention

[0007] In view of the shortcomings of the prior art described above, the purpose of the present application is to provide an intelligent picking robot and a monitoring system to solve the technical problems of poor flexibility, poor adaptability, poor ability to identify and locate fruits and low picking efficiency of the existing intelligent picking robots.

[0008] To achieve the above-mentioned purpose and other related purposes, the first aspect of the present application provides an intelligent picking robot, which includes: a robot body, including: a walking structure, used to move the intelligent picking robot to a specified position; a mechanical arm structure, fixedly installed on the walking structure through a supporting structure; wherein the supporting structure includes: a plurality of support rods and a plurality of chassis, each chassis is detachably connected to a plurality of support rods around the periphery, and can move up and down along each support rod or be locked in a specified position; the mechanical arm structure includes: a plurality of mechanical arms and actuators correspondingly installed at the end of each mechanical arm, used to perform picking tasks according to picking instructions, and place the picked fruit and vegetable objects in a storage structure; each mechanical arm has multiple degrees of freedom, including: a fixed component and a plurality of telescopic structures, one end of which is fixed to each chassis through a fixed component; an environmental monitoring module, arranged on the robot body, used to collect field environmental monitoring data in real time; a picking object recognition module, arranged on the robot body, connected to the environmental monitoring module, used to collect image data of each object to be picked in real time According to the collected image data and the received field environment monitoring data, the pre-trained picking object recognition model is used to analyze the variety, size, shape and maturity of each object to be picked in a specific field environment, identify one or more target picking objects, and determine the picking priority and picking control parameters of each target picking object; a picking navigation module is arranged on the robot body and connected to the picking object recognition model, and is used to collect the position information and image data of each obstacle and each object to be picked in the current environment in real time, and generate a three-dimensional map of the area to be picked based on the collected information, so as to obtain the target picking position of each target picking object according to the three-dimensional map, and generate an optimal picking path in combination with the picking priority of each target picking object; a picking execution control module is arranged on the robot body, connected to the picking object recognition module and the picking navigation module, and is used to control the walking structure to reach each target picking position in turn according to the optimal picking path, and control the mechanical arm structure to perform a picking task on each target picking object according to each picking control parameter.

[0009] In some embodiments of the first aspect of the present application, the picking object recognition model is obtained by training based on the DT-DETR real-time target detection model; the DT-DETR real-time target detection model includes: a hybrid encoder, which is used to perform multi-scale feature extraction on each collected image data and the field environment monitoring data, generate multiple multi-scale feature data of each object to be picked, and perform self-attention operation on each multi-scale feature data based on the attention mechanism, perform feature fusion on each multi-scale feature data based on a convolutional neural network, and select the optimal feature data of each object to be picked based on the uncertainty minimum query selection mechanism and output it to the transformer decoder; the transformer decoder is used to generate multiple bounding boxes according to the optimal feature data of each object to be picked to select each object to be picked, and analyze the maturity of each object to be picked, classify each object to be picked based on maturity, determine one or more target picking objects suitable for picking, and the picking priority and picking control parameters of each target picking object.

[0010] In some embodiments of the first aspect of the present application, the method of training the DT-DETR real-time target detection model to obtain the picking object recognition model includes: obtaining picking object training samples under multiple specific field environments; wherein each picking object training sample includes: field environment historical monitoring data and historical image data; performing data enhancement on the field environment historical monitoring data and historical image data in each picking object training sample to generate a corresponding picking object enhanced data set; using an adaptive hyperparameter customization method to determine the optimizer hyperparameters of the DT-DETR real-time target detection model, and training the DT-DETR real-time target detection model with the picking object enhanced data set to obtain a primary picking object recognition model; dynamically verifying the generalization ability of the DT-DETR real-time target detection model, and according to the verification results, dynamically adjusting the intensity and type of data enhancement for each picking object training sample, optimizing and updating the model structure and optimizer hyperparameters of the DT-DETR real-time target detection model, and obtaining a converged picking object recognition model.

[0011] In some embodiments of the first aspect of the present application, a pressure sensor is provided on each actuator of the robotic arm structure, which is used to monitor in real time the pressure applied by the actuator to each target picking object when the robotic arm structure performs a picking task.

[0012] In some embodiments of the first aspect of the present application, the environmental monitoring module includes: one or more of a light sensor, a temperature sensor, a humidity sensor, and a soil moisture sensor, which are used to collect light monitoring data, air temperature monitoring data, air humidity monitoring data, and soil moisture monitoring data in real time, respectively, to generate field environment monitoring data.

[0013] In some embodiments of the first aspect of the present application, the picking navigation module includes: one or more of an inertial measurement unit, a lidar sensor, and an ultrasonic sensor, which are used to collect the position information of the intelligent picking robot in real time, and scan the current environment to obtain the position information and shape information of each obstacle and each object to be picked, so as to generate a three-dimensional map of the area to be picked by using multi-sensor fusion technology, and based on the three-dimensional map, obtain the target picking position of each target picking object, and generate the optimal picking path in combination with the picking priority of each target picking object.

[0014] In some embodiments of the first aspect of the present application, the intelligent picking robot also includes: a picking process monitoring module, which is arranged on the robot body, connected to the picking execution control module, the picking object identification module and the picking navigation module, and is used to monitor the picking process in real time and generate picking data, and send the picking data to the picking object identification module to optimize the picking object identification model, and send it to the picking navigation module to update the three-dimensional map of the area to be picked; wherein the picking data includes but is not limited to: actual picking position, actual picking control parameters, total picking amount, number of erroneous picking and number of damaged picking objects.

[0015] In some embodiments of the first aspect of the present application, the intelligent picking robot also includes: a wireless communication module, which is arranged on the robot body and connected to the picking process monitoring module, and is used to upload the generated picking data to an external monitoring center for analyzing the working status, picking progress, picking results and picking efficiency of the intelligent picking robot, and remotely managing and optimizing the intelligent picking robot.

[0016] In some embodiments of the first aspect of the present application, the intelligent picking robot further includes: a power module, arranged on the robot body, including: an energy storage battery and a solar panel, for providing continuous power for the intelligent picking robot; an energy management module, arranged on the robot body, for automatically adjusting the working mode of the intelligent picking robot according to the remaining target number of picking objects of the current picking task and the remaining power of the intelligent picking robot; wherein the working modes include: a first working mode, a second working mode and a charging mode; when the remaining power of the intelligent picking robot is greater than a preset first power threshold, the working mode is set to the first working mode, so that the intelligent picking robot The intelligent picking robot continues to perform the picking task according to the target picking strategy; when the remaining power of the intelligent picking robot is less than the preset first power threshold, the working mode is set to the second working mode, and one or more picking objects with the first priority level are selected from the remaining target picking objects as new target picking objects, the target picking strategy is updated, and the intelligent picking robot performs the picking task according to the updated target picking strategy; when the remaining power of the intelligent picking robot is less than the preset second power threshold, the working mode is set to the charging mode, and the intelligent picking robot returns to the charging warehouse for charging; wherein, the first power threshold is greater than the second power threshold.

[0017] To achieve the above-mentioned purpose and other related purposes, the second aspect of the present application provides an intelligent picking robot monitoring system, which includes: multiple intelligent picking robots as described in any one of the above embodiments, used to automatically perform picking tasks and generate corresponding picking data to be uploaded to a monitoring center through a wireless communication module; a monitoring center, connected to each of the intelligent picking robots, used to analyze the working status, picking progress, picking results and picking efficiency of each of the intelligent picking robots according to the picking data generated by each of the intelligent picking robots, and remotely manage and optimize each of the intelligent picking robots; a mobile terminal application end, connected to the monitoring center, used for users to monitor the working status, picking progress and picking results of a specified intelligent picking robot on the mobile terminal, and remotely control the operation of the intelligent picking robot.

[0018] As described above, the present application provides an intelligent picking robot and an intelligent picking robot monitoring system, which collects the position information and image data of each obstacle and each object to be picked in the current environment in real time through an image acquisition module, and generates a three-dimensional map of the area to be picked and a target picking strategy through a picking control module according to the collected position information and image data, so as to control the walking structure of the intelligent picking robot to move to the target picking position, and control its mechanical arm structure to perform the picking task; wherein the mechanical arm structure includes a plurality of mechanical arms with multiple degrees of freedom. The present application has the following beneficial effects: the intelligent picking robot described in the present application can adapt to the picking tasks of fruits and vegetables of different types and maturity, and can realize accurate identification, positioning and picking of target picking objects in a complex agricultural environment, improve the efficiency and quality of fruit and vegetable picking, and solve the problems of poor flexibility, poor adaptability and poor obstacle avoidance ability of existing intelligent picking robots.

[0019] As described above, the present application provides an intelligent picking robot and monitoring system, which generates field environment monitoring data through an environmental monitoring module; identifies one or more target picking objects based on the pre-trained picking object recognition model according to the field environment monitoring data and the collected image data of each object to be picked through a picking object recognition module; generates a three-dimensional map of the area to be picked and the optimal picking path through a picking navigation module; controls the walking structure to reach each target picking position in turn through a picking execution control module, and controls the mechanical arm structure to perform the picking task. Therefore, the present application has the following beneficial effects: the intelligent picking robot can adapt to the picking tasks of different types of fruits and vegetables with different maturity, and can accurately identify, locate and pick the target picking objects in a complex agricultural environment, thereby improving the efficiency and quality of fruit and vegetable picking. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 Shown is a schematic diagram of the structure of an intelligent picking robot in one embodiment of the present application.

[0021] Figure 2 Shown is a schematic diagram of the structure of the robot body in one embodiment of the present application.

[0022] Figure 3 Shown is a schematic diagram of the structure of a robotic arm in one embodiment of the present application.

[0023] Figure 4 Shown is a structural schematic diagram of an intelligent picking robot monitoring system in one embodiment of the present application. DETAILED DESCRIPTION

[0024] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0025] In order to solve the problems in the above-mentioned background technology, the present invention provides an intelligent picking robot and monitoring system, which aims to solve the technical problems of poor flexibility, poor adaptability, poor fruit identification and positioning ability and low picking efficiency of existing intelligent picking robots, and can accurately identify and locate target picking objects of fruits and vegetables of different types, shapes, sizes and maturity, and have efficient obstacle avoidance and path planning capabilities, and can adapt to the picking tasks of fruits and vegetables of different types, shapes, sizes and maturity in complex agricultural environments, thereby improving the efficiency and quality of fruit and vegetable picking. At the same time, in order to make the purpose, technical solutions and advantages of the present invention clearer, the technical solutions in the embodiments of the present invention are further described in detail through the following embodiments and in combination with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the invention.

[0026] like Figure 1 As shown, a schematic diagram of the structure of the intelligent picking robot in an embodiment of the present invention is shown. The intelligent picking robot in this embodiment includes: a robot body 1, an environment monitoring module 2, a picking object recognition module 3, a picking navigation module 4 and a picking execution control module 5. Among them, the environment monitoring module 2, the picking object recognition module 3, the picking navigation module 4 and the picking execution control module 5 are all arranged in the robot body 1, and are used to drive the robot body 1 to automatically perform the picking task.

[0027] The robot body 1 is as follows Figure 2 As shown, it includes: a walking structure 11, a mechanical arm structure 12 and a supporting structure 13.

[0028] Specifically, the walking mechanism 11 is used to move the intelligent picking robot to a designated position.

[0029] The mechanical arm structure 12 is fixedly mounted on the walking structure 11 via a supporting structure 13, and includes: a plurality of mechanical arms 121 and an actuator 122 correspondingly mounted at the end of each mechanical arm 121, for performing picking tasks according to picking instructions and placing the picked fruits and vegetables in a storage structure (not shown). Each mechanical arm 121 has multiple degrees of freedom.

[0030] like Figure 2 As shown, the support structure 13 includes a plurality of support rods 131 and a plurality of chassis 132. The periphery of each chassis 132 is detachably connected to the plurality of support rods 131, and can move up and down along the support rods 131 or be locked in a fixed position, so as to adjust the height of each chassis 132, so that each mechanical arm 121 installed on the chassis 132 can adapt to the picking environment at different heights and perform picking tasks for different kinds of fruits and vegetables, such as apples concentrated in higher tree crowns, strawberries distributed in the lower part of lower plants, and grapes distributed on trellises at different heights.

[0031] In one embodiment, if Figure 3 As shown, the robotic arm 121 includes: a fixing component 123 and a plurality of telescopic structures, such as a first telescopic structure 124, a second telescopic structure 125 and a third telescopic structure 126. One end of the robotic arm 121 is fixed to each chassis 132 of the support structure 13 through the fixing component 123. In addition, each chassis 132 includes a plurality of installation areas, on which a plurality of robotic arms 121 can be installed, and each robotic arm 121 can be installed at a different position. Each telescopic structure of the robotic arm 121 can be adjusted to be telescopic and can rotate in the horizontal direction, so that the robotic arm 121 has multiple degrees of freedom, can achieve a larger range of activities, and can adjust its angle, height and position according to different picking objects, thereby improving the adaptability and flexibility of the intelligent picking robot, so that it can support the picking of more types of fruits and vegetables.

[0032] In one embodiment, a pressure sensor is provided on each actuator 122 of the mechanical arm structure 12, which is used to monitor the pressure applied by the actuator 122 to each target picking object in real time when the mechanical arm structure 12 performs a picking task. In this embodiment, the purpose of such a design is that the intelligent picking robot can adjust the force applied by the actuator 122 according to the type, size and maturity of the object to be picked, thereby avoiding damage to the picked fruits and vegetables and maintaining the appearance, quality and commercial value of the fruits and vegetables.

[0033] The environmental monitoring module 2 is used to collect field environmental monitoring data in real time. In a specific embodiment, the environmental monitoring module 2 includes: one or more of a light sensor, a temperature sensor, a humidity sensor, and a soil humidity sensor, which are respectively used to collect light monitoring data, air temperature monitoring data, air humidity monitoring data, and soil humidity monitoring data in real time to generate field environmental monitoring data, and send the field detection data to the picking object identification module 3.

[0034] The picking object recognition module 3 is connected to the environmental monitoring module 2, and is used to collect image data of each object to be picked in real time. Based on the pre-trained picking object recognition model, according to the collected image data and the received field environment monitoring data, the module analyzes the variety, size, shape and maturity of each object to be picked in a specific field environment, identifies one or more target picking objects, and determines the picking priority and picking control parameters of each target picking object.

[0035] In one embodiment, the picking object recognition module 3 includes: one or more depth cameras for collecting image data of each object to be picked in real time, so as to combine the field environment monitoring data collected by the environment monitoring module 2, and accurately identify the target picking objects of the specified type based on the pre-trained picking object recognition model.

[0036] In a preferred embodiment, the picking object recognition model is obtained by training based on a DT-DETR real-time target detection model. The DT-DETR real-time target detection model includes: a hybrid encoder and a transformer decoder.

[0037] The hybrid encoder is used to perform multi-scale feature extraction on the collected image data and the field environment monitoring data respectively, generate multiple multi-scale feature data of each object to be picked, perform self-attention operation on each multi-scale feature data based on the attention mechanism, perform feature fusion on each multi-scale feature data based on the convolutional neural network, and select the optimal feature data of each object to be picked based on the uncertainty minimum query selection mechanism and output it to the transformer decoder.

[0038] In a specific embodiment, the hybrid encoder includes an internal scale feature interaction unit and a cross-scale feature fusion unit. The internal scale feature interaction unit adopts an attention mechanism, which can perform self-attention operations on multiple multi-scale feature data of each picking object, and dynamically adjust the degree of attention to each element by calculating the correlation scores between elements at different positions in the sequence data, thereby improving the flexibility and expressiveness of the DT-DETR real-time target detection model. The cross-scale feature fusion unit is based on a convolutional neural network and adopts multiple fusion blocks to perform feature fusion on each multi-scale feature data, and finally generate fused feature data of each object to be picked.

[0039] In this embodiment, the hybrid encoder also adopts the uncertainty minimum query selection mechanism, by defining the uncertainty by the difference between the predicted positioning distribution (P) and the classification distribution (C), and then minimizing this uncertainty during the training process to select the features that are most consistent with the classification and positioning predictions as the optimal feature data, so as to select the optimal feature data with the minimum uncertainty as the initial query. Thus, the uncertainty in the transformer decoder optimization process will be reduced, thereby improving the performance of the DT-DETR real-time target detection model.

[0040] The transformer decoder is used to generate multiple bounding boxes based on the optimal feature data of each object to be picked, to select each object to be picked, and to analyze the maturity of each object to be picked, to classify each object to be picked based on maturity, to determine one or more target picking objects suitable for picking, and the picking priority and picking control parameters of each target picking object. In a specific embodiment, the transformer encoder can also be connected to an auxiliary prediction head, which can effectively improve the learning effect of the DT-DETR real-time target detection model and improve the detection accuracy and detection capability of the model.

[0041] In one embodiment, the method of training the DT-DETR real-time target detection model to obtain the picking object recognition model includes the following steps.

[0042] ① Obtaining multiple picking object training samples in specific field environments; wherein each picking object training sample includes: field environment historical monitoring data and historical image data.

[0043] ② Perform data enhancement on the field environment historical monitoring data and historical image data in each picking object training sample to generate the corresponding picking object enhanced data set.

[0044] ③ An adaptive hyperparameter customization method is used to determine the optimizer hyperparameters of the DT-DETR real-time target detection model, and the DT-DETR real-time target detection model is trained by the picking object enhanced dataset to obtain a primary picking object recognition model.

[0045] ④ Dynamically verify the generalization ability of the DT-DETR real-time target detection model, and according to the verification results, dynamically adjust the intensity and type of data enhancement for each picking object training sample, optimize and update the model structure and optimizer hyperparameters of the DT-DETR real-time target detection model, and obtain a converged picking object recognition model.

[0046] In this embodiment, the DT-DETR real-time target detection model adopts a dynamic data enhancement strategy during training, which can adjust the intensity and type of data enhancement according to the training status of the model, avoid the uncontrollable variability of the enhanced data, effectively reduce underfitting and overfitting, optimize the model training strategy, and enhance the generalization ability of the model. At the same time, the DT-DETR real-time target detection model adopts an adaptive hyperparameter customization method during training, which further increases the flexibility and practicality of model training, so that the model can better adapt to different training data and training tasks.

[0047] Therefore, the picking object recognition model obtained by training the DT-DETR real-time target detection model can continuously learn the correlation between the field environment monitoring data and the image data of each object to be picked, accurately identify multiple target picking objects that are mature and suitable for picking, and determine the picking priority and picking control parameters of each target picking object, so that the intelligent picking robot can pick fruits and vegetables at a safe and appropriate angle and strength without affecting the quality of the fruits and vegetables, and perform more, faster, better and smoother picking tasks, effectively solving the technical problems of low recognition accuracy and low picking efficiency of existing intelligent picking robots, and is more suitable for large-scale agricultural production.

[0048] It should be noted that the picking object recognition model described in this application is obtained based on the DT-DETR real-time target detection model training, which is only a preferred embodiment of this application. Users can choose other neural network models according to their needs, such as training based on convolutional neural networks, residual networks, recurrent neural networks, deep learning networks, generative adversarial networks, and autoencoders. That is, this application does not limit what kind of neural network model the picking object recognition model is trained on.

[0049] The picking navigation module 4 is connected to the picking object recognition model 3, and is used to collect the position information and image data of each obstacle and each object to be picked in the current environment in real time, and generate a three-dimensional map of the area to be picked based on the collected information, so as to obtain the target picking position of each target picking object according to the three-dimensional map, and generate the optimal picking path in combination with the picking priority of each target picking object.

[0050] In one embodiment, the picking navigation module 4 includes: one or more of an inertial measurement unit, a lidar sensor, and an ultrasonic sensor, which are respectively used to collect the position information of the intelligent picking robot in real time, and scan the current environment to obtain the position information and shape information of each obstacle and each object to be picked, so as to use multi-sensor fusion technology to generate a three-dimensional map of the area to be picked, and based on the three-dimensional map, obtain the target picking position of each target picking object, and generate the optimal picking path in combination with the picking priority of each target picking object.

[0051] Specifically, the picking navigation module 4 uses multi-sensor fusion technology to generate a three-dimensional map of the area to be picked and the most efficient picking path, including: measuring the posture information of the intelligent picking robot through an inertial measurement unit, and using a lidar sensor or an ultrasonic sensor, using global satellite navigation GNSS technology and real-time dynamic measurement RTK technology to scan the surrounding environment, measure the distance between the intelligent picking robot and each obstacle, the object to be picked and other crops, obtain the position information and shape information of each obstacle and each object to be picked, so as to construct a three-dimensional map of the area to be picked according to the obtained information; In the three-dimensional map, the target picking position of each target picking object is obtained, and in combination with the picking priority of each target picking object, an optimal picking path is generated through a path planning algorithm; during the picking process of the intelligent picking robot, it continues to perceive the surrounding environment in real time based on an inertial measurement unit, a lidar sensor or an ultrasonic sensor, including terrain, obstacles, objects to be picked and other crops, and uses a dynamic obstacle avoidance algorithm to continuously update the generated three-dimensional map and optimize the generated optimal picking path, so as to accurately avoid obstacles when encountering obstacles and avoid collisions with obstacles (such as other machines and equipment or personnel).

[0052] In a preferred embodiment, the present application can also be combined with the depth camera in the picking object recognition module 3 to collect image data of each obstacle and each object to be picked for map matching; then the inertial navigation system is combined to navigate and locate the intelligent picking robot, and the information obtained by scanning with the lidar sensor or ultrasonic sensor is integrated to further optimize the three-dimensional map of the picking area and the optimal picking path, thereby improving the path planning ability and obstacle avoidance ability of the intelligent picking robot and improving its efficiency in picking fruits and vegetables.

[0053] The picking execution control module 5 is connected to the picking object identification module 3 and the picking navigation module 4, and is used to control the walking structure to reach each target picking position in turn according to the optimal picking path, and control the mechanical arm structure to perform picking tasks on each target picking object according to each picking control parameter.

[0054] In one embodiment, the intelligent picking robot further includes: a picking process monitoring module, a power module and an energy management module.

[0055] Specifically, the picking process monitoring module is arranged in the robot body 1, connected to the picking execution control module 5, the picking object identification module 3 and the picking navigation module 4, and is used to monitor the picking process in real time and generate picking data, and send the picking data to the picking object identification module 3 to optimize the picking object identification model, and send it to the picking navigation module 4 to update the three-dimensional map of the area to be picked.

[0056] The power module includes: an energy storage battery and a solar panel, which are used to provide continuous power for the intelligent picking robot.

[0057] The energy management module is used to automatically adjust the working mode of the intelligent picking robot according to the number of remaining target picking objects of the current picking task and the remaining power of the intelligent picking robot. Wherein, the working modes include: a first working mode, a second working mode and a charging mode; when the remaining power of the intelligent picking robot is greater than the preset first power threshold, the working mode is set to the first working mode, so that the intelligent picking robot continues to perform the picking task according to the target picking strategy; when the remaining power of the intelligent picking robot is less than the preset first power threshold, the working mode is set to the second working mode, one or more picking objects with a first priority as the new target picking object are selected from the remaining target picking objects, the target picking strategy is updated, and the intelligent picking robot performs the picking task according to the updated target picking strategy; when the remaining power of the intelligent picking robot is less than the preset second power threshold, the working mode is set to the charging mode, so that the intelligent picking robot returns to the charging warehouse for charging; wherein, the first power threshold is greater than the second power threshold.

[0058] In this embodiment, the design of the power module and the energy management module of the intelligent harvesting robot described in this application is mainly based on the low-power design principle, and the electronic components used in other modules are also low-power electronic components, and optimized software algorithms are used, which can greatly reduce unnecessary calculations and communications and reduce energy consumption.

[0059] In summary, it should be understood that the intelligent harvesting robot described in the present application adopts a modular design and can be freely combined and replaced according to specific application scenarios. For example, according to the application scenario, you can choose to add or remove the energy management module, the picking process monitoring module, etc.; the mechanical arm structure 12 can set a corresponding number of mechanical arms 121 according to the type of fruits and vegetables picked and the field environment, or select a high-light mechanical arm 121, or select a flexible mechanical arm 121. As a result, the intelligent harvesting robot has high flexibility, and it will be more convenient to maintain and upgrade, and one of the modules or units can be maintained separately, thereby reducing the maintenance cost of the robot.

[0060] In order to better describe the working mode and applicable scenarios of the intelligent picking robot, the present application provides the following specific embodiments to further illustrate that the intelligent picking robot can flexibly adapt to the picking of various types of fruits and vegetables.

[0061] Embodiment 1: An intelligent picking robot for picking apples.

[0062] In an apple orchard, ripe apples are usually distributed in various parts of the tree canopy, and manual picking often requires the use of ladders or other tools, which is inefficient. In this embodiment, the intelligent picking robot includes: a walking structure, a support structure, a mechanical arm structure, a storage structure, an environmental monitoring module, a picking object recognition module, a picking navigation module, and a picking execution control module. The mechanical arm structure includes a plurality of mechanical arms with multiple degrees of freedom and actuators correspondingly installed at the end of each mechanical arm, and each actuator is installed with a pressure sensor. The picking object recognition module includes a high-precision visual sensor.

[0063] The working method of the intelligent picking robot includes: first, scanning the canopy area through the visual sensor, and inputting the image data and the field environment monitoring data collected by the environment monitoring module into the pre-trained DT-DETR real-time target detection model to identify ripe apples; then, the picking navigation module analyzes the position and surrounding environment of the ripe apples and generates the optimal picking path, so that the picking execution control module controls the walking mechanism to reach each target picking position in turn along the optimal picking path, and controls the mechanical arm structure to perform the picking task; in the process of performing the picking task, the pressure sensors installed on each actuator monitor the force applied to the apple in real time to ensure that the apple will not be squeezed or otherwise damaged; after the picking is completed, the apple can be transported to the storage structure via a conveyor belt. In addition, the intelligent picking robot also has an obstacle avoidance function, which can avoid branches and other obstacles in a complex canopy environment, and continuously optimize the optimal picking path during the movement.

[0064] Embodiment 2: An intelligent picking robot for picking strawberries.

[0065] The strawberry plants in the strawberry greenhouse are relatively short, and the fruits are usually distributed in the middle and lower parts of the plants. Since strawberry fruits are relatively fragile, the surface of the fruits is easily damaged during manual picking, affecting the quality and shelf life. In this embodiment, the intelligent picking robot includes: a walking structure, a support structure, a mechanical arm structure, a storage structure, an environmental monitoring module, a picking object recognition module, a picking navigation module, and a picking execution control module. The mechanical arm structure includes a plurality of flexible mechanical arms and actuators correspondingly installed at the end of each mechanical arm, and each actuator includes a shearing tool. The picking object recognition module includes a high-precision visual sensor.

[0066] The working method of the intelligent picking robot includes: identifying ripe strawberries through the picking object recognition module and removing immature or damaged fruits; adjusting the operating force of the mechanical arm and the shearing tool according to the size and shape of the fruit, and when picking, the mechanical arm gently holds the handle of the strawberry and uses the shearing tool to pick the fruit. In this way, the intelligent picking robot can accurately pick strawberries without damaging the fruit.

[0067] Embodiment 3: An intelligent picking robot for picking grapes.

[0068] Grapes in vineyards usually grow on trellises with different heights, and the grape bunches are relatively fragile and easily damaged. In this embodiment, the intelligent picking robot includes: a walking structure, a supporting structure, a mechanical arm structure, a storage structure, an environmental monitoring module, a picking object recognition module, a picking navigation module, and a picking execution control module. The mechanical arm structure includes a plurality of mechanical arms with multiple degrees of freedom and actuators correspondingly installed at the end of each mechanical arm, and each actuator is installed with a pressure sensor. The picking object recognition module includes a high-precision visual sensor.

[0069] The working method of the intelligent picking robot includes: first, scanning the grape vines through the visual sensor, identifying the ripe grape bunches, and determining their positions; then, adjusting and controlling the angle and strength of the mechanical arm and the actuator according to the size and position of the grape bunches through the picking execution control module, so that grapes can be picked at different heights and angles of the vine frame, and it is ensured that the grapes will not be damaged during the picking process; after the picking is completed, the grape bunches are gently placed in the storage structure through the mechanical arm to avoid collision. In addition, the intelligent picking robot also has a path optimization function, which can optimize the optimal picking path in real time and continuously in a complex vine frame structure, thereby improving the picking efficiency.

[0070] Embodiment 4: An intelligent picking robot for picking citrus fruits.

[0071] In a citrus orchard, citrus fruits are usually relatively hard, but the branches and leaves are dense and the fruits are widely distributed, so the traditional manual picking method is inefficient. In this embodiment, the intelligent picking robot includes: a walking structure, a support structure, a mechanical arm structure, a storage structure, an environmental monitoring module, a picking object recognition module, a picking navigation module, and a picking execution control module. The mechanical arm structure includes a plurality of high-strength mechanical arms with multiple degrees of freedom and corresponding actuators installed at the end of each mechanical arm, and each actuator is equipped with a pressure sensor. The picking object recognition module includes a high-precision visual sensor.

[0072] The working method of the intelligent picking robot includes: first, identifying and locating mature citrus fruits through visual sensors; then controlling the walking structure to quickly reach the target picking position through the picking execution control module, and controlling the robotic arm to extend to the citrus fruits to be picked, using the actuator to grab the citrus fruits and perform rotation or shearing operations to separate the citrus fruits from the branches. During the entire picking process, the intelligent picking robot can quickly pick citrus fruits distributed in different positions, and can adjust the strength of the picking action in real time to avoid damage to the fruits. In addition, the intelligent picking robot can also set the picking priority according to the maturity of the fruit, automatically adjust the picking order, and ensure that the picked fruits reach the optimal maturity.

[0073] In the embodiments of the present application, words such as "first" and "second" are used to distinguish the same or similar items with substantially the same functions and effects. For example, the first telescopic structure and the second telescopic structure are only used to distinguish different telescopic structures, and do not limit their order. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit them to be different.

[0074] It should be noted that in the embodiments of the present application, words such as "exemplary" or "for example" represent examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0075] In the embodiments of the present application, "at least one" refers to one or more, and "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can represent: a, b, c, ab, ac, bc or abc, where a, b, c can be single or multiple.

[0076] like Figure 4 , which shows a schematic diagram of the structure of an intelligent picking robot monitoring system 400 in an embodiment of the present invention. The intelligent picking robot monitoring system 400 includes: a plurality of intelligent picking robots 401, a monitoring center 402 and a mobile terminal application 403.

[0077] The intelligent picking robot 401, such as the intelligent picking robot provided in the above embodiments, is used to automatically perform picking tasks and generate corresponding picking data; the intelligent picking robot 401 includes a wireless communication module, which is used to upload the picking data to the monitoring center 402. The picking data includes but is not limited to: the actual picking position, actual picking control parameters, total picking amount, number of wrong pickings, and number of damaged picking objects generated by the current intelligent picking robot 401 when performing the picking task.

[0078] The monitoring center 402 is connected to each of the intelligent picking robots 401, and is used to analyze the working status, picking progress, picking results and picking efficiency of each of the intelligent picking robots 401 according to the picking data generated by each of the intelligent picking robots 401, and remotely manage and optimize each of the intelligent picking robots 401.

[0079] In a specific embodiment, after the monitoring center 402 obtains the picking data generated by each intelligent picking robot 401, it can obtain picking experience based on these data and analyze the performance of each intelligent picking robot 401, such as analyzing the motion trajectory of the mechanical arm of each intelligent picking robot 401, the actual target picking position, the force control of the actuator, etc. And according to these experience data, the pre-built picking object recognition model can be trained and optimized, so that the picking object recognition model can more quickly and accurately identify the mature objects to be picked as the target picking objects, and determine the picking mode, picking priority and picking control parameters of each target picking object, such as the mechanical arm activity parameters and the picking pressure parameters. At the same time, according to these picking experience data, the three-dimensional map of the area to be picked is further updated, and the optimal picking path is optimized to avoid collisions with some obstacles. Thus, each of the intelligent picking robots 401 can complete the picking task better, more accurately identify mature fruits and vegetables as target picking objects, reduce the picking error rate; control the robot arm to pick various types of fruits and vegetables with safer and more suitable movement speed and strength, improve the robot's picking efficiency; and ensure that the picked fruits and vegetables are not damaged; so as to improve the robot's path planning ability and obstacle avoidance ability. In addition, the monitoring center 402 can also analyze the crop growth data under different climatic conditions based on the collected data, so as to be used for long-term agricultural research and production optimization, and help agricultural managers adjust planting strategies to improve crop yields and quality.

[0080] The mobile terminal application end 403 is connected to the monitoring center 402, and is used for the user to monitor the working status, picking progress and picking results of the designated intelligent picking robot 401 on the mobile terminal, and remotely control the operation of the intelligent picking robot 401. Therefore, agricultural managers can use mobile terminals, such as mobile phones or computer terminals, to view the field environment monitoring data, picking data, the working status, remaining power, picking progress, etc. of the intelligent picking robot 401 obtained by the designated intelligent picking robot 401 in real time, thereby controlling the operation of the intelligent picking robot 401, including adjusting its working mode to give priority to picking the target picking objects of the first priority, or directly return to the charging warehouse for charging, etc., to achieve intelligent management and remote management of each intelligent picking robot 401.

[0081] It should be understood that the specific process of each module executing the above corresponding steps has been described in detail in the various embodiments of the above intelligent picking robot, and will not be repeated here for the sake of brevity.

[0082] It should also be understood that the division of modules in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional module in each embodiment of the present application may be integrated into a processor, or may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0083] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0084] In summary, the present application provides an intelligent picking robot and a monitoring system, which generates field environment monitoring data through an environmental monitoring module; identifies one or more target picking objects based on a pre-trained picking object recognition model according to the field environment monitoring data and the collected image data of each object to be picked through a picking object recognition module; generates a three-dimensional map of the area to be picked and an optimal picking path through a picking navigation module; controls the walking structure to reach each target picking position in turn through a picking execution control module, and controls the robotic arm structure to perform the picking task; thereby enabling the intelligent picking robot to adapt to the picking tasks of different types and maturity of fruits and vegetables, and to achieve accurate identification, positioning and picking of target picking objects in a complex agricultural environment, thereby improving the efficiency and quality of fruit and vegetable picking.

[0085] Therefore, the present application effectively overcomes various shortcomings in the prior art and has high industrial utilization value.

[0086] The above embodiments are merely illustrative of the principles and effects of the present application and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.

Claims

1. An intelligent picking robot, characterized in that: include: The robot body comprises: a walking structure for moving the intelligent picking robot to a designated position; a mechanical arm structure, which is fixedly mounted on the walking structure through a supporting structure; wherein the supporting structure comprises: a plurality of support rods and a plurality of chassis, each chassis is detachably connected to a plurality of support rods around the periphery, and can move up and down along each support rod or be locked at a designated position; the mechanical arm structure comprises: a plurality of mechanical arms and an actuator correspondingly mounted at the end of each mechanical arm, which is used to perform picking tasks according to picking instructions and place the picked fruit and vegetable objects in a storage structure; each mechanical arm has multiple degrees of freedom, including: a fixed component and a plurality of telescopic structures, one end of which is fixed to each chassis through a fixed component; An environmental monitoring module, provided on the robot body, for collecting field environmental monitoring data in real time; A picking object recognition module is arranged on the robot body and connected to the environment monitoring module, and is used to collect image data of each object to be picked in real time, and based on a pre-trained picking object recognition model, analyzes the variety, size, shape and maturity of each object to be picked in a specific field environment according to each collected image data and the received field environment monitoring data, identifies one or more target picking objects, and determines the picking priority and picking control parameters of each target picking object; A picking navigation module is provided on the robot body and connected to the picking object recognition model, and is used to collect the position information and image data of each obstacle and each object to be picked in the current environment in real time, and generate a three-dimensional map of the area to be picked based on the collected information, so as to obtain the target picking position of each target picking object according to the three-dimensional map, and generate an optimal picking path in combination with the picking priority of each target picking object; A picking execution control module is arranged on the robot body, connected to the picking object identification module and the picking navigation module, and is used to control the walking structure to reach each target picking position in sequence according to the optimal picking path, and control the mechanical arm structure to perform picking tasks on each target picking object according to each picking control parameter.

2. The intelligent picking robot according to claim 1, characterized in that: The picking object recognition model is obtained by training based on the DT-DETR real-time target detection model; the DT-DETR real-time target detection model includes: A hybrid encoder is used to perform multi-scale feature extraction on each collected image data and the field environment monitoring data, generate multiple multi-scale feature data of each object to be picked, perform self-attention operation on each multi-scale feature data based on an attention mechanism, perform feature fusion on each multi-scale feature data based on a convolutional neural network, and select the optimal feature data of each object to be picked based on an uncertainty minimum query selection mechanism and output them to a transformer decoder; The transformer decoder is used to generate multiple bounding boxes based on the optimal feature data of each object to be picked to select each object to be picked, analyze the maturity of each object to be picked, classify each object to be picked based on maturity, determine one or more target picking objects suitable for picking, and the picking priority and picking control parameters of each target picking object.

3. The intelligent picking robot according to claim 2, characterized in that: The method of training the DT-DETR real-time target detection model to obtain the picking object recognition model includes: Acquire a plurality of picking object training samples in specific field environments; wherein each picking object training sample includes: field environment historical monitoring data and historical image data; Perform data enhancement on the field environment historical monitoring data and historical image data in each picking object training sample to generate the corresponding picking object enhanced data set; Adopting an adaptive hyperparameter customization method to determine the optimizer hyperparameters of the DT-DETR real-time target detection model, and training the DT-DETR real-time target detection model with the picking object enhanced data set to obtain a primary picking object recognition model; The generalization ability of the DT-DETR real-time target detection model is dynamically verified, and according to the verification results, the intensity and type of data enhancement for each picking object training sample are dynamically adjusted, the model structure and optimizer hyperparameters of the DT-DETR real-time target detection model are optimized and updated, and a converged picking object recognition model is obtained.

4. The intelligent picking robot according to claim 1, characterized in that: A pressure sensor is provided on each actuator of the mechanical arm structure, and is used for real-time monitoring of the pressure applied by the actuator to each target picking object when the mechanical arm structure performs a picking task.

5. The intelligent picking robot according to claim 1, characterized in that: The environmental monitoring module includes: one or more of a light sensor, a temperature sensor, a humidity sensor and a soil moisture sensor, which are used to collect light monitoring data, air temperature monitoring data, air humidity monitoring data and soil moisture monitoring data in real time to generate field environment monitoring data.

6. The intelligent picking robot according to claim 1, characterized in that: The picking navigation module includes: one or more of an inertial measurement unit, a lidar sensor and an ultrasonic sensor, which are used to collect the position information of the intelligent picking robot in real time, and scan the current environment to obtain the position information and shape information of each obstacle and each object to be picked, so as to generate a three-dimensional map of the area to be picked by using multi-sensor fusion technology, and obtain the target picking position of each target picking object based on the three-dimensional map, and generate the optimal picking path in combination with the picking priority of each target picking object.

7. The intelligent picking robot according to claim 1, characterized in that: Also includes: A picking process monitoring module is arranged on the robot body, connected to the picking execution control module, the picking object recognition module and the picking navigation module, and is used to monitor the picking process in real time, generate picking data, send the picking data to the picking object recognition module to optimize the picking object recognition model, and send the picking data to the picking navigation module to update the three-dimensional map of the area to be picked; The picking data include but are not limited to: actual picking position, actual picking control parameters, total picking amount, number of erroneous picking and number of damaged picking objects.

8. The intelligent picking robot according to claim 7, characterized in that: Also includes: A wireless communication module is arranged on the robot body and connected to the picking process monitoring module, and is used to upload the generated picking data to an external monitoring center for analyzing the working status, picking progress, picking results and picking efficiency of the intelligent picking robot, and remotely managing and optimizing the intelligent picking robot.

9. The intelligent picking robot according to claim 1, characterized in that: Also includes: A power module, arranged on the robot body, includes: an energy storage battery and a solar panel, for providing continuous power for the intelligent picking robot; An energy management module, disposed on the robot body, for automatically adjusting the working mode of the intelligent picking robot according to the number of remaining target picking objects of the current picking task and the remaining power of the intelligent picking robot; Wherein, the working modes include: a first working mode, a second working mode and a charging mode; When the remaining power of the intelligent picking robot is greater than a preset first power threshold, the working mode is set to the first working mode, so that the intelligent picking robot continues to perform the picking task according to the target picking strategy; when the remaining power of the intelligent picking robot is less than the preset first power threshold, the working mode is set to the second working mode, and one or more picking objects with a first priority level are selected from the remaining target picking objects as new target picking objects, the target picking strategy is updated, and the intelligent picking robot performs the picking task according to the updated target picking strategy; when the remaining power of the intelligent picking robot is less than the preset second power threshold, the working mode is set to the charging mode, so that the intelligent picking robot returns to the charging warehouse for charging; wherein, the first power threshold is greater than the second power threshold.

10. An intelligent picking robot monitoring system, characterized in that: include: A plurality of intelligent picking robots as claimed in claims 1 to 9, used to automatically perform picking tasks and generate corresponding picking data to be uploaded to a monitoring center via a wireless communication module; A monitoring center connected to each of the intelligent picking robots, for analyzing the working status, picking progress, picking results and picking efficiency of each of the intelligent picking robots according to the picking data generated by each of the intelligent picking robots, and remotely managing and optimizing each of the intelligent picking robots; The mobile terminal application end is connected to the monitoring center, and is used for the user to monitor the working status, picking progress and picking results of the specified intelligent picking robot on the mobile terminal, and remotely control the operation of the intelligent picking robot.

Images