A visual servo system and control method for an agricultural picking robot

Abstract

The present invention provides a visual servo system and control method for an agricultural harvesting robot. The visual servo system includes a visual sensor unit and an air gun unit. The visual sensor unit is mounted on the robot's elbow joint and can move with the elbow joint. The air gun unit is also mounted on the elbow joint and is configured to blow air in a picking direction to dislodge obstructions in that direction, facilitating image acquisition by the visual sensor unit. This invention addresses the issues with existing agricultural harvesting robots, such as the tendency to miss fruit and the complex fruit recognition algorithms that strain computing resources and thus affect the system's real-time performance.

Description

Technical Field

[0001] The present invention relates to the technical field of agricultural robots, and in particular to a visual servo system and a control method of an agricultural picking robot. Background Art

[0002] Agricultural harvesting robots are devices designed to automatically harvest fruit, replacing manual labor and improving harvesting efficiency. Traditional agricultural harvesting robots primarily rely on various sensors to assist in harvesting. These sensors, such as vision sensors and laser sensors, perform tasks such as path navigation, fruit identification, fruit location, fruit picking, and fruit placement. Traditional agricultural harvesting robots also utilize vision sensor systems to distinguish between fruit and leaves, and use vision sensors or binocular vision sensors to determine the spatial distance between the sensor and the fruit.

[0003] In existing technologies, when agricultural harvesting robots are harvesting fruit, if the fruit is obscured by leaves, they often cannot effectively identify or cannot identify the fruit behind the leaves, resulting in missed harvests. Furthermore, when the terminal structure of an agricultural harvesting robot that uses traditional image recognition processing technology to identify agricultural fruit approaches the target fruit, the robot often executes the same fruit recognition algorithm within different visual ranges. For example, the fruit maturity judgment algorithm is still used in scenarios where fruit maturity identification is unnecessary. This undoubtedly wastes the computing power resources of the agricultural harvesting robot controller hardware and affects the controller's performance.

[0004] At present, there is no effective solution to the problems that existing agricultural picking robots are prone to missing fruits and the fruit recognition algorithms are complex, resulting in tight computing resources and affecting the real-time performance of the system. Summary of the Invention

[0005] In view of this, an embodiment of the present invention provides a visual servo system and control method for an agricultural picking robot to solve the problem that existing agricultural picking robots are prone to missing fruits and the fruit recognition algorithm is complex, resulting in tight computing resources and affecting the real-time performance of the system.

[0006] A visual servo system for an agricultural harvesting robot according to an embodiment of the present invention is installed on an elbow joint unit of the agricultural harvesting robot, comprising:

[0007] a visual sensor unit, the visual sensor unit being mounted on the elbow joint unit and being capable of moving with the elbow joint unit;

[0008] an air gun unit, the air gun unit being arranged on the elbow joint unit;

[0009] The air gun unit is used to blow air in the picking direction to blow away the obstructions in the picking direction;

[0010] The visual sensor unit is used to obtain image information in the picking direction when the air gun unit blows air or not, and send the image information to the agricultural picking robot;

[0011] The agricultural picking robot is used to identify and process the image information to form an image processing result, which includes whether there is a target object in the picking direction and the distance between the target object and the visual sensor unit. The agricultural picking robot can move and pick the target object based on the image processing result.

[0012] Furthermore, the air gun unit comprises:

[0013] A valve element is provided in the air gun unit and is used to control the air gun unit to perform pulsed air blowing along the picking direction.

[0014] A visual servo control method for an agricultural harvesting robot according to an embodiment of the present invention includes:

[0015] obtaining a position instruction, and controlling the agricultural picking robot to perform a movement action based on the position instruction;

[0016] During the movement, a visual processing result is obtained based on current visual information obtained by the visual sensor unit, and when the visual processing result is greater than a preset threshold, it is determined that the agricultural picking robot is located in a far field of view;

[0017] When the agricultural harvesting robot is located in the far field of view, identifying a target object and current canopy depth information of the target object based on a far field of view image sequence acquired by the visual sensor unit, and controlling the agricultural harvesting robot to approach the target object based on the current canopy depth information of the target object, wherein the far field of view image sequence includes N frames of images;

[0018] During the process of the agricultural picking robot approaching the target object, if the visual processing result obtained based on the current visual information acquired by the visual sensor unit is less than or equal to the preset threshold, it is determined that the agricultural picking robot is located in the near field of view;

[0019] When the agricultural picking robot is located in the near field of view, the target object is identified based on a near field of view image sequence acquired by the visual sensor unit, and surface feature information of the target object and current canopy depth information of the target object are acquired, wherein the near field of view image includes M frames of images, and M is greater than N;

[0020] The posture of the end effector of the agricultural harvesting robot is adjusted based on the surface feature information, and the end effector is controlled to perform a harvesting action based on the target object identified by the near-field image sequence and the current canopy depth information of the target object.

[0021] Further, judging that the agricultural picking robot is located in the far field of view based on the visual processing result includes:

[0022] During the movement action, a far field image sequence is obtained based on the visual sensor unit, and when the current canopy depth information obtained based on the far field image sequence is greater than a preset distance threshold, it is determined that the agricultural picking robot is located in the far field.

[0023] Further, judging that the agricultural picking robot is located in the near field of view based on the visual processing result includes:

[0024] In the process of the agricultural picking robot approaching the target object, the far field image sequence is continuously acquired based on the visual sensor unit, and when the current canopy depth information acquired based on the far field image sequence is less than or equal to a preset distance threshold, it is determined that the agricultural picking robot is located in the near field of view.

[0025] Further, judging that the agricultural picking robot is located in the far field of view based on the visual processing result includes:

[0026] During the moving action, the number of partitions within the field of view and the number of targets within the partitions are obtained based on the field of view of the visual sensor unit. When the number of partitions is greater than a preset partition number threshold and the number of targets within all the partitions is greater than the preset target number threshold, it is determined that the end effector is located in the far field of view.

[0027] Further, judging that the agricultural picking robot is located in the near field of view based on the visual processing result includes:

[0028] In the process of the agricultural picking robot approaching the target object, the number of partitions within the field of view and the number of targets in all the partitions are obtained based on the field of view of the visual sensor unit. When the number of partitions is less than or equal to a preset partition number threshold and the number of targets in all the partitions is less than or equal to the preset target number threshold, it is determined that the end effector is located in the near field of view.

[0029] Furthermore, identifying the target object based on the far field image sequence includes:

[0030] Performing image edge detection on each of the N frames of images in the far field image sequence;

[0031] The maximum gradient value of each image in the N frames of images is obtained. When the maximum gradient value is greater than a set value A, the target object is roughly detected.

[0032] Furthermore, when the maximum gradient value of an image in N frames of images is less than or equal to the set value A, image edge detection is performed on the next image in the N frames of images.

[0033] Furthermore, identifying the target object based on the near-field image sequence acquired by the visual sensor unit includes:

[0034] Performing image edge detection on each of the M frames of image in the near-field image sequence;

[0035] The maximum gradient value of each image in the M frames of images is obtained. When the maximum gradient value is greater than a set value B, the target object is carefully detected, wherein the set value B is greater than the set value A.

[0036] Furthermore, when the maximum gradient value of an image in the M frames of images is less than or equal to the set value B, image edge detection is performed on the next image in the M frames of images.

[0037] Furthermore, acquiring surface feature information of the target object based on the near-field image sequence includes:

[0038] Preprocessing the M frames of images in the near-field image sequence in sequence, and dividing each of the M frames of images into a plurality of subdivided areas;

[0039] The surface feature information is extracted based on a plurality of the subdivided areas.

[0040] Furthermore, when it is determined that the end effector is located in the near field of view, the visual servo control method further includes:

[0041] controlling the air gun unit of the agricultural picking robot to perform pulsed air blowing;

[0042] Delaying for a specified period of time, and controlling the visual sensor unit to acquire image information in a dynamic image capture manner;

[0043] stopping the air gun unit to perform pulsed air blowing, and identifying the image information to obtain a recognition result, wherein the recognition result includes whether the target object is contained;

[0044] In a case where the recognition result includes the target object, the agricultural picking robot is controlled to pick the target object.

[0045] Compared with the prior art, the present invention has at least the following technical effects:

[0046] (1) The present invention provides a visual sensor unit and an air gun unit on the elbow joint unit, so that the air gun unit can blow away the obstruction to expose the target object, thereby preventing the target object from being obscured and missed, thus solving the problem that existing agricultural picking robots are prone to missing fruits due to obstruction by branches and leaves;

[0047] (2) The visual sensor unit uses a dynamic image capture method to acquire images, thereby facilitating the collection of image information of dynamically moving obstructions or targets, thereby improving the reliability of identifying dynamically moving obstructions or targets;

[0048] (3) The present invention obtains image information of different frames in the far field of view and the near field of view through the visual sensor unit, and sets different gradient setting values when performing edge recognition in the far field of view and when performing edge recognition in the near field of view, thereby achieving coarse recognition of the target object in the far field of view and fine recognition of the target object in the near field of view, solving the problem in the prior art that the fruit recognition algorithm is complex, resulting in tight computing resources and thus affecting the real-time performance of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 This is a schematic diagram of the partial structure of an agricultural harvesting robot according to an embodiment of the present invention;

[0050] Figure 2 Flowchart (1) of the visual servo control method according to an embodiment of the present invention;

[0051] Figure 3 This is a block diagram of a visual servo control system in which the end effector of an agricultural harvesting robot according to an embodiment of the present invention is located in a far field of view;

[0052] Figure 4 Schematic diagram of the change in the field of view of the visual sensor unit from far field to near field of view according to an embodiment of the present invention;

[0053] Figure 5 This is a block diagram of a visual servo control system in which the end effector of an agricultural harvesting robot according to an embodiment of the present invention is located in a near field of view;

[0054] Figure 6 This is a schematic structural diagram of a robot end effector located in different field of view areas according to an embodiment of the present invention;

[0055] Figure 7 Flowchart (2) of the visual servo control method according to an embodiment of the present invention;

[0056] Figure 8 This is a flow chart of a rough inspection of fruit edges according to an embodiment of the present invention;

[0057] Figure 9 Flowchart (3) of the visual servo control method according to an embodiment of the present invention;

[0058] Figure 10 This is a flow chart of a detailed inspection of fruit edges according to an embodiment of the present invention;

[0059] Figure 11 Flowchart (4) of the visual servo control method according to an embodiment of the present invention;

[0060] Figure 12 Flowchart of target feature (surface feature information) extraction according to an embodiment of the present invention;

[0061] Figure 13 Flowchart (5) of the visual servo control method according to an embodiment of the present invention;

[0062] Figure 14 A schematic diagram of a robot end effector blowing air to expose a partially obscured target object according to an embodiment of the present invention;

[0063] Figure 15 A schematic diagram of a robot end effector blowing air to reveal a completely obscured target object according to an embodiment of the present invention;

[0064] Figure 16 The figure is a flow chart of a specific implementation of the present invention. DETAILED DESCRIPTION

[0065] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0066] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that, in the absence of conflict, the features in the following embodiments and embodiments can be combined with each other. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without making creative work are within the scope of protection of this application.

[0067] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on this application, it should be understood by those skilled in the art that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number and aspect described herein can be used to implement an apparatus and / or practice a method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this apparatus and / or practice this method.

[0068] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application. The illustrations only show components related to the present application and are not drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component can be changed at will, and the component layout type may also be more complicated.

[0069] Additionally, in the following description, specific details are provided to provide a thorough understanding of the examples, however, one skilled in the art will appreciate that the examples can be practiced without these specific details.

[0070] Example 1

[0071] like Figure 1 As shown, this embodiment provides a visual servo system for an agricultural harvesting robot, which is installed on the elbow joint unit of the agricultural harvesting robot 10. The visual sensor unit 20 is installed on the elbow joint unit of the agricultural harvesting robot 10 and can move with the elbow joint unit; the air gun unit 30 is installed on the elbow joint unit of the agricultural harvesting robot 10 and is used to blow air in the picking direction to blow away obstructions in the picking direction.

[0072] The visual sensor unit 20 is used to obtain image information in the picking direction when the air gun unit 30 blows air or not.

[0073] The agricultural picking robot 10 is used to identify and process the image information sent by the visual sensor unit 20 to form an image processing result, which includes whether there is a target object in the picking direction and the distance between the target object and the visual sensor unit 20; the agricultural picking robot 10 is used to move according to the image processing result to form visual servo control, and pick the target object when it moves to the target object position.

[0074] For example, the target object may be an agricultural fruit.

[0075] Among them, the visual sensor unit 20 and the air gun unit 30 are both communicatively connected to the controller of the agricultural picking robot 10, so that the controller of the agricultural picking robot 10 can obtain the image information obtained by the visual sensor unit 20 and can also control the air gun unit 30 to blow air.

[0076] Among them, after the agricultural picking robot 10 obtains the image information sent by the visual sensor unit 20, the image data processing module in the agricultural picking robot 10 can perform image processing on the image information obtained by the visual sensor unit 20 to form an image processing result.

[0077] Among them, the image data processing module is set on the controller of the agricultural picking robot 10.

[0078] The image processing result also includes information such as surface feature information of the target object and whether the surface feature information of the target object matches the maturity feature of the target object.

[0079] Among them, the air gun unit 30 can blow air when the visual sensor unit 20 does not detect the target object, so as to blow away the obstruction, so that the visual sensor unit 20 obtains image information behind the obstruction; the air gun unit 30 can blow air when the target object is partially obscured, so as to completely or partially expose the target object, so that the visual sensor unit 20 obtains all image information of the target object.

[0080] Preferably, the air gun unit 30 may be disposed at the lower portion of the visual sensor unit 20 , that is, the air gun unit 30 is disposed between the visual sensor unit 20 and the elbow joint unit.

[0081] Specifically, when it is necessary to pick a target object that is not blocked by an obstruction, the visual sensor unit 20 obtains image information in the picking direction and transmits the image data to the agricultural picking robot 10, and then the image data processing module running in the agricultural picking robot controller identifies and processes the image information to form an image processing result. When the agricultural picking robot judges that there is a target object in the picking direction based on the image processing result, the agricultural picking robot 10 approaches the target object based on the image processing result and picks the target object; when it is necessary to pick a target object that is completely or partially blocked, the air gun unit 30 blows air along the picking direction, and then the visual sensor unit 20 obtains the image information after the air gun blows, and identifies and processes the image information through the image data processing module to form an image processing result. Finally, the agricultural picking robot 10 approaches the target object based on the image processing result to pick the target object or directly executes the next target picking program.

[0082] Furthermore, the air gun unit 30 includes a valve element, which is arranged in the air gun unit 30 and is used to control the air gun unit 30 to perform pulsed blowing along the picking direction to reduce gas consumption.

[0083] Wherein, the valve element is an electrically controllable gas valve.

[0084] The air gun unit 30 is an air gun, and is connected to the air outlet of an external air pump to obtain gas.

[0085] For example, a portable air pump is installed on the robot body, and the air pump is connected to the air gun unit 30 , thereby supplying air to the air gun unit 30 .

[0086] The valve element can realize pulsed air blowing of the air gun unit 30 by controlling the opening and closing of the air inlet pipe of the air gun unit 30 .

[0087] Part of the working principles of the embodiment of the present invention is as follows:

[0088] (1) Picking the target objects that are not blocked by the obstruction

[0089] The visual sensor unit 20 acquires image information in the picking direction. The image data processing module then acquires the image information sent by the visual sensor unit 20 and performs recognition processing to form an image processing result. The image processing result includes one or more of whether there is a target in the picking direction, the location information of the target, whether the target is ripe, and whether the target is blocked.

[0090] Based on the image processing results, the agricultural harvesting robot 10 approaches the target object and harvests the target object.

[0091] (2) Picking partially obscured objects

[0092] When the visual sensor unit 20 determines that a target object is partially obscured, the air gun unit 30 performs pulsed air blowing in the picking direction to blow away the obstruction and expose the target object;

[0093] The visual sensor unit 20 obtains image information of the exposed target object, and then the image data processing module obtains the image information sent by the visual sensor unit 20 and performs recognition processing to form an image processing result;

[0094] Based on the image processing results, the agricultural harvesting robot 10 approaches the target object and harvests the target object.

[0095] (3) Picking completely obscured objects

[0096] In the case that the visual sensor unit 20 does not detect the target object based on the image information, the air gun unit 30 still performs pulsed air blowing along the picking direction to test whether the target object is completely blocked;

[0097] When a target object is completely blocked, the visual sensor unit 20 obtains image information of the blocked target object after the air gun unit 30 blows air, and then the image data processing module obtains the image information sent by the visual sensor unit 20 and performs recognition processing to form an image processing result;

[0098] Based on the image processing results, the agricultural harvesting robot 10 approaches the target object and harvests the target object.

[0099] Example 2

[0100] Figure 2 Flowchart (1) of the visual servo control method according to an embodiment of the present invention is as follows: Figure 2 As shown in FIG, this method realizes visual servo control of an agricultural picking robot based on far and near fields of view. The visual servo control method includes:

[0101] Step S102: obtaining a position instruction, and controlling the agricultural picking robot to perform a moving action based on the position instruction;

[0102] Step S104: during the movement, obtaining a visual processing result based on the current visual information obtained by the visual sensor unit, and determining that the agricultural picking robot is located in a far field of view when the visual processing result is greater than a preset threshold;

[0103] Step S106: When the agricultural harvesting robot is in a far field of view, identifying a target object and current canopy depth information of the target object based on a far field of view image sequence acquired by the visual sensor unit, and controlling the agricultural harvesting robot to approach the target object based on the current canopy depth information of the target object, wherein the far field of view image sequence includes N frames of images;

[0104] Step S108: When the agricultural picking robot approaches the target object, if the visual processing result obtained based on the current visual information obtained by the visual sensor unit is less than or equal to a preset threshold, it is determined that the agricultural picking robot is located in the near field of view;

[0105] Step S110: When the agricultural harvesting robot is in a near-field of view, the target object is identified based on a near-field image sequence acquired by the visual sensor unit, and surface feature information and current canopy depth information of the target object are acquired, wherein the near-field image includes M frames of images, and M is greater than N.

[0106] Step S112: adjusting the posture of the end effector of the agricultural harvesting robot based on the surface feature information, and controlling the end effector to perform a harvesting action based on the target object identified by the near-field image sequence and the current canopy depth information of the target object.

[0107] In step S102, the agricultural picking robot may obtain the position instruction output by the trajectory planning module, and then the agricultural picking robot moves under the action of the position controller to approach or move away from the target object.

[0108] In step S104, the current visual information includes the far field image sequence acquired by the visual sensor unit and the field of view of the visual sensor unit, and the visual processing result includes the current canopy depth information acquired based on the far field image sequence and the number of partitions within the field of view and the number of targets within all partitions.

[0109] In some of the embodiments, when the current visual information includes a far field image sequence acquired by a visual sensor, the visual processing result includes current canopy depth information acquired based on the far field image sequence, thereby determining that the agricultural harvesting robot is located in the far field when the current canopy depth information is greater than a preset distance threshold.

[0110] Among them, the visual sensor unit obtains a long-field image sequence based on a dynamic capture method to solve the problem in the existing technology that the traditional image recognition processing technology is used to identify agricultural fruits, resulting in low reliability in identifying leaves and fruits that are dynamically shaking.

[0111] Among them, the current canopy depth information is used to determine the distance between the agricultural picking robot and the target object in real time, that is, to determine the distance between the visual sensor unit and the target object.

[0112] In some of the embodiments, when the current visual information includes the field of view of the visual sensor unit, the number of partitions within the field of view and the number of all targets in all partitions are obtained, and then, when the number of partitions within the field of view is greater than a preset partition number threshold and the number of targets in all partitions is greater than a preset target number threshold, it is determined that the end effector is located in the far field of view.

[0113] For example, when the number of partitions within the field of view is greater than 1 and the number of targets in all partitions is greater than 1, the end effector is judged to be in the far field of view; in addition, when the number of targets in all partitions is less than 1, the agricultural picking robot is controlled to execute the next fruit picking procedure regardless of whether the number of partitions within the field of view is greater than the preset partition number threshold.

[0114] In step S106, the image data processing module in the agricultural harvesting robot can perform recognition processing on the far-field image sequence to preliminarily identify the target object, that is, to preliminarily distinguish the target object, the obstruction or the background object.

[0115] Among them, the current canopy depth information is used to facilitate the agricultural picking robot to approach the target object according to the current canopy depth information to achieve distance-based visual servo control or to determine whether the agricultural picking robot is in the far field of view in conjunction with a preset distance threshold.

[0116] The N frames of images in the far-field image sequence are N frames of images continuously acquired by the visual sensor unit within a continuous time period close to the target object.

[0117] Specifically, when the agricultural picking robot approaches the target object, the agricultural picking robot continuously acquires images, and after acquiring N frames of images, the agricultural picking robot acquires the current canopy depth information of the target object based on the acquired N frames of images. Finally, the agricultural picking robot approaches the target object based on the current canopy depth information of the target object and determines the field of view of the agricultural picking robot.

[0118] In step S108, when the visual processing result includes current canopy depth information obtained based on the far field image sequence, if the current canopy depth information obtained based on the far field image sequence is less than or equal to the preset distance threshold, it is determined that the agricultural picking robot is located in the near field of view.

[0119] The visual sensor unit includes a real-time edge detection module, which is used to perform image edge detection on the image.

[0120] For example, Figure 3 As shown, Figure 3 This is a block diagram of the visual servo control system of an agricultural picking robot in a far field of view. When the robot (agricultural picking robot) determines that it is in the far field of view based on the extracted canopy depth information (current canopy depth information), the robot obtains the position instruction output by the trajectory planning module, and then the position controller controls the robot to move based on the position instruction. The terminal visual sensor (visual sensor unit) obtains an N-frame sequence (N-frame image) based on a dynamic image capture method, and then the real-time edge detection module in the terminal visual sensor identifies the target object according to the N-frame sequence and obtains the canopy depth information of the target object. Then the position controller controls the robot to approach the target fruit according to the canopy depth information, so as to realize distance-based visual servo control of the agricultural picking robot, until the robot determines that the end effector is in the near field of view according to the obtained canopy depth information, at which time step S110 is executed.

[0121] In some embodiments, in step S108, when the visual information includes the field of view of the visual sensor unit, the number of partitions within the field of view and the number of targets in all partitions are obtained based on the field of view of the visual sensor unit. When the number of partitions is less than or equal to a preset partition number threshold and the number of targets in all partitions is less than or equal to the preset target number threshold, it is determined that the end effector of the agricultural picking robot is located in the near field of view.

[0122] For example, when the number of partitions within the field of view of the visual sensor unit is 1 and there is only one target object in the partition, if the preset partition number threshold and the preset target object number threshold are both 1, it is determined that the end effector of the agricultural harvesting robot is located in the near field of view.

[0123] More specifically, in the case of judging whether the visual sensor unit is located in the near field of view based on the partitions within the field of view, first, several partitions within the field of view of the visual sensor unit and at least one target object within the several partitions are obtained. Then, during the movement of the agricultural picking robot, if there is only one partition within the field of view of the visual sensor unit and there is only one target object within the partition, it is judged that the end effector is located in the near field of view.

[0124] like Figure 4 As shown, when the end effector is located in the first far field of view X1, there are 6*6 partitions in the field of view of the visual sensor unit. At this time, the visual sensor unit selects a target object and the partition where the target object is located, and the field of view of the visual sensor unit gradually becomes smaller in the process of approaching the target object. When the end effector is located in the second far field of view X2, there are 4*4 partitions in the field of view of the visual sensor unit. At this time, the end effector continues to approach the selected target object until the end effector is located in the near field of view. There is only one partition in the field of view of the visual sensor unit, and there is a target object in the partition. At this time, it can be determined that the end effector is located in the near field of view X3.

[0125] In step S110, when the end effector of the agricultural harvesting robot is located in the near field of view, the visual sensor unit continues to acquire a near field of view image sequence to finely identify the target object based on the near field of view image sequence and obtain the surface feature information and current canopy depth information of the target object.

[0126] Among them, identifying the target based on the near-field image sequence is to finely judge the edge of the target based on the near-field image sequence, so as to finely distinguish the target from the obstruction / background object, thereby facilitating the agricultural picking robot to perform picking actions according to the finely identified target.

[0127] The near-field image sequence includes M frames of images, and M is greater than N. The edge of the target object detected by the visual sensor unit in the near-field is finer than the edge of the target object detected in the far-field.

[0128] Among them, the surface feature information of the target object is obtained so that the agricultural picking robot can adjust the posture of the end effector of the agricultural picking robot based on the surface feature information to facilitate the end effector to pick the target object.

[0129] Among them, the end effector of the agricultural picking robot can perform picking actions to pick fruits based on the current canopy depth information obtained based on the near-field image sequence and the finely identified target objects.

[0130] For example, Figure 5 As shown, Figure 5 This is the visual servo control process when the end effector is in the near field of view. When the robot (agricultural picking robot) is in the near field of view, the robot adjusts the position of the end effector based on the Jacobian matrix. Then the end vision sensor (visual sensor unit) obtains an M-frame sequence (M-frame image) based on real-time dynamic image capture. After the real-time edge detection module accurately identifies the target object, the end vision sensor extracts the target features (surface feature information) through the target feature extraction module. Finally, the robot adjusts the position of the end effector according to the target features.

[0131] Among them, the agricultural picking robot continuously obtains a near-field image sequence of the target object in the near-field, and obtains the surface feature information of the target object based on the near-field image sequence. Then, the agricultural picking robot continuously adjusts the posture of the end effector of the agricultural picking robot according to the surface feature information of the target object, so that when the relative position relationship between the agricultural picking robot and the target object changes, the end effector can still maintain a suitable posture to perform the picking action.

[0132] Specifically, when the agricultural picking robot obtains the surface feature information of the target object, the agricultural picking robot obtains the error between the surface feature information and the given image surface feature within itself, and inputs the error into the visual servo controller. Finally, the visual servo controller controls the agricultural picking robot to continue moving, so that the two surface feature information and the given image surface feature gradually approach each other, thereby reducing the error.

[0133] In step S112, after the visual sensor unit identifies the target object and obtains the current canopy depth information, the end effector has adjusted its posture according to the surface feature information, so that the end effector can perform a picking action according to the current canopy depth information, that is, pick the target object.

[0134] In the process of the end effector performing the picking action, the end effector can also continue to obtain the image of the target object to obtain the surface feature information of the target object, so as to adjust the posture of the end effector during the picking process of the end effector.

[0135] For example, Figure 6 As shown, Figure 6 This is a schematic diagram of the end effector located in different far fields of view and near fields of view. When approaching the target object, the end effector determines whether it is in the first far field of view X1, the second far field of view X2 or the near field of view X3 based on the current canopy depth information obtained by the visual sensor unit. The near field of view X4 is the area where the end effector is located when performing the picking action in the near field of view X3.

[0136] In steps S102 to S110, in the vision-based servo control, a small amount of image information is obtained in the far field of view by the visual sensor unit, and more image information is obtained in the near field of view, so that the visual sensor unit processes fewer images in the far field of view and more images in the near field of view, thereby solving the problem that the existing agricultural picking robot's fruit recognition algorithm is complex, resulting in tight computing resources and then affecting the real-time performance of the system, and also solving the problem that the existing agricultural picking robot executes the same recognition algorithm in different visual ranges, resulting in a waste of computing resources.

[0137] Figure 7 Flowchart (II) of the visual servo control method according to an embodiment of the present invention is as follows: Figure 7 As shown, the following steps are included in the recognition of the target based on the long-field image sequence to roughly recognize the target:

[0138] Step S202: performing image edge detection on each of the N frames of images in the far field image sequence;

[0139] Step S204 : Obtain the maximum gradient value of each image in the N frames of images. If the maximum gradient value is greater than a set value A, the target object is roughly detected.

[0140] Among them, the real-time edge detection module in the visual sensor unit is used to perform image filtering, image edge enhancement, image edge positioning and image edge detection on N frames of images in sequence to roughly detect the edge of the target object, and when the maximum gradient value of the N frames of images is greater than the set value A, the edge of the target object is roughly detected.

[0141] Furthermore, when the maximum gradient value of an image in the N frames of images is less than or equal to the set value A, image edge detection is performed on the next image in the N frames of images.

[0142] For example, Figure 8As shown, the visual sensor unit performs image filtering, image edge enhancement, image edge positioning, and image edge detection on the image frame sequence (far field image sequence) (f(0), f(1), f(2), ..., f(N-1)) in sequence, and then obtains the maximum gradient of one of the frames of the image. When the maximum gradient is greater than the set value A, the edge of the fruit is roughly detected; when the edge of the fruit is not roughly detected, the edge detection processing is performed on the next image.

[0143] In steps 202 to S204, by adopting a vision-based position servo control method and a dynamic image capture method, the rough edge detection method is used to preliminarily identify the fruit, so that a smaller number of image frame sequences can be used, and the rough recognition of the fruit target edge can be achieved with lower data processing and computational complexity, thereby improving the real-time performance of the system control.

[0144] Figure 9 Flowchart (3) of the visual servo control method according to an embodiment of the present invention is shown in FIG. Figure 9 As shown, the target object recognition based on the near-field image sequence obtained by the visual sensor includes:

[0145] Step S302: performing image edge detection on each of the M frames in the near-field image sequence;

[0146] Step S304 : Obtain the maximum gradient value of each image in the M frames of images. If the maximum gradient value is greater than a set value B, the target object is carefully detected, wherein the set value B is greater than the set value A.

[0147] Among them, the real-time edge detection module in the visual sensor unit is also used to perform image filtering, image edge enhancement, image edge positioning and image edge detection on the M frames of images in the near-field image sequence in sequence to finely detect the edges of the target objects.

[0148] Furthermore, when the maximum gradient value of an image in the M frames of images is less than or equal to the set value B, image edge detection is performed on the next image in the M frames of images.

[0149] For example, Figure 10 As shown, the visual sensor unit performs image filtering, image edge enhancement, image edge positioning, and image edge detection on the image frame sequence (near field of view image sequence) (f(0), f(1), f(2), ..., f(M-1)) in sequence to obtain the maximum gradient of one frame of the image. When the maximum gradient is greater than the set value B, the edge of the fruit is finely detected; when the edge of the fruit is not finely detected, the detection continues on the next image.

[0150] Steps S302 to S304 improve image quality and enrich image details by processing a larger number of frames compared to steps S202 to S204. Although the computational efficiency and execution efficiency are reduced to a certain extent, the success rate of target object recognition is improved.

[0151] In steps S302 to S304, when the field of view sensor unit is located in the near field of view, the visual sensor unit uses a fine edge detection method based on dynamic image capture to identify the fruit and extract surface feature information, so that the end effector can perform precise operations to pick the fruit; and the visual sensor unit captures a large number of dynamic image frames in the near field of view, and performs edge detection and refined identification, which is conducive to finely identifying the edges of the target object and obtaining the explicit features of the surface feature information of the target object, which is conducive to improving the accuracy of system operation.

[0152] Figure 11 Flowchart (four) of the visual servo control method according to an embodiment of the present invention is shown in FIG. Figure 11 As shown, obtaining the surface feature information of the target object based on the near-field image sequence includes:

[0153] Step S402: pre-processing the M frames of images in the near-field image sequence in sequence, and dividing each of the M frames into a plurality of subdivided regions;

[0154] Step S404: extracting surface feature information based on a number of subdivided areas.

[0155] Among them, the surface feature information refers to the features of the fruit surface, such as the dominant features of the fruit.

[0156] For example, Figure 12 As shown in FIG, the visual sensor unit preprocesses the acquired M frames of images, and then subdivides the area within the viewing angle of each frame of the image to extract the target features (surface feature information). Then, if the target features match the maturity features, the target position (current canopy depth information) and target feature data are updated based on the data; if the target features do not match the maturity features, the next image is preprocessed.

[0157] Figure 13 FIG. 5 is a flow chart of the visual servo control method according to an embodiment of the present invention, as shown in FIG. Figure 13 As shown, when it is determined that the end effector is located in the near field of view, the visual servo control method further includes:

[0158] Step S502: Control the air gun unit of the agricultural picking robot to perform pulsed air blowing;

[0159] Step S604: Delay for a specified period of time, and control the visual sensor unit to acquire image information in a dynamic image capture mode;

[0160] Step S506: Stop the air gun unit to perform pulsed air blowing, and recognize image information to obtain a recognition result, wherein the recognition result includes whether the target object is contained;

[0161] Step S508: When the recognition result includes the target object, control the agricultural picking robot to pick the target object.

[0162] In step S502, the pulsed air blowing is a blow-once-stop mode, not a continuous air blowing mode, thereby reducing the air blowing volume of the air gun unit and reducing gas consumption.

[0163] In step S504, by controlling the visual sensor unit to obtain image information during a specified delay period, the visual sensor unit can obtain image information after the air gun unit blows the obstruction, thereby avoiding the situation where the image capture response speed of the visual sensor unit is much faster than the air gun blowing action and airflow speed, making it difficult for the visual sensor unit to capture image information of the obstruction being blown.

[0164] In step S506 , the recognition result at least includes whether the image information includes a target object.

[0165] Furthermore, when the recognition result does not include the target object, the agricultural picking robot is controlled to execute the next fruit picking procedure.

[0166] For example, Figure 14 As shown, when the agricultural picking robot is located in the near field of view, the visual sensor unit approaches and picks the first fruit 1. After the robot's end effector picks the first fruit 1, there is a second fruit 2 partially blocked by leaves in the field of view of the visual sensor unit. At this time, the visual sensor unit recognizes the second fruit 2 and determines that it needs to be picked. The visual sensor unit obtains the image information of the second fruit 2 to determine the distance information of the second fruit 2, and then determines the field of view area where the fruit is located based on the distance information. Finally, the second fruit 2 is picked based on the field of view area where the second fruit 2 is located.

[0167] For example, Figure 15 As shown, when the agricultural picking robot is located in the near field of view, the third fruit 3 is completely blocked by the leaves 4. Even if the visual sensor unit 20 recognizes that there is no third fruit 3 but only leaves 4 in the front field of view, steps S502 to S506 are still executed. If it is judged based on the recognition result that there is fruit, step S508 is executed. If it is judged based on the recognition result that there is no fruit, the next fruit recognition and picking program is executed.

[0168] Steps S502 to S510 can blow away the obstruction to reveal the target object, thereby solving the problem that existing agricultural picking robots are prone to missing objects and improving picking efficiency.

[0169] like Figure 16 As shown, the following is a specific implementation of this embodiment:

[0170] Step S602, start pulse blowing;

[0171] Step S604, delay Td;

[0172] Step S606: The visual sensor unit takes pictures during the blowing period;

[0173] Step S608: image recognition;

[0174] Step S610: Determine whether there is fruit to be picked. If there is fruit to be picked, execute step S612; if there is no fruit to be picked, execute step S616;

[0175] Step S612: start the extraction process;

[0176] Step S614: This picking is successful;

[0177] Step S616: Enter the next fruit picking procedure.

[0178] Among them, step S604 is equivalent to step S502.

[0179] Among them, steps S604 to S606 are equivalent to step S504.

[0180] Among them, steps S608 to S610 are equivalent to step S506.

[0181] Among them, steps S612 to S614 are equivalent to step S508.

[0182] Specifically, the airflow blown out by the air gun unit will cause the leaves in front to shake, and then delay for a specified period of time. The visual sensor unit quickly and dynamically captures the image within the field of view, and first determines whether there is a target fruit by edge detection and recognition. If there is a target fruit, the feature point extraction program is used to extract the feature information of the fruit surface, and finally guides the visual servo to approach and perform the picking action; in addition, during the blowing period and the moment the leaves shake, the visual sensor unit dynamically and quickly captures the image. Even if a small part of the fruit rather than the complete fruit image is captured, the edge detection module determines that there is fruit behind the leaves by distinguishing the edge information of the leaves and the fruit. Therefore, there is no need for the airflow to blow the leaves completely or expose the entire picture of the fruit to the visual sensor unit.

[0183] In this specification, references to the same or similar parts between the various embodiments can be made to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the product embodiments described later, since they correspond to the methods, the description is relatively simple, and the relevant parts can be referred to the partial description of the system embodiment.

[0184] The above are only specific embodiments of the present application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A visual servo system for an agricultural harvesting robot, installed on the elbow joint unit of the agricultural harvesting robot, characterized in that: include: a visual sensor unit, the visual sensor unit being mounted on the elbow joint unit and being capable of moving with the elbow joint unit; an air gun unit, the air gun unit being arranged on the elbow joint unit; The air gun unit is used to blow air in the picking direction to blow away the obstructions in the picking direction; The visual sensor unit is used to obtain image information in the picking direction when the air gun unit blows air or not, and send the image information to the agricultural picking robot; The agricultural picking robot is used to perform recognition processing on the image information to form an image processing result, wherein the image processing result includes whether a target object is contained in the picking direction and the distance between the target object and the visual sensor unit. The agricultural picking robot can move and pick the target object based on the image processing result; The visual servo system is configured to perform the following steps: obtaining a position instruction, and controlling the agricultural picking robot to perform a movement action based on the position instruction; During the movement action, a visual processing result is obtained based on current visual information obtained by the visual sensor unit. When the visual processing result is greater than a preset threshold, it is determined that the agricultural picking robot is located in a far field of view, wherein the visual processing result includes current canopy depth information or the number of partitions within the field of view and the number of targets within all partitions obtained based on a far field of view image sequence. The current canopy depth information is used to determine the distance between the visual sensor unit and the target object. When the agricultural harvesting robot is located in the far field of view, identifying a target object and current canopy depth information of the target object based on a far field of view image sequence acquired by the visual sensor unit, and controlling the agricultural harvesting robot to approach the target object based on the current canopy depth information of the target object, wherein the far field of view image sequence includes N frames of images; During the process of the agricultural picking robot approaching the target object, if the visual processing result obtained based on the current visual information acquired by the visual sensor unit is less than or equal to the preset threshold, it is determined that the agricultural picking robot is located in the near field of view; When the agricultural picking robot is located in the near field of view, the target object is identified based on a near field of view image sequence acquired by the visual sensor unit, and surface feature information of the target object and current canopy depth information of the target object are acquired, wherein the near field of view image includes M frames of images, and M is greater than N; The posture of the end effector of the agricultural harvesting robot is adjusted based on the surface feature information, and the end effector is controlled to perform a harvesting action based on the target object identified by the near-field image sequence and the current canopy depth information of the target object.

2. The visual servo system according to claim 1, wherein: The air gun unit comprises: A valve element is provided in the air gun unit and is used to control the air gun unit to perform pulsed air blowing along the picking direction.

3. A visual servo control method for an agricultural picking robot, characterized in that: include: obtaining a position instruction, and controlling the agricultural picking robot to perform a movement action based on the position instruction; During the movement action, a visual processing result is obtained based on current visual information obtained by the visual sensor unit. When the visual processing result is greater than a preset threshold, it is determined that the agricultural picking robot is located in a far field of view, wherein the visual processing result includes current canopy depth information or the number of partitions within the field of view and the number of targets within all partitions obtained based on a far field of view image sequence. The current canopy depth information is used to determine the distance between the visual sensor unit and the target object. When the agricultural harvesting robot is located in the far field of view, identifying a target object and current canopy depth information of the target object based on a far field of view image sequence acquired by the visual sensor unit, and controlling the agricultural harvesting robot to approach the target object based on the current canopy depth information of the target object, wherein the far field of view image sequence includes N frames of images; During the process of the agricultural picking robot approaching the target object, if the visual processing result obtained based on the current visual information acquired by the visual sensor unit is less than or equal to the preset threshold, it is determined that the agricultural picking robot is located in the near field of view; When the agricultural picking robot is located in the near field of view, the target object is identified based on a near field of view image sequence acquired by the visual sensor unit, and surface feature information of the target object and current canopy depth information of the target object are acquired, wherein the near field of view image includes M frames of images, and M is greater than N; The posture of the end effector of the agricultural harvesting robot is adjusted based on the surface feature information, and the end effector is controlled to perform a harvesting action based on the target object identified by the near-field image sequence and the current canopy depth information of the target object.

4. The visual servo control method according to claim 3, wherein: Determining, based on the visual processing result, that the agricultural picking robot is located in the far field of view includes: During the movement, a far field image sequence is acquired based on the visual sensor unit, and when current canopy depth information acquired based on the far field image sequence is greater than a preset distance threshold, it is determined that the agricultural picking robot is located in the far field; or Determining, based on the visual processing result, that the agricultural picking robot is located in the near field of view includes: In the process of the agricultural picking robot approaching the target object, the far field image sequence is continuously acquired based on the visual sensor unit, and when the current canopy depth information acquired based on the far field image sequence is less than or equal to a preset distance threshold, it is determined that the agricultural picking robot is located in the near field of view.

5. The visual servo control method according to claim 3, wherein: Determining, based on the visual processing result, that the agricultural picking robot is located in the far field of view includes: During the movement, the number of partitions within the field of view and the number of targets within the partitions are obtained based on the field of view of the visual sensor unit. If the number of partitions is greater than a preset threshold value for the number of partitions and the number of targets within all the partitions is greater than a preset threshold value for the number of targets, it is determined that the agricultural picking robot is located in the far field of view; or Determining, based on the visual processing result, that the agricultural picking robot is located in the near field of view includes: In the process of the agricultural picking robot approaching the target object, the number of partitions within the field of view and the number of targets in all the partitions are obtained based on the field of view of the visual sensor unit. When the number of partitions is less than or equal to a preset partition number threshold and the number of targets in all the partitions is less than or equal to the preset target number threshold, it is determined that the end effector is located in the near field of view.

6. The visual servo control method according to claim 3, wherein: Identifying the target object based on the far field image sequence includes: Performing image edge detection on each of the N frames of images in the far field image sequence; Obtaining the maximum gradient value of each image in the N frames of images, and if the maximum gradient value is greater than a set value A, roughly detecting the target object; or When the maximum gradient value of an image in N frames of images is less than or equal to the set value A, image edge detection is performed on the next image in the N frames of images.

7. The visual servo control method according to claim 6, wherein: Identifying the target object based on the near-field image sequence acquired by the visual sensor unit includes: Performing image edge detection on each of the M frames of image in the near-field image sequence; Obtaining a maximum gradient value of each image in the M frames of images, and carefully detecting the target object when the maximum gradient value is greater than a set value B, wherein the set value B is greater than the set value A; or When the maximum gradient value of an image in the M frames of images is less than or equal to the set value B, image edge detection is performed on the next image in the M frames of images.

8. The visual servo control method according to claim 3, wherein: Acquiring surface feature information of the target object based on the near-field image sequence includes: Preprocessing the M frames of images in the near-field image sequence in sequence, and dividing each of the M frames of images into a plurality of subdivided areas; The surface feature information is extracted based on a plurality of the subdivided areas.

9. The visual servo control method according to any one of claims 3 to 8, characterized in that: When it is determined that the end effector is located in the near field of view, the visual servo control method further includes: controlling the air gun unit of the agricultural picking robot to perform pulsed air blowing; Delaying for a specified period of time, and controlling the visual sensor unit to acquire image information in a dynamic image capture manner; stopping the air gun unit to perform pulsed air blowing, and identifying the image information to obtain a recognition result, wherein the recognition result includes whether the target object is contained; In a case where the recognition result includes the target object, the agricultural picking robot is controlled to pick the target object.

Images