Intelligent picking control method based on machine vision and related robot system

Abstract

The invention provides an intelligent picking control method based on machine vision and a related robot system, and the method comprises the steps: shooting a first position of a target fruit tree through a visual system, obtaining a first image, obtaining the first attribute information of the target fruit tree, carrying out the target recognition of the first image according to the first attribute information, and obtaining a second image; obtaining m to-be-collected targets and m pieces of second attribute information; determining n picking points according to the m pieces of second attribute information and the first image; determining a picking sequence of the m to-be-collected targets according to the n picking points and the m pieces of second attribute information; acquiring third attribute information of the n picking points to obtain n pieces of third attribute information; determining a first control parameter according to the picking sequence and the n pieces of third attribute information; and determining a second control parameter according to the first control parameter, the picking sequence and the n picking points. According to the embodiment of the invention, the intelligence of the picking robot can be improved.

Description

Technical Field

[0001] The present application relates to the field of robotics technology or computer technology, and specifically to an intelligent picking control method based on machine vision and a related robot system. Background Art

[0002] With the rapid development of science and technology, robots have also become a "hot spot" in the market. Taking agricultural robots as an example, they are the application of robots in agricultural production. Specifically, they can be understood as a new generation of unmanned automatic operating machinery that can be controlled by different program software to adapt to various operations, and can sense and adapt to crop types or environmental changes, and have artificial intelligence skills in intelligent detection (such as vision, etc.) functions and calculation functions.

[0003] The application of agricultural robots in picking can also be called picking robots. The current picking robots are not intelligent enough, so the problem of how to improve the intelligence of picking robots needs to be solved urgently. Summary of the invention

[0004] The embodiments of the present application provide an intelligent picking control method based on machine vision and a related robot system, thereby improving the intelligence of the picking robot.

[0005] In a first aspect, an embodiment of the present application provides an intelligent picking control method based on machine vision, which is applied to a picking robot, wherein the picking robot includes a first mechanical arm, a tool system, a second mechanical arm, a moving device, a storage device, and a visual system, wherein the first mechanical arm is connected to the tool system; the second mechanical arm is used to connect to the storage device; the method includes:

[0006] Controlling the picking robot to move by the mobile device, and detecting the distance between the target fruit tree and the picking robot by the visual system;

[0007] When the distance is within a preset range, the first position of the target fruit tree is photographed by the visual system to obtain a first image, first attribute information of the target fruit tree is obtained, and target recognition is performed on the first image according to the first attribute information to obtain m targets to be collected and m second attribute information, and each target to be collected corresponds to one second attribute information; m is a positive integer;

[0008] Determine n picking points according to the m second attribute information and the first image, where n is a positive integer less than or equal to m;

[0009] Determining a picking order of the m objects to be collected according to the n picking points and the m second attribute information;

[0010] Obtain the third attribute information of the n picking points to obtain n pieces of third attribute information;

[0011] Determine a first control parameter according to the picking order and the n pieces of third attribute information, where the first control parameter includes: a first movement parameter of the first robotic arm and a first action parameter of the tool system;

[0012] Determine a second control parameter according to the first control parameter, the picking order, and the n picking points, where the second control parameter includes: a second movement parameter of the second robotic arm and a second action parameter of the storage device;

[0013] Control the first robotic arm and the tool system to perform picking through the first control parameter, and control the second robotic arm and the storage device to store the m targets to be collected through the second control parameter.

[0014] In a second aspect, an intelligent picking control system based on machine vision provided by an embodiment of the present application is applied to a picking robot. The picking robot includes a first robotic arm, a tool system, a second robotic arm, a moving device, a storage device, and a vision system. The first robotic arm is connected to the tool system; the second robotic arm is used to connect the storage device. The intelligent picking control system based on machine vision includes: a detection unit, an identification unit, a determination unit, an acquisition unit, and a picking unit, where,

[0015] The detection unit is used to control the picking robot to move through the moving device and detect the distance between the target fruit tree and the picking robot through the vision system;

[0016] The identification unit is used to, when the distance is within a preset range, capture a first position of the target fruit tree through the vision system to obtain a first image, obtain the first attribute information of the target fruit tree, and perform target recognition on the first image according to the first attribute information to obtain m targets to be collected and m pieces of second attribute information, and each target to be collected corresponds to a piece of second attribute information; m is a positive integer;

[0017] The determination unit is used to determine n picking points according to the m pieces of second attribute information and the first image, where n is a positive integer less than or equal to m; determine the picking order of the m targets to be collected according to the n picking points and the m pieces of second attribute information;

[0018] The acquisition unit is used to obtain the third attribute information of the n picking points to obtain n pieces of third attribute information;

[0019] The determining unit is further configured to determine a first control parameter according to the picking sequence and the n third attribute information, where the first control parameter includes: a first movement parameter of the first robotic arm and a first action parameter of the tool system; and determine a second control parameter according to the first control parameter, the picking sequence and the n picking points, where the second control parameter includes: a second movement parameter of the second robotic arm and a second action parameter of the storage device.

[0020] The picking unit is configured to control the first robotic arm and the tool system to perform picking through the first control parameter, and control the second robotic arm and the storage device to store the m to-be-picked targets through the second control parameter.

[0021] In a third aspect, an embodiment of the present application provides an electronic device, including a processor, a memory, a communication interface, and one or more programs, where the one or more programs are stored in the memory and are configured to be executed by the processor, and the programs include instructions for executing the steps in the first aspect of the embodiment of the present application.

[0022] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, where the computer-readable storage medium stores a computer program for electronic data exchange, and the computer program causes a computer to execute some or all of the steps described in the first aspect of the embodiment of the present application.

[0023] In a fifth aspect, an embodiment of the present application provides a computer program product, where the computer program product includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute some or all of the steps described in the first aspect of the embodiment of the present application. The computer program product may be a software installation package.

[0024] Implementing the embodiments of the present application has the following beneficial effects:

[0025] It can be seen that the machine vision-based intelligent picking control method and related system described in the embodiments of the present application are applied to a picking robot. The picking robot includes a first robotic arm, a tool system, a second robotic arm, a mobile device, a storage device, and a vision system. The first robotic arm is connected to the tool system; the second robotic arm is used to connect the storage device. The picking robot is controlled to move through the mobile device. The distance between the target fruit tree and the picking robot is detected through the vision system. When the distance is within a preset range, the first position of the target fruit tree is photographed through the vision system to obtain a first image. The first attribute information of the target fruit tree is obtained, and the first image is subjected to target recognition according to the first attribute information to obtain m to-be-picked targets and m pieces of second attribute information. Each to-be-picked target corresponds to one piece of second attribute information; m is a positive integer. n picking points are determined according to the m pieces of second attribute information and the first image, where n is a positive integer less than or equal to m. The picking order of the m to-be-picked targets is determined according to the n picking points and the m pieces of second attribute information. The third attribute information of the n picking points is obtained to get n pieces of third attribute information. The first control parameter is determined according to the picking order and the n pieces of third attribute information. The first control parameter includes: the first movement parameter of the first robotic arm and the first action parameter of the tool system. The second control parameter is determined according to the first control parameter, the picking order, and the n picking points. The second control parameter includes: the second movement parameter of the second robotic arm and the second action parameter of the storage device. The first robotic arm and the tool system are controlled to pick through the first control parameter, and the second robotic arm and the storage device are controlled to store the m to-be-picked targets through the second control parameter. First, the distance between the picking robot and the target fruit tree is appropriate, which can enable efficient picking and ensure the clarity of the fruits. Since the first attribute information reflects the characteristics of the target fruit tree, the accuracy of fruit recognition can be guaranteed. Second, to a certain extent, the second attribute information reflects the fruit characteristics and the surrounding environment characteristics of the fruits. The first image reflects the connection relationship between fruits and between fruits and their surrounding environment, that is, the corresponding picking points can be determined based on the fruit characteristics and the surrounding environment characteristics of the fruits. Picking through these picking points will not affect the fruit quality and can minimize the damage to the fruit tree as soon as possible. Third, the corresponding picking order can be determined based on the actual fruit characteristics and the surrounding environment characteristics of the fruits. Therefore, the subsequent fruit picking efficiency can be guaranteed, and the subsequent fruit quality classification efficiency can be guaranteed to a certain extent. Fourth, the movement coordination and consistency between the two robotic arms can be guaranteed, which helps to ensure the subsequent fruit picking efficiency and can also guarantee the subsequent fruit quality classification efficiency to a certain extent, that is, the intelligence of the picking robot can be improved. Description of the Drawings

[0026] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the accompanying drawings required for the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can also be obtained based on these drawings.

[0027] Figure 1 is a schematic flowchart of an intelligent picking control method based on machine vision provided by an embodiment of the present application;

[0028] Figure 2 is a schematic structural diagram of a picking robot provided by an embodiment of the present application;

[0029] Figure 3 is a schematic structural diagram of an electronic device provided by an embodiment of the present application;

[0030] Figure 4 is a block diagram of the functional units of an intelligent picking control system based on machine vision provided by an embodiment of the present application. Detailed implementation manners

[0031] The terms "first", "second", etc. in the specification and claims of the present application and the above accompanying drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may also include unlisted steps or units in a possible example, or other steps or units inherent to these processes, methods, products or devices in a possible example.

[0032] Referring to "embodiment" herein means that a specific feature, structure or characteristic described in connection with the embodiment may be included in at least one embodiment of the present application. The phrase appears in various places in the specification and does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described herein may be combined with other embodiments.

[0033] In order to enable those skilled in the art to better understand the solution of the present application, the following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.

[0034] The electronic devices involved in the embodiments of the present application may include but are not limited to: picking robots, servers, controllers of picking robots, mobile terminals (such as mobile phones, tablets, etc.), etc. There is no limitation here. For example, the electronic device may include a picking robot. Another example is that when the electronic device does not include a picking robot, a communication connection can be established between the electronic device and the picking robot.

[0035] Among them, the mobile device may include at least one of the following: crawler device, tire device, mechanical leg device, etc. There is no limitation here.

[0036] Among them, the picking robot may also include any one of the following: humanoid robot, or animal-shaped robot, or car-shaped robot, or flying robot.

[0037] Please refer to Figure 1 , Figure 1 , which is a schematic flow chart of an intelligent picking control method based on machine vision provided by the embodiments of the present application, and is applied to a picking robot. The picking robot includes a first robotic arm, a tool system, a second robotic arm, a mobile device, a storage device, and a vision system. The first robotic arm is connected to the tool system; the second robotic arm is used to connect the storage device. This intelligent picking control method based on machine vision includes:

[0038] 101. Control the picking robot to move through the mobile device, and detect the distance between the target fruit tree and the picking robot through the vision system.

[0039] In specific implementation, as Figure 2 shown, the picking robot may include a first robotic arm, a tool system, a second robotic arm, a robot body, a mobile device, a storage device, and a vision system. The first robotic arm is connected to the tool system; the second robotic arm is used to connect the storage device. The tool system may include any tool for realizing fruit picking. The tool system may include at least one of the following tools: clamp, fruit shear, cutter, saw, etc. There is no limitation here, and different tools can be switched. The storage device may include at least one of the following: storage bag, storage box, storage net, etc. There is no limitation here. Both the first robotic arm and the second robotic arm can extend a certain length, and the extension lengths of the first robotic arm and the second robotic arm are different. Both the first robotic arm and the second robotic arm can be fixed on the mobile device; or, the picking robot may include a robot body, and both the first robotic arm and the second robotic arm can be fixed on the robot body, and the robot body can be arranged above the mobile device.

[0040] Among them, the vision system can be set on at least one of the following components: the first robotic arm, the second robotic arm, the mobile device, etc., which is not limited here.

[0041] Among them, the mobile device can be used to control the movement of the picking robot. The tool system can be used to pick the fruits of the target fruit tree. The storage device can be used to receive the picked fruits.

[0042] In specific implementation, the target fruit tree can be any fruit tree, and the fruit tree can include at least one of the following: apple tree, pear tree, banana tree, persimmon tree, pomegranate tree, jujube tree, peach tree, chestnut tree, longan tree, grape tree, etc., which is not limited here.

[0043] Among them, the vision system can include at least one of the following: camera, ultrasonic sensor, laser sensor, radar sensor, etc., which is not limited here.

[0044] Among them, the camera can include one or more cameras. For example, the camera can include a dual camera, or a depth camera. In specific implementation, the distance between the target fruit tree and the picking robot can be detected at preset time intervals. The preset time interval can be set in advance or be the system default. For example, the preset time interval can be related to the moving speed of the mobile device. The greater the moving speed, the smaller the preset time interval. On the contrary, the smaller the moving speed, the larger the preset time interval. Also, for example, the preset time interval can be related to the distance between the mobile device and the target fruit tree. If the distance is greater, the preset time interval is larger; if the distance is smaller, the preset time interval is smaller.

[0045] In the embodiments of this application, the movement of the picking robot can be controlled by the mobile device, and the distance between the target fruit tree and the picking robot can be detected by the vision system at preset time intervals. Of course, the vision system can also observe the terrain to find a suitable position to fix the mobile device, and then carry out the picking work at this position, so as to ensure the stable picking of the picking robot.

[0046] Of course, in specific implementation, the target fruit tree can also be scanned as a whole by the vision system to obtain a 3D model of the target fruit tree. At least one area can be determined based on the 3D model, and each area corresponds to a collection area of the target fruit tree. The mobile device can be used to control the picking robot to move to each area of the at least one area to pick the corresponding collection area.

[0047] 102. When the distance is within a preset range, capture the first position of the target fruit tree through the vision system to obtain a first image, acquire the first attribute information of the target fruit tree, and perform target recognition on the first image according to the first attribute information to obtain m targets to be collected and m pieces of second attribute information, with each target to be collected corresponding to one piece of second attribute information; m is a positive integer.

[0048] Among them, the targets to be collected may include a cluster of fruits or a single fruit.

[0049] Among them, the preset range can be set in advance or be the system default, and the preset range may be related to at least one of the following factors: the height of the target fruit tree, the length of the first robotic arm, the length of the second robotic arm, the girth of the target fruit tree, etc., which are not limited herein.

[0050] Among them, the first attribute information may include at least one of the following: the variety of the target fruit tree, the height of the target fruit tree, the tree rings (age) of the target fruit tree, the girth of the target fruit tree, the growth trend of the target fruit tree, the growth environment information of the target fruit tree, the color of the fruits of the target fruit tree, the shape of the fruits of the target fruit tree, the size of the target fruits, etc., which are not limited herein. The growth environment information of the target fruit tree may include at least one of the following: geographical location, soil environment parameters, fertilization situation, pesticide application situation, etc., which are not limited herein.

[0051] In the embodiments of the present application, when the distance is within the preset range, it indicates that the distance between the picking robot and the target fruit tree is appropriate and efficient picking can be carried out. Furthermore, the first position of the target fruit tree can be captured through the vision system to obtain a first image, and the clarity of the fruits can also be ensured.

[0052] Among them, the second attribute information of the targets to be collected may include at least one of the following: color space related parameters, position, shape, connection relationship parameters, size, etc., which are not limited herein. The color space related parameters may include at least one of the following: color, external color and luster, gloss, etc., which are not limited herein. The connection relationship parameters refer to the connection relationship between different parts of the fruit tree, and this connection relationship parameter can be understood as the connection relationship between the fruit and the fruit stalk, or the connection relationship between the fruit and the surrounding leaves, or the connection relationship between the fruit and the root or leaf stalk. For example, an apple is connected to the branch of an apple tree through the fruit stalk.

[0053] In specific implementation, the first attribute information of the target fruit tree can be acquired. Since the first attribute information reflects the characteristics of the target fruit tree, target recognition is performed on the first image according to the first attribute information to obtain m targets to be collected and m pieces of second attribute information, which can ensure the accuracy of fruit recognition. For example, fruits with the desired maturity can be recognized, and each target to be collected corresponds to one piece of second attribute information, where m is a positive integer.

[0054] Optionally, in step 102 above, target recognition is performed on the first image according to the first attribute information to obtain m targets to be collected and m pieces of second attribute information, which may include the following steps:

[0055] Determine a first image segmentation algorithm and a first target recognition algorithm corresponding to the first attribute information;

[0056] Obtain a first reference control parameter of the first image segmentation algorithm and a second reference control parameter of the first target recognition algorithm corresponding to the first attribute information;

[0057] Obtain a first shooting parameter of the first image;

[0058] Determine a first control parameter according to the first shooting parameter and the first reference control parameter;

[0059] Perform target segmentation on the first image according to the first control parameter and the first image segmentation algorithm to obtain k target regions, where k is an integer greater than or equal to m;

[0060] Determine the region size parameters of the k target regions to obtain k region size parameters;

[0061] Determine a second control parameter according to the k region size parameters and the second reference control parameter;

[0062] Perform recognition on the k target regions according to the second control parameter and the first target recognition algorithm to obtain the m targets to be collected.

[0063] In specific implementation, a mapping relationship between the attribute information of a preset fruit tree and an image segmentation algorithm can be stored in advance. Furthermore, based on this mapping relationship, the first image segmentation algorithm corresponding to the first attribute information can be determined. Since the first attribute information reflects the characteristics of the target fruit tree, in this way, an image segmentation algorithm corresponding to the target fruit tree can be selected, which can ensure the accuracy of image segmentation of the fruits of the target fruit tree.

[0064] Among them, the first image segmentation algorithm can be any algorithm for implementing the image segmentation function, and the first target recognition algorithm can be any algorithm for implementing the target recognition function.

[0065] Among them, the control parameter of the first image segmentation algorithm is used to control the segmentation effect of the fruits of the target fruit tree. This segmentation effect may include at least one of the following: segmentation speed, segmentation degree, segmented target size, segmented target color, segmented target gloss, etc., which are not limited here. For example, some smaller fruits can be filtered out, or some bad fruits can be filtered out, or some unripe fruits can be filtered out.

[0066] Among them, the control parameters of the first target recognition algorithm are used to control the target recognition effect of the fruits of the target fruit tree. The target recognition effect may include at least one of the following: target recognition speed, target recognition degree, recognized target size, target color, target gloss, etc., which are not limited herein. For example, some smaller fruits, or some bad fruits, or some unripe fruits can be filtered out.

[0067] Correspondingly, the mapping relationship between the attribute information of the preset fruit tree and the target recognition algorithm can also be pre-stored. Furthermore, the first target recognition algorithm corresponding to the first attribute information of the target fruit can be determined based on this mapping relationship. Since the first attribute information reflects the characteristics of the target fruit tree, in this way, the target recognition algorithm corresponding to the target fruit tree can be selected, which can not only ensure the accuracy of target recognition of the fruits of the target fruit tree, but also ensure the accuracy of recognition of ripe fruits.

[0068] Of course, the mapping relationship between the attribute information of the preset target fruit tree and the fruit tree evaluation parameters can also be pre-stored. The fruit tree evaluation parameters are used to evaluate the quality of the fruit tree. For example, the fruit tree evaluation parameters are used to evaluate the growth degree of the fruit tree, or the growth trend of the fruit tree, or the economic value of the fruit tree, etc., which are not limited herein. Furthermore, the first attribute information corresponding to the first attribute information can be determined based on this mapping relationship. The mapping relationship between the preset fruit tree evaluation parameters and the control parameters of the first image segmentation algorithm can also be pre-stored. Furthermore, the first reference control parameter of the first image segmentation algorithm corresponding to the first fruit tree evaluation parameter can be determined based on this mapping relationship. Correspondingly, the mapping relationship between the preset fruit tree evaluation parameters and the control parameters of the first target recognition algorithm can also be pre-stored. Furthermore, the second reference control parameter of the first target recognition algorithm corresponding to the first fruit tree evaluation parameter can be determined based on this mapping relationship. In this way, the characteristics of the fruit tree can be deeply considered to determine the corresponding image segmentation algorithm and the corresponding control parameters, and the target recognition algorithm and the corresponding control parameters.

[0069] Among them, the first shooting parameter may include at least one of the following: shooting mode, sensitivity, exposure duration, focal length, white balance parameter, etc., which are not limited herein. In specific implementation, the first shooting parameter of the first image can be obtained. Since the first shooting parameter reflects the shooting effect, furthermore, the first control parameter can be determined according to the first shooting parameter and the first reference control parameter, that is, the control parameter is further optimized based on the shooting effect, so that the image segmentation effect not only deeply conforms to the characteristics of the target fruit tree, but also can deeply conform to its actual shooting effect.

[0070] Among them, the region size parameter may include at least one of the following: region contour parameter, region area, region perimeter, etc., which are not limited herein.

[0071] Next, the first image can be subjected to target segmentation according to the first control parameter and the first image segmentation algorithm to obtain k target regions, where k is an integer greater than or equal to m. Then, the region size parameters of the k target regions are determined to obtain k region size parameters, and the second control parameter is determined according to the k region size parameters and the second reference control parameter. The k region size parameters reflect the correlation between regions, that is, the consistency (integrity) between regions at different positions of the target fruit tree. Based on this consistency, the accuracy of target recognition can be further grasped. For example, some unripe fruits, or some bad fruits, or some small fruits can be not recognized. Finally, the k target regions are recognized according to the second control parameter and the first target recognition algorithm to obtain m target fruits to be collected.

[0072] In this example, first, an image segmentation algorithm corresponding to the target fruit tree can be selected, which can ensure the accuracy of image segmentation of the fruits of the target fruit tree, and an target recognition algorithm corresponding to the target fruit tree can be selected, which can not only ensure the accuracy of target recognition of the fruits of the target fruit tree, but also ensure the accuracy of recognition of ripe fruits. Second, the control parameter is further optimized based on the shooting effect, so that the image segmentation effect not only deeply conforms to the characteristics of the target fruit tree, but also deeply conforms to its actual shooting effect. Third, the k region size parameters reflect the correlation between regions, the consistency (integrity) between regions at different positions of the target fruit tree. Based on this consistency, the accuracy of target recognition can be further grasped. For example, some unripe fruits, or some bad fruits, or some small fruits can be not recognized. In this way, the accuracy and quality of target segmentation and target recognition can be ensured, the subsequent fruit picking efficiency can be guaranteed, and the subsequent fruit quality classification efficiency can also be guaranteed to a certain extent.

[0073] Of course, target segmentation can not only be understood as preliminary fruit filtering, but also the results of target segmentation can be used to guide target recognition to further ensure the accuracy of target recognition. Target recognition is also equivalent to further fruit filtering, which can not only ensure the accuracy of fruit recognition, but also ensure the quality of fruit recognition.

[0074] Optionally, for the above steps, determining the first control parameter according to the first shooting parameter and the first reference control parameter can be implemented in the following manner:

[0075] Determine a first adjustment parameter corresponding to the first shooting parameter;

[0076] Adjust the first reference control parameter according to the first adjustment parameter to obtain the first control parameter.

[0077] In specific implementation, the first mapping relationship between the shooting parameters and the adjustment parameters can be pre-stored. Furthermore, the first adjustment parameter corresponding to the first shooting parameter can be determined based on this mapping relationship. Furthermore, the first reference control parameter can be adjusted according to the first adjustment parameter to obtain the first control parameter. For example, the first control parameter = (1 + the first adjustment parameter) * the first reference control parameter. In this way, the control parameter is further optimized based on the shooting effect, so that the image segmentation effect not only conforms to the characteristics of the target fruit tree in depth, but also conforms to its actual shooting effect in depth, which helps to ensure the subsequent fruit recognition accuracy and fruit recognition quality.

[0078] Optionally, the above step of determining the second control parameter according to the k regional size parameters and the second reference control parameter may include at least one of the following:

[0079] Determine k area values according to the k regional size parameters;

[0080] Determine the first standard deviation of the k area values;

[0081] Determine k perimeters according to the k regional size parameters;

[0082] Determine the second standard deviation between the k perimeters;

[0083] Determine the second adjustment parameter corresponding to the first standard deviation;

[0084] Determine the first fine-tuning parameter corresponding to the second standard deviation;

[0085] Determine the second control parameter according to the second adjustment parameter, the first fine-tuning parameter and the second reference control parameter.

[0086] In specific implementation, k area values can be determined according to the k regional size parameters, and each regional size parameter corresponds to an area value. The standard deviation operation can also be performed on the k area values to obtain the first standard deviation. The first standard deviation reflects to a certain extent the consistency of the fruit sizes of the target fruit tree. Correspondingly, each regional size parameter corresponds to a perimeter. k perimeters can be determined according to each regional size parameter among the k regional size parameters, and then the standard deviation operation is performed on the k perimeters to obtain the second standard deviation. The second standard deviation reflects to a certain extent the consistency of the fruit shapes of the target fruit tree.

[0087] Next, the mapping relationship between the preset standard deviation and the adjustment parameter can be pre-stored, and based on this mapping relationship, the second adjustment parameter corresponding to the first standard deviation can be determined. The mapping relationship between the preset standard deviation and the fine-tuning parameter can also be pre-stored. Furthermore, based on this mapping relationship, the first fine-tuning parameter corresponding to the second standard deviation can be determined. Finally, the second control parameter can be determined according to the second adjustment parameter, the first fine-tuning parameter, and the second reference control parameter, that is, the second control parameter = (1 + the second adjustment parameter) * (1 + the first fine-tuning parameter) * the second reference control parameter. In this way, since the first standard deviation reflects to a certain extent the consistency of the fruit size of the target fruit tree, and the second standard deviation reflects to a certain extent the consistency of the fruit shape of the target fruit tree, the fruit size and the fruit shape represent the fruit quality to a certain extent. Furthermore, based on this consistency, the accuracy of target recognition can be further grasped. For example, some unripe fruits can be not recognized, or some bad fruits can be not recognized, or some smaller fruits can be not recognized. In this way, the accuracy and recognition quality of target segmentation and target recognition can be guaranteed, the subsequent fruit picking efficiency can be ensured, and the subsequent fruit quality classification efficiency can also be guaranteed to a certain extent.

[0088] 103. Determine n picking points according to the m second attribute information and the first image, where n is a positive integer less than or equal to m.

[0089] In the embodiment of the present application, the second attribute information reflects the fruit characteristics and the surrounding environment characteristics of the fruit to a certain extent, and the first image reflects the connection relationship between the fruits and between the fruits and their surrounding environment. That is, n picking points can be determined according to the m second attribute information and the first image, where n is a positive integer less than or equal to m. Since it is possible to pick one fruit at a time, or pick multiple fruits at a time, the corresponding picking points can be determined based on the fruit characteristics and the surrounding environment characteristics of the fruit. Picking through these picking points will not affect the fruit quality and will also minimize the damage to the fruit tree as soon as possible.

[0090] Optionally, the second attribute information includes connection relationship parameters, position, shape, size, color space related parameters; the above step 103 of determining n picking points according to the m second attribute information and the first image can be implemented in the following manner:

[0091] Determine a reference picking points according to the connection relationship parameters and positions in the m second attribute information, where a is a positive integer greater than or equal to n;

[0092] Divide the m targets to be collected into a target sets to be collected according to the a reference picking points;

[0093] Determine the evaluation value of each target collection to be picked in the a target collections to be picked according to the m pieces of second attribute information, and obtain a evaluation values;

[0094] Select the evaluation values that meet the preset conditions from the a evaluation values to obtain n evaluation values, and obtain the picking points corresponding to the n evaluation values to obtain the n picking points.

[0095] Among them, the preset conditions can be set in advance or be the system default. For example, if the evaluation value is within the preset range, it means that the evaluation value meets the preset conditions; on the contrary, if the evaluation value is not within the preset range, it means that the evaluation value does not meet the preset conditions. The preset range can be set in advance or be the system default.

[0096] In specific implementation, a reference picking points can be determined according to the connection relationship parameters and positions in the m pieces of second attribute information. a is a positive integer greater than or equal to n. For example, based on the connection relationship parameters and positions, the corresponding petiole can be locked, that is, a specified position on the petiole can be used as the picking point. The specified position can be set in advance or be the system default, and the specified position can be the end of the petiole.

[0097] Among them, each target collection to be picked can be understood as one or more targets to be picked that can be picked at one time.

[0098] Next, the m targets to be picked can be divided into a target collections to be picked according to the a reference picking points. Each target collection to be picked can include at least one target to be picked. Then, determine the evaluation value of each target collection to be picked in the a target collections to be picked according to the m pieces of second attribute information (such as: shape, size, color space related parameters) to obtain a evaluation values. The shape, size, and color space related parameters reflect the weight, quality, and maturity of the fruit to a certain extent. Furthermore, for any target collection to be picked, the fruit is evaluated accordingly based on the shape, size, and color space related parameters to obtain multiple evaluation parameters, and then these evaluation parameters are weighted and calculated to obtain the evaluation value of this target collection to be picked. This evaluation value reflects the weight, quality, and maturity of the fruit, and also reflects the characteristics of the fruit corresponding to a target collection to be picked. Select the evaluation values that meet the preset conditions from the a evaluation values to obtain n evaluation values, and obtain the picking points corresponding to the n evaluation values to obtain the n picking points. For example, the smaller the evaluation value, the worse the possible quality or the less mature it may be, so it can be temporarily not picked. Based on this method, the picking points can be initially determined based on the connection relationship of the fruit, and the picking points can also be screened according to the weight, quality, and maturity of the fruit. Thus, the subsequent fruit picking efficiency can be guaranteed, and the subsequent fruit quality classification efficiency can also be guaranteed to a certain extent.

[0099] Among them, for the corresponding evaluation of fruits based on shape, size, and color space-related parameters to obtain multiple evaluation parameters, that is, the mapping relationship between each dimension (shape, size, color space-related parameters) and the evaluation parameters can be pre-stored. Furthermore, based on this mapping relationship, the evaluation parameters of each dimension are determined to obtain multiple evaluation parameters. Each dimension corresponds to a weight set, and the sum of the weight set is 1. The weight set is related to the branch-related parameters of the target fruit tree. The branch-related parameters may include at least one of the following: branch type, branch position, branch length, branch thickness, etc., which are not limited here.

[0100] 104. Determine the picking order of the m to-be-picked targets according to the n picking points and the m second attribute information.

[0101] In the embodiments of the present application, the second attribute information reflects the fruit characteristics and the surrounding environment characteristics of the fruit to a certain extent. The picking order of the m to-be-picked targets can be determined according to the n picking points and the m second attribute information. In this way, the corresponding picking order can be determined based on the actual fruit characteristics and the surrounding environment characteristics of the fruit. Therefore, the subsequent fruit picking efficiency can be guaranteed, and the subsequent fruit quality classification efficiency can also be guaranteed to a certain extent.

[0102] Optionally, in the above step 106, determining the picking order of the m to-be-picked targets according to the n picking points and the m second attribute information can be implemented in the following manner:

[0103] Determine the trajectory length between each picking point in the n picking points and the first robotic arm to obtain n first trajectory lengths;

[0104] Determine the perturbation parameter of the evaluation value corresponding to each picking point in the n picking points to obtain n perturbation parameters;

[0105] Determine n second trajectory lengths according to the n first trajectory lengths and the n perturbation parameters;

[0106] Sort according to the n second trajectory lengths to obtain the picking order.

[0107] In specific implementation, fitting can be performed based on the actual space environment to determine the trajectory length between each picking point in the n picking points and the first robotic arm to obtain n first trajectory lengths. The trajectory length can be understood as the trajectory length of the first robotic arm from the current position to the picking point. Then, the mapping relationship between the evaluation value of the preset picking point and the perturbation parameter can also be pre-stored. Furthermore, based on this mapping relationship, the perturbation parameter of the evaluation value corresponding to each picking point in the n picking points can be determined to obtain n perturbation parameters. The perturbation parameter can be between 0 and 1. The larger the evaluation value, the smaller the perturbation parameter.

[0108] Next, n second trajectory lengths can be determined based on the n first trajectory lengths and the n perturbation parameters. The second trajectory length = (1 - perturbation parameter) * first trajectory length. Then, sort the n second trajectory lengths from smallest to largest to obtain the picking order.

[0109] The corresponding picking order can be determined based on the actual fruit characteristics and the environmental characteristics around the fruit. This can not only ensure the priority picking of fruits that are close, heavy, and of good quality. Of course, since the number of fruits on the fruit tree decreases after picking, the subsequent picking pressure can also be reduced. In addition, since the heavier fruits are placed at the bottom of the storage device, the extrusion between fruits can also be reduced to ensure the fruit quality. Thus, the subsequent fruit picking efficiency can be guaranteed, and to a certain extent, the subsequent fruit quality classification efficiency can also be ensured.

[0110] 105. Obtain the third attribute information of the n picking points to obtain n pieces of third attribute information.

[0111] In the embodiments of the present application, the third attribute information may include at least one of the following: the position of the picking point, the connection relationship of the picking points, the environmental information around the picking point, etc., which is not limited herein.

[0112] In specific implementation, image recognition technology can be used to obtain the third attribute information of the n picking points to obtain n pieces of third attribute information.

[0113] 106. Determine a first control parameter according to the picking order and the n pieces of third attribute information. The first control parameter includes: the first movement parameter of the first robotic arm and the first action parameter of the tool system.

[0114] Among them, the first control parameter includes: the first movement parameter of the first robotic arm and the first action parameter of the tool system. The first movement parameter of the first robotic arm may include at least one of the following: movement speed, movement trajectory, movement action, movement direction, movement acceleration, etc., which is not limited herein. The first action parameter of the tool system may include at least one of the following: tool type, the action corresponding to the tool, the control force corresponding to the work, the picking speed corresponding to the work, the control direction corresponding to the tool, etc., which is not limited herein.

[0115] Optionally, for step 106 above, determining the first control parameter according to the picking order and the n pieces of third attribute information can be implemented in the following manner:

[0116] Determine the first movement parameter according to the picking order;

[0117] Determine the first working parameter according to the picking order, the first movement parameter, and the n pieces of third attribute information.

[0118] In a specific implementation, since the picking order is fixed, each picking point can be regarded as a position, and path planning technology can be used for planning to obtain the corresponding movement trajectory. Based on this movement trajectory, the first movement parameter can be determined. Then, according to the picking order, the first movement parameter, and the n third attribute information, the first working parameter can be determined. Not only can the corresponding first working parameter be determined based on the surrounding environment characteristics of the picking point and the corresponding connection relationship, that is, corresponding controls are adopted for different fruits to ensure the fruit picking efficiency and reduce the damage to the fruit tree.

[0119] 107. Determine a second control parameter according to the first control parameter, the picking order, and the n picking points. The second control parameter includes: the second movement parameter of the second robotic arm and the second action parameter of the storage device.

[0120] Among them, the second control parameter includes: the second movement parameter of the second robotic arm and the second action parameter of the storage device. The second movement parameter of the second robotic arm may include at least one of the following: movement speed, movement trajectory, movement action, etc., which are not limited here. The second action parameter of the storage device may include at least one of the following: the position of the storage device, the height of the storage device, the angle of the storage device, the movement speed of the storage device, the movement action of the storage device, etc., which are not limited here.

[0121] Optionally, in step 107 above, determining the second control parameter according to the first control parameter, the picking order, and the n picking points can be implemented as follows:

[0122] Determine the second movement parameter according to the picking order and the first control parameter;

[0123] Determine the second action parameter according to the picking order, the n picking points, and the second movement parameter.

[0124] In a specific implementation, since the storage device is to cooperate with the tool system, that is, the storage device is used to receive the fruits picked by the working system. That is, the second movement parameter is determined according to the picking order and the first control parameter, so as to ensure the degree of cooperation between the two robotic arms. Next, the second action parameter is determined according to the picking order, the n picking points, and the second movement parameter. In this way, the action coordination and consistency between the two robotic arms can be ensured, which helps to ensure the subsequent fruit picking efficiency and can also ensure the subsequent fruit quality classification efficiency to a certain extent.

[0125] 108. Control the first robotic arm and the tool system to pick through the first control parameter, and control the second robotic arm and the storage device to store the m to-be-collected targets through the second control parameter.

[0126] In the embodiments of the present application, the first robotic arm and the tool system can be controlled by the first control parameter for picking, and the second robotic arm and the storage device can be controlled by the second control parameter to store m target objects to be collected. In this way, the coordination and consistency of the actions between the two robotic arms can be ensured, which helps to ensure the subsequent fruit picking efficiency and can also ensure the subsequent fruit quality classification efficiency to a certain extent.

[0127] It can be seen that the intelligent picking control method based on machine vision described in the embodiments of the present application is applied to a picking robot. The picking robot includes a first robotic arm, a tool system, a second robotic arm, a mobile device, a storage device, and a vision system. The first robotic arm is connected to the tool system; the second robotic arm is used to connect the storage device, and the picking robot is controlled to move through the mobile device. The distance between the target fruit tree and the picking robot is detected through the vision system. When the distance is within a preset range, the vision system takes a picture of the first position of the target fruit tree to obtain a first image, and the first attribute information of the target fruit tree is obtained. The first image is subjected to target recognition according to the first attribute information to obtain m to-be-picked targets and m pieces of second attribute information, and each to-be-picked target corresponds to one piece of second attribute information; m is a positive integer. n picking points are determined according to the m pieces of second attribute information and the first image, where n is a positive integer less than or equal to m. The picking order of the m to-be-picked targets is determined according to the n picking points and the m pieces of second attribute information, and the third attribute information of the n picking points is obtained to get n pieces of third attribute information. A first control parameter is determined according to the picking order and the n pieces of third attribute information. The first control parameter includes: the first movement parameter of the first robotic arm and the first action parameter of the tool system. A second control parameter is determined according to the first control parameter, the picking order, and the n picking points. The second control parameter includes: the second movement parameter of the second robotic arm and the second action parameter of the storage device. The first robotic arm and the tool system are controlled to pick through the first control parameter, and the second robotic arm and the storage device are controlled to store the m to-be-picked targets through the second control parameter. First, the distance between the picking robot and the target fruit tree is appropriate, enabling efficient picking and ensuring the clarity of the fruits. Since the first attribute information reflects the characteristics of the target fruit tree, the accuracy of fruit recognition can be guaranteed. Second, to a certain extent, the second attribute information reflects the characteristics of the fruits and the characteristics of the surrounding environment of the fruits. The first image reflects the connection relationship between fruits and between fruits and their surrounding environment. That is, the corresponding picking points can be determined based on the fruit characteristics and the characteristics of the surrounding environment of the fruits, and picking through these picking points will not affect the quality of the fruits and will also minimize the damage to the fruit tree as soon as possible. Third, the corresponding picking order can be determined based on the actual fruit characteristics and the characteristics of the surrounding environment of the fruits. Thus, the subsequent fruit picking efficiency can be guaranteed, and the subsequent fruit quality classification efficiency can also be guaranteed to a certain extent. Fourth, the coordination and consistency of the actions between the two robotic arms can be guaranteed, which helps to ensure the subsequent fruit picking efficiency and can also guarantee the subsequent fruit quality classification efficiency to a certain extent, that is, the intelligence of the picking robot can be improved.

[0128] Consistent with the above embodiments, please refer to Figure 3 , Figure 3It is a schematic structural diagram of an electronic device provided by an embodiment of the present application. As shown in the figure, the electronic device includes a processor, a memory, a communication interface, and one or more programs. Among them, the above one or more programs are stored in the above memory and are configured to be executed by the above processor. In the embodiment of the present application, the electronic device includes a picking robot, or the electronic device is used to control the picking robot. The picking robot includes a first robotic arm, a tool system, a second robotic arm, a moving device, a storage device, and a vision system. The first robotic arm is connected to the tool system; the second robotic arm is used to connect the storage device; the above programs include instructions for performing the following steps:

[0129] Control the picking robot to move through the moving device, and detect the distance between the target fruit tree and the picking robot through the vision system;

[0130] When the distance is within a preset range, take a picture of the first position of the target fruit tree through the vision system to obtain a first image, obtain the first attribute information of the target fruit tree, and perform target recognition on the first image according to the first attribute information to obtain m to-be-collected targets and m pieces of second attribute information. Each to-be-collected target corresponds to one piece of second attribute information; m is a positive integer;

[0131] Determine n picking points according to the m pieces of second attribute information and the first image, where n is a positive integer less than or equal to m;

[0132] Determine the picking order of the m to-be-collected targets according to the n picking points and the m pieces of second attribute information;

[0133] Obtain the third attribute information of the n picking points to obtain n pieces of third attribute information;

[0134] Determine a first control parameter according to the picking order and the n pieces of third attribute information. The first control parameter includes: the first movement parameter of the first robotic arm and the first action parameter of the tool system;

[0135] Determine a second control parameter according to the first control parameter, the picking order, and the n picking points. The second control parameter includes: the second movement parameter of the second robotic arm and the second action parameter of the storage device;

[0136] Control the first robotic arm and the tool system to pick through the first control parameter, and control the second robotic arm and the storage device to store the m to-be-collected targets through the second control parameter.

[0137] Optionally, in the aspect of performing target recognition on the first image according to the first attribute information to obtain m targets to be collected and m pieces of second attribute information, the above program includes instructions for performing the following steps:

[0138] Determine a first image segmentation algorithm and a first target recognition algorithm corresponding to the first attribute information;

[0139] Obtain a first reference control parameter of the first image segmentation algorithm and a second reference control parameter of the first target recognition algorithm corresponding to the first attribute information;

[0140] Obtain first shooting parameters of the first image;

[0141] Determine a first control parameter according to the first shooting parameters and the first reference control parameter;

[0142] Perform target segmentation on the first image according to the first control parameter and the first image segmentation algorithm to obtain k target regions, where k is an integer greater than or equal to m;

[0143] Determine region size parameters of the k target regions to obtain k region size parameters;

[0144] Determine a second control parameter according to the k region size parameters and the second reference control parameter;

[0145] Perform recognition on the k target regions according to the second control parameter and the first target recognition algorithm to obtain the m targets to be collected.

[0146] Optionally, in the aspect of determining the first control parameter according to the first shooting parameters and the first reference control parameter, the above program includes instructions for performing the following steps:

[0147] Determine a first adjustment parameter corresponding to the first shooting parameters;

[0148] Adjust the first reference control parameter according to the first adjustment parameter to obtain the first control parameter.

[0149] Optionally, in the aspect of determining the second control parameter according to the k region size parameters and the second reference control parameter, the above program includes instructions for performing the following steps:

[0150] Determine k area values according to the k region size parameters;

[0151] Determine a first standard deviation of the k area values;

[0152] Determine k perimeters according to the k region size parameters;

[0153] Determine a second standard deviation among the k perimeters;

[0154] Determine a second adjustment parameter corresponding to the first standard deviation;

[0155] Determine a first fine-tuning parameter corresponding to the second standard deviation;

[0156] Determine the second control parameter according to the second adjustment parameter, the first fine-tuning parameter, and the second reference control parameter.

[0157] Optionally, the second attribute information includes connection relation parameters, position, shape, size, color space related parameters; in terms of determining n picking points according to the m pieces of second attribute information and the first image, the above program includes instructions for performing the following steps:

[0158] Determine a reference picking points according to the connection relation parameters and position in the m pieces of second attribute information, where a is a positive integer greater than or equal to n;

[0159] Divide the m targets to be collected into a sets of targets to be collected according to the a reference picking points;

[0160] Determine an evaluation value for each set of targets to be collected in the a sets of targets to be collected according to the m pieces of second attribute information, obtaining a evaluation values;

[0161] Select the evaluation values that meet the preset conditions from the a evaluation values, obtaining n evaluation values, and obtain the picking points corresponding to the n evaluation values, obtaining the n picking points.

[0162] Optionally, in terms of determining the picking order of the m targets to be collected according to the n picking points and the m pieces of second attribute information, the above program includes instructions for performing the following steps:

[0163] Determine the trajectory length between each picking point in the n picking points and the first robotic arm, obtaining n first trajectory lengths;

[0164] Determine the perturbation parameter of the evaluation value corresponding to each picking point in the n picking points, obtaining n perturbation parameters;

[0165] Determine n second trajectory lengths according to the n first trajectory lengths and the n perturbation parameters;

[0166] Sort according to the n second trajectory lengths to obtain the picking order.

[0167] It can be seen that in the embodiments of the present application, the electronic device includes a picking robot, or the electronic device is used to control the picking robot. First, the distance between the picking robot and the target fruit tree is appropriate, enabling efficient picking and ensuring the clarity of the fruits. Since the first attribute information reflects the characteristics of the target fruit tree, the recognition accuracy of the fruits can be guaranteed. Second, the second attribute information reflects the fruit characteristics and the surrounding environment characteristics of the fruits to a certain extent. The first image reflects the connection relationship between the fruits and between the fruits and their surrounding environment, that is, the corresponding picking points can be determined based on the fruit characteristics and the surrounding environment characteristics of the fruits, and picking through these picking points will not affect the quality of the fruits and can also minimize the damage to the fruit tree as soon as possible. Third, the corresponding picking order can be determined based on the actual fruit characteristics and the surrounding environment characteristics of the fruits. Thus, the subsequent fruit picking efficiency can be guaranteed, and the subsequent fruit quality classification efficiency can also be guaranteed to a certain extent. Fourth, the action coordination and consistency between the two robotic arms can be guaranteed, which helps to ensure the subsequent fruit picking efficiency and can also guarantee the subsequent fruit quality classification efficiency to a certain extent, that is, the intelligence of the picking robot can be improved.

[0168] Figure 4 It is a functional unit composition block diagram of an intelligent picking control system 400 based on machine vision involved in the embodiments of the present application. The intelligent picking control system 400 based on machine vision is applied to a picking robot. The picking robot includes a first robotic arm, a tool system, a second robotic arm, a mobile device, a storage device, and a vision system. The first robotic arm is connected to the tool system; the second robotic arm is used to connect the storage device; the intelligent picking control system 400 based on machine vision includes: a detection unit 401, an identification unit 402, a determination unit 403, an acquisition unit 404, and a picking unit 405, where,

[0169] The detection unit 401 is used to control the movement of the picking robot through the mobile device and detect the distance between the target fruit tree and the picking robot through the vision system;

[0170] The identification unit 402 is used to, when the distance is within a preset range, take a picture of the first position of the target fruit tree through the vision system to obtain a first image, acquire the first attribute information of the target fruit tree, and perform target recognition on the first image according to the first attribute information to obtain m to-be-collected targets and m pieces of second attribute information, and each to-be-collected target corresponds to one piece of second attribute information; m is a positive integer;

[0171] The determining unit 403 is configured to determine n picking points according to the m pieces of second attribute information and the first image, where n is a positive integer less than or equal to m; and determine the picking order of the m targets to be collected according to the n picking points and the m pieces of second attribute information.

[0172] The obtaining unit 404 is configured to obtain third attribute information of the n picking points, obtaining n pieces of third attribute information.

[0173] The determining unit 403 is further configured to determine a first control parameter according to the picking order and the n pieces of third attribute information, where the first control parameter includes: a first movement parameter of the first robotic arm and a first action parameter of the tool system; and determine a second control parameter according to the first control parameter, the picking order, and the n picking points, where the second control parameter includes: a second movement parameter of the second robotic arm and a second action parameter of the storage device.

[0174] The picking unit 405 is configured to control the first robotic arm and the tool system to perform picking through the first control parameter, and control the second robotic arm and the storage device to store the m targets to be collected through the second control parameter.

[0175] Optionally, in terms of performing target recognition on the first image according to the first attribute information to obtain m targets to be collected and m pieces of second attribute information, the recognition unit 402 is specifically configured to:

[0176] Determine a first image segmentation algorithm and a first target recognition algorithm corresponding to the first attribute information;

[0177] Obtain a first reference control parameter of the first image segmentation algorithm and a second reference control parameter of the first target recognition algorithm corresponding to the first attribute information;

[0178] Obtain first shooting parameters of the first image;

[0179] Determine a first control parameter according to the first shooting parameters and the first reference control parameter;

[0180] Perform target segmentation on the first image according to the first control parameter and the first image segmentation algorithm, obtaining k target regions, where k is an integer greater than or equal to m;

[0181] Determine region size parameters of the k target regions, obtaining k region size parameters;

[0182] Determine a second control parameter according to the k region size parameters and the second reference control parameter;

[0183] Identify the k target regions according to the second control parameter and the first target recognition algorithm to obtain the m targets to be collected.

[0184] Optionally, in terms of determining the first control parameter according to the first shooting parameter and the first reference control parameter, the recognition unit 402 is specifically configured to:

[0185] Determine a first adjustment parameter corresponding to the first shooting parameter;

[0186] Adjust the first reference control parameter according to the first adjustment parameter to obtain the first control parameter.

[0187] Optionally, in terms of determining the second control parameter according to the k region size parameters and the second reference control parameter, the recognition unit 402 is specifically configured to:

[0188] Determine k area values according to the k region size parameters;

[0189] Determine the first standard deviation of the k area values;

[0190] Determine k perimeters according to the k region size parameters;

[0191] Determine the second standard deviation between the k perimeters;

[0192] Determine a second adjustment parameter corresponding to the first standard deviation;

[0193] Determine a first fine-tuning parameter corresponding to the second standard deviation;

[0194] Determine the second control parameter according to the second adjustment parameter, the first fine-tuning parameter, and the second reference control parameter.

[0195] Optionally, the second attribute information includes connection relationship parameters, position, shape, size, color space related parameters; in terms of determining n picking points according to the m second attribute information and the first image, the determining unit 403 is specifically configured to:

[0196] Determine a reference picking points according to the connection relationship parameters and positions in the m second attribute information, where a is a positive integer greater than or equal to n;

[0197] Divide the m targets to be collected into a sets of targets to be collected according to the a reference picking points;

[0198] Determine the evaluation value of each set of targets to be collected in the a sets of targets to be collected according to the m second attribute information to obtain a evaluation values;

[0199] Select the evaluation values that meet the preset conditions from the a evaluation values to obtain n evaluation values, and obtain the picking points corresponding to the n evaluation values to obtain the n picking points.

[0200] Optionally, in terms of determining the picking order of the m to-be-collected targets according to the n picking points and the m second attribute information, the determining unit 403 is specifically configured to:

[0201] Determine the trajectory length between each picking point in the n picking points and the first robotic arm to obtain n first trajectory lengths;

[0202] Determine the perturbation parameter of the evaluation value corresponding to each picking point in the n picking points to obtain n perturbation parameters;

[0203] Determine n second trajectory lengths according to the n first trajectory lengths and the n perturbation parameters;

[0204] Sort according to the n second trajectory lengths to obtain the picking order.

[0205] It can be seen that the intelligent picking control system based on machine vision described in the embodiments of the present application is applied to a picking robot. First, the distance between the picking robot and the target fruit tree is appropriate, which can perform efficient picking and ensure the clarity of the fruit. Since the first attribute information reflects the characteristics of the target fruit tree, the recognition accuracy of the fruit can be guaranteed. Second, the second attribute information reflects the fruit characteristics and the surrounding environment characteristics of the fruit to a certain extent. The first image reflects the connection relationship between fruits and between fruits and their surrounding environment, that is, the corresponding picking points can be determined based on the fruit characteristics and the surrounding environment characteristics of the fruit. Picking through this picking point will not affect the quality of the fruit and will also minimize the damage to the fruit tree as soon as possible. Third, the corresponding picking order can be determined based on the actual fruit characteristics and the surrounding environment characteristics of the fruit. Therefore, the subsequent fruit picking efficiency can be guaranteed, and the subsequent fruit quality classification efficiency can be guaranteed to a certain extent. Fourth, the action coordination and consistency between the two robotic arms can be guaranteed, which helps to ensure the subsequent fruit picking efficiency and can also guarantee the subsequent fruit quality classification efficiency to a certain extent, that is, the intelligence of the picking robot can be improved.

[0206] The embodiments of the present application further provide a computer storage medium, wherein the computer storage medium stores a computer program for electronic data exchange, and the computer program enables a computer to execute part or all of the steps of any method recorded in the above method embodiments. The above computer includes an electronic device.

[0207] The embodiments of the present application also provide a computer program product. The computer program product includes a non-transitory computer-readable storage medium storing a computer program. The computer program is operable to cause a computer to execute some or all of the steps of any of the methods described in the foregoing method embodiments. The computer program product may be a software installation package, and the computer includes an electronic device.

[0208] It should be noted that, for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the present application is not limited by the described action sequence, because according to the present application, certain steps may be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the present application.

[0209] In the above embodiments, the descriptions of the respective embodiments have their own emphases. For the parts not detailed in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

[0210] In several embodiments provided by the present application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the above division of units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces. The indirect coupling or communication connection of the device or unit may be in an electrical or other form.

[0211] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0212] In addition, in each embodiment of the present application, the functional units can be integrated into one processing unit, or each unit exists physically alone, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0213] If the above integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the above methods in various embodiments of the present application. The aforementioned memory includes: various media such as USB flash drives, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), external hard drives, magnetic disks, or optical discs that can store program codes.

[0214] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing relevant hardware through a program. This program can be stored in a computer-readable memory, and the memory can include: flash drives, read-only memories (English: Read-Only Memory, abbreviated: ROM), random access memories (English: Random Access Memory, abbreviated: RAM), magnetic disks, or optical discs, etc.

[0215] The above has introduced the embodiments of the present application in detail. Specific examples are used in this article to elaborate on the principle and implementation manner of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application; at the same time, for those of ordinary skill in the art, according to the idea of the present application, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present application.

Claims

1. An intelligent picking control method based on machine vision, characterized in that: Applied to a picking robot, the picking robot comprises a first mechanical arm, a tool system, a second mechanical arm, a moving device, a storage device and a visual system, wherein the first mechanical arm is connected to the tool system; The second mechanical arm is used to connect to the storage device; the method comprises: Controlling the picking robot to move by the mobile device, and detecting the distance between the target fruit tree and the picking robot by the visual system; When the distance is within a preset range, the first position of the target fruit tree is photographed by the visual system to obtain a first image, first attribute information of the target fruit tree is acquired, and target recognition is performed on the first image according to the first attribute information to obtain m targets to be collected and m second attribute information; Determine n picking points according to the m second attribute information and the first image; Determining a picking order of the m objects to be collected according to the n picking points and the m second attribute information; Acquire the third attribute information of the n picking points to obtain n third attribute information; Determine a first control parameter according to the picking order and the n third attribute information; Determine a second control parameter according to the first control parameter, the picking order and the n picking points; The first control parameter is used to control the first robotic arm and the tool system to perform picking, and the second control parameter is used to control the second robotic arm and the storage device to store the m objects to be collected.

2. The method according to claim 1, characterized in that The performing target recognition on the first image according to the first attribute information to obtain m targets to be collected and m second attribute information includes: Determining a first image segmentation algorithm and a first target recognition algorithm corresponding to the first attribute information; Acquire a first reference control parameter of the first image segmentation algorithm and a second reference control parameter of the first target recognition algorithm corresponding to the first attribute information; Acquire a first shooting parameter of the first image; determining a first control parameter according to the first shooting parameter and the first reference control parameter; Performing target segmentation on the first image according to the first control parameter and the first image segmentation algorithm to obtain k target regions, where k is an integer greater than or equal to m; Determine the region size parameters of the k target regions to obtain k region size parameters; Determine a second control parameter according to the k region size parameters and the second reference control parameter; The k target areas are identified according to the second control parameter and the first target recognition algorithm to obtain the m targets to be collected.

3. The method according to claim 2, characterized in that The determining the first control parameter according to the first shooting parameter and the first reference control parameter includes: Determining a first adjustment parameter corresponding to the first shooting parameter; The first reference control parameter is adjusted according to the first adjustment parameter to obtain the first control parameter.

4. The method according to claim 3, characterized in that The determining the second control parameter according to the k region size parameters and the second reference control parameter comprises: Determine k area values ​​according to the k area size parameters; determining a first standard deviation of the k area values; Determine k perimeters according to the k area size parameters; determining a second standard deviation between the k perimeters; determining a second adjustment parameter corresponding to the first standard deviation; determining a first fine-tuning parameter corresponding to the second standard deviation; The second control parameter is determined according to the second adjustment parameter, the first fine-tuning parameter and the second reference control parameter.

5. The method according to any one of claims 1 to 4, characterized in that: The second attribute information includes connection relationship parameters, position, shape, size, and color space related parameters; and determining n picking points according to the m second attribute information and the first image includes: Determine a reference picking points according to the connection relationship parameters and positions in the m second attribute information, where a is a positive integer greater than or equal to n; Dividing the m objects to be collected into a sets of objects to be collected according to the a reference picking points; Determine the evaluation value of each of the a sets of targets to be collected according to the m second attribute information to obtain a evaluation values; Select the evaluation values ​​that meet the preset conditions from the a evaluation values ​​to obtain n evaluation values, and obtain the picking points corresponding to the n evaluation values ​​to obtain the n picking points.

6. The method according to claim 5, characterized in that The step of determining the picking order of the m objects to be collected according to the n picking points and the m second attribute information includes: Determine the trajectory length between each of the n picking points and the first robotic arm to obtain n first trajectory lengths; Determine a disturbance parameter of the evaluation value corresponding to each of the n picking points to obtain n disturbance parameters; Determining n second trajectory lengths according to the n first trajectory lengths and the n disturbance parameters; The picking order is obtained by sorting the n second trajectories according to their lengths.

7. An intelligent picking control system based on machine vision, characterized in that: Applied to a picking robot, the picking robot comprises a first mechanical arm, a tool system, a second mechanical arm, a moving device, a storage device and a visual system, wherein the first mechanical arm is connected to the tool system; The second mechanical arm is used to connect to the storage device; The intelligent picking control system based on machine vision includes: a detection unit, a recognition unit, a determination unit, an acquisition unit and a picking unit, wherein: The detection unit is used to control the movement of the picking robot through the mobile device, and detect the distance between the target fruit tree and the picking robot through the visual system; The recognition unit is used to, when the distance is within a preset range, photograph a first position of a target fruit tree through the visual system to obtain a first image, obtain first attribute information of the target fruit tree, perform target recognition on the first image according to the first attribute information, and obtain m targets to be collected and m second attribute information; The determining unit is used to determine n picking points according to the m second attribute information and the first image; and determine the picking order of the m objects to be collected according to the n picking points and the m second attribute information; The acquiring unit is used to acquire the third attribute information of the n picking points to obtain n pieces of third attribute information; The determination unit is further configured to determine a first control parameter according to the picking order and the n third attribute information; and determine a second control parameter according to the first control parameter, the picking order and the n picking points; The picking unit is used to control the first robotic arm and the tool system to perform picking through the first control parameters, and to control the second robotic arm and the storage device to store the m objects to be collected through the second control parameters.

8. The intelligent picking control system based on machine vision according to claim 7 is characterized in that: In the aspect of performing target recognition on the first image according to the first attribute information to obtain m targets to be collected and m second attribute information, the recognition unit is specifically used for: Determining a first image segmentation algorithm and a first target recognition algorithm corresponding to the first attribute information; Acquire a first reference control parameter of the first image segmentation algorithm and a second reference control parameter of the first target recognition algorithm corresponding to the first attribute information; Acquire a first shooting parameter of the first image; determining a first control parameter according to the first shooting parameter and the first reference control parameter; Performing target segmentation on the first image according to the first control parameter and the first image segmentation algorithm to obtain k target regions, where k is an integer greater than or equal to m; Determine the region size parameters of the k target regions to obtain k region size parameters; Determine a second control parameter according to the k region size parameters and the second reference control parameter; The k target areas are identified according to the second control parameter and the first target recognition algorithm to obtain the m targets to be collected.

9. The intelligent picking control system based on machine vision according to claim 8 is characterized in that: In the aspect of determining the first control parameter according to the first shooting parameter and the first reference control parameter, the identification unit is specifically used for: Determining a first adjustment parameter corresponding to the first shooting parameter; The first reference control parameter is adjusted according to the first adjustment parameter to obtain the first control parameter.

10. The intelligent picking control system based on machine vision according to claim 9 is characterized in that: In the aspect of determining the second control parameter according to the k region size parameters and the second reference control parameter, the identification unit is specifically used for: Determine k area values ​​according to the k area size parameters; determining a first standard deviation of the k area values; Determine k perimeters according to the k area size parameters; determining a second standard deviation between the k perimeters; determining a second adjustment parameter corresponding to the first standard deviation; determining a first fine-tuning parameter corresponding to the second standard deviation; The second control parameter is determined according to the second adjustment parameter, the first fine-tuning parameter and the second reference control parameter.

Images