Intelligent picking method, agricultural picking robot, chip system and medium

Abstract

The invention provides an intelligent picking method, an agricultural picking robot, a chip system and a medium, and the method comprises the steps: collecting a first image, and processing the first image to obtain a second image, so as to recognize a first region which is an approximate region where a target fruit is located; a first region may be determined through a feature intensity relationship between pixel points. And then determining a target detection area based on the first area, and transmitting laser pulses to the target detection area through a laser transmitter to obtain measurement information of a plurality of pixel points corresponding to the target detection area. And finally, determining an upper boundary point of a target fruit on the target fruit tree from the plurality of pixel points, and determining a picking point according to the upper boundary point. And picking the target fruits based on the picking points. According to the scheme, the fruit picking precision and efficiency can be improved.

Description

Technical Field

[0001] The present application belongs to the field of smart agriculture, and in particular relates to an intelligent picking method, an agricultural picking robot, a chip system and a medium. Background Art

[0002] With the development of agricultural modernization, traditional crop picking methods can no longer meet the growing demand for agricultural products. For example, more and more plantations have begun to plant apple trees on a large scale. When the fruits are ripe, in order to pick the ripe apples as quickly as possible within the optimal picking window, a large number of pickers are usually required, which is not only inefficient but also has high labor costs.

[0003] In view of this, how to achieve fully automatic intelligent harvesting of crops is a problem that needs to be solved at present. Summary of the invention

[0004] The embodiments of the present application provide an intelligent picking method, an agricultural picking robot, a chip system and a medium, which can improve the accuracy and efficiency of fruit picking.

[0005] In a first aspect, an intelligent picking method is provided, which is applied to an agricultural picking robot, and the method comprises:

[0006] Acquire a first image, wherein the image content of the first image includes a target fruit tree, and the target fruit tree includes target fruits;

[0007] The first image is processed based on a first neural network model to obtain a second image, wherein the first neural network model is trained based on multiple training data sets, each training data set includes multiple images augmented from an original image, the original image includes fruits of the same type as the target fruit, and the first neural network model is trained for the purpose of reducing a first loss function value, a second loss function value, and a third loss function value, wherein the first loss function value is used to characterize the feature difference between the input image and the output image of the first neural network model, the second loss function value is a loss function value against a fourth loss function value, the fourth loss function value is used to characterize the feature difference between multiple output images corresponding to each training data set, and the third loss function value is used to characterize the difference between a fruit recognition result and a target recognition result for the training data set;

[0008] Identify a first area in the second image, the second image is composed of the first area and the second area, the first image is composed of the third area and the fourth area, the first area corresponds to the third area, the second area corresponds to the fourth area, the characteristic intensity deviations of the corresponding pixels of the first area and the third area are both less than or equal to a first threshold, and the characteristic intensity deviations of the corresponding pixels of the second area and the fourth area are both greater than the first threshold;

[0009] Determine a target detection area based on the first area, and emit laser pulses to the target detection area through a laser transmitter;

[0010] The laser receiving array receives the echo signal corresponding to the laser pulse to obtain the measurement information of multiple pixel points corresponding to the target detection area, wherein the number of receiving units in the laser receiving array is less than or equal to a preset threshold;

[0011] Determine an upper boundary point of the target fruit on the target fruit tree from the multiple pixel points, and determine a picking point according to the upper boundary point;

[0012] The target fruit is picked based on the picking point.

[0013] Optionally, determining the upper boundary point of the target fruit on the target fruit tree from the multiple pixel points includes:

[0014] Detecting each pixel point in the candidate boundary points, where the candidate boundary points are part or all of the multiple boundary points;

[0015] When the i-th pixel point among the candidate boundary points meets the preset conditions, the i-th pixel point is determined to be the upper boundary point of the target fruit, and the preset conditions include: the difference between the measured distance corresponding to the x pixels above the i-th pixel point and the measured distance corresponding to the i-th pixel point is greater than a first threshold, and the difference between the measured distance corresponding to the y pixels below the i-th pixel point and the measured distance corresponding to the i-th pixel point is less than a second threshold, and x, y, and i are all positive integers, and the values ​​of x and y are determined based on the type of the target fruit tree.

[0016] Optionally, the method further comprises:

[0017] P rows of pixel points among the multiple pixel points are selected as the candidate boundary points, and the measured distances corresponding to more than m pixel points in the p rows of pixel points are greater than the third threshold, and the measured distances corresponding to more than n pixel points are less than the fourth threshold.

[0018] Optionally, the number of receiving units in the receiving array is determined based on the type of the target fruit tree and the model of the agricultural harvesting robot.

[0019] Optionally, different receiving units in the receiving array correspond to different measurement ranges, and the measurement range corresponding to each receiving unit is determined based on a historical maximum measurement range of the receiving unit.

[0020] Optionally, channels in the first neural network model whose dependence coefficients are less than or equal to a fifth threshold are deleted in advance, and parameters of the first neural network model are adjusted in advance based on a loss amount, wherein the loss amount is used to characterize the degree of structural change of the first neural network model, and the dependence coefficient is used to characterize the dependency relationship between the channel and the model.

[0021] In a second aspect, a terminal device is provided, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the computer program, the terminal device implements the steps of the method described in any one of the above-mentioned first aspects.

[0022] In a third aspect, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the steps of the method described in any one of the first aspects are implemented.

[0023] In a fourth aspect, a computer program product is provided. When the computer program product is executed on a terminal device, the terminal device executes any one of the methods described in the first aspect.

[0024] In a fifth aspect, a chip system is provided, which includes a processor coupled to a memory, and the processor executes a computer program stored in the memory to implement the method described in any one of the first aspects above.

[0025] The chip system may be a single chip or a chip module composed of multiple chips.

[0026] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 An exemplary flow chart of an intelligent picking method for fruit trees provided in an embodiment of the present application;

[0028] Figure 2 A schematic diagram of a training method for the first neural network model;

[0029] Figure 3 is an example of the first image in this application;

[0030] Figure 4 is an example diagram of the second image in this application;

[0031] Figure 5 A schematic diagram of a first area and a second area of ​​a second image in the present application;

[0032] Figure 6is a schematic diagram of the third area and the fourth area of ​​the first image in this application;

[0033] Figure 7 is an exemplary structure of a laser radar;

[0034] Figure 8 A schematic diagram of the target fruit;

[0035] Fig. 9 A schematic diagram of the laser radar detection results. DETAILED DESCRIPTION

[0036] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0037] With the development of agricultural modernization, traditional crop picking methods can no longer meet the growing demand for agricultural products. For example, more and more plantations have begun to plant apple trees on a large scale. When the fruits are ripe, in order to pick the ripe apples as quickly as possible within the optimal picking window, a large number of pickers are usually required, which is not only inefficient but also has high labor costs.

[0038] Therefore, many researchers have begun to study agricultural harvesting robots that can achieve intelligent harvesting. However, there are still many problems with current agricultural harvesting robots. For example, the accuracy of current harvesting robots in identifying fruits is not high enough, and when picking with mechanical arms, the fruits are often damaged, affecting the value of the fruits. For another example, the accuracy of current harvesting robots in identifying fruits is affected by many factors, such as changes in light, which may also greatly reduce the recognition accuracy.

[0039] In view of this, the embodiment of the present application provides an agricultural picking robot that can accurately identify target fruits (such as apples, pears, peaches, etc.). Specifically, in the scheme provided by the present application, after processing the captured image by a pre-trained artificial intelligence model, the approximate area where the target fruit is located is determined, and then the area is scanned by a laser radar. The position of the target fruit is identified by the point cloud data obtained by the laser radar scanning, which is not only highly accurate, but also not affected by environmental factors. In addition, the present application also creatively proposes a method of determining the picking point based on the upper boundary point of the target fruit in combination with the characteristics of the target fruit, because the fruit stalk of the target fruit is usually located on its upper boundary, and picking by grabbing the fruit stalk can avoid damaging the surface of the fruit, thereby improving the commercial value of the fruit.

[0040] Combine the following Figure 1The method 100 in the embodiment of the present application exemplifies the intelligent picking method for fruit trees provided in the embodiment of the present application. The method 100 can be executed by an agricultural picking robot, or by an internal module or component of the agricultural picking robot, which is not limited in the present application. The agricultural picking robot execution method 100 is used as an example for explanation below. The agricultural picking robot described in the embodiment of the present application is a fully automatic or semi-automatic robot that can intelligently pick fruits from fruit trees. The agricultural picking robot can be a special robot, that is, it is specifically used for picking, or the agricultural picking robot can also be a multifunctional robot with other functions besides the picking function, which is not limited in the present application. The present application does not limit the type of fruit trees to which the agricultural picking robot is applicable. Theoretically, the agricultural picking robot can be used to pick fruits of various types of fruit trees, such as apples, pears, lychees, etc. Even, the agricultural picking robot can also be used to pick fruits of other types of crops, such as cucumbers, tomatoes, etc.

[0041] S110: Acquire a first image.

[0042] Exemplarily, the agricultural harvesting robot in the present application includes a visual recognition system, which may specifically include one or more cameras. These cameras may be RGB cameras, ToF cameras, hyperspectral cameras, or a combination of different types of cameras, which is not limited in the present application.

[0043] The agricultural picking robot can control the camera through the control system to shoot the orchard to obtain a first image. The image content of the first image includes the target fruit tree, and the target fruit tree includes one or more target fruits. In other words, the first image is an image obtained by shooting the target fruit tree.

[0044] In the embodiments of the present application, "target fruit tree" is used to refer to the fruit tree to be picked. "Target fruit" is used to refer to the fruit to be picked on the target fruit tree. The target fruit tree in the present application can be an apple tree, a pear tree, a lychee tree, etc., and the corresponding target fruit can be apples, pears, and lychees.

[0045] It should be understood that the target fruit tree in the first image is used to refer to one of the fruit trees in the orchard.

[0046] It should also be understood that the first image in the present application may be one or more images.

[0047] Figure 3 An exemplary first image is shown. Figure 3 It can be seen from the figure that the first image includes the target fruit tree and the target fruit hanging on the branches of the target fruit tree. In addition to the target fruit tree and the target fruit, the first image may also include other interference features such as weeds and leaves.

[0048] It is understandable that the purpose of capturing the first image in this application is not to identify the position of the target fruit based on the first image. Fruit recognition in the prior art is basically based on images, but the recognition effect is not good, mainly because there is a visual similarity between leaves and fruits, and lighting effects, image clarity, dew, etc. can also form visual interference. Therefore, fruit recognition based on images often results in recognition errors or missed recognition. Even if the position of the fruit is identified based on the captured image, the recognition accuracy problem may cause damage to the fruit skin during the subsequent picking operation, thereby shortening the shelf life of the fruit and affecting the commercial value of the fruit.

[0049] Based on the above analysis, this application does not use visual images to identify the position of the target fruit, but uses laser radar to identify the position of the target fruit (the specific process can be referred to the description of the subsequent S140~S160 part, which will not be repeated here). However, it is actually quite difficult to identify the position of the fruit through the point cloud data obtained by laser radar scanning, and it will also consume more energy. Therefore, there is no related technology in the prior art to apply laser radar to the scene of fruit recognition. The main reason is that laser radar can obtain the measurement distance, reflectivity and other information corresponding to each pixel point in the target detection area through scanning of the laser array. If a large area is scanned, the scanning result will be very complicated, because the leaves, trunks, fruits, etc. of the fruit trees are intertwined, and the measurement distances corresponding to different positions are alternating and staggered. If you want to detect the position of relatively small fruits in such a result, you need to use a very complex algorithm, and the recognition accuracy is difficult to guarantee.

[0050] In view of the above analysis, the embodiment of the present application adopts a combination of visual perception and laser radar to detect the target fruit. The processing idea of ​​the present application is: first identify the approximate location of the fruit through a visual solution, and then let the laser radar emit laser pulses purposefully, that is, only scan the fruit and a small area around the fruit, and then identify the location of the target fruit from this scanning result. It will be much easier. Therefore, the main invention of the present application solution mainly includes two parts of the solution. The first solution is used to determine how to detect the approximate area where the target fruit is located from the image. The present application refers to the approximate area where the target fruit is located as the target detection area. The size of the target detection area should be less than the preset value, because if the target detection area is too large, it will be very difficult to use the laser radar for detection later; and accordingly, the smaller the target detection area, the higher the accuracy of subsequent fruit recognition, but the difficulty of determining the target detection area will increase; the second solution is used to determine how to use the laser radar to accurately identify the picking point from the target detection area. It should be understood that identifying the picking point is not as simple as identifying the position of the target fruit. A large number of cases in reality have proved that the picking method of identifying the position of the fruit and grabbing the target fruit with a robotic arm can easily cause damage to the surface peel. Therefore, the embodiment of the present application hopes to accurately identify the fruit stalk of the target fruit and use the fruit stalk as the picking point to complete the picking of the target fruit with minimal damage. The following introduces a possible implementation method corresponding to scheme one in combination with S120~S130; and introduces a possible implementation method corresponding to scheme two in combination with S140~S160.

[0051] S120. Process the first image based on the first neural network model to obtain a second image.

[0052] Exemplarily, after the first image is captured, the first image is not directly used for fruit detection, but the first image is first processed using the first neural network model to obtain the second image. The first neural network model is used to enhance the first image so that the features related to the fruit in the first image are enhanced, while the features unrelated to the fruit (such as leaf features, trunk features, light features, weed features, etc.) are weakened, which makes it easier to locate the specific position of the target fruit. Because the first neural network model can weaken the features unrelated to the fruit, these unrelated features will interfere with the recognition accuracy when identifying the fruit area. For example, leaves and trunks related to the fruit will interfere with the recognition accuracy, and the light scattered on the crops at different times will also produce some shadows or highlights, which will also interfere with the recognition accuracy. The first neural network model involved in the present application can weaken the influence of these features so that the area where the target fruit is located can be located more accurately.

[0053] That is to say, the second image is obtained by enhancing the first image, so the feature intensity of the fruit-related features in the second image is higher than that of the fruit-related features in the first image, and the feature intensity of the non-fruit-related features in the second image is lower than that of the non-fruit-related features in the first image.

[0054] The first neural network model in this application is a fruit feature enhancement model creatively proposed in combination with the fruit recognition scenario. Figure 2 An example of a training method for the first neural network model is given.

[0055] The first neural network model is trained based on multiple training data sets. It can be understood that each training data set is obtained by augmenting an original image, and each training data set includes multiple images. The original image contains the target fruit tree and the target type of fruit. The type of crops in the image is the same as the type of crops to be detected, and the target type is also the same as the type of fruit to be detected. For example, if we want to detect apples on apple trees, then we need to prepare several apple tree images, which include several apples, and then augment these apple tree images to obtain more images. The augmentation here refers to the image obtained after some processing operations are performed on the original image, such as rotating the rotated image, adding shadows, adding light at different angles, scaling, adding occlusion, applying blur effects, etc. In other words, for each image in the training set, they all come from the same original image, or are augmented from the same original image.

[0056] It should be understood that establishing different training data sets instead of mixing all images together as a large data set is to facilitate the subsequent characterization of the differences between images obtained by augmenting the same image. For details, please refer to the subsequent description, which will not be repeated here.

[0057] In the process of executing model training, the images in the training data set are input as input images into the first neural network model. The first neural network model extracts feature vectors, which are used to describe the characteristics of different contents in the image. The extracted features are then processed and the processed images are finally output.

[0058] The training process for the first neural network model is essentially a process of training and adjusting the parameters of the first neural network model. The following is an exemplary description of a training method for model parameters:

[0059] The first neural network model is trained for the purpose of reducing the first loss function value, the second loss function value, and the third loss function value.

[0060] Among them, the first loss function value is used to characterize the feature difference between the input image and the output image of the first neural network model, specifically, the difference between the input image of the first neural network model and the output image corresponding to the input image. The smaller the first loss function value, the smaller the difference between the input image and the output image. In other words, during the training of the model parameters, we hope that the change of the image is as small as possible, that is, try to keep it as original as possible. But this does not meet our expectations, because we hope that the fruit-related features in the image remain the same, while other features are weakened or even eliminated. In view of this, we introduced the third loss function value and the fourth loss function value. We hope that these two loss function values ​​can make the features other than the fruit in the image change significantly, and the change results significantly improve the accuracy of fruit recognition. The implementation process is explained below.

[0061] Among them, the fourth loss function value is used to characterize the feature differences between multiple output images corresponding to each training data set. The smaller the fourth loss function value is, for a training data set, the smaller the difference between different output images corresponding to different input images in the training data set is; on the contrary, the larger the fourth loss function value is, for a training data set, the greater the difference between different output images corresponding to different input images in the training data set is, and accordingly, the smaller the adversarial loss function value corresponding to the fourth loss function value is. The present application uses the second loss function value to characterize the adversarial loss function of the fourth loss function value.

[0062] It should be understood that the fourth loss function value in the present application represents the difference between multiple output images corresponding to a training data set. Therefore, each data set will theoretically have a corresponding fourth loss function value, and accordingly, there will also be a corresponding second loss function value. The reason for doing this is that for a training data set, the images in this training data set are all obtained by augmenting an original image, so for the images in a data set, the postures of the fruits therein are basically the same. And for a training image, most of the content is non-fruit features, because the volume of the fruit is usually small, for a training data set, amplifying the difference between the output images is more about amplifying non-fruit features, because non-fruit features are the vast majority of features in the image, and if these features do not change, the requirement of reducing the second loss function value cannot be met. Of course, if only the second loss function value is reduced, the fruit features in the image will also change greatly, so it is also necessary to combine the first loss function value and the third loss function value.

[0063] The third loss function value is used to characterize the difference between the fruit recognition result and the target recognition result for the training data set. Therefore, the smaller the third loss function value is, the higher the recognition accuracy is, wherein the fruit recognition result here refers to the actual recognition result after the input image in the training data set is subjected to feature extraction and the fruit recognition is performed using the second neural network model, the second neural network model can be any existing fruit recognition model, and the target recognition result refers to the actual fruit recognition result corresponding to the input image, which can be obtained by manual calibration.

[0064] Thus, the second loss function value forms a confrontation with the first loss function value and the third loss function value. This confrontation makes the second loss function value lowered so that when the image changes, it does not change without rules, but changes the non-fruit features, and the change results can improve the accuracy of the fruit recognition results. Therefore, this change gradually weakens or even eliminates the non-fruit features in the image, and the remaining features (i.e., fruit features) are naturally retained or even enhanced. After such a processing process, the third image obtained after being processed by the first neural network model will become an image that is more conducive to fruit recognition, thereby improving the fruit recognition results.

[0065] It is understandable that the first image may include only one image or multiple images (such as m images, where m is an integer greater than 1), which is not limited in this application. If the first image includes multiple images, step S120 may be performed on the multiple images in sequence, and then the multiple second images obtained may be fused to improve the effect of the final enhanced image. Alternatively, the image with the best effect (such as the image with the most uniform light and the highest clarity) may be selected from the multiple images to perform step S120, and the other images may be discarded. Alternatively, before executing step S120, the multiple images may be fused, and step S120 may be performed on the fused image. It is understandable that performing fusion processing of multiple images may refer to fusing image features to achieve the purpose of feature equalization and improve the effect of subsequent image processing.

[0066] Optionally, the size of the first neural network model can be reduced by channel pruning to increase the lightweight degree of the picking robot and improve the efficiency of model processing. For example, the channels in the first neural network model whose dependency coefficient is less than or equal to the fifth threshold are deleted in advance, and the parameters of the first neural network model are adjusted in advance based on the loss amount, the loss amount is used to characterize the degree of structural change of the first neural network model, and the dependency coefficient is used to characterize the dependency relationship between the channel and the model.

[0067] S130: Identify a first area in the second image.

[0068] For example, after the processing flow of S120, a second image is finally obtained, in which most non-fruit features have been weakened or even filtered, while the fruit features have been basically retained or even strengthened. Figure 3 and Figure 4 For example: Assume Figure 3 is the first image, wherein the first image includes the target fruit tree and the target fruit. Figure 3 In the example shown, the first image includes the complete target fruit, but in actual application scenarios, the first image may only include a partial area of ​​the target fruit tree, and this application does not limit this. Figure 3 In the example shown, the first image includes one target fruit. However, in actual application scenarios, the first image may also include two or more target fruits, which is not limited in this application. In addition, in addition to the target fruit tree and the target fruit, the first image may also include other elements, such as weeds, soil, light spots, etc. ( Figure 3 Since the purpose of this application is to pick the target fruit, all features except the target fruit in the first image are interference features, that is, they may cause interference when identifying the position of the target fruit. By processing the first image with the first neural network model, these interference features can be weakened or even eliminated. The specific effects are as follows: Figure 4 As shown. It is understandable that Figure 4 yes Figure 3 The corresponding effect diagram after processing, but Figure 4 The effect diagram is only a schematic diagram, which is a relatively ideal state. It is mainly used to understand the processing effect of the first neural network model. The actual processing effect of the first neural network model may be affected by many factors.

[0069] Since the interference features in the second image are weakened, the position of the target fruit can be identified using the second image to improve the recognition effect. In the embodiment of the present application, the first area is used to characterize the approximate area where the target fruit is located. Among them, the first area can be a regular shape or an irregular shape, which is not limited in the present application. However, it should be understood that the first area at least includes the complete target fruit.

[0070] In the embodiment of the present application, it is not necessary to use a special neural network model for fruit recognition to process the second image to determine the first area, but the first area is determined directly based on the first image and the second image. Because the solution in the present application does not intend to directly use images and artificial intelligence models to determine the location of the target fruit and further pick the fruit, because fruit picking must be able to very accurately identify the picking point, preferably on the fruit stalk, otherwise the target fruit may be damaged during picking. Using images and artificial intelligence models to identify the picking point, the recognition accuracy is very easily affected by environmental factors, and the power consumption will also be relatively high.

[0071] The following is an exemplary description of a possible implementation method: the first area is determined based on the first image and the second image, wherein the second image is composed of the first area and the second area, the first image is composed of the third area and the fourth area, the first area corresponds to the third area, the second area corresponds to the fourth area, the characteristic intensity deviations of the corresponding pixels of the first area and the third area are both less than or equal to the first threshold, and the characteristic intensity deviations of the corresponding pixels of the second area and the fourth area are both greater than the first threshold. The following is a detailed explanation of the scheme.

[0072] Since the volume of the fruit is relatively small, the area occupied by the entire image will also be relatively small, that is, there will be a lot of interference information in the second image. In view of this, we can find a way to crop the second image, and the cropped image retains the image content corresponding to the fruit, and the size of the image obtained after cropping is much smaller than the size of the second image, so that the laser radar can be used for accurate scanning. But the key to the problem is how to retain the fruit on the cropped image (i.e., the first area mentioned above). For this reason, we think of the second image obtained after the first neural network model processes the first image in step S120, and the first neural network model can retain or even enhance the feature intensity of the features related to the fruit in the image, while other features are weakened, so the first area mentioned above can be determined based on this characteristic. For example, the first image and the second image are compared. Since the second image is obtained by processing the first image, each pixel in the second image actually corresponds to each pixel of the first image one by one, except that the pixels in the second image are obtained after processing the pixels in the first image. By comparing each pixel in turn, it can be found that the variation range of the feature intensity of different pixels is different. The reason is that the first neural network model can weaken non-fruit features and retain or even strengthen fruit features. Therefore, a numerical value can be used to measure the change in the feature intensity of corresponding pixels in the first image and the second image. For convenience, this value is called the feature change. If the feature change of a pixel in the second image exceeds a preset threshold (such as greater than or equal to the first threshold), it is divided into the second area. If the feature change of a pixel is less than the preset threshold, it is divided into the first area. The second image consists of the first area and the second area. In other words, we divide the second image into two parts based on the feature change. One part (the first area) includes pixels with relatively small feature changes, and the other part (the second area) includes pixels with relatively large feature changes. From the characteristics of the first neural network model, it can be seen that the image content corresponding to the fruit is most likely included in the first area. In this way, the second area can be discarded to reduce information interference. Figure 5 and Figure 6 For example ( Figure 5 and Figure 6 The examples in Figure 4 and Figure 3 ) Figure 5 A distribution diagram of the first area and the second area in the second image is shown, wherein the first area covers the target fruit; Figure 6 A distribution diagram of the third area and the fourth area in the first image is shown. In addition, the position of the third area corresponds to the position of the first area, and the position of the fourth area corresponds to the position of the second area.

[0073] It is understandable that since the first neural network model is only used for feature processing, it is difficult to directly use the above method to accurately determine the boundary of the target crop, so we hope that the first area can at least cover the target fruit, even if it includes some interference features. In specific implementation, the first threshold can be set slightly larger so that the first area is large enough to cover the target fruit.

[0074] S140: Determine a target detection area based on the first area, and emit laser pulses to the target detection area through a laser emitter.

[0075] S150, receiving an echo signal corresponding to the laser pulse through a laser receiving array to obtain measurement information of a plurality of pixel points corresponding to the target detection area.

[0076] Exemplarily, the main purpose of the above steps S110 to S130 is to determine the approximate area where the target fruit is located, which is represented by the first area. The size of the first area is usually smaller than a preset value (the size of the first area can be controlled by setting the size of the first threshold), so the laser radar can be used to accurately scan the first area without the need for a large-scale scan of the entire area within the visible range. For example, after determining the first area, the three-dimensional space corresponding to the first area is used as the target detection area of ​​the laser radar, and then a laser pulse is emitted to the target detection area by the laser transmitter of the laser radar.

[0077] In order to facilitate the understanding of the solution provided by this application, the working principle of LiDAR is briefly introduced below: LiDAR is an active remote sensing device that uses a laser as a transmitting light source and adopts photoelectric detection technology. It can be used to detect the position, speed and other characteristic quantities of each object in the target scene. Its working principle is to transmit laser pulses (detection signals) to the target detection object in the target scene, and compare the received echo signal reflected from the target detection object with the transmitted laser pulse, output the corresponding electrical signal, and then the signal processing unit processes the electrical signal appropriately to form a point cloud. By processing the point cloud, the distance, direction, height, speed, posture and shape and other parameters of the target object can be obtained, thereby realizing the laser detection function, and then it can be applied to navigation avoidance, obstacle recognition, ranging, speed measurement, automatic driving and other scenarios of products such as automobiles, robots, logistics vehicles and inspection vehicles. The following is combined with Figure 7 The following is an example of a structure of a laser radar. Figure 7 As shown, the laser radar includes a transmitting array (i.e., an array transmitting device) and a receiving array (i.e., an array receiving device). The transmitting array is composed of multiple transmitting units ( Figure 7 Each small square in the transmitting array represents a transmitting unit), and the receiving array consists of multiple receiving units ( Figure 7Each small square in the receiving array represents a receiving unit). In one scenario, the number of transmitting units can be set to be the same as the number of receiving units, that is, one transmitting unit corresponds to one receiving unit. In another scenario, it can also be designed that one transmitting unit corresponds to multiple receiving units, or multiple transmitting units correspond to one receiving unit. It can be understood that Figure 1 This is just a schematic diagram of a correspondence between the transmitting unit and the receiving unit of the laser radar. The specific correspondence between the transmitting and receiving units in this application is not limited to Figure 7 In this way, the laser radar in the present application can optionally have a rotating component, such as a rotating drive platform, based on such a transmitting and receiving channel; or a scanning component, such as a rotating mirror, a galvanometer, or a combination of the two. It is understood that the present application does not limit the type and number of scanning components. It is understood that the present application does not limit the specific architecture of the laser radar system.

[0078] Based on the above introduction, the laser radar can transmit laser pulses to the target detection area through the transmitting array, and then the corresponding echo signal is received by the receiving array. Since the target detection area is determined by the first area, and the first area includes the complete target fruit, the point cloud data generated according to the above echo signal includes the point cloud data of the target fruit.

[0079] It is understandable that the size of the first area is smaller than the preset value, that is, in order to achieve accurate measurement, the area of ​​the target detection area is generally not very large, therefore, the number of receiving units in the laser receiving array of the laser radar can be set to be less than or equal to the preset threshold. It is understandable that the more receiving units there are, the more echo signals can be received, the larger the measurement range, and the higher the measurement accuracy, but correspondingly, the power consumption will be higher and the data processing time will be longer. In view of the fact that the present application has limited the target detection area to a relatively small size through the aforementioned steps, there is no need to use a high-precision and high-energy consumption laser radar. Therefore, a laser radar with fewer receiving units than the preset threshold can be used to implement the measurement work required by this application, which can not only meet the measurement requirements, but also maintain relatively low power consumption, and improve measurement efficiency.

[0080] In a possible implementation, the specific number of receiving units in the receiving array of the laser radar can be determined based on the type of the target fruit tree and the model of the agricultural harvesting robot. Because different fruit trees have different shapes, the corresponding target fruits have different sizes, shapes and colors, and the first neural network model has different processing effects on the first image in different scenarios. Therefore, the size of the first area finally determined in different application scenarios is different, so a unified standard cannot be used to determine the number of receiving units in the laser radar, but it can be determined according to the actual scenario.

[0081] When determining the number of receiving units, the influence of the model of the agricultural harvesting robot can also be considered. Because different models of agricultural harvesting robots have different detection ranges, mechanical arm lengths, and laser radar installation locations, these factors will affect the size of the first area measured, and also affect the specific measurement method. When determining the number of receiving units, considering the model of the agricultural harvesting robot at the same time can meet the detection needs of the current scene while minimizing the number of receiving units.

[0082] Optionally, in a possible implementation, different measurement ranges can be configured for the receiving units in the receiving array. It is understandable that, for a receiving unit in the receiving array, its measurement range refers to the laser detection range, which is determined by the transmission power of the transmitting unit corresponding to the receiving unit. When other conditions remain unchanged, the greater the transmission power, the larger the measurement range, and the smaller the transmission power, the smaller the measurement range. However, increasing the transmission power consumes more resources. In the scenario corresponding to the embodiment of the present application, the laser radar is used to measure the target detection area, and the positions corresponding to different receiving units are also different. Since the first area is larger than the target fruit, for the receiving units in the receiving array, some of the echo signals received by the receiving units are reflected back by the target fruit, some are reflected back by leaves or trunks, and some are emitted back by objects behind the target fruit tree. In this application, the agricultural picking robot usually detects at a preset distance from the target fruit tree. Therefore, when detecting different fruit trees, the measurement ranges corresponding to the different receiving units are actually within a certain range. For example, the measurement ranges corresponding to the receiving units in the middle position are relatively small, because most of them are reflected back by the target fruit, while the measurement ranges corresponding to the receiving units at the edge positions are relatively large, because most of them are reflected back by other objects behind the target fruit tree. If all receiving units (and their corresponding transmitting units) use a relatively large measurement range, this can certainly cope well with the measurement work described in this application, but it will also lead to a waste of detection resources. Therefore, we can set the current measurement range of a receiving unit based on its historical maximum measurement range. In this way, the waste of detection resources can be reduced.

[0083] S160, determining the upper boundary point of the target fruit on the target fruit tree from multiple pixel points, and determining the picking point according to the upper boundary point.

[0084] For example, the measurement information of multiple pixel points corresponding to the target detection area, that is, the point cloud data corresponding to the target detection area, can be obtained through step S150. Then, the picking point needs to be determined according to the measurement information.

[0085] In the embodiment of the present application, the fruit stalk of the target fruit is used as the picking point, and picking by the fruit stalk can minimize the damage to the fruit caused by picking.

[0086] In a possible implementation, the upper boundary point is determined from the measured multiple pixel points, and then the picking point is determined based on the upper boundary point. It can be understood that the receiving array includes multiple receiving units, each receiving unit corresponds to a pixel point measurement information, and the measurement information at least measures distance, reflectivity and other information. These multiple pixel points are obtained by the laser radar through repeated retracement measurement, so these multiple pixel points will also form an array. Assume that the array corresponding to the multiple pixel points is as follows Fig. 9 As shown in the squares in .

[0087] For the convenience of explanation, Fig. 9 In the figure, the pixel points corresponding to the boundary of the target fruit are filled with shadows, that is, these pixel points are the boundary points corresponding to the target fruit, including the upper boundary points.

[0088] Since the fruit stalk is connected to the upper boundary point of the fruit body, accurately identifying the position of the upper boundary point is the key to identifying the fruit stalk. A possible implementation method is exemplified below.

[0089] First, determine the candidate boundary points, which include the upper boundary points, and the candidate boundary points are part of all the pixels. The reason for determining the candidate boundary points is that when detecting the upper boundary points, one pixel needs to be detected one pixel at a time. If all the pixels are detected once, it will be too time-consuming and power consumption will be relatively high. Therefore, we hope to select some pixels (i.e., candidate boundary points) first, and then determine the upper boundary points from these pixels, so as to reduce the number of detections and reduce power consumption.

[0090] In a possible implementation, p rows of pixels among the multiple pixels are selected as the candidate boundary points, and the measured distances corresponding to more than m pixels in the p rows are greater than the third threshold, and the measured distances corresponding to more than n pixels are less than the fourth threshold. It should be understood that the measured distance corresponding to the target fruit must be smaller than the measured distances corresponding to other surrounding areas, that is, the measured distance corresponding to the upper boundary point is smaller than the measured distance corresponding to the area above the upper boundary. If the measured distances corresponding to more than m pixels in the p rows of pixels are greater than the third threshold, it means that the pixels corresponding to more than m are not the target fruit, and the measured distances corresponding to more than n pixels are less than the fourth threshold, it means that the pixels corresponding to more than n are the target fruit. As long as the values ​​of m and n are set, it can be determined that the upper boundary point is included in the p rows of pixels. In this way, the candidate boundary points can be easily determined.

[0091] After determining the candidate boundary points, each pixel point in the candidate boundary points is detected. Take the i-th pixel point as an example: when the i-th pixel point in the candidate boundary points meets the preset conditions, the i-th pixel point is determined to be the upper boundary point of the target fruit, and the preset conditions include: the difference between the measured distance corresponding to the x pixels above the i-th pixel point and the measured distance corresponding to the i-th pixel point is greater than the first threshold, and the difference between the measured distance corresponding to the y pixels below the i-th pixel point and the measured distance corresponding to the i-th pixel point is less than the second threshold, and x, y, and i are all positive integers, and the values ​​of x and y are determined based on the type of the target fruit tree. From the previous analysis, it can be seen that the measured distance corresponding to the upper boundary point is less than the measured distance corresponding to the area above the upper boundary. Therefore, by analyzing whether the measured distances of the corresponding pixels above and below a pixel point meet the threshold conditions, the upper boundary point can be determined.

[0092] After determining the upper boundary point, the position of the picking point can be directly estimated based on the upper boundary point. The pixel points above the boundary point can also be analyzed one by one to determine the specific location of the picking point. For example, the location of the picking point can be determined by analyzing the measured distance. This will not be explained in detail here.

[0093] S170: Pick the target fruit based on the picking point.

[0094] After the picking point is determined, the target fruit is picked based on the picking point. The specific picking method is not limited here, which depends on the type of mechanical arm of the picking robot. It can be picked by gripping with a claw-type arm, or by pruning the fruit stalk with a scissor-type arm, so that the target fruit falls to the soft mat or net bag laid on the ground. It will not be described in detail here.

[0095] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0096] An embodiment of the present application provides a computer program product. When the computer program product runs on a device, the device can implement the steps in the above-mentioned method embodiments when the device is executed.

[0097] The embodiment of the present application provides a chip, the chip is used to execute instructions, when the chip is running, the technical solution in the above embodiment is executed. The implementation principle and technical effect are similar, and will not be repeated here.

[0098] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0099] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0100] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0101] It should be understood that the "embodiment" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, the various embodiments in the entire specification do not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0102] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

[0103] In addition, it should be noted that the various numerical numbers involved in this application (such as the terms "first", "second", "third", "fourth" and other various terminology labels (if any) in the specification and claims and the above-mentioned drawings) are only distinguished for the convenience of description and are not used to limit the scope of this application. The size of the sequence number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic.

Claims

1. An intelligent picking method, characterized in that: The method is applied to an agricultural picking robot, and the method comprises: Acquire a first image, wherein the image content of the first image includes a target fruit tree, and the target fruit tree includes a target fruit; The first image is processed based on a first neural network model to obtain a second image, wherein the first neural network model is trained based on multiple training data sets, each training data set includes multiple images obtained by augmenting an original image, and the original image contains fruits of the same type as the target fruit. The first neural network model is trained for the purpose of reducing a first loss function value, a second loss function value, and a third loss function value, wherein the first loss function value is used to characterize the feature difference between the input image and the output image of the first neural network model, the second loss function value is a loss function value against a fourth loss function value, and the fourth loss function value is used to characterize the feature difference between multiple output images corresponding to each training data set, and the third loss function value is used to characterize the difference between a fruit recognition result and a target recognition result for the training data set; Identify a first region in the second image, the second image is composed of the first region and the second region, the first image is composed of the third region and the fourth region, the first region corresponds to the third region, the second region corresponds to the fourth region, feature intensity deviations of corresponding pixels of the first region and the third region are both less than or equal to a first threshold, and feature intensity deviations of corresponding pixels of the second region and the fourth region are both greater than the first threshold; Determine a target detection area based on the first area, and emit laser pulses to the target detection area through a laser transmitter; Receiving the echo signal corresponding to the laser pulse through a laser receiving array to obtain measurement information of a plurality of pixel points corresponding to the target detection area, wherein the number of receiving units in the laser receiving array is less than or equal to a preset threshold; Determine the upper boundary point of the target fruit on the target fruit tree from the multiple pixel points, and determine the picking point according to the upper boundary point; The target fruit is picked based on the picking points.

2. The method according to claim 1, characterized in that: Determining the upper boundary point of the target fruit on the target fruit tree from the multiple pixel points includes: Detecting each pixel point in the candidate boundary points, where the candidate boundary points are part or all of the multiple boundary points; When the i-th pixel point among the candidate boundary points meets preset conditions, the i-th pixel point is determined to be the upper boundary point of the target fruit, and the preset conditions include: the difference between the measured distances corresponding to the x pixels above the i-th pixel point and the measured distances corresponding to the i-th pixel point is greater than a first threshold, and the difference between the measured distances corresponding to the y pixels below the i-th pixel point and the measured distances corresponding to the i-th pixel point is less than a second threshold, and x, y, and i are all positive integers, and the values ​​of x and y are determined based on the type of the target fruit tree.

3. The method according to claim 2, characterized in that The method further comprises: P rows of pixel points from the multiple pixel points are selected as the candidate boundary points, and the measured distances corresponding to more than m pixel points in the p rows of pixel points are greater than a third threshold, and the measured distances corresponding to more than n pixel points in the p rows of pixel points are less than a fourth threshold.

4. The method according to claim 1, characterized in that: The number of receiving units in the receiving array is determined based on the type of the target fruit tree and the model of the agricultural harvesting robot.

5. The method according to claim 4, characterized in that Different receiving units in the receiving array correspond to different measurement ranges, and the measurement range corresponding to each receiving unit is determined based on the historical maximum measurement range of the receiving unit.

6. The method according to claim 1, characterized in that Channels in the first neural network model whose dependency coefficients are less than or equal to a fifth threshold are deleted in advance, and parameters of the first neural network model are adjusted in advance based on the amount of loss, wherein the amount of loss is used to characterize the degree of structural change of the first neural network model, and the dependency coefficient is used to characterize the dependency relationship between the channel and the model.

7. An agricultural harvesting robot, characterized in that: The agricultural picking robot comprises: An image acquisition module, used for acquiring a first image, wherein the image content of the first image includes a target fruit tree, and the target fruit tree includes a target fruit; A model processing module, for processing the first image based on a first neural network model to obtain a second image, wherein the first neural network model is trained based on multiple training data sets, each training data set includes multiple images obtained by augmenting an original image, the original image includes fruits of the same type as the target fruit, and the first neural network model is trained for the purpose of reducing a first loss function value, a second loss function value, and a third loss function value, wherein the first loss function value is used to characterize the feature difference between the input image and the output image of the first neural network model, the second loss function value is a loss function value against a fourth loss function value, the fourth loss function value is used to characterize the feature difference between the multiple output images corresponding to each training data set, and the third loss function value is used to characterize the difference between the fruit recognition result and the target recognition result for the training data set; an identification module, configured to identify a first region in the second image, the second image being composed of the first region and the second region, the first image being composed of the third region and the fourth region, the first region corresponding to the third region, the second region corresponding to the fourth region, feature intensity deviations of corresponding pixels of the first region and the third region being less than or equal to a first threshold, and feature intensity deviations of corresponding pixels of the second region and the fourth region being greater than the first threshold; A detection module, configured to determine a target detection area based on the first area, and to emit a laser pulse to the target detection area through a laser transmitter; and to receive an echo signal corresponding to the laser pulse through a laser receiving array to obtain measurement information of a plurality of pixel points corresponding to the target detection area, wherein the number of receiving units in the laser receiving array is less than or equal to a preset threshold; A detection module, used to determine the upper boundary point of the target fruit on the target fruit tree from the multiple pixel points, and determine the picking point according to the upper boundary point; A picking module is used to control the mechanical device of the agricultural picking robot to pick the target fruit based on the picking point.

8. An agricultural harvesting robot comprising one or more processors and a memory; The memory is coupled to the one or more processors, and the memory is used to store computer program codes, wherein the computer program codes include computer instructions, and the one or more processors call the computer instructions to enable the agricultural harvesting robot to perform the method as described in any one of claims 1 to 7.

9. A chip system, characterized in that: The chip system is applied to an agricultural harvesting robot, and the chip system includes one or more processors, and the one or more processors are used to call computer instructions so that the agricultural harvesting robot executes the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium comprises instructions, which, when executed on an electronic device, cause the electronic device to perform the method as claimed in any one of claims 1 to 7.

Images