A fruit picking optimization model method of deep learning and swarm intelligence algorithm

Abstract

The application relates to the technical field of artificial intelligence picking, and discloses a fruit picking optimization model method of deep learning and swarm intelligence algorithm, which comprises the following steps: S1, collecting fruit pictures in a natural environment as training data, and preprocessing the data; S2, designing a CA U-Net semantic segmentation network of an improved U-Net model, and replacing the backbone network VGG16 of the U-Net model with a lightweight MobileNet-V3 structure; and S3, replacing the SE attention mechanism module in the MoblieNet-v3 backbone feature network with a Coordinate Attention attention mechanism. The fruit picking optimization model method of deep learning and swarm intelligence algorithm uses computer vision technology to realize a fruit picking sequence algorithm of image segmentation, improves fruit picking efficiency and fruit positioning accuracy, solves the problems of waste and damage in the picking process in the prior art, can avoid fruit rotting and damage, and solves the problems of being unable to improve the shelf life and market value of the fruit.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence picking, in particular to a fruit picking optimization model method based on deep learning and swarm intelligence algorithm. BACKGROUND

[0002] Fruit picking is one of the key links in agricultural production, and the picking sequence can affect picking efficiency and fruit quality; traditional fruit picking methods are usually manual picking, but this method is time-consuming and labor-intensive, and the efficiency is low; with the development of computer vision technology and the improvement of mechanical automation level, automatic fruit picking has become a new research direction; among them, the research on fruit picking sequence algorithm based on image segmentation is one of the important directions; at present, it is difficult to pick pears, which requires a large amount of manpower and material resources, and in a more complex orchard environment, the picking efficiency is not ideal, which is not suitable for the development of large-scale pear planting industry, so realizing picking automation has great significance for improving the efficiency of large-scale picking scenes.

[0003] At present, agricultural robots still have many deficiencies in adaptability, especially in the face of complex agricultural environment in reality, agricultural robots are difficult to adapt to the complex and changeable environment in the orchard and the changeable weather; in addition, although modern agricultural robots are of various types, they are more inclined to realize simple agricultural production activities, and the research and production costs of specialized agricultural robots are relatively high, and the efficiency is lower compared with single-function agricultural equipment, which leads to the difficulty of agricultural robots in being widely used in actual scenes. SUMMARY

[0004] (I) Technical problems to be solved

[0005] In view of the deficiencies of the prior art, the present application provides a fruit picking optimization model method based on deep learning and swarm intelligence algorithm, which uses computer vision technology to realize image segmentation fruit picking sequence algorithm, improves fruit picking efficiency and fruit positioning accuracy, solves the waste and damage in the picking process in the prior art, and also avoids fruit rotting and damage, and cannot improve the shelf life and market value of fruits.

[0006] (II) Technical scheme

[0007] In order to achieve the above purpose, the present application provides the following technical scheme: a fruit picking optimization model method based on deep learning and swarm intelligence algorithm, comprising the following steps:

[0008] S1, collect fruit pictures in natural environment as training data, and pretreat the data;

[0009] S2, design a CA U-Net semantic segmentation network improved U-Net model, replace the backbone network VGG16 of U-Net model with lightweight MobileNet-V3 structure;

[0010] S3, replace the SE attention mechanism module in MoblieNet-v3 backbone feature network with CoordinateAttention attention mechanism, so that better results can be achieved in complex orchard background;

[0011] S4, first group the fruits using K-means clustering algorithm, then use clustering grouping to optimize, combine clustering evolution mechanism with multi-objective locust optimization algorithm to determine the picking order optimization model;

[0012] S5, optimize the picking order of fruits based on the two principles of shortest picking distance and smallest fruit occluded area, and reasonably allocate factor influence weight according to different picking scenarios and different fruits;

[0013] S6, finally, use the picking order optimization model to optimize the picking order of fruits, and use agricultural robots to pick;

[0014] The mathematical model of the minimum picking distance in S5 is as follows:

[0015]

[0016] Where, m is the total m sub-populations in the picking scenario, n is the total n fruits in each sub-population, i is the corresponding population number, and the populations are numbered in order as i = 1, 2, …, m, j is the number of fruits in each sub-population, and the fruits are sorted in order as j = 1, 2, …, n;

[0017] According to the occluded area of the fruit, the external naked condition of the fruit is judged to reduce the damage of the fruit, and the naked area of the fruit is calculated and compared according to the overlapping degree of the detection frame of the target detection model, and the mathematical model is as follows:

[0018] ;

[0019] Combining the above two target effects with the actual picking scene, the model is solved by using algorithm, so that f1 and f2 are minimized at the same time, and the multi-objective picking order optimization algorithm model is established as .

[0020] Preferably, the SE attention mechanism module in S3 requires two fully connected layers, and the CoordinateAttention attention mechanism only requires one fully connected layer and one learnable position encoder.

[0021] Preferably, the Coordinate Attention is a lightweight attention mechanism for reducing computational cost and parameter quantity, and the Coordinate Attention learns the relationship between features by using coordinate information to improve the performance of the network, and the Coordinate Attention attention mechanism models the relationship between spatial position channels as the interaction between 2D coordinates and channel embeddings, thereby enabling the network to better capture the spatial and channel dependencies between features.

[0022] Preferably, the K-means clustering in S4 divides the population into different sub-populations, and the best solution in each sub-population is compared to determine the optimal parameter determination scheme in the whole.

[0023] (Three) beneficial effects

[0024] Compared with the prior art, the present application provides a fruit picking optimization model method based on deep learning and swarm intelligence algorithm, which has the following beneficial effects:

[0025] 1、The present application adopts deep learning algorithm to detect and locate pear fruit pictures and image segmentation, uses binocular camera to locate the three-dimensional coordinates of the picking point of the fruit, optimizes the picking order of each fruit, replaces the large parameter backbone feature extraction network VGG16 in U-Net with a lightweight MobileNet-V3 network, then improves the SE module in MobileNet-V3 to Coordinate Attention attention module, improves the image segmentation accuracy in complex background, applies multi-object locust optimization algorithm to the scene of fruit picking order optimization, optimizes the grouping mechanism of the fruit, uses K-means clustering algorithm to group the fruit first, then optimizes the picking order of the fruit based on the two principles of shortest picking distance and smallest fruit occluded area, and reasonably allocates the factor influence weight according to different picking scenes and different fruits, and uses the improved swarm intelligence optimization algorithm to optimize the picking order of the fruit.

[0026] 2、The method of the fruit picking optimization model of deep learning and swarm intelligence algorithm provided by the present application has the advantages that the network structure is lightened, the attention mechanism of spatial perception is paid more attention to, fruit pictures in a natural environment are used as a data set for model training, and a swarm intelligence algorithm is used to optimize the fruit picking sequence. BRIEF DESCRIPTION OF DRAWINGS

[0027] Fig. 1 A principle diagram of the method of the fruit picking optimization model of deep learning and swarm intelligence algorithm provided by the present application is shown in the figure.

[0028] Fig. 2 A step diagram of the method of the fruit picking optimization model of deep learning and swarm intelligence algorithm provided by the present application is shown in the figure. DETAILED DESCRIPTION

[0029] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0030] Referring to the accompanying drawings Figs. 1-2 A method of the fruit picking optimization model of deep learning and swarm intelligence algorithm, comprising the following steps:

[0031] S1, collect fruit pictures in a natural environment as training data, and pre-process the data;

[0032] S2, design a CA U-Net semantic segmentation network improved from a U-Net model, and replace the backbone network VGG16 of the U-Net model with a light-weight MobileNet-V3 structure;

[0033] S3, replace the SE attention mechanism module in the MoblieNet-v3 backbone feature network with a CoordinateAttention attention mechanism, so that better results can be achieved in a complex orchard background;

[0034] S4, using the K-means clustering algorithm to first group the fruits, and then using the clustering grouping to optimize, combining the clustering evolution mechanism with the multi-objective locust optimization algorithm to determine the picking order optimization model;

[0035] S5, using the shortest picking distance and the smallest fruit occluded area as the two principles to optimize the picking order of the fruits, and reasonably allocating the factor influence weight according to different picking scenes and different fruits;

[0036] S6, finally, using the picking order optimization model to optimize the picking order of the fruits, and using the agricultural robot to pick;

[0037] A semantic segmentation network is designed, which adopts a lightweight network structure and pays more attention to the spatial perception attention mechanism, and uses fruit pictures in natural environment as data set for model training. Secondly, the swarm intelligence algorithm is used to optimize the fruit picking order, and the shortest picking distance and the smallest fruit occluded area are used as optimization indexes according to different scenes and fruit characteristics, and the factor influence weight is reasonably allocated;

[0038] The fruit picking order algorithm based on image segmentation is realized by using computer vision technology, which improves the fruit picking efficiency and fruit positioning accuracy, solves the waste and damage in the picking process in the prior art, and also avoids the problems of fruit rotting and damage, thereby improving the shelf life and market value of fruits. Therefore, the fruit picking order algorithm based on image segmentation has important practical application value.

[0039] The SE attention mechanism module in S3 needs two fully connected layers, so the parameter quantity is relatively large, which requires additional calculation cost and parameter quantity, which may cause certain burden to the lightweight mobile device, and the CoordinateAttention attention mechanism only needs one fully connected layer and one learnable position encoder, the parameter quantity is smaller, the calculation speed of the attention is relatively faster, and it is more suitable for real-time picking scene in orchard.

[0040] The Coordinate Attention is a lightweight attention mechanism, which aims to provide the ability of enhanced feature representation similar to SE attention, for reducing the calculation cost and parameter quantity, and the Coordinate Attention learns the relationship between features by using coordinate information, improves the performance of the network, the Coordinate Attention attention mechanism models the relationship between spatial position channels as the interaction between 2D coordinates and channel embeddings, so as to realize the network to better capture the spatial and channel dependence between features.

[0041] According to the size of the picking distance, the moving distance of the machine can be effectively reduced, and the picking efficiency is increased. The target is optimized to minimize the picking distance. The mathematical model of the minimum picking distance in S5 is as follows:

[0042]

[0043] Where, m is the total m sub-populations in the picking scene, n is the total n fruits in each sub-population, i is the corresponding population number, the populations are numbered in turn i = 1, 2, …, m, j is the number of fruits in each sub-population, and the fruits are sorted in turn j = 1, 2, …, n;

[0044] According to the area of the fruit being blocked, the outer surface of the fruit is determined to reduce the damage of the fruit. The exposed area of the fruit is calculated and compared according to the overlapping degree of the target detection model detection frame. The mathematical model is as follows:

[0045]

[0046] Combining the above two target influences, combining the actual picking scene, using algorithm to solve the model, so that f1 and f2 are minimized at the same time, and the multi-objective picking sequence optimization algorithm model is established as .

[0047] In S4, the total population is divided into different sub-populations by using K-means clustering, and the best solution in each sub-population is compared to determine the optimal parameter determination scheme in the whole.

[0048] It should be noted that the term "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or apparatus including a series of elements includes not only those elements, but also other elements not explicitly listed, or inherent to such a process, method, article or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus including the element.

[0049] Although embodiments of the present application have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made thereto without departing from the principles and spirit of the present application, and the scope of the present application is defined by the appended claims and their equivalents.​

Claims

1. A method for optimizing fruit harvesting using deep learning and swarm intelligence algorithms, characterized in that: Includes the following steps: S1. Collect images of fruits in their natural environment as training data and preprocess the data. S2. Design an improved U-Net model CA U-Net semantic segmentation network, replacing the U-Net model's backbone network VGG16 with a lightweight MobileNet-V3 structure; S3. Replace the SE attention mechanism module in the MobileNet-v3 backbone feature network with the CoordinateAttention attention mechanism to achieve better results in complex orchard backgrounds. S4. First, the fruits are grouped using the K-means clustering algorithm, and then the clustering grouping is used for optimization. The clustering evolution mechanism is combined with the multi-objective grasshopper optimization algorithm to determine the optimal model for the picking order. S5. Optimize the fruit picking order based on the two principles of shortest picking distance and minimum fruit shading area, and reasonably allocate the influence weight of factors according to different picking scenarios and different fruits. S6. Finally, after optimizing the fruit picking order using the picking order optimization model, agricultural robots are used for picking. The mathematical model for minimizing the picking distance in S5 is shown below: Wherein, 𝑚 represents the 𝑚 subpopulations in the picking scene, 𝑛 represents the 𝑛 fruits in each subpopulation, 𝑖 represents the corresponding population number, the populations are numbered sequentially as 𝑖 = 1, 2, ... , 𝑚, and 𝑗 represents the number of fruits in each subpopulation, the fruits are sorted sequentially as 𝑗 = 1, 2, ... , 𝑛; The degree of fruit exposure is determined by the area of ​​the fruit that is covered, in order to reduce fruit damage. The exposed area of ​​the fruit is calculated and compared based on the degree of overlap of the detection boxes in the target detection model. The mathematical model is as follows: ； Considering the combined effects of the two objectives mentioned above and the actual harvesting scenario, an algorithm is used to solve the model, minimizing both 𝑓1 and 𝑓2 simultaneously. A multi-objective harvesting order optimization algorithm model is established as follows: .

2. The method for optimizing fruit harvesting using deep learning and swarm intelligence algorithms according to claim 1, characterized in that: The SE attention mechanism module in S3 requires two fully connected layers, while the Coordinate Attention mechanism only requires one fully connected layer and a learnable position encoder.

3. The method for optimizing fruit harvesting using deep learning and swarm intelligence algorithms according to claim 2, characterized in that: The Coordinate Attention is a lightweight attention mechanism designed to reduce computational cost and the number of parameters. It learns the relationships between features by using coordinate information, thereby improving network performance. The Coordinate Attention mechanism models the relationships between spatial location channels as the interaction between 2D coordinates and channel embeddings, enabling the network to better capture the spatial and channel dependencies between features.

4. The method for optimizing fruit harvesting using deep learning and swarm intelligence algorithms according to claim 1, characterized in that: In step S4, K-means clustering is used to divide the population into different subpopulations. The best solution in each subpopulation is compared to determine the optimal parameter determination scheme for the whole.

Images