Bagging method, device, storage medium and electronic device

Abstract

The present disclosure relates to a bagging method, device, storage medium and electronic device, the method comprising: acquiring an image of a fruit to be bagged in real time; performing fruit recognition on the image to identify the outline of each fruit; extracting the coordinates of the center of each fruit in the image according to the outline of the fruit; identifying the position of the connection point between the fruit and the branch from the image, and determining the coordinates of the position of each connection point; generating a bagging position of a robotic arm and a bagging direction of the robotic arm for the fruit according to the center coordinates of any fruit and the coordinates of the corresponding connection point position; generating a bagging route for the robotic arm from the current position to the bagging position; controlling the robotic arm to reach the bagging position according to the bagging route, adjusting the robotic arm to the above-mentioned bagging direction, and bagging the fruit.

Description

Technical Field

[0001] The present invention belongs to the technical field of agricultural cultivation, and in particular relates to a bagging method, a device, a storage medium and an electronic device. Background Art

[0002] Fruit bagging is a key technology for producing pollution-free fruits, and it is also an effective way to improve fruit quality and farmers' income. In the fruit planting process, fruit bagging has always been a time-consuming and labor-intensive process. At present, the method and time planning of fruit bagging still rely mainly on manual labor, and the fruit planting area is generally in rural areas. In recent years, the rural population has been aging seriously, and a large part of the young and middle-aged people have gone out to work, which has led to problems such as insufficient rural labor and rising labor costs.

[0003] Moreover, at this stage, the degree of modernization of many fruit forests is not high, and the bagging of fruits is mainly done by manpower. When there are many fruits that need to be bagged, there will be problems such as heavy workload, inconvenient working hours, and insufficient labor, which will lead to mistakes in a hurry. In addition, due to the randomness of fruit tree growth, the bagging efficiency is low and manual bagging is difficult, which can be replaced by machines. In this way, there is no need for technicians to conduct multiple supervision inspections and a large amount of labor to carry out bagging, which greatly reduces the burden on fruit farmers and technicians, and reduces the error rate. The yield and quality of fruits will also be greatly improved, which will promote fruit farmers to increase production and income and promote regional economic development. Summary of the invention

[0004] The purpose of the present invention is to provide a bagging method, device, storage medium and electronic device, which are used to solve the problems of low bagging efficiency, difficulty in manual bagging and waste of labor when bagging fruits manually in the prior art.

[0005] In order to achieve the above object, the first aspect of the present disclosure provides a bagging method, the method comprising:

[0006] A bagging method, characterized by comprising:

[0007] Acquire the image of the fruit to be bagged in real time;

[0008] Performing fruit recognition on the image to identify the outline of each fruit;

[0009] Extracting the center coordinates of each fruit in the image according to the fruit outline;

[0010] Identify the connection points between the fruit and the branch from the image, and determine the coordinates of each connection point;

[0011] According to the center coordinates of any fruit and the coordinates of the corresponding connection point position, the bagging position and bagging direction of the robot arm for the fruit are generated;

[0012] Generate a bagging route for the robot arm from its current position to the bagging position;

[0013] The robot arm is controlled to reach the bagging position according to the bagging route, and the robot arm is adjusted to the above-mentioned bagging direction to bag the fruit.

[0014] Optionally, performing fruit recognition on the image to identify each fruit outline includes:

[0015] Based on the trained Faster-RCNN neural network model, the image is subjected to fruit recognition and a frame is selected to select any complete fruit in the image;

[0016] Performing blackening processing on the area other than the framed area in the image;

[0017] Get the canny contour layer from the blackened image.

[0018] Optionally, extracting the center coordinates of each fruit in the image according to the fruit outline includes:

[0019] Obtain a two-dimensional point set of a closed fruit contour edge in the canny contour layer;

[0020] Traverse all points of the closed contour and calculate the first distance between any two points;

[0021] The maximum first distance is taken as the diameter value of the fruit, and the coordinates of the center of the fruit are determined based on the coordinates of the two points at which the maximum first distance is calculated.

[0022] Optionally, the step of identifying the connection point positions between the fruit and the branch from the image and determining the coordinates of each connection point position comprises:

[0023] The trained YOLOv3 network backbone network is used to extract features from the image of the fruit to be bagged that is obtained in real time, and a 13×13 scale feature map is obtained;

[0024] A part of the feature map of the 13×13 scale is extracted from the image of the fruit to be bagged currently acquired in real time to obtain a feature map of 52×52×64 scale, and the feature is fused using the FPN network of the YOLOv3 network to obtain a feature map of 26×26 scale;

[0025] The prediction results of the connection points between the fruit and the branches corresponding to the two scales are obtained;

[0026] All prediction results are processed using the classification network and non-maximum suppression module to obtain the detection results of the connection points between fruits and branches.

[0027] Optionally, the step of generating the bagging position and bagging direction of the robot arm for any fruit according to the center coordinates of the fruit and the coordinates of the corresponding connection point position includes:

[0028] A linear equation Ax+By+Cz=0 is established according to the center coordinates and the connection point coordinate data, and the position where the center coordinates are moved by 1 / 2 of the maximum distance along the slope direction of the linear equation is determined as the bagging position of the robot arm for the fruit;

[0029] The bagging angle of the bagging robot arm is generated according to the slope of the established linear equation.

[0030] Optionally, after the step of determining and generating the bagging position of the mechanical arm for the fruit, the step further includes:

[0031] Determine the distance between each bagging position according to the bagging position;

[0032] According to the basic principle of ant colony algorithm, an ant colony algorithm for bagging position sequence planning is constructed. The advantages of ant colony algorithm are used to optimize the bagging position sequence planning. The shortest path is used as the evaluation index to find the optimal path sequence among many feasible bagging path sequences.

[0033] In a second aspect, a bagging device is provided, the device comprising:

[0034] An acquisition module is used to acquire the image of the fruit to be bagged in real time;

[0035] A recognition module, used to perform fruit recognition on the image and identify the outline of each fruit;

[0036] A parameter generation module, used to identify the connection points between the fruit and the branch from the image and determine the coordinates of each connection point; and

[0037] According to the center coordinates of any fruit and the coordinates of the corresponding connection point position, the bagging position and bagging direction of the robot arm for the fruit are generated;

[0038] Route generation module, used to generate the bagging route of the robot arm from its current position to the bagging position

[0039] The control module is used to control the robotic arm to reach the bagging position according to the bagging route, adjust the robotic arm to the above-mentioned bagging direction, and bag the fruit.

[0040] According to a third aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method according to the first aspect are implemented.

[0041] In a fourth aspect, an electronic device is provided, comprising: a memory having a computer program stored thereon;

[0042] A processor is used to execute the computer program in the memory to implement the steps of the method described in the first aspect above.

[0043] In the above technical scheme, an image of the fruit to be bagged is obtained in real time; fruit recognition is performed on the image to identify the outline of each fruit; the center coordinates of each fruit in the image are extracted according to the fruit outline; the connection point positions between the fruit and the branch are identified from the image, and the coordinates of each connection point position are determined; the bagging position and bagging direction of the robotic arm for the fruit are generated according to the center coordinates of any fruit and the coordinates of the corresponding connection point position; a bagging route for the robotic arm from its current position to the bagging position is generated; the robotic arm is controlled to reach the bagging position according to the bagging route, and the robotic arm is adjusted to the above-mentioned bagging direction to bag the fruit. Through the above technical scheme, the real-time image of the fruit to be bagged can be identified, analyzed and processed to identify the outline of each fruit, and the coordinates of the center of each fruit in the image can be extracted according to the outline, and the coordinates of each connection point position can be determined by identifying the connection point position between the fruit and the branch from the image, and then the bagging position and bagging direction of the robotic arm can be determined according to the obtained position and connection point coordinate parameters, and then the robotic arm can be controlled to bag the fruit according to the shortest path obtained by the ant colony algorithm, so that the fruit can be bagged automatically according to machine vision, thereby solving the problems of low bagging efficiency, difficulty in manual bagging, and waste of labor when bagging fruits manually in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the present disclosure but do not constitute a limitation of the present disclosure. In the accompanying drawings:

[0045] Figure 1 It is a schematic diagram of a bagging method provided by an embodiment of the present disclosure;

[0046] Figure 2 is a schematic diagram of a process of performing fruit recognition on the image and identifying the outline of each fruit provided by an embodiment of the present disclosure;

[0047] Figure 3is a schematic diagram of a process for extracting the center coordinates of each fruit in the image according to the fruit outline provided by an embodiment of the present disclosure;

[0048] Figure 4 The present disclosure provides a flow chart of identifying the connection point positions between fruits and branches from an image and determining the coordinates of each connection point position;

[0049] Figure 5 It is a flow chart of generating a bagging position and a bagging direction of a robotic arm for any fruit according to the center coordinates of any fruit and the coordinates of the corresponding connection point position provided by an embodiment of the present disclosure;

[0050] Figure 6 is a flow chart after the step of determining and generating a bagging position of the mechanical arm for the fruit provided by an embodiment of the present disclosure;

[0051] Figure 7 It is a structural schematic diagram of a bagging device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0052] The specific implementation of the present disclosure is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described herein is only used to illustrate and explain the present disclosure, and is not used to limit the present disclosure.

[0053] Before introducing the embodiments of the present disclosure in detail, the application scenario of the present disclosure is first described. The present disclosure can be applied to the bagging scenario of fruits, such as apples, pears and other fruits. In this application scenario, during the growth of fruits, due to the randomness of fruit growth on fruit trees, manual bagging cannot complete multi-target bagging while planning the bagging sequence according to the shortest path, which will result in low bagging efficiency and difficulty in manual bagging.

[0054] In the prior art, fruit bagging can generally be achieved in two ways. One is to bag the fruits growing on fruit trees based on manual bagging; the other is to use traditional visual recognition to segment and identify the fruits, and then use a robotic arm to bag the identified fruits. However, the traditional visual segmentation method can only complete the recognition task under limited conditions, and its stability is low. It cannot simultaneously overcome the difficulties in fruit recognition on trees, such as being greatly affected by external environmental factors, having many interference targets, and having high real-time requirements.

[0055] In order to solve the above problems, the present application proposes a bagging method, device, storage medium and electronic device. When bagging fruits on fruit trees, firstly, a camera is used to take a color image of the bagged fruit tree from a distance, and then the image is used to identify the fruit according to the trained neural network model, and the contour of each fruit is identified. The center coordinates of each fruit in the image are extracted according to the contour, and the coordinates of each connection point position are determined from the image and the connection point position between the fruit and the branch are identified, and then the bagging position and bagging direction of the robot arm are determined according to the obtained position and connection point coordinate parameters, and then the robot arm is controlled to bag the fruit according to the shortest path obtained by the ant colony algorithm, which can realize the automatic bagging of the fruit according to machine vision, thereby solving the problems of low bagging efficiency, difficulty in manual bagging, and waste of labor when bagging fruits manually in the prior art.

[0056] The present disclosure is described below in conjunction with specific embodiments.

[0057] Figure 1 is a flow chart of a precise quantitative watering method according to an exemplary embodiment of the present disclosure. The method can be applied to electronic devices, such as computers, mobile phones, tablet computers, etc. Figure 1 As shown, the method may include the following steps:

[0058] In step S11, an image of the fruit to be bagged is obtained in real time.

[0059] This image acquisition is based on ordinary color images. The target of image acquisition is the fruit to be bagged under actual natural growth conditions. The image acquisition is based on CCD camera acquisition. During the shooting process, the shooting angle of view and the shooting height are parallel to the fruit tree, so that the fruit tree growing the fruit to be bagged remains in the center of the image.

[0060] Among them, the image can be directly obtained by taking pictures with a CCD camera, or it can be stored in a terminal device after being taken by a CCD camera, and then obtained by calling the image stored in the terminal device. The terminal in the present disclosure can be a computer, a smart phone, a smart watch, a tablet computer, etc.

[0061] In step S12, fruit recognition is performed on the image to identify the outline of each fruit.

[0062] In this step, after obtaining the image of the fruit to be bagged, firstly, the image is subjected to fruit recognition based on the trained Faster-RCNN neural network model to select any complete fruit on the image, then the unselected part of the selected image is blackened, and then the canny layer is obtained from the blackened image.

[0063] In step S21, a fruit recognition frame is performed on the image based on the trained Faster-RCNN neural network model to select any complete fruit in the image.

[0064] Before executing step S21, the Faster-RCNN neural network model is first trained. The training method is as follows:

[0065] a. Build a Faster-RCNN neural network model based on the VGG16 feature extraction network.

[0066] b. Initialize the weights of the feature extraction network, and then train the initialized Faster-RCNN neural network model through the training data sequence; obtain the Faster-RCNN neural network model and network weight parameters after training.

[0067] c. Input the test dataset into the trained Faster-RCNN neural network model.

[0068] Further, the step a is specifically as follows:

[0069] The Faster-RCNN neural network model consists of three parts: the first part is the feature extraction network layer, whose main task is to extract image feature maps. It is designed based on the VGG16 network structure. The CNN network uses the VGG16 network, which has a good effect on identifying the features of large-scale images; a deep network consisting of 13 convolutional layers, 13 ReLU activation functions, 4 pooling layers and 3 fully connected layers is used. The pooling layer uses the maximum pooling layer to retain texture features; the second part is the RPN layer, which is used to generate predicted target boxes; the third part is the ROI pooling layer, which is used to regress and classify the size of the generated predicted target boxes.

[0070] Furthermore, a sample input sequence is obtained, and the sample input sequence is divided into two parts, one part is used as a training data sequence, and the other part is used as a test data sequence. The sample input sequence is preprocessed.

[0071] For example, first use a CCD camera to obtain the corresponding batch of pictures, then delete the repeated samples and unclear samples in this batch of pictures, and then use the homomorphic filtering algorithm to enhance the image. After the enhancement, use Python language to call the interface of the OpenCV open source function library to resize along the shortest dimension of the image width and height, and then crop the image with the center as the origin to obtain an RGB image with an input of 375×500, so that the aspect ratio of the image can be maintained, and the image will not be distorted and twisted. Manually annotate this batch of pictures. In order to ensure the accuracy of the annotation and reduce personal subjective errors, another professional can be used to review the annotated images to obtain more accurate images of the annotation results. Part of the batch is used as a training data sequence, and the other part is used as a test data sequence.

[0072] Furthermore, in order to ensure that the annotated content can be easily seen, a rectangular frame with a shortest side greater than 3 mm can be used for annotation, and the rectangular frame should completely surround the fruit in the image.

[0073] After obtaining the sample input sequence, the training data sequence is used to train the Faster-RCNN model. The images of the training data sequence are used as the input of the CNN network to extract the fruit position features. The CNN network outputs a feature map that is compatible with the image of the fruit to be bagged. The CNN network uses the VGG16 network.

[0074] For example, an RGB image with an aspect ratio of 375×500 is used as input for training the Faster-RCNN model. The RGB image is first input into the VGG16 network in the Faster-RCNN subnetwork. The input feature map is activated by a Relu activation function each time it passes through a convolution layer. The convolution layer is followed by a pooling layer for maximum pooling. Each convolution layer uses a standard convolution of size 3×3, a padding number (Padding) of 1, and a step size of 2. Each pooling layer uses maximum pooling, a pooling kernel size of 2×2, and a step size of 2. After the VGG16 network extracts the fruit position features in the image of the fruit to be bagged, a feature map with the fruit position features in the image of the fruit to be bagged is obtained.

[0075] The feature map is then input into the RPN network for training. The RPN network selects the predicted target box including the fruit from the feature map to obtain the initial RPN network. The RPN network consists of a fully connected layer and a convolutional layer. First, the convolutional layer is convolved with the feature map to obtain a candidate area. The candidate area includes the location and probability of the existence of the target fruit position feature. Then the convolutional layer is convolved with the candidate area, and the candidate area is predicted to obtain a predicted target box. For a feature map with an aspect ratio of W*H, W*H*3*3 predicted target boxes can be obtained.

[0076] It should be further explained that the target area is represented by a four-dimensional vector (x, y, w, h), where x, y, w and h represent the center coordinates of the sliding window and the width and height of the sliding window respectively. The sliding window can automatically correct the border of the candidate area, so as to obtain a target area close to the actual target area by using border regression correction. The RoI Pooling layer uses the predicted target box generated by RPN and the previously obtained feature map to obtain a predicted target box of a fixed size.

[0077] The fixed-size predicted target box is fully connected using the fully connected layer, and the probability value of each area is calculated through the fully connected layer and softmax. The positioning accuracy in each area is calculated through the fully connected layer and the sigmoid function, and the offset of each area is obtained by bounding box regression, and finally the precise position of the detection box is obtained.

[0078] At the same time, the loss function of the training model to calculate the predicted target box is obtained:

[0079]

[0080] In formula (1), i is the i-th anchor box, p i is the probability that the anchor box is predicted as the target, p i * is the label, which is 1 when the sample frame is true, otherwise it is zero, N cls is the minimum batch normalization value, N reg is the number of anchor box positions, λ is to set the score loss and position loss in approximately equal proportion, L cls It is the loss function to determine whether there is a target in the anchor box, L reg is the position regression loss function, t i To predict the four coordinates of the sliding box, t i * The coordinates of the target box of the real box corresponding to the anchor box.

[0081] The training effect can be judged by the loss rate. The lower the loss rate, the better the training effect, that is, the better the classification effect of the region of interest. The convolutional layers of the initial FasterRCNN network and the initial RPN network are shared, and the initial RPN network is trained based on the initial FasterRCNN network. At this time, the network parameters shared by the initial FasterRCNN network and the initial RPN network are kept unchanged, and the unique parameters of the initial RPN network are further updated; the unique parameters of the FasterRCNN network are updated while the convolutional layers remain unchanged; and the training is stopped when the maximum number of iterations exceeds 50,000.

[0082] After stopping, the trained neural network model is recognized using the test data sequence until the recognition accuracy is not less than 90% and tends to be stable, then the training can be terminated; if the recognition accuracy of the test data sequence is less than 90%, the weight decay term is modified and adjusted to repeat the above training steps until the recognition accuracy of the test data sequence is not less than 90%.

[0083] The image of the fruit to be bagged is input into the unified network, and the unified network outputs the fruit position detection result of the image of the fruit to be bagged, and outputs the fruit position in the image of the fruit to be bagged and the image of the fruit position selected.

[0084] In step S22, the area other than the framed area in the image is blackened.

[0085] For example, the image of the selected fruit position is processed by reverse blackening except for the portion other than the selected portion, and the cv2 module in Python is called to process the image to highlight the selected fruit position, which is helpful for obtaining the radius and coordinates of the target fruit later.

[0086] In step S23, a canny contour layer is obtained from the blackened image.

[0087] The canny contour layer can be obtained by performing a canny edge detection algorithm on the blackened image.

[0088] In step S13, the center coordinates of each fruit in the image are extracted according to the fruit outline.

[0089] In this step, a two-dimensional point set of a closed fruit contour edge in the canny contour layer is first obtained. First, a sampling contour layer obtained by connecting the preset sampling points nearby is obtained from the blackened image. The preset sampling points are fitted to each fruit edge in the canny contour layer.

[0090] Next, all points of the closed contour are traversed and the first distance between any two points is calculated.

[0091] Then, the maximum first distance is used as the diameter value of the fruit, and the coordinates of the center of the fruit are determined based on the coordinates of the two points at which the maximum first distance is calculated.

[0092] For example, the distances between two points are calculated respectively, and each calculated distance is compared, and the maximum distance compared is used as the diameter of the fruit; and the midpoint of the coordinates of the two points for calculating the diameter is used as the center of the fruit.

[0093] A spatial rectangular coordinate system is established, with the connection point between the fruit tree and the ground as the absolute origin, and the upward growth direction of the fruit tree as the Z axis of the spatial rectangular coordinate system. The unit vectors of the coordinate axes of the spatial rectangular coordinate system are: X = (1, 0, 0) T 、Y=(0,1,0) T 、Z=(0,0,1) T Then, the coordinate relationship between the center of each fruit circle and the spatial rectangular coordinate system is established, and the coordinates of the center of the circle are determined.

[0094] In step S14, the connection points between the fruit and the branch are identified from the image, and the coordinates of each connection point are determined.

[0095] Firstly, the backbone network of the trained YOLOv3 network is used to extract features of the image of the fruit to be bagged obtained in real time to obtain a feature map of 13×13 scale.

[0096] Among them, the YOLOv3 network adopted in the present invention is an improved YOLOv3 network. The original YOLOv3 network adopts the network structure of Darknet-53. The Darknet-53 network uses data in 256×256×3 format as the input of the model and uses the coco data set for training. There are 80 categories in the data set. Since the present invention only needs to identify the connection between the fruit and the branch, a network model that is too deep will increase the detection time. Therefore, we improve the network structure based on the YOLOv3 deep convolutional neural network and use a network model similar to VGG to build a 13-layer network.

[0097] The overall network architecture of the YOLOv3 network used in the present disclosure specifically includes 13 convolutional layers, 7 pooling layers, 2 fusion layers, 1 upsampling layer, and 2 output layers of different scales. Compared with YOLOv3, its overall architecture reduces the residual layer and adds a series of pooling layers instead. It also deletes some convolutional layers and FPN networks used to extract features, thereby simplifying the network, reducing computational complexity, and improving recognition speed.

[0098] At the same time, three anchor points of different sizes are set in the original YOLOv3 network, and the grid sizes of the three anchor points of different sizes are 13, 26, and 52 respectively. In order to improve the detection capability of small objects without increasing the detection time, the present invention adopts an anchor point of 13×13 scale.

[0099] When training the YOLOv3 network model, 32 samples are used as a processing unit. BN (batch normalization) is used for regularization each time the weight is updated, and a dropout layer is added to each layer to prevent overfitting. The dropout ratio is 0.5, the momentum is set to 0.9, the weight decay is set to 0.0005, the initial learning rate is set to 0.001, and then the Adam optimizer is used to update the learning rate.

[0100] When training a neural network, the loss function is optimized. The model is continuously updated through back propagation to continuously improve the model. The loss function is as follows:

[0101]

[0102] Where λ coord and λ noobj The weights are set to balance the proportion of each loss, which are 4 and 0.4 respectively; x i and i To predict the x and y axis coordinates of the target, and are the x and y axis coordinates of the actual target; w i and h i To predict the width and height of the target, and is the actual width and height of the target; c i is the category of the predicted target, is the category of the actual target. The fourth item is the algorithm's prediction of the confidence of the background; the fifth item is the prediction of the object category, p i (c) and is the confidence of the predicted target 548C is the confidence of the actual target; S and B are each grid cell and bounding box respectively, obj and noobj are the ones containing the target and not containing the target respectively.

[0103] When the Loss function reaches about 0.2, when there is a slight oscillation and the Loss function gradually tends to be stable, the training can be terminated. When the Loss function cannot reach about 0.2, adjust the number of training times. Too few training times will lead to underfitting, and too many training times will lead to overfitting.

[0104] Next, a part of the feature map of the 13×13 scale is combined with the image of the fruit to be bagged obtained in real time to obtain a feature map of 52×52×64 scale, and the FPN network of the YOLOv3 network is used to perform feature fusion to obtain a feature map of 26×26 scale.

[0105] For example, the format of the image of the fruit to be bagged is fixed to 416×416×3 format and used as the input of the YOLOv3 network. A single convolutional neural network is used to traverse the sample input sequence. This can achieve the detection of the target position in a complex environment while ensuring both efficiency and accuracy. The first prediction tensor is obtained through the convolutional layers in sequence. When the feature map is 52×52×64 in size, the feature map is upsampled and passed through a 1×1×128 convolutional layer to obtain a 13×13×128 feature map. After feature fusion with the previously obtained 52×52×64 feature map, the second prediction tensor is output through the output layer.

[0106] Then the prediction results of the connection points between fruits and branches corresponding to the two scales are obtained.

[0107] All prediction tensors can be obtained by taking the union of the first prediction tensor and the second prediction tensor. All the above prediction tensors are all prediction results of the position of the connection point between the fruit and the branch.

[0108] Then, all the prediction results are processed using the classification network and non-maximum suppression module to obtain the detection results of the position of the connection point between the fruit and the branch, and the output position information of the connection point between the fruit and the branch is synchronized with the above-mentioned spatial rectangular coordinate system to output the position coordinates of the connection point between the fruit and the branch.

[0109] In step S15, the bagging position and bagging direction of the robotic arm for any fruit are generated according to the center coordinates of any fruit and the coordinates of the corresponding connection point positions.

[0110] In this step, firstly, a linear equation Ax+By+Cz+D=0 is established according to the center coordinates and the connection point coordinate data, and the position where the center coordinates are moved 1 / 2 of the maximum distance along the slope direction of the linear equation is determined as the bagging position of the robotic arm for the fruit.

[0111] Then, the bagging angle of the bagging robot arm is generated according to the slope of the established linear equation.

[0112] The slope of the linear equation established according to the coordinates of the center of the circle and the coordinate data of the connection point is used as the angle of the bagging robot when bagging.

[0113] In step S16, a bagging route is generated from the current position of the robot arm to the bagging position.

[0114] In this step, firstly, the distance between each bagging position is determined according to the bagging position.

[0115] Then, according to the basic principle of ant colony algorithm, an ant colony algorithm for each bagging position sequence planning is constructed. The advantages of ant colony algorithm are used to optimize the bagging position sequence planning. The shortest path is used as the evaluation index to find the optimal path sequence among many feasible bagging path sequences.

[0116] According to the principle of ant colony algorithm and the characteristics of path planning of apple bagging, a corresponding mathematical model can be established. First, place ants at different apple target points, and set the number of individuals in the entire ant population to be L = ∑b m (t), where b m (t) is the number of ants at the apple target point m at time t. mn (m,n=1,2,...i) is the distance between target point m and target point n, i is the number of apple target points. The probability of an ant moving between apple targets is:

[0117]

[0118] P in the formula mn j (t) represents the probability of ant j moving from target point m to target point n at time t, τ mn (t) represents the ant hormone concentration on the path (m,n) at time t, η mn (t) is the heuristic function, which is the inverse of the distance of the path (m, n), representing the expectation of ant j to choose target point n from target point m, allowed i =(1,2,...i) is the set of all apple target points waiting to be bagged. At the beginning, it includes i-1 target points in addition to the ant's starting point, allowed i The number of target points will gradually decrease to zero. is the weight factor of ant hormones, and its value determines the degree to which ants are affected by hormones. β is the weight factor of the heuristic function, also known as the expected heuristic factor.

[0119] In the ant colony algorithm, the volatility of ant hormones is an important factor for the ant colony to distinguish the length of the path. ρ is often used to represent the volatility of ant hormones. After the ants have completed a traversal or search, the concentration of ant hormones on the path between each target point is:

[0120] τ mn (t+1)=_(1-ρ)τ mn (t)+Δτmn ,0<ρ<1 (4)

[0121] Δτ mn =∑Δτ j mn (5)

[0122] Δτ j mn represents the hormone increment of the jth ant on the path between the target point m and the target point n, Δτ mn Represents the increment of ant hormone released by all ants on the path between target point m and target point n. Δτ j mn The basic calculation method commonly used is the AntCycle ant cycle model:

[0123]

[0124] The path visited by ant j this time is (m, n), where Q represents the total amount of hormones released by the ant in a complete cycle; L j is the total distance traveled by the jth ant.

[0125] In the apple bagging path planning, for i apple target points P = {p1, p2, p3...p i The access order of} is T = {t1, t2, t3...t i}. Among them, the target point t m ∈P,m=(1,2,...i), let t m The coordinates of the target point are x m ,y m ,z m , that is, the distance moved by the end of the robot arm is:

[0126]

[0127] Where D is a path plan that traverses all target points.

[0128] The execution steps of the ant colony algorithm are:

[0129] Initialize the number of ant colony iterations, the total number of ants L, and the weight factor of ant hormones The weight factor β of the heuristic function, the volatility of ant hormone ρ, the total amount of ant hormone Q, and the iteration upper limit IC max , iterative initial value IC = 1;

[0130] The iteration begins. The result of each iteration will be used as the initial value for the next iteration, and is also an important reference factor for the next batch of ants to select paths.

[0131] Add the ant's current target point to the set of passed points.

[0132] IC max Calculate and update P mn j The value of (t).

[0133] Increase the number of ants.

[0134] Update the set of waypoints.

[0135] (If the current number of ants is less than the total number of ants L, return to step 4. If the current number of ants is equal to the total number of ants L, continue to update the total amount of ant hormones Q.

[0136] If the current number of iterations is equal to the iteration limit IC max , then the operation ends and the result is output; if the number of iterations is less than the iteration upper limit IC max , then initialize the waypoint set and iterate again.

[0137] For example, assuming the initial values ​​are L=10, Q=10, IC max =5, the total number of apples is 12. After we obtain the three-dimensional coordinates of these 12 apples, we randomly place two ants on the coordinates of the apples (set multiple ants according to the specific problem). We agree that each ant can only pass through each apple once. At the beginning of the first iteration, each ant chooses the next apple to reach based on the pheromone size and the distance between the two points. When both ants have searched once, it is an iteration. Each iteration requires a pheromone update on all edges. After the update is completed, the original ants are discarded, and two new ants are put in for a new round of search. All edges continue to be updated with pheromones. When the predetermined number of iterations is reached or all ants choose the same path, the algorithm ends and the current optimal solution is used as the optimal solution to the problem.

[0138] In step S17, the robot arm is controlled to reach the bagging position according to the bagging route, and the robot arm is adjusted to the above-mentioned bagging direction to bag the fruit.

[0139] Optionally, FIG. 1 is a block diagram of a bagging device according to an exemplary embodiment of the present disclosure. As shown in FIG. 1 , the device may include:

[0140] An acquisition module 201 is used to acquire the image of the fruit to be bagged in real time;

[0141] The recognition module 202 is used to perform fruit recognition on the image and identify the outline of each fruit;

[0142] The parameter generation module 203 is used to identify the connection points between the fruit and the branch from the image and determine the coordinates of each connection point; and

[0143] According to the center coordinates of any fruit and the coordinates of the corresponding connection point position, the bagging position and bagging direction of the robot arm for the fruit are generated;

[0144] A route generation module 204 is used to generate a bagging route for the robot arm from its current position to the bagging position;

[0145] The control module 205 is used to control the robotic arm to reach the bagging position according to the bagging route, adjust the robotic arm to the above-mentioned bagging direction, and bag the fruit.

[0146] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0147] By adopting the above device, the real-time image of the fruit to be bagged can be identified, analyzed and processed to identify the outline of each fruit, and the coordinates of the center of each fruit in the image can be extracted according to the outline, and the coordinates of each connection point position can be determined by identifying the connection point position between the fruit and the branch from the image, and then the bagging position and bagging direction of the robotic arm can be determined according to the obtained position and connection point coordinate parameters, and then the robotic arm can be controlled to bag the fruit according to the shortest path obtained by the ant colony algorithm, so that the fruit can be bagged automatically according to machine vision, thereby solving the problems of low bagging efficiency, difficulty in manual bagging and waste of labor when bagging fruits manually in the prior art.

[0148] The preferred embodiments of the present disclosure are described in detail above in conjunction with the accompanying drawings; however, the present disclosure is not limited to the specific details in the above embodiments. Within the technical concept of the present disclosure, a variety of simple modifications can be made to the technical solution of the present disclosure, and these simple modifications all fall within the protection scope of the present disclosure.

[0149] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present disclosure will not further describe various possible combinations.

[0150] In addition, various embodiments of the present disclosure may be arbitrarily combined, and as long as they do not violate the concept of the present disclosure, they should also be regarded as the contents disclosed by the present disclosure.

Claims

1. A bagging method, characterized in that: include: Acquire the image of the fruit to be bagged in real time; Performing fruit recognition on the image to identify the outline of each fruit; Extracting the center coordinates of each fruit in the image according to the fruit outline; Identify the connection points between the fruit and the branch from the image, and determine the coordinates of each connection point; According to the center coordinates of any fruit and the coordinates of the corresponding connection point position, the bagging position and bagging direction of the robot arm for the fruit are generated; Generate a bagging route for the robot arm from its current position to the bagging position; The robot arm is controlled to reach the bagging position according to the bagging route, and the robot arm is adjusted to the above-mentioned bagging direction to bag the fruit.

2. The method according to claim 1, characterized in that The performing fruit recognition on the image to identify each fruit outline comprises: Based on the trained Faster-RCNN neural network model, the image is subjected to a fruit recognition frame to select any complete fruit in the image; Performing blackening processing on the area other than the framed area in the image; Get the canny contour layer from the blackened image.

3. The method according to claim 2, characterized in that The step of extracting the center coordinates of each fruit in the image according to the fruit contour comprises: Obtain a two-dimensional point set of a closed fruit contour edge in the canny contour layer; Traverse all points of the closed contour and calculate the first distance between any two points; The maximum first distance is taken as the diameter value of the fruit, and the center coordinates of the fruit are determined based on the coordinates of the two points at which the maximum first distance is calculated.

4. The method according to claim 1, characterized in that: The step of identifying the connection point positions between the fruit and the branch from the image and determining the coordinates of each connection point position comprises: The trained YOLOv3 network backbone network is used to extract features from the image of the fruit to be bagged that is acquired in real time, and a 13×13 scale feature map is obtained; A part of the feature map of the 13×13 scale is extracted from the image of the fruit to be bagged currently acquired in real time to obtain a feature map of 52×52×64 scale, and the feature is fused using the FPN network of the YOLOv3 network to obtain a feature map of 26×26 scale; The prediction results of the connection points between the fruit and the branches corresponding to the two scales are obtained; All prediction results are processed using the classification network and non-maximum suppression module to obtain the detection results of the connection points between fruits and branches.

5. The method according to claim 1, characterized in that The step of generating the bagging position and bagging direction of the robot arm for any fruit according to the center coordinates of the fruit and the coordinates of the corresponding connection point position comprises: A linear equation Ax+By+Cz=0 is established according to the center coordinates and the connection point coordinate data, and the position where the center coordinates are moved by 1 / 2 of the maximum distance along the slope direction of the linear equation is determined as the bagging position of the robot arm for the fruit; The bagging angle of the bagging robot arm is generated according to the slope of the established linear equation.

6. The method according to claim 5, characterized in that After the step of determining and generating the bagging position of the mechanical arm for the fruit, the method further includes: Determine the distance between each bagging position according to the bagging position; According to the basic principle of ant colony algorithm, an ant colony algorithm for bagging position sequence planning is constructed. The advantages of ant colony algorithm are used to optimize the bagging position sequence planning. The shortest path is used as the evaluation index to find the optimal path sequence among many feasible bagging path sequences.

7. A bagging device, characterized in that: The device comprises: An acquisition module is used to acquire the image of the fruit to be bagged in real time; A recognition module, used to perform fruit recognition on the image and identify the outline of each fruit; A parameter generation module, used to identify the connection points between the fruit and the branch from the image and determine the coordinates of each connection point; and According to the center coordinates of any fruit and the coordinates of the corresponding connection point position, the bagging position and bagging direction of the robot arm for the fruit are generated; Route generation module, used to generate the bagging route of the robot arm from its current position to the bagging position The control module is used to control the robotic arm to reach the bagging position according to the bagging route, adjust the robotic arm to the above-mentioned bagging direction, and bag the fruit.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a computer program, wherein the computer program includes instructions for executing the method according to any one of claims 1 to 6.

9. An electronic device, characterized in that: include: The computer readable storage medium as claimed in claim 8; and one or more processors, configured to execute the program in the computer-readable storage medium.

Images