A system and method for estimating the adaptive initial grasping force of flexible fruits

Abstract

The present invention relates to the field of agricultural engineering technology, and specifically to a system and method for estimating the initial grasping force of flexible fruits. The invention obtains image data through a visual camera and, based on the image data, uses image processing and neural network methods to obtain the contour features of the fruit to be grasped and the hardness of the fruit, thereby completing the estimation of the initial grasping force of the fruit. The beneficial effects are: the present invention completes the estimation of the initial grasping force of the flexible fruit only through machine vision without too many sensors, has a simple structure, and saves costs. At the same time, based on the prior knowledge of industry applications, a fruit category and maturity recognition model is designed to indirectly obtain fruit hardness information, reduce the difficulty of model reasoning, and reduce the requirements for computer computing power while meeting the recognition accuracy requirements.

Description

Technical Field

[0001] The invention relates to the technical field of agricultural engineering, and in particular to a system and method for estimating a flexible fruit adaptive initial grasping force. Background Art

[0002] With the global population growth and rising agricultural labor costs, traditional manual fruit picking can no longer meet the needs of large-scale fruit cultivation. In addition, the fragility of fruit skin and the diversity of shapes make it difficult for rigid manipulators to adapt, so flexible grasping technology came into being.

[0003] In the field of fruit grasping, how to achieve stable grasping of fruits and vegetables of different shapes, sizes and hardness by accurately controlling the size and direction of the grasping force while taking into account the manufacturing and operating costs has become the focus of field applications. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a system and method for estimating the adaptive initial grasping force of flexible fruits. The present invention only collects data through machine vision to complete the estimation of the adaptive initial grasping force of flexible fruits.

[0005] To achieve the above object, the present invention provides the following technical solution: a flexible fruit adaptive initial grasping force estimation system, comprising:

[0006] A visual module, which is fixedly installed and collects image data of the fruit to be grasped;

[0007] A data processing module, wherein the data processing module uses an image processing algorithm to obtain a contour feature c of the fruit to be grasped and predicts a deformation variable ε when the flexible claw grasps, and uses a neural network model algorithm to obtain an estimated hardness h of the fruit to be grasped, wherein the contour feature c is represented with the flexible claw base coordinate system as a reference system;

[0008] The force estimation module calculates the basic force value F0=f(ε) of the flexible claw grasping based on the predicted deformation variable ε of the flexible claw during grasping and the "deformation-force" mapping equation f of the flexible claw, calculates the flexible claw grasping pre-tightening force value dF=g(h) based on the estimated hardness h of the fruit to be grasped and the "fruit hardness-flexible claw grasping force" mapping equation g, and finally calculates the initial grasping force estimate F=F0+dF=f(ε)+g(h) of the current fruit.

[0009] In particular, the data processing module also calculates the center position coordinates of the flexible claw when grasping the fruit according to the contour feature c of the fruit to be grasped. , where x, y represent the horizontal and vertical coordinates of the center position of the flexible claw when grasping, xi, yi represent the horizontal and vertical coordinates of each feature point on the fruit contour, i represents the i-th feature point, and n represents the total number of feature points on the contour. Finally, the flexible claw grasps the fruit based on L(x, y) and F value.

[0010] Preferably, the data processing module also calculates the best posture L_best when the flexible claw grasps and the best fruit grasping position according to the contour feature c of the fruit to be grasped and the parameters of the flexible claw itself, specifically:

[0011] S301. Based on the fruit contour c, search for the straight line connecting the longest two points on the contour c, establish the x1 axis, search for the two contour tangent points A (xa, ya), B (xb, yb) farthest from the x1 axis parallel to the x1 axis, and establish The coordinate system P1 is the origin and the straight line parallel to the x1 axis and passing through the P0 point is the horizontal axis;

[0012] S302. The best posture L_best is the position where the flexible claw coordinate system coincides with the P1 coordinate system;

[0013] S303. The best fruit grabbing positions are the locations of points A and B.

[0014] In particular, the predicted deformation amount ε of the flexible claw during grasping is: when the flexible claw is at position P0, the deformation amount ε of the flexible claw corresponding to the contraction of the flexible claw when the flexible claw just contacts the optimal grasping position.

[0015] Furthermore, the data processing module obtains the estimated hardness h of the fruit to be grasped, specifically:

[0016] S501. Using a neural network model to classify the image of the fruit to be captured, and obtain the category t of the fruit to be captured;

[0017] S502. Using a neural network model to extract features based on the fruit image to be captured and using a regression algorithm to obtain the current maturity of the fruit r;

[0018] S503. Based on the category t of the fruit to be grabbed, the maturity r and the "maturity - hardness" mapping equation s of different fruits, the hardness of the current fruit is calculated h = s (t, r);

[0019] Preferably, the neural network model for obtaining the estimated hardness h of the fruit to be grasped is: using a convolutional neural network CNN to perform feature extraction, and finally outputting N+1 dimensional data, wherein N represents the number of soft fruit types that the model can identify, completing the S501 classification task, 1 represents the corresponding fruit maturity, completing the S502 maturity r prediction task, and the model loss function , where Lcls and Lripe represent the category loss and maturity regression loss, α and β are weight parameters, zk represents the true label of the kth dimension in N dimensions, pk represents the model output of the kth dimension in N dimensions, and rpre and rreal represent the fruit maturity output value and the true value of the model.

[0020] Furthermore, the present invention also provides a method for estimating an adaptive initial grasping force of flexible fruits. The initial grasping force estimation method is based on image data of fruits to be grasped, uses an image processing algorithm to obtain contour features c of the fruits to be grasped, and obtains a basic grasping force value F0 of the flexible claw based on the fruit contour c, uses a neural network model algorithm to obtain an estimated hardness h of the fruits to be grasped, and obtains a flexible claw grasping preload force value dF based on the estimated hardness h, and finally calculates an estimated value F=F0+dF of the initial grasping force of the current fruit.

[0021] Preferably, the initial grasping force estimation method uses percentage relative coordinates to convert pixel coordinates in the image data into actual coordinates with the flexible claw base coordinate system as a reference system.

[0022] Compared with the prior art, the present invention has the following beneficial effects:

[0023] 1. The present invention estimates the initial grasping force of flexible fruits by only collecting data through a visual camera, without the need to arrange a large number of sensors like traditional mechanical claws to collect diverse data to complete the calculation, and the manufacturing and application costs are low.

[0024] 2. Based on the prior knowledge of industry applications, the present invention designs a CNN-based fruit classification and maturity regression neural network model. Since the image classification and regression tasks are relatively simple, the model can not only process classification and regression problems end-to-end, but also has the characteristics of simple structure. Therefore, under the premise of meeting the accuracy requirements, the computer computing power requirements are relatively low. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a schematic diagram of the overall framework of the system of the present invention;

[0026] Figure 2 It is a schematic diagram of the principle of conversion from image coordinates to actual coordinates of the present invention;

[0027] Figure 3 This is a schematic diagram of the result of extracting contour features through image processing according to the present invention;

[0028] Figure 4 It is a schematic diagram of the structure of the fruit classification and maturity regression neural network model of the present invention. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] like Figure 1 As shown, the flexible fruit adaptive initial grasping force estimation system of the present invention is mainly used for grasping common flexible fruits, and is suitable for the automated harvesting and sorting of fragile fruits such as strawberries, bananas, peaches, etc. The flexible claws generally adopt a two-claw form, and the material is usually a flexible material such as silicone and rubber. It includes three modules: a visual module, a data processing module and a force estimation module. The visual module collects image data of the fruit to be grasped, the data processing module processes the image data and obtains the feature information required for force estimation, and the force estimation module adaptively estimates the size of the initial grasping force of the fruit according to the type and maturity of the current fruit. The data processing module includes an image processing algorithm, a model and model parameters to meet the larger computing power application requirements in the system.

[0031] (1) Vision module

[0032] The vision module is fixedly installed on the fruit grasping production line to avoid the need to compensate for the position changes of the flexible claw and correct the image in real time when following the movement of the flexible claw, while increasing the load on the flexible claw and affecting the grasping sensitivity.

[0033] After the visual module is installed, it needs to be calibrated to determine the lens installation position, lens field of view angle, and flexible claw base installation position, and to match the lens scene with the actual scene. The width and height of the actual scene corresponding to the lens scene can be measured, so that the relative coordinate method can be used to complete the conversion of the pixel coordinates of the fruit position based on the visual lens in the image to the coordinates of the flexible claw base based on the actual position, such as Figure 2 As shown, assuming that the width and height of the image collected by the visual module are Wp and Hp, the unit pixel, the corresponding actual scene width and height are Wr and Hr, the offset from the visual lens to the flexible claw base along the production line direction is dx, and the offset in the vertical production line direction is dy, then the mapping relationship between the target fruit center relative to the upper left corner point (Xp, Yp) in the image and the coordinates (Xr, Yr) of the flexible claw base coordinates is: At the same time, the height of the fruit grasping production line from the flexible claw base is measured. When discussing the grasping of fruits in three-dimensional space, only the positional relationship between the targets on the horizontal plane needs to be discussed, which reduces the computational complexity.

[0034] (2) Data processing module

[0035] The data processing module completes the conversion from image unstructured data to mathematical structured data, completing two major functions.

[0036] (2.1) Extraction of the contour c of the fruit image data to be captured

[0037] In practical applications, the background of the fruit grabbing production line is simple, and the threshold segmentation method can be used to achieve the segmentation of the fruit to be grabbed, such as Figure 3 As shown, the image of the fruit to be grasped is binarized, and the contour line of the fruit is obtained by using the contour tracking algorithm. The best posture L_best position and the best grasping point of the fruit are obtained based on the longest distance connection line in two directions.

[0038] At the same time, the flexible claw shape variable ε is predicted when the flexible claw contacts the best grasping point at the L_best position. In order to simplify the calculation and minimize the use of sensors, the flexible claw grasping height is selected to be close to the production line, and the three-dimensional space problem is converted into a two-dimensional space problem without considering the height. When the flexible claw contracts to the bottom to reach the best grasping point, it is considered to have reached the ε deformation position. At this time, for smaller fruits, the middle wide part of the flexible claw may not contact the fruit (in fact, the size of the flexible claw is not suitable). After a certain force is added to form the final grasping force, due to the small size of the fruit, the middle wide part of the flexible claw is likely to be clamped. When it is not clamped, the fruit will move downward when it is picked up and will eventually be grasped by the bottom claw. Due to the small size of the fruit, its quality is also low, and the fruit is unlikely to fall.

[0039] As a simple implementation method of this example, the average value can be directly calculated based on the obtained fruit contour line point coordinates to obtain the center of gravity position of the fruit horizontal contour line, that is, , when grasping, the fruit is grasped directly based on the L(x,y) position and the grasping force F value.

[0040] (2.2) Type of fruit to be picked and determination of hardness

[0041] Based on the collected fruit image data, a neural network model is established, such as Figure 4 Considering the prior knowledge, the hardness of different fruits will change at different maturity stages, as shown in the following table:

[0042] Table of hardness changes of common soft fruits (hardness unit: kg / cm 2 )

[0043] fruit Immature stage (high hardness) Maturity (hardness decreases) Overripe period (very low hardness) Key maturity period reference strawberry 2-3 (green fruit) 1-1.5 (red fruit) Perishable Field strawberries: late May banana 4-6 (green skin) 1-3 (yellow skin) Pasty (dark spot stage) Spring banana planting: November-December peach 8-12 4-6 ＜ 3 Mid-June to early October

[0044] As shown in the table above, the maturity period of common soft fruits is quantified. In actual production, over-ripe soft fruits are generally not involved in production. Therefore, only the immature-mature process of fruits is quantified. The quantification range is 0-1, where 0 represents completely immature and 1 represents fully mature. Different maturity levels are represented by floating-point numbers in the range of 0-1. Correspondingly, the hardness of fruits at different maturity stages can be filled in by linear interpolation. The mathematical formula expression is a linear equation. For example, if the category is strawberry (t=strawberry), the mapping relationship s between maturity r and hardness h is h=-2*r+3.

[0045] Based on this, the neural network model to be established can indirectly judge the hardness of the fruit by judging the fruit category and maturity. By having a stronger correlation between the fruit appearance and the fruit maturity, the difficulty of model reasoning is reduced and the reasoning accuracy is improved. At the same time, when business needs require the grading and classification of fruits, the fruits can be graded directly according to different maturity levels.

[0046] Therefore, a neural network model can be established with an architecture such as Figure 4 As shown in the figure, the entire neural network architecture can be divided into three layers: feature extraction layer, classification judgment layer, and maturity regression layer. The feature extraction layer uses convolutional neural network CNN to obtain image feature information, and the feature information is finally given to the last layer (output layer) of the model. The number of neural nodes in the last layer is N+1, where the first N neural nodes constitute the classification judgment layer, N represents the number of categories of soft fruits (such as N=3), and the last node constitutes the maturity regression layer, which regresses the maturity floating point number of the current category of fruit. When training the model, the classification and regression loss functions L are constructed separately. cls , L ripe , where the classification loss can be defined using the cross entropy loss function, formula: , the regression loss can be defined using the mean absolute error loss, formula: , since the maturity value range is [0, 1], the maturity regression loss has the same dimension as the fruit classification loss, and the final model loss Loss can be directly merged. , α and β are weight parameters (such as α=β= 0.5). Finally, the Loss is continuously reduced through optimization methods such as gradient descent, and the parameters of the neural network model are optimized through back propagation to complete the classification of soft fruits and the calculation of maturity values.

[0047] It should be further explained that since the task types of the neural network of the present invention are: image classification and data regression, the task types are simple, so the early feature extraction network layer can select simpler networks such as mobilenet, shufflenet, ResNet, etc. for fine-tuning. In this embodiment, training is based on ResNet and Fruit-Dataset fruit data set, and the final model classification accuracy is greater than 90%, and the relative absolute error of regression is less than 0.1.

[0048] (3) Force estimation module

[0049] (3.1) Based on the contour feature c of the fruit, calculate the basic force value F0 of the flexible claw grasping

[0050] a. The optimal posture position L_best of the flexible claw, the optimal grasping points A and B, and the predicted contraction deformation ε of the flexible claw calculated by the data processing module.

[0051] b. A mapping equation f between the contraction amount ε of the flexible claw and the force F_ε applied to the flexible claw is established according to the parameters of the flexible claw itself. In applications, the flexible claw can be pneumatic, hydraulic or electric.

[0052] c. According to the mapping equation f and the contraction deformation variable ε of the flexible claw, the basic grasping force value F0=f(ε) is directly calculated.

[0053] (3.2) Calculate the preload force dF of the flexible claw based on the fruit hardness h

[0054] According to the specifications of the selected flexible claw, establish the fruit hardness h and the gripping force equation g(h) of the flexible claw to reach this hardness, such as considering the minimum contact area of ​​1mm 2 ,g(h)=0.098*h, unit: Newton.

[0055] (3.3) Calculate the estimated value F of the initial grasping force of the final fruit: F = F0 + dF = f(ε) + g(h).

[0056] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A flexible fruit adaptive initial grasping force estimation system, characterized in that: The estimation system comprises: A visual module, which is fixedly installed and collects image data of the fruit to be grasped; A data processing module, wherein the data processing module uses an image processing algorithm to obtain a contour feature c of the fruit to be grasped and predicts a deformation variable ε when the flexible claw grasps, and uses a neural network model algorithm to obtain an estimated hardness h of the fruit to be grasped, wherein the contour feature c is represented with the flexible claw base coordinate system as a reference system; The force estimation module calculates the basic force value F0=f(ε) of the flexible claw grasping based on the predicted deformation variable ε of the flexible claw during grasping and the "deformation-force" mapping equation f of the flexible claw, calculates the flexible claw grasping pre-tightening force value dF=g(h) based on the estimated hardness h of the fruit to be grasped and the "fruit hardness-flexible claw grasping force" mapping equation g, and finally calculates the initial grasping force estimate F=F0+dF=f(ε)+g(h) of the current fruit.

2. A flexible fruit adaptive initial grasping force estimation system according to claim 1, characterized in that: The data processing module also calculates the center position coordinates of the flexible claw when grasping the fruit according to the contour feature c of the fruit to be grasped. , where x and y represent the horizontal and vertical coordinates of the center position of the flexible claw when grasping. i , y i represents the horizontal and vertical coordinates of each feature point on the fruit contour, i represents the i-th feature point, n represents the total number of feature points on the contour, and finally the flexible claw grasps the fruit based on L(x, y) and F value.

3. The flexible fruit adaptive initial grasping force estimation system according to claim 1, characterized in that: The data processing module also calculates the best posture L_best and the best fruit grabbing position of the flexible claw when grabbing according to the contour feature c of the fruit to be grabbed and the parameters of the flexible claw itself, specifically: S301. Based on the fruit contour c, search for the straight line connecting the longest two points on the contour c, establish the x1 axis, search for the two contour tangent points A (xa, ya), B (xb, yb) farthest from the x1 axis parallel to the x1 axis, and establish The coordinate system P1 is the origin and the straight line parallel to the x1 axis and passing through the P0 point is the horizontal axis; S302. The best posture L_best is the position where the flexible claw coordinate system coincides with the P1 coordinate system; S303. The best fruit grabbing positions are the locations of points A and B.

4. A flexible fruit adaptive initial grasping force estimation system according to claim 3, characterized in that: The predicted deformation variable ε of the flexible claw during grasping is: the deformation variable ε of the flexible claw corresponding to the contraction of the flexible claw when the flexible claw is at position P0 and the flexible claw just contacts the optimal grasping position.

5. The flexible fruit adaptive initial grasping force estimation system according to claim 1, characterized in that: The data processing module obtains the estimated hardness h of the fruit to be grasped, specifically: S501. Using a neural network model to classify the image of the fruit to be captured, and obtain the category t of the fruit to be captured; S502. Using a neural network model to extract features based on the fruit image to be captured and using a regression algorithm to obtain the current maturity of the fruit r; S503. Based on the category t of the fruit to be grabbed, the maturity r and the "maturity-hardness" mapping equation s of different fruits, the hardness of the current fruit is calculated h = s (t, r).

6. A flexible fruit adaptive initial grasping force estimation system according to claim 5, characterized in that: The neural network model for obtaining the estimated hardness h of the fruit to be grasped is: using a convolutional neural network CNN to extract features and finally output N+1 dimensional data, where N represents the number of soft fruit types that the model can identify, completing the S501 classification task, 1 represents the corresponding fruit maturity, completing the S502 maturity r prediction task, and the model loss function , where L cls , L ripe represents the category loss and maturity regression loss, α and β are weight parameters, and z k represents the true label of the kth dimension in N dimensions, p k represents the model output of the kth dimension in N dimensions, r pre 、r real Represents the model's output value of fruit maturity and the true value.

7. A method for estimating the adaptive initial grasping force of flexible fruit, the method for estimating the adaptive initial grasping force of flexible fruit being applied to the system for estimating the adaptive initial grasping force of flexible fruit according to any one of claims 1 to 6, characterized in that: The initial grasping force estimation method is based on the image data of the fruit to be grasped, adopts an image processing algorithm to obtain the contour feature c of the fruit to be grasped and obtains the flexible claw grasping basic force value F0 based on the fruit contour c, adopts a neural network model algorithm to obtain the estimated hardness h of the fruit to be grasped and obtains the flexible claw grasping pre-tightening force value dF based on the estimated hardness h, and finally calculates the current fruit initial grasping force estimation value F=F0+dF.

8. The method for estimating the adaptive initial grasping force of flexible fruit according to claim 7, characterized in that: The initial grasping force estimation method uses a percentage relative coordinate method to convert pixel coordinates in the image data into actual coordinates with the flexible claw base coordinate system as a reference system.

Images