Vision and touch integrated sensing equipment for fruit maturity analysis and analysis method

Abstract

The invention provides vision and touch integrated sensing equipment for fruit maturity analysis and an analysis method. The system comprises a mechanical arm, a manipulator, a binocular camera, a PDMS film, 199UV stamp-pad ink, an ultraviolet lamp, a light source and a controller. A visual touch integrated sensing device is built; whether the maturity of the fruit is suitable for being judged through the color of the outer skin or not is judged through a fruit recognition and positioning module, and three-dimensional space positioning is conducted on the fruit; if yes, the controller executes a visual detection method; on the contrary, the PDMS film is adopted to simulate the human palm, the ultraviolet lamp is turned on, the invisible speckles on the film are shown, the binocular camera replaces the human eyes, the controller controls the mechanical arm to press the fruits on the film, the displacement process of the speckles on the surface of the film is recorded, and the digital image related technology is utilized. Tactile information sensed by a palm is shown through displacement of random speckles, finally maturity classification of fruits such as tomatoes is achieved, the structure is simple and compact, work is stable, and the classification accuracy is high.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent agriculture, and specifically relates to a visual-touch integrated sensing device and an analysis method for fruit maturity analysis. Background Art

[0002] Fruit maturity is an important indicator affecting fruit quality. Traditional fruit maturity detection methods mainly rely on manual experience or chemical analysis, and have many disadvantages, such as strong subjectivity, low efficiency, and high misjudgment rate. Destructive testing technology requires the destruction of the fruit, which is inefficient, costly, and difficult to achieve long-term storage. Nowadays, non-destructive testing technology is developing day by day, and most of them are aimed at measuring certain characteristics of the fruit's outer skin color, smell or texture, such as electroacoustic method, machine vision method, electronic nose or infrared spectroscopy. The latter two devices are expensive, and the machine vision method has a single judgment condition, resulting in low accuracy. Summary of the invention

[0003] In view of the above problems, the present invention proposes a visual-touch integrated sensing device for fruit maturity analysis. Compared with a single detection method, when visual detection cannot successfully determine the degree of maturity, a tactile method is used to perform tactile determination of the fruit maturity, simulating the soft and hard information perceived by the human palm. The operation is simple and the judgment accuracy is high, which saves manpower and ensures the accuracy of fruit maturity determination.

[0004] The present invention uses a controller to control a binocular camera to use a fruit recognition system to identify the type of fruit, and combines a fruit positioning module to spatially locate the fruit to determine whether it is suitable to use a visual method to determine the maturity; if the outer skin of the fruit presents different colors with different maturity stages, it is suitable, otherwise it is not suitable, and a tactile method is used, in which the controller controls the magnitude of the applied force, a film is used to simulate a human palm, and the fruit is slowly pressed on the film, and the maturity level is indirectly determined according to the deformation displacement information of the speckle on the surface of the film, so as to achieve effective judgment of the maturity of fruits such as peaches, and the structure is simple and compact, the operation is stable, and the judgment accuracy is high.

[0005] The present invention achieves the above technical objectives through the following technical means.

[0006] A visual-touch integrated sensing device for fruit maturity analysis, comprising a robotic arm, a robotic hand, a binocular camera, a PDMS film, 199UV printing ink, an ultraviolet lamp, a light source and a controller;

[0007] The 199UV printing ink is sprayed evenly on the PDMS film. When the tactile system is used for judgment, the ultraviolet lamp is turned on and the image is collected by the binocular camera.

[0008] The ultraviolet lamps and light sources are arranged in an interlaced manner on the inner wall of the sensor to form a circle, with a total of 8.

[0009] Each joint of the mechanical arm and the manipulator is provided with a motor for rotation, and the motor is connected to the controller; the binocular camera, the ultraviolet lamp, and the light source are also connected to the controller;

[0010] The binocular camera uses a fruit identification and positioning system to identify the type of fruit and locate the location of the fruit to determine whether it is suitable to use a visual method to determine the maturity; if the outer skin of the fruit presents different colors at different maturity stages, it is suitable, otherwise it is not suitable; then a tactile method is used to indirectly determine the maturity level.

[0011] In the above scheme, the binocular camera uses the fruit recognition system to identify the type of fruit and determine whether it is suitable to use visual methods to judge the maturity; if the outer skin of the fruit presents different colors with different maturity stages, it is applicable, otherwise it is not applicable; then a tactile method is used, and the robot applies different forces to slowly press the fruit on the film. According to the deformation displacement information of the speckle on the surface of the film, an SVM classifier is constructed to indirectly judge the maturity level.

[0012] The PDMS film with 199UV speckles is fixed on the palm of the robot gripper, and a binocular camera is fixed inside the robot arm at a vertical position of about 20 cm away from the film. Eight light sources are fixed in a circle between the binocular camera and the film, that is, on the inner wall of the robot arm, in an interlaced manner, with four ultraviolet lamps and four light sources.

[0013] The binocular camera is used to collect the color of the outer skin of the fruit under the visual method to judge the maturity and the deformation of the speckle on the surface of the film after squeezing under the tactile method.

[0014] Further, the controller includes a fruit recognition module, a fruit positioning module, a visual detection maturity module, a 3D-DIC module and a maturity discrimination module under a tactile system;

[0015] The fruit recognition module is used to identify and determine whether the fruit is suitable for judging the maturity by visual detection methods. Images with different maturity are tracked and collected, and the annotation data is revised to improve the model's detection ability for fruits of adjacent maturity. In this module, after collecting new category image data, the fruit and vegetable positioning is saved to the known category library to complete the new category function. The features of the fruit and vegetable image data to be identified are extracted, and the features of the fruit and vegetable image data in the known category library are extracted and saved in the feature vector library. The extracted fruit and vegetable features to be identified are compared with the vectors in the feature vector library for similarity, and the feature vector with the highest feature similarity is obtained and its category is returned.

[0016] The fruit positioning module is used to accurately identify and locate the position of the fruit, so that the robot arm can accurately approach and grab the fruit. The controller controls the binocular camera to capture the depth image of the fruit, generates the distance from each pixel to the camera, uses Gaussian filtering to remove noise and erroneous data points, combines the color information in the RGB image with the information in the depth image, and generates color data; uses the point cloud segmentation algorithm to separate the fruit from the background and identify the point cloud where the fruit is located; calculates the position of the fruit in three-dimensional space, and uses the center of mass of the fruit as the three-dimensional coordinate of the fruit; finally, determines the three-dimensional position coordinates of the fruit in the robot arm workspace through coordinate conversion.

[0017] The visual detection maturity module is used to track and collect partial images with different degrees of maturity, and revise the annotation data to improve the model's detection ability for fruits of adjacent maturity. In this module, after collecting new category image data, it is saved in the known category library after fruit and vegetable positioning, and the new category function can be completed. The features of the fruit and vegetable image data to be identified are extracted, and the features of the fruit and vegetable image data in the known category library are extracted and saved in the feature vector library. The extracted features of the fruit and vegetable to be identified are compared with the vectors in the feature vector library for similarity, and the feature vector with the highest feature similarity is obtained and its category is returned.

[0018] The 3D-DIC module is used to use a binocular camera to collect images before and after speckle deformation, select an ROI area in the image before deformation as a template, use a normalized cross-correlation coefficient in the image after deformation to judge the matching degree between the template and each position of the image after deformation, determine the matching points, calculate the transformation relationship before and after deformation through the matching points, and then combine the internal and external parameters of the camera to use a triangulation method to convert the image coordinates into coordinates in a three-dimensional space, calculate the three-dimensional displacement of the points before and after deformation, thereby obtaining a three-dimensional deformation field, and then realizing three-dimensional reconstruction of the surface to be measured.

[0019] The maturity discrimination module under the tactile system is used to construct a maturity classifier using the SVM algorithm. The relationship between all applied forces and speckle shape variables is collected and plotted into a curve. The curves fitted by the three different maturity fruits in the sample set are re-fitted into three new curves, namely "unripe", "ripe" and "overripe". The maturity state of the fruits in the test set is judged by the three characterization curves obtained, and the classification interval is maximized by minimizing the norm of the weight vector. In order to consider a certain degree of misclassification tolerance, a slack variable is introduced to make the model more robust.

[0020] An analysis method of a visual-touch integrated sensing device for fruit maturity analysis comprises the following steps:

[0021] Fruit recognition: The binocular camera collects color images of the fruit for image processing, identifies the fruit type, and determines whether it can be judged based on the different ripening colors of the fruit skin at different maturity stages.

[0022] Fruit positioning: The binocular camera collects the depth information of the fruit. After denoising, the color information in the RGB image is combined with the information in the depth image to generate color data. The DBSCAN point cloud segmentation algorithm is used to separate the fruit from the background, and the coordinates of the fruit in three-dimensional space are calculated. The external parameter calibration technology is used to determine the relative position and posture between the camera and the robotic arm. The coordinate conversion is performed through the secondary transformation matrix, and the three-dimensional coordinates of the fruit in the robotic arm workspace are output.

[0023] Visual detection of maturity: The controller will be able to process images of fruits that can be visually detected according to the color of the outer skin, and revise the labeled data to improve the model's ability to detect adjacent mature fruits. Then, the backbone feature extraction network of Yolo-v7 is replaced with a lightweight network MobileNetV3, which greatly reduces the number of parameters and calculations of the model. 1×1 convolution is used for dimensionality increase, and the inverse residual structure of the linear bottleneck is introduced. Then, 3×3 depth-separable convolution is performed, and the weights between channels are adjusted through the attention model SE-Net. Finally, the swish function is modified to reduce the amount of calculation. The GAM attention module is introduced in the feature fusion module to enhance the feature expression ability of the network, and the sequential channel-spatial attention mechanism in CBAM is used to redesign the submodule; the channel attention submodule uses 3D combination to retain information across three dimensions, and uses a multi-layer perceptron to allow feature vectors to fully interact between different dimensions, so that the model obtains more nonlinear features and combined feature information, reduces the impact of the decline in detection accuracy caused by the small number of model parameters, and improves the detection accuracy of adjacent mature fruits and occluded fruits.

[0024] 3D-DIC: The controller slowly applies pressure to the fruits that cannot be successfully visually inspected through the manipulator, and turns on the ultraviolet lamp at the same time. The manipulator controls the force at a uniform speed until it reaches 10N. The binocular camera simultaneously captures the fluorescent spots of invisible ink under the irradiation of the ultraviolet lamp. Ten images are taken to record the degree of change of the film during the pressing process of fruits of different maturity and the displacement of the surface spots. The ROI area is selected as the template in the image before deformation. The template size is determined according to the characteristics of the target area. A search area is defined in the image after deformation, and the normalized correlation coefficient (C cc ) is used to judge the matching degree between the template and each position in the search area of ​​the deformed image, C ccThe value is between -1 and 1, and the closer it is to 1, the higher the matching degree. After determining the matching points, the transformation relationship before and after deformation is calculated through the matching points. Then, combined with the internal and external parameters of the camera, the image coordinates are converted into coordinates in three-dimensional space using the triangulation method, and the three-dimensional displacement of the points before and after deformation is calculated to obtain the three-dimensional deformation field, thereby realizing the three-dimensional reconstruction of the surface to be measured.

[0025] Maturity discrimination under the tactile system: The controller obtains the speckle displacement-pressure curves of different fruits based on the images taken by the binocular camera after analysis by the DIC algorithm module. The data set is divided into a training set and a test set in a ratio of 3:1. An SVM classifier is constructed to judge the maturity level. The classification interval is maximized by minimizing the norm of the weight vector. In order to consider a certain degree of misclassification tolerance, a slack variable is introduced to make the model more robust.

[0026] In the above scheme, the fruit identification module includes the following steps:

[0027] Image acquisition: track and collect some images with different degrees of maturity, and revise the annotation data to improve the model's ability to detect fruits of adjacent maturity. Fruit positioning: select the main body of fruits and vegetables in the picture, which can filter out unnecessary background interference and lay the foundation for subsequent feature extraction. Add new categories: collect new category image data and save it to the known category library after fruit positioning to complete the function of adding new categories. Feature extraction: extract the features of the fruit image data to be identified, and at the same time extract the features of the fruit image data in the known category library and save them in the feature vector library. Compare the similarity between the extracted features of the fruit to be identified and the vectors in the feature vector library, obtain the feature vector with the highest feature similarity and return its category.

[0028] In the above scheme, the fruit positioning module includes the following steps:

[0029] The controller controls the binocular camera to capture the depth image of the fruit, generates the distance from each pixel to the camera, uses Gaussian filtering to remove noise and erroneous data points, combines the color information in the RGB image with the information in the depth image, and generates color data; uses the DBSCAN point cloud segmentation algorithm to separate the fruit from the background and identify the point cloud where the fruit is located; calculates the position of the fruit in three-dimensional space, and uses the center of mass of the fruit as the three-dimensional coordinate of the fruit; finally, determines the three-dimensional position coordinates of the fruit in the robotic arm workspace through coordinate transformation.

[0030] In the above scheme, the visual detection maturity module includes the following steps:

[0031] The fruit images were collected using a binocular camera and image preprocessing was performed, including denoising, image enhancement and resizing, to improve the accuracy and efficiency of subsequent detection. The preprocessed images were feature extracted using a lightweight backbone network, and multi-level feature maps were extracted to capture the feature information of the fruit at different maturity levels. 1×1 convolution was used for dimensionality increase, and the inverse residual structure of the linear bottleneck was introduced. Then, 3×3 depth-separable convolution was performed, and the weights between channels were adjusted through the attention model SE-Net. Finally, the swish function was modified to reduce the amount of calculation. The GAM attention mechanism was introduced in the feature fusion stage of the YOLO v7 network, and the sequential channel-spatial attention mechanism in CBAM was used to redesign the submodules; the channel attention submodule used 3D combination to retain information across three dimensions, and a multi-layer perceptron was used to allow feature vectors to fully interact between different dimensions. The improved network model was used to detect targets on the feature maps after feature fusion. The detection results were post-processed to screen out detection boxes with high confidence. Indicators such as detection accuracy, recall rate and average detection time were calculated to evaluate the model performance. The detection results are output, including the detection box and category probability of the fruit maturity classification output, which are used to identify the maturity state of the fruit. The feature fusion effect is enhanced through the global attention mechanism, and the interactive features of the global dimension are fully utilized to improve the expression ability of the model.

[0032] In the above solution, the 3D-DIC module includes the following steps:

[0033] Use a binocular camera to capture speckle images on the film surface, use Gaussian filtering to remove noise from the image and enhance image quality, use a feature matching algorithm to find corresponding feature point pairs in images captured under different pressures, and further fine-tune the image. Divide the reference image into multiple small areas, and search for the corresponding area of ​​each sub-image in the deformed image. Use bilinear interpolation to improve the accuracy of displacement calculation to the sub-pixel level. Calculate the disparity map between cameras through a stereo matching algorithm, extract depth information from the disparity map, and obtain the three-dimensional shape of the fruit surface based on the disparity map and camera calibration parameters. Use three-dimensional point cloud data to analyze the deformation field and displacement field of the fruit surface, calculate the correlation coefficient of each sub-image, find the position of the maximum correlation coefficient, and determine the displacement vector.

[0034] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention adopts a visual-touch integrated detection method, and through image recognition of the fruit color image collected by the binocular camera, it is determined whether to use a visual detection method or a tactile detection method. If it is a visual detection, the fruits of different maturity are tracked and collected, and the backbone feature extraction network of yolo-v7 is replaced with a lightweight network, and the GAM attention module is introduced to reduce the factors affecting the detection accuracy. If it is a tactile detection, it is different from the situation in which the maturity is judged by the color of the outer skin in visual detection. The displacement change of the invisible speckle on the surface of the film captured by the binocular camera is combined with digital image correlation technology. When the force of different sizes is applied, the degree of squeezing of the film by soft and hard fruits is different, thereby calculating the three-dimensional displacement of the speckle and indirectly judging the maturity of the fruit. The method realizes the non-destructive detection of the maturity of the fruit, which is not only suitable for judging whether the fruit is suitable for picking before picking, reducing labor and speeding up the picking speed, but also suitable for food processing and supply chain management after picking, to control the quality of the produced products, and better arrange procurement and sales plans. It solves the problem of misjudgment due to skin color in traditional visual inspection, improves detection accuracy, has a simple and compact structure and is easy to operate.

[0035] Note that the description of these effects does not prevent the existence of other effects. One embodiment of the present invention does not necessarily have all of the above effects. Effects other than the above can be clearly seen and extracted from the description of the specification, drawings, claims, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a three-dimensional structural diagram of an apparatus according to one embodiment of the present invention.

[0037] Figure 2 It is a detailed structural diagram of a manipulator according to one embodiment of the present invention.

[0038] Figure 3 It is a detailed internal structure diagram of a manipulator according to one embodiment of the present invention.

[0039] Figure 4 It is an overall flow chart of fruit maturity analysis according to one embodiment of the present invention.

[0040] Figure 5 It is a fruit identification flow chart of one embodiment of the present invention.

[0041] Figure 6 It is a fruit positioning flow chart of one embodiment of the present invention.

[0042] Figure 7 1 is a flow chart of visually detecting maturity according to one embodiment of the present invention.

[0043] Figure 8This is a flow chart of tactile detection maturity according to one embodiment of the present invention.

[0044] In the figure: 1-upper arm, 2-middle arm, 3-lower arm, 4-manipulator, 5-conveyor belt, 6-PDMS film, 7-speckle sprayed by 199UV printing ink, 8-ultraviolet lamp, 9-light source, 10-binocular camera. DETAILED DESCRIPTION

[0045] Figure 1 The figure shows a preferred embodiment of the visual-touch integrated sensing device for fruit maturity analysis. The sensor for fruit maturity analysis includes a robotic arm, a robotic hand, a binocular camera, a PDMS film, 199UV printing ink, an ultraviolet lamp, a light source and a controller.

[0046] The 199UV printing ink is sprayed evenly on the PDMS film. When the tactile system is used for judgment, the ultraviolet lamp is turned on and the image is collected by the binocular camera.

[0047] Each joint of the mechanical arm and the manipulator is provided with a motor for rotation, and the motor is connected to the controller; the binocular camera, the ultraviolet lamp, and the light source are also connected to the controller;

[0048] The binocular camera uses a fruit recognition system to identify the type of fruit and determine whether it is suitable to use a visual method to judge the maturity; if the outer skin of the fruit presents different colors at different maturity stages, it is suitable, otherwise it is not suitable; then a tactile method is used, and the controller controls the magnitude of the applied force to slowly and evenly press the fruit on the film, and indirectly judge the maturity level based on the deformation displacement information of the speckle on the surface of the film.

[0049] According to this embodiment, preferably, the color image of the fruit collected by the binocular camera is subjected to image recognition and positioning, and the fruit is judged to be suitable for visual inspection or tactile inspection, and the fruit is positioned; if it is visual inspection, the inspection is performed based on the standard that the outer skin color of the fruit with different maturity is different. If it is tactile inspection, different from the situation in which the maturity is judged by the outer skin color in visual inspection, the displacement change of the invisible speckle on the surface of the film captured by the binocular camera is combined with digital image correlation technology. When different forces are applied, the degree of squeezing of the film by soft and hard fruits is different, so the three-dimensional displacement of the speckle is calculated to indirectly judge the maturity of the fruit.

[0050] According to this embodiment, preferably, the RGB frame resolution of the binocular camera is 4416*1242.

[0051] According to this embodiment, preferably, the visual-touch integrated sensing device includes an upper arm 1, a middle arm 2, a lower arm 3, a manipulator 4, a conveyor belt 5, a transparent PDMS film 6, invisible speckles 7 sprayed with 199UV printing ink, an ultraviolet lamp 8, a light source 9, a binocular camera 10 and a controller.

[0052] When the visual-touch integrated sensing device judges the maturity of the fruit, there are two maturity detection methods. One is to judge directly based on the color of the outer skin, and the other is to apply different forces to the fruit through a robot arm, record the deformation of the speckle before and after captured by the binocular camera, and indirectly judge the maturity.

[0053] According to this embodiment, preferably, the analysis method of the visual-touch integrated sensing device includes at least a fruit recognition module, a fruit positioning module, a visual maturity detection module, a 3D-DIC module, and a maturity discrimination module under the tactile system;

[0054] The fruit recognition module is used to determine whether the outer skin color of the fruit changes as the maturity increases during the ripening process.

[0055] The fruit positioning module is used to locate the three-dimensional position of the fruit relative to the working space of the robotic arm for subsequent path planning and robotic arm control.

[0056] The visual maturity detection module is used to track and collect partial images with different maturity levels to extract features of the fruit and vegetable image data to be identified, obtain the feature vector with the highest feature similarity and return its category.

[0057] The 3D-DIC module is used to use a binocular camera to collect images before and after speckle deformation, use a DIC algorithm to match feature points in the images before and after deformation, and then combine the internal and external parameters of the camera to use a triangulation method to convert the image coordinates into coordinates in a three-dimensional space, calculate the three-dimensional displacement of the points before and after deformation, thereby obtaining a three-dimensional deformation field, and then realizing three-dimensional reconstruction of the surface to be measured.

[0058] The maturity discrimination module under the tactile system is used to construct a maturity classifier based on SVM, and the displacement-pressure fitting curve is calculated to calculate the fitting rate, and the norm of the weight vector is minimized to achieve the maximum classification interval, and the relaxation variable is introduced to make the model more robust.

[0059] A visual-touch integrated sensing device and analysis method for fruit maturity analysis, comprising the following steps:

[0060] Fruit recognition: The binocular camera collects color images of the fruit for image processing, identifies the fruit type, and determines whether it can be judged based on the different ripening colors of the fruit skin at different maturity stages.

[0061] Visual detection of maturity: The controller will be able to process images of fruits that can be visually detected according to the color of the outer skin, and revise the labeled data to improve the model's ability to detect adjacent mature fruits. Then, the backbone feature extraction network of Yolo-v7 is replaced with a lightweight network MobileNetV3. First, 1×1 convolution is used for dimension increase, and the inverse residual structure of the linear bottleneck is introduced. Then, 3×3 depth-separable convolution is performed, and the weights between channels are adjusted through the attention model SE-Net. Finally, the swish function is modified to greatly reduce the number of parameters and calculations of the model. The GAM attention module is introduced in the feature fusion module to enhance the feature expression ability of the network. By adopting the sequential channel-spatial attention mechanism in CBAM, the submodule is redesigned. The channel attention submodule uses 3D combination to retain information across three dimensions, and uses a multi-layer perceptron to allow feature vectors to fully interact between different dimensions, so that the model obtains more nonlinear features and combined feature information, reduces the impact of the decline in detection accuracy caused by the small number of model parameters, and improves the detection accuracy of adjacent mature fruits and occluded fruits.

[0062] 3D-DIC: The controller slowly applies pressure to the fruits that cannot be successfully inspected by visual inspection, and turns on the UV lamp at the same time. The controller controls the manipulator to uniformly apply different forces up to 10N. The binocular camera simultaneously captures the fluorescent spots of invisible ink under the UV lamp. Ten images are taken to record the degree of change of the film during the pressing process of fruits of different maturity and the displacement of the surface spots. The ROI area is selected as the template in the image before deformation. The template size is determined according to the characteristics of the target area. A search area is defined in the image after deformation, and the normalized correlation coefficient (C cc ) is used to judge the matching degree between the template and each position in the search area of ​​the deformed image, C cc The value is between -1 and 1, and the closer it is to 1, the higher the matching degree. After determining the matching points, the transformation relationship before and after deformation is calculated through the matching points. Then, combined with the internal and external parameters of the camera, the image coordinates are converted into coordinates in three-dimensional space using the triangulation method, and the three-dimensional displacement of the points before and after deformation is calculated to obtain the three-dimensional deformation field, thereby realizing the three-dimensional reconstruction of the surface to be measured.

[0063] Maturity discrimination under the tactile system: The controller obtains the displacement-pressure curves of different fruits based on the images taken by the binocular camera after analysis by the DIC algorithm module. The data set is divided into a training set and a test set in a ratio of 3:1. An SVM classifier is constructed to judge the maturity level. The classification interval is maximized by minimizing the norm of the weight vector. In order to consider a certain degree of misclassification tolerance, a slack variable is introduced to make the model more robust.

[0064] In combination with 5, according to this embodiment, preferably, the fruit identification module includes the following process:

[0065] 1. The binocular camera collects color images of fruits of different maturity and uses Gaussian filtering to denoise the images;

[0066] 2. Track and collect some images with different maturity levels, and revise the annotation data to improve the model's ability to detect fruits of adjacent maturity levels.

[0067] 3. Add a new category. After collecting the image data of the new category, save it to the known category library after fruit positioning to complete the function of adding a new category.

[0068] 4. Feature extraction: extract the features of the fruit image data to be identified, and at the same time extract the features of the fruit image data in the known category library and save them into the feature vector library.

[0069] 5. Compare the extracted features of the fruit to be identified with the vectors in the feature vector library for similarity, obtain the feature vector with the highest feature similarity and return its category.

[0070] Combination Figure 6 As shown, according to this embodiment, preferably, the fruit positioning module includes the following processes:

[0071] 1. Use Gaussian filtering to remove noise and erroneous data points in the fruit depth image.

[0072] 2. Combine the color information in the RGB image with the information in the depth image to generate color data.

[0073] 3. Use the DBSCAN point cloud segmentation algorithm to separate the fruit from the background and identify the point cloud where the fruit is located.

[0074] 4. Calculate the position of the fruit in three-dimensional space and use the center of mass of the fruit as the three-dimensional coordinate of the fruit; finally, determine the three-dimensional position coordinates of the fruit in the working space of the robotic arm through coordinate transformation.

[0075] Combination Figure 7 As shown, according to this embodiment, preferably, the visual detection maturity module includes the following process:

[0076] 1. Use a binocular camera to collect fruit images and perform image preprocessing, including denoising, image enhancement, and size adjustment.

[0077] 2. Use a lightweight backbone network to perform feature extraction on the preprocessed image and extract multi-level feature maps to capture the characteristic information of fruits at different maturity levels.

[0078] 3. Introduce the GAM attention mechanism in the feature fusion stage of the network, and use the improved network model to perform target detection on the feature map after feature fusion.

[0079] 4. Post-process the detection results and filter out the detection frames with higher confidence. Output the detection results, including the fruit maturity classification output detection frame and category probability, which are used to identify the maturity state of the fruit.

[0080] Combination Figure 8 As shown, according to this embodiment, preferably, the tactile maturity detection method includes the following steps:

[0081] Select a reference point P0(x0,y0) on the initial image (i.e., the reference image), and the grayscale value of this point is f(x0,y0). Select a reference point P1(x1,y1) on the image after force is applied (i.e., the deformed image), and the grayscale value of this point is g(x1,y1). In the initial image, a square area of ​​(2M+1,2M+1) in size with P0(x0,y0) as the center is the reference sub-area, and there must be a deformed area with P1(x1,y1) as the center in the reference image, and the correlation between the two areas is the largest, then P0 and P1 are considered to be corresponding matching points. The correlation function is a function that describes the correlation between the initial target area and the deformed area after force is applied, and its mathematical formula is:

[0082]

[0083] Where: M refers to a parameter used to describe the size of the square area centered at the reference point P0 (x0, y0); i is the index of the pixel position on the horizontal axis (x-axis) in the two-dimensional image; j is the index of the pixel position on the vertical axis (y-axis) in the two-dimensional image, ——are the average grayscale values ​​of the pixels in the sub-region before and after the force is applied, and the calculation formula is:

[0084] Since the force on the film is not uniform during the pressing process, resulting in uneven deformation of the film, the first-order shape function model cannot accurately express it. Therefore, the second-order shape function model is used. There is a certain functional relationship between a point Q(x, y) in the initial area and the corresponding point Q1(x1, y1) in the deformation area after the force is applied. The shape function is:

[0085]

[0086] p=(u,u x ,u y ,u xx ,u xy ,uyy ,v,v x ,v y ,v xx ,v xy ,v yy ) T

[0087] Where: x' and y' are the distances from point (x1, y1) to P0 (x0, y0); u and v are the displacements of the center point of the reference sub-region in the x and y directions; u xx ,u xy ,u yy and v xx , v xy , v yy --Displacement gradient of the sub-region.

[0088] In order to obtain the full-field displacement, it is necessary to first divide the region of interest of the reference image into a uniform virtual grid, then select a seed point as the center position of the first reference subset for DIC analysis, calculate the corresponding deformation parameters, and calculate the sum of the subset located at the center of this point C LS , and activate the point. According to the correlation coefficient, the four adjacent points are inserted into the queue in descending order, the position of the seed point and the position in the queue are stored, the position of the seed point is deactivated, and then the program enters a loop. In the loop, the top element (that is, the smallest element C LS The point is copied and deleted from the queue, the displacement result after loading is plotted in the data graph, and the four surrounding points are judged to be valid points that have not been calculated. Then the four remaining scattered points around it that have not been calculated are calculated using the previous deformation parameters; if these points can be successfully copied, they will be deactivated. And so on, until the queue is empty.

[0089]

[0090] In the above formula, C cc represents the normalized cross-correlation coefficient, which is used to measure the similarity between two image regions; are the coordinates of the initial reference subset points; is the coordinate of the final current subset point; f and g are the reference image and current image functions respectively, and return the grayscale value corresponding to the specified (x, y) point, f m and g m are the average grayscale values ​​of the final reference and the current subset, respectively, where

[0091] In the above scheme, the maturity determination module under the tactile system includes the following steps:

[0092] The controller obtains the displacement-pressure curves of different fruits after DIC analysis based on the images captured by the binocular camera, divides the data set into training set and test set in a ratio of 3:1, builds a SVM classifier to judge the maturity grade, maximizes the classification interval by minimizing the norm of the weight vector, and introduces slack variables to make the model more robust.

[0093] The present invention combines the method of visual detection of maturity with touch, simulates the skin of human palm, replaces the human palm with PDMS film, controls the binocular camera through the controller to collect the color image of the fruit for image processing, identifies the fruit type, and calculates the position of the fruit in the workspace of the mechanical arm. The controller selects visual detection or tactile detection according to the information judged by the fruit recognition module. Visual detection judges the maturity according to the color of the fruit skin during the ripening period, and tactile detection grabs the fruit by the manipulator and imitates the pressure exerted on the fruit by humans. The binocular camera is used to replace the human eyes to indirectly feel the softness and hardness of the fruit surface and analyze the maturity level. The method realizes accurate detection of the maturity of the fruit and reduces the waste of resources. In the food processing industry, the method can replace the staff to check whether the maturity of the fruit is suitable for processing jams, beverages and other products to ensure the quality and taste of the processed products. In terms of supply chain management, the method can help suppliers, wholesalers and retailers determine the maturity of the fruit, so as to better arrange procurement, transportation and sales plans, give priority to the sale of slightly soft fruits, and store slightly hard fruits for a short time, which helps to reduce inventory backlogs and cargo losses.

[0094] It should be understood that although this specification is described according to various embodiments, not every embodiment contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art. The series of detailed descriptions listed above are only specific descriptions of the feasible embodiments of the present invention. They are not intended to limit the scope of protection of the present invention. All equivalent embodiments or changes that do not deviate from the technical spirit of the present invention should be included in the scope of protection of the present invention.

Claims

1. A visual-touch integrated sensing device for fruit maturity analysis, characterized in that: It comprises a mechanical arm and a mechanical hand (4) which are connected in sequence; a conveyor belt (5) is arranged below the mechanical hand (4); and a binocular camera (10), a PDMS film (6), 199UV printing ink (7), an ultraviolet lamp (8), and a light source (9) are arranged on the mechanical hand (4); The robot arm comprises a large arm (1), a middle arm (2), and a small arm (3) which are connected in sequence. At the upper part of the robot arm (4), 199UV printing ink (7) is evenly sprayed on the PDMS film (6). At the lower part of the robot arm (4), when the tactile system is used for judgment, the ultraviolet lamp (8) is turned on and the image is collected by the binocular camera (10); the ultraviolet lamps (8) and light sources (9) are arranged in a circle on the inner wall of the sensor in an interlaced manner, with a total of 8; The joints of the upper arm (1), the middle arm (2), the lower arm (3) and the manipulator (4) are provided with motors for rotation, and the motors are connected to a controller; the binocular camera (10), the ultraviolet lamp (8) and the light source (9) are also connected to the controller; The binocular camera (10) uses a fruit identification and positioning system to identify the type of fruit and locate the location of the fruit to determine whether it is suitable to use a visual method to determine the maturity; if the outer skin of the fruit presents different colors at different maturity stages, it is suitable, otherwise it is not suitable; the mechanical arm controls the magnitude of the applied force to slowly and evenly press the fruit onto the PDMS film (6), and indirectly determine the maturity level based on the deformation displacement information of the ink spots (7) on the surface of the film.

2. The device according to claim 1, characterized in that The binocular camera (10) is connected to the controller to collect color images of the fruit for image processing, locate the location of the fruit, identify the type of fruit, and determine whether it is possible to make a determination based on the characteristic that the ripe color of the fruit skin at different maturity stages is different.

3. The device according to claim 2, characterized in that The binocular camera (10) is connected to a controller and comprises a fruit recognition module, a fruit positioning module, a visual maturity detection module, a 3D-DIC module and a maturity discrimination module under a tactile system; The fruit recognition module adds new category image data, extracts multi-level feature information of the fruit, selects the fruit region of interest, reduces redundant background information, and improves recognition accuracy; The fruit positioning module, the controller controls the binocular camera to capture the fruit image, and calculates the depth information to obtain the spatial three-dimensional coordinates and size of the fruit, and obtains the position of the coordinates of the robot arm positioning; The visual detection maturity module tracks and collects partial images with different maturity levels, and revises the annotated data to improve the model's detection capability for fruits of adjacent maturity levels; The visual detection maturity module replaces the backbone feature extraction network of Yolo-v7 with a lightweight network, which greatly reduces the number of parameters and calculations of the model; uses 1×1 convolution to increase the dimension, introduces the inverse residual structure of the linear bottleneck, and then performs 3×3 depth-separable convolution, adjusts the weights between channels through the attention model SE-Net, and finally modifies the swish function to reduce the amount of calculation; The visual detection maturity module introduces a GAM attention module in the feature fusion module to reduce the impact of reduced detection accuracy caused by a small number of model parameters; the sequential channel-spatial attention mechanism in CBAM is adopted to redesign the submodules; the channel attention submodule adopts 3D combination to retain information across three dimensions, and uses a multi-layer perceptron to allow feature vectors to fully interact between different dimensions, so that the model can obtain more nonlinear features and combined feature information; The 3D-DIC module uses a binocular camera to collect images before and after speckle deformation, selects an ROI area in the image before deformation as a template, uses a normalized cross-correlation coefficient in the image after deformation to judge the degree of matching between the template and each position of the image after deformation, determines matching points, and calculates the transformation relationship before and after deformation through the matching points; then, in combination with the internal and external parameters of the camera, uses a triangulation method to convert the image coordinates into coordinates in a three-dimensional space, calculates the three-dimensional displacement of the points before and after deformation, thereby obtaining a three-dimensional deformation field, and then realizing three-dimensional reconstruction of the surface to be measured; The maturity discrimination module under the tactile system uses a maturity classifier constructed using a support vector machine (SVM) algorithm module. Based on the images captured by the binocular camera, the displacement-pressure curves of different fruits are obtained after analysis by the DIC algorithm module. The obtained data set is divided into a training set and a test set at a ratio of 3:

1. The test set curve is fitted, and the classification interval is maximized by minimizing the norm of the weight vector. A slack variable is introduced to make the model more robust.

4. The device according to claim 3, characterized in that The DIC algorithm module divides the image before and after deformation into multiple sub-regions, and then determines the region in the deformed image that best matches the reference subset through the normalized cross-correlation criterion and the cross-correlation coefficient. After obtaining the matching displacement of each subset before and after deformation, the displacement field of the entire image can be calculated.

5. An analysis method of a visual-touch integrated sensing device for fruit maturity analysis, characterized in that: The following steps are involved: Step 1, fruit recognition: The binocular camera collects fruit color images for image processing, identifies the fruit type, and determines whether it can be judged based on the different ripening colors of the fruit skin at different maturity stages; Step 2, visual detection of maturity: The controller processes the images of fruits whose maturity can be visually detected based on the color of the outer skin, and revises the labeled data to improve the model's ability to detect adjacent mature fruits. Then, the backbone feature extraction network of Yolo-v7 is replaced with the lightweight network MobileNetV3, which greatly reduces the number of model parameters and the amount of calculation. The GAM attention module is introduced into the feature fusion module to enhance the network's feature expression ability, reduce the impact of the decrease in detection accuracy caused by the small number of model parameters, and improve the detection accuracy of adjacent mature fruits and occluded fruits. Step 3: The controller slowly applies pressure to the fruits that cannot be successfully visually inspected, and turns on the ultraviolet lamp at the same time. The controller controls the manipulator to uniformly apply different forces up to 10N. The binocular camera simultaneously captures the fluorescent spots of invisible ink under the irradiation of the ultraviolet lamp, and captures ten images to record the degree of change of the film during the pressing process of fruits of different maturity and the degree of displacement of the surface spots. The DIC algorithm module is used to match the feature points in the images before and after deformation. Then, the image coordinates are converted into coordinates in three-dimensional space using the triangulation method in combination with the internal and external parameters of the camera, and the three-dimensional displacement of the points before and after deformation is calculated, thereby obtaining a three-dimensional deformation field, and then realizing three-dimensional reconstruction of the surface to be measured. Step 4, maturity discrimination under the tactile system: The controller obtains the displacement-pressure curves of different fruits based on the images captured by the binocular camera after analysis by the DIC algorithm module. The data set is divided into a training set and a test set in a ratio of 3:

1. An SVM classifier is constructed to judge the maturity level. The classification interval is maximized by minimizing the norm of the weight vector. In order to consider a certain degree of misclassification tolerance, a slack variable is introduced to make the model more robust.

Images