A non-destructive hardness testing device and testing line for kiwifruit production line based on visual-tactile perception

Abstract

The present invention relates to the technical field of nondestructive testing of kiwifruit, and provides a nondestructive hardness testing device for a kiwifruit production line based on visual touch and a testing production line. The testing device includes a support frame, a roller, an information collection unit and an information processing unit. The roller is rotatably arranged on the support frame. The roller is a polygon, and each side is an open structure. The opening is provided with a flexible membrane, and the flexible membrane is provided with a plurality of synapses. The information collection unit collects the motion signal of the synapse and transmits it to the information processing unit, and the information processing unit predicts the hardness value of the kiwifruit to be tested. The testing production line includes a testing device and a transmission device. The transmission device transmits the kiwifruit to be tested to the bottom of the testing device, and the movement speed of the transmission device is consistent with the rotation speed of the roller. The testing device and the testing production line of the present invention will not damage the kiwifruit when testing the hardness of the kiwifruit, and have a simple structure and low equipment cost, and can realize fast and repeated testing.

Description

Technical Field

[0001] The invention relates to the technical field of nondestructive testing of kiwifruit, and in particular to a nondestructive hardness testing device for a kiwifruit production line based on vision and touch, and a testing production line. Background Art

[0002] Kiwifruit, also known as kiwi fruit, is rich in glucose, fructose, vitamins and minerals, which are essential nutrients for the human body. It is known as the "king of vitamin C". In the past 10 years, the scale of kiwifruit cultivation has grown rapidly and has become the mainstream consumer fruit in the world. China's total kiwifruit planting area and total output are both ranked first in the world. With the expansion of the kiwifruit consumer market, people's demand for high-quality kiwifruit is increasing day by day. In order to take into account the storage time and taste of kiwifruit, it is usually necessary to pick it when it is not fully ripe. After picking, kiwifruit needs a certain amount of time to reach physiological maturity. During the ripening process, insoluble starch is gradually converted into soluble solids, the sugar content of glucose, fructose and other sugars in the fruit increases, and the organic acid that produces sour taste and the protease that produces astringency decrease. At the same time, the hardness of the fruit is significantly reduced. Therefore, the fruit hardness of kiwifruit is an important indicator that determines the quality and maturity of kiwifruit. With the expansion of the kiwifruit industry and the increase in consumer demand, it is urgent to test the hardness of kiwifruit.

[0003] At present, the methods for testing the hardness of kiwifruit are divided into destructive testing and non-destructive testing. Destructive testing uses a texture analyzer, which inserts a probe into the kiwifruit pulp and uses a sensor to derive its interaction force. This method is accurate, but it will cause irreversible damage to the kiwifruit, and the tested kiwifruit cannot be sold. Non-destructive testing usually uses near-infrared spectroscopy equipment, which can quickly perform non-destructive testing on kiwifruit on the production line. Chinese patent application No. 2021113000809 discloses an online nondestructive detection device and method for kiwifruit quality based on spectral technology, including a probe detection device, a spectrometer, a light source, a conveying platform, a photoelectric sensor, and a dark box; a probe detection device is installed on the top of the dark box, and the probe detection device includes a follow-up stray light elimination mechanism and a reflective fiber probe; the input and output ends of the reflective fiber probe are respectively connected to the light source and the spectrometer, and a receiving fiber is arranged in the middle of the detection end of the reflective fiber probe, and 6-12 transmitting fibers are arranged at equal intervals along the circumference of the receiving fiber. The reflective fiber probe detects kiwifruit samples through a surrounding multi-point reflection; a conveying platform is installed at the bottom of the dark box, and the conveying platform includes a conveying track, a slide installed on the conveying track, and a fruit cup for holding kiwifruit samples is fixed above the slide; photoelectric sensors are installed on both sides of the dark box. The problems with this method are: the spectral equipment has poor repeatability, the instrument is expensive, and the cost is high. The construction of the experimental platform and the collection of spectral data require professional and technical personnel to complete, and the operation steps are complicated and difficult to promote. Summary of the invention

[0004] In order to overcome the shortcomings of the above-mentioned prior art, the present invention provides a non-destructive hardness detection device and detection production line for a kiwifruit production line based on vision and touch, which will not damage the kiwifruit when performing hardness detection on the kiwifruit, and has a simple structure and low equipment cost, and can realize fast and repeated detection.

[0005] In order to achieve the above purpose, this application adopts the following technical solutions:

[0006] A non-destructive hardness detection device for a kiwifruit production line based on vision and touch, the detection device comprises a support frame, a roller, an information collection unit and an information processing unit, the roller is rotatably arranged on the support frame, the roller is a polygon, each side is an opening structure, a flexible film is arranged at the opening, a plurality of synapses are arranged on the flexible film, the information collection unit collects motion signals of the synapses and transmits them to the information processing unit, and the information processing unit predicts the hardness value of the kiwifruit to be detected.

[0007] In the above technical solution, the information collection unit is a camera, a horizontal axis is provided at the lower end of the support frame, the front and back sides of the drum pass through the horizontal axis and can rotate around the horizontal axis, and the camera is arranged on the lower end surface of the horizontal axis to collect images of the kiwi fruit and the synapse when they are in contact with each other in real time.

[0008] In the above technical solution, the synapses are evenly distributed over the flexible membrane.

[0009] In the above technical solution, a transmission mechanism is provided on the drum, and the transmission mechanism includes two cylindrical roller bearings and a motor gear. The two cylindrical roller bearings are concentrically arranged on the bearing mounting holes located on the front and back sides of the drum. The motor gear is connected to one of the cylindrical roller bearings to drive it to rotate and then drive the drum to rotate.

[0010] In the above technical solution, the information processing unit is a GPU processor, and the camera transmits the collected image information to the GPU processor. A deep learning model based on the Python language runs on the GPU processor, which performs calculations based on the received image information to predict the hardness value of the kiwi fruit.

[0011] In the above technical solution, the deep learning model is a deep learning model based on CNN-LSTM joint learning of image sequences.

[0012] In the above technical solution, the specific operation steps of the deep learning model are as follows:

[0013] A. Image acquisition stage: the camera communicates with the GPU processor through wireless data transmission, and records in real time the relative movement of the synapse generated by the interaction force between the kiwi fruit and the synapse. The GPU processor selects the video record when the kiwi fruit and the synapse just start to contact as the first frame image, selects the video record when the contact ends as the last frame image, and selects the intermediate frame contact image at the same time. At this point, all image sequences are collected;

[0014] B. Preprocessing stage: grayscale all image sequences, convert the images into grayscale images, and convert them into images of the same resolution size for normalization;

[0015] C. Kiwi fruit contour semantic segmentation stage: First, the image enters the semantic segmentation model trained by labeled data. The semantic segmentation model divides the image into kiwi fruit and background. Then, the area of ​​the kiwi fruit is obtained by calculating the pixel accumulation value of the kiwi fruit. Secondly, the perimeter of the kiwi fruit is obtained by calculating the accumulated alternating points of the kiwi fruit and the background. Finally, the above semantic segmentation process is performed on all images to obtain the area and perimeter features of the kiwi fruit.

[0016] D. Synaptic displacement feature extraction stage: While semantically segmenting the kiwifruit contour, extract the synaptic displacement features in the image. Use the VGG16 network for transfer learning and output the feature value of each image. The output feature value contains the displacement and deformation information of the synapse.

[0017] E. Hardness prediction stage: The semantic segmentation features and VGG16 features extracted from each frame of the image are sent to the LSTM network. Each LSTM layer outputs 128 features. All LSTM internal units are interconnected. The output features of the previous layer are correlated with the features of this layer to output the output features of this layer. After passing through the last layer of LSTM units, all LSTM unit output features are expanded into a fully connected layer through the ReLU activation function to obtain the hardness value of the kiwi fruit.

[0018] In the above technical solution, in step A, the GPU processor calls the OpenCV library through Python language, and determines whether the contact starts or ends by setting the threshold. When the contact starts, the synapse will be displaced, and the threshold of the image frame will change. The GPU processor uses this moment as the first frame of the video record, and obtains the last frame of the image when the threshold is stable. OpenCV takes the middle value of the image obtained in this process to obtain another three frames of contact images. At this point, the five-image sequence is collected.

[0019] A kiwifruit non-destructive hardness detection production line based on vision and touch comprises a detection device and a transmission device. The transmission device transmits the kiwifruit to be detected to the bottom of the detection device. The detection device is the above detection device. The movement speed of the transmission device is consistent with the rotation speed of the drum.

[0020] In the above technical solution, the transmission device is a conveyor belt, a fruit cup is provided on the conveyor belt, the detection device is arranged directly above the fruit cup, and the movement speed of the conveyor belt is consistent with the rotation speed of the drum.

[0021] The present invention has the following beneficial effects: the detection device and the detection production line of the present invention will not damage the kiwifruit when performing hardness detection on the kiwifruit, and the structure is simple and the equipment cost is low, and rapid and repeated detection can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a schematic diagram of the production line structure for this application;

[0023] Figure 2 This is a schematic diagram of the structure of the detection device of this application;

[0024] Figure 3 A partial cross-sectional view of the production line for this application test;

[0025] In the figure, 1. support frame; 2. roller; 3. conveyor belt; 4. fruit cup; 5. flexible membrane; 6. synapse; 7. cylindrical roller bearing; 8. motor gear; 9. camera; 10. horizontal axis. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0027] like Figure 1-3 As shown, the present application provides a non-destructive hardness detection device for a kiwifruit production line based on visual touch, the detection device comprises a support frame 1, a roller 2, an information collection unit and an information processing unit, and the roller 2 is rotatably arranged on the support frame 1. The roller 2 is a polygon, each side of which is an open structure, and a flexible film 5 is provided at the opening, specifically, the back of the flexible film 5 is fixed to the top surface of the opening. A plurality of synapses 6 are provided on the flexible film 5. The information collection unit collects the motion signal of the synapse 6 and transmits it to the information processing unit, and the information processing unit predicts the hardness value of the kiwifruit to be detected.

[0028] The roller 2 can be configured as a regular pentagon, a regular hexagon, a regular octagon, etc. A flexible film 5 is provided on each side. The flexible film 5 is preferably a silicone sensing flexible film, which is a soft translucent silicone with a thickness of 1mm, a size of 110mm*110mm, and a hardness of 10A. Synapses 6 are provided on the silicone sensing flexible film. The synapses 6 are preferably black silicone synapses with a synapse hardness of 20A, a diameter of 2.5mm, and a thickness of 2mm. The synapses 6 are evenly distributed on the flexible film 5, so that the synapses 6 can be fully in contact with the kiwi fruit, thereby improving the accuracy of the detection. Figure 1 As shown, as one embodiment, the roller 2 is configured as a regular hexagon, and the synapse 6 array is disposed on the flexible membrane 5.

[0029] In the above technical solution, the information collection unit can be a camera or a displacement sensor or other device that can record the motion signal of the synapse 6. When the displacement sensor is selected, it can be set at the end of the synapse 6 to record the position change of the synapse 6 during the contact between the synapse 6 and the kiwi fruit to be detected.

[0030] like Figure 3 As shown, the information collection unit can also select a camera 9, one of which is set as follows: a horizontal axis 10 is provided at the lower end of the support frame 1, the front and back sides of the drum 2 pass through the horizontal axis 10 and can rotate around the horizontal axis 10, and the camera 9 is set at the lower end surface of the horizontal axis 10 to collect images of the kiwi fruit and the synapse 6 in real time when they are in contact with each other. In addition, the camera 9 can also be set at any other position that can record images of the kiwi fruit and the synapse 6 when they are in contact with each other, such as using a support frame set on the side of the contact point between the kiwi fruit and the synapse 6.

[0031] As one embodiment, the connection structure between the drum 2 and the support frame 1 can be: a transmission mechanism is provided on the drum 2, and the transmission mechanism includes two cylindrical roller bearings 7 and a motor gear 8. Bearing mounting holes are provided on the front and back of the drum 2, and the two bearing mounting holes are concentrically arranged. The two cylindrical roller bearings 7 are respectively arranged on the bearing mounting holes. The horizontal shaft 10 located at the lower end of the support frame 1 passes through the two bearing mounting holes and the center of the two cylindrical roller bearings 7, and the two ends of the horizontal shaft 10 are fixed on the support frame 1. The motor gear 8 is connected to one of the cylindrical roller bearings 7 by transmission, and the motor drives the motor gear 8 to rotate, thereby driving the drum 2 to rotate. In addition, other connection structures can be selected between the drum 2 and the support frame 1, as long as the rotation of the drum 2 can be realized.

[0032] In the above technical solution, the information processing unit is a GPU processor, and the camera 9 transmits the collected image information to the GPU processor. A deep learning model based on the Python language runs on the GPU processor, which performs calculations based on the received image information to predict the hardness value of the kiwi fruit.

[0033] In the above technical solution, the deep learning model is a deep learning model based on CNN-LSTM joint learning of image sequences.

[0034] In the above technical solution, the specific operation steps of the deep learning model are as follows:

[0035] A. Image acquisition stage: the camera 9 communicates with the GPU processor through wireless data transmission, and records in real time the relative movement of synapse 6 generated by the interaction force between kiwifruit and synapse 6. The GPU processor calls the OpenCV library through the Python language, and determines whether the contact starts or ends by setting a threshold. When the contact starts, synapse 6 will be displaced, and the threshold of the image frame will change. The GPU processor uses this moment as the first frame of the video record, and obtains the last frame of the image when the threshold is stable. OpenCV uses the middle value of the image obtained in this process to obtain another three frames of contact images. At this point, the five image sequences are collected.

[0036] B. Preprocessing stage: grayscale the five image sequences, convert the images into grayscale images, and convert them into images with a resolution of 224*224 pixels for normalization.

[0037] C. Kiwi contour semantic segmentation stage: First, the image enters the semantic segmentation model trained by labeled data. The semantic segmentation model divides the image into kiwi and background classes. Then, the area of ​​the kiwi is obtained by calculating the cumulative pixel value of the kiwi class. Secondly, the circumference of the kiwi is obtained by calculating the cumulative alternating points of kiwi (clusters with pixel values ​​of 1) and background (clusters with pixel values ​​of 0). Finally, the above semantic segmentation process is performed on five sequential RGB images to obtain the area and circumference features of the five kiwis.

[0038] D. Synaptic displacement feature extraction stage: While semantically segmenting the kiwifruit contour, synaptic displacement features in the image are extracted. The VGG16 network is used for transfer learning to output the feature values ​​of each image. The output feature values ​​contain the displacement and deformation information of the synapse.

[0039] E. Hardness prediction stage: The semantic segmentation features and VGG16 features extracted from each frame of the image are sent to the LSTM network. Each LSTM layer will output 128 features. The five internal units of the LSTM are interconnected. The output features of the previous layer are correlated with the features of this layer to output the output features of this layer. After passing through the last layer of LSTM units, all the output features of the LSTM units are expanded into a fully connected layer through the ReLU activation function to obtain the hardness value of the kiwi fruit.

[0040] Verification of the prediction effect of the deep learning model: The kiwifruit to be tested is placed on the detection platform of a kiwifruit hardness detection texture analyzer. The hardness value obtained by inserting a kiwifruit test probe into the kiwifruit pulp is used as the true hardness label. The hardness value predicted by the deep learning model of the present invention is compared with the hardness label and statistically analyzed. There is no significant difference in the results of the two detection methods, and the prediction effect of the deep learning model of the present invention is good.

[0041] A kiwifruit non-destructive hardness detection production line based on vision and touch includes a detection device and a transmission device. The transmission device transmits the kiwifruit to be detected to the bottom of the detection device. The detection device is a detection device of the above structure. The movement speed of the transmission device is consistent with the rotation speed of the drum 2.

[0042] As one embodiment, the transmission device is a conveyor belt 3, a fruit cup 4 is provided on the conveyor belt 3, and the detection device is arranged just above the fruit cup 4, and the movement speed of the conveyor belt 3 is consistent with the rotation speed of the drum 2. In addition, the transmission device can also be a manipulator or an intelligent robot, which transmits the kiwi fruit to be detected to the bottom of the detection device, and the detection device detects the hardness of the kiwi fruit.

[0043] Working principle: The drum 2 rotates driven by the motor gear 8, while the production line at the lower end moves in a straight line (the conveyor belt 3 moves in a straight line), and the movement speeds of the two are consistent. The kiwis on the production line contact the synapse 6 one by one, and the displacement and interaction force generated are dispersed on the flexible membrane 5. The hardness of the kiwis is different, and the forces acting on the synapse 6 are also different. The camera 9 records and captures the movement trajectory of the synapse 6, and transmits the collected video to the GPU processor. The GPU processor runs a deep learning model based on the python language. The model performs semantic segmentation on the contour of the kiwi, extracts its size features, and further processes the segmented image with the model convolution layer, extracts the feature values ​​of the synapse 6, and performs calculations based on the deep learning network model to predict the hardness value of the kiwi. With a simple and low-cost device, the hardness of the kiwi is quickly and non-destructively detected under real-time online conditions, which is convenient for the next step of sorting or other processing.

[0044] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A non-destructive hardness detection production line for kiwifruit production line based on visual touch, including a detection device and a transmission device, characterized in that: The detection device comprises a support frame (1), a roller (2), an information collection unit and an information processing unit, wherein the roller (2) is rotatably arranged on the support frame (1), the roller (2) is a polygon, each side of which is an open structure, a flexible film (5) is provided at the opening, a plurality of synapses (6) are provided on the flexible film (5), the information collection unit collects motion signals of the synapses (6) and transmits them to the information processing unit, and the information processing unit makes a prediction on the hardness value of the kiwi fruit to be detected; The information collection unit is a camera (9); a horizontal axis (10) is provided at the lower end of the support frame (1); the front and back sides of the roller (2) pass through the horizontal axis (10) and can rotate around the horizontal axis (10); the camera (9) is arranged on the lower end surface of the horizontal axis (10) to collect images of the kiwi fruit and the synapse (6) in real time when they are in contact with each other; the synapse (6) is evenly distributed over the flexible membrane (5); The roller (2) is provided with a transmission mechanism, the transmission mechanism comprising two cylindrical roller bearings (7) and a motor gear (8), the two cylindrical roller bearings (7) being concentrically arranged on bearing mounting holes located on the front and back sides of the roller (2), the motor gear (8) being in transmission connection with one of the cylindrical roller bearings (7) to drive the cylindrical roller bearing to rotate, thereby driving the roller (2) to rotate; The transmission device transmits the kiwi fruit to be detected to the bottom of the detection device, and the movement speed of the transmission device is consistent with the rotation speed of the drum (2); the transmission device is a conveyor belt (3), and a fruit cup (4) is provided on the conveyor belt (3). The detection device is arranged directly above the fruit cup (4), and the movement speed of the conveyor belt (3) is consistent with the rotation speed of the drum (2).

2. The detection production line according to claim 1, characterized in that: The information processing unit is a GPU processor, and the camera (9) transmits the collected image information to the GPU processor. A deep learning model based on the Python language runs on the GPU processor, and performs calculations based on the received image information to predict the hardness value of the kiwi fruit.

3. The detection production line according to claim 2, characterized in that: The deep learning model is a deep learning model based on CNN-LSTM joint learning of image sequences.

4. The detection production line according to claim 2, characterized in that: The specific operation steps of the deep learning model are as follows: Image acquisition stage: the camera (9) communicates with the GPU processor via wireless data transmission, and records in real time the relative movement of the synapse (6) generated by the interaction force between the kiwi fruit and the synapse (6). The GPU processor selects the video record when the kiwi fruit and the synapse (6) just start to contact as the first frame image, selects the video record when the contact ends as the last frame image, and selects the intermediate frame contact image at the same time. At this point, all image sequences are collected; B. Preprocessing stage: grayscale all image sequences, convert the images into grayscale images, and convert them into images of the same resolution size for normalization; C. Kiwi fruit contour semantic segmentation stage: First, the image enters the semantic segmentation model trained by labeled data. The semantic segmentation model divides the image into kiwi fruit and background. Then, the area of ​​the kiwi fruit is obtained by calculating the pixel accumulation value of the kiwi fruit. Secondly, the perimeter of the kiwi fruit is obtained by calculating the accumulated alternating points of the kiwi fruit and the background. Finally, the above semantic segmentation process is performed on all images to obtain the area and perimeter features of the kiwi fruit. D. Synaptic displacement feature extraction stage: While semantically segmenting the kiwifruit contour, extract the synaptic displacement features in the image. Use the VGG16 network for transfer learning and output the feature value of each image. The output feature value contains the displacement and deformation information of the synapse. E. Hardness prediction stage: The semantic segmentation features and VGG16 features extracted from each frame of the image are sent to the LSTM network. Each LSTM layer outputs 128 features. All LSTM internal units are interconnected. The output features of the previous layer are correlated with the features of this layer to output the output features of this layer. After passing through the last layer of LSTM units, all LSTM unit output features are expanded into a fully connected layer through the ReLU activation function to obtain the hardness value of the kiwi fruit.

5. The detection production line according to claim 4, characterized in that: In step A, the GPU processor calls the OpenCV library through the Python language and determines whether the contact starts or ends by setting a threshold. When the contact starts, the synapse (6) will be displaced and the threshold of the image frame will change. The GPU processor uses this moment as the first frame of the video record. When the threshold is stable, the last frame of the image is obtained. OpenCV takes the middle value of the image obtained in this process to obtain another three frames of contact images. At this point, the five image sequences are collected.

Images