Fruit hardness detection device and method based on machine vision and acoustic vibration technology

Abstract

The invention discloses a fruit hardness detection device and method based on a machine vision and acoustic vibration technology. The device comprises a conveying mechanism, an image acquisition mechanism, a vibration signal acquisition mechanism and a data processing module, the conveying mechanism is arranged on the ground, the image collecting mechanism is installed on the conveying mechanism, the vibration signal collecting mechanism is arranged at the tail end of the conveying mechanism, and the data processing module is electrically connected with the image collecting mechanism and the vibration signal collecting mechanism. The method comprises the following steps: respectively processing to obtain contour feature parameters and vibration feature parameters, then measuring hardness labels to jointly form a fruit hardness data set, and inputting the fruit hardness data set into a constructed fruit hardness prediction model for training to obtain a fruit hardness prediction result. And inputting the fruit surface image and the vibration signal of the to-be-detected fruit into a data processing module for processing and detecting to obtain the hardness of the to-be-detected fruit. The invention provides a novel fruit hardness detection method, which solves the problem that acoustic vibration nondestructive detection neglects fruit shape influence, and is high in precision.

Description

Technical Field

[0001] The present invention belongs to the field of fruit detection, and particularly relates to a fruit hardness detection device and method based on machine vision and acoustic vibration technology. Background Art

[0002] With the continuous innovation of agricultural technology, new planting methods, irrigation technologies, etc. have continuously improved the yield and quality of fruits; at the same time, consumers' requirements for fruit quality have also risen, and they pay more attention to taste, freshness, etc. In this context, fruit online detection technology has become increasingly crucial in fruit production and sales.

[0003] Traditional fruit hardness detection mainly relies on manual detection and laboratory analysis. When manually detecting, the detector judges the hardness by pressing the fruit by hand based on experience. Not only is the speed slow, but also different people have different hand feelings, and the results are easily affected subjectively; although laboratory analysis is more accurate, the fruit needs to be sent to a professional place and detected by complex processes and instruments, which takes a long time and has a high cost, and it is difficult to meet the requirements of real-time feedback and high-throughput detection in modern production.

[0004] When the existing acoustic vibration technology is used for fruit hardness detection, factors such as the shape, size, and internal structure of the fruit will interfere with the vibration response, and the problem of ignoring the influence of fruit shape will be ignored, affecting the detection accuracy. Moreover, the equipment is complex, difficult to maintain, and has a high cost. For traditional machine vision technology, it is easily interfered by the surface color, texture, and defects of the fruit, and can only observe the surface and cannot deeply detect the internal structure and hardness, and there are also limitations. Therefore, it is urgent to develop a more efficient and accurate fruit hardness online detection technology. Summary of the Invention

[0005] In order to solve the problems existing in the background art, the purpose of the present invention is to provide a fruit hardness detection device and method based on machine vision and acoustic vibration technology. Based on machine vision and acoustic vibration technology, the corrugated plate rolling method is used to quickly obtain the vibration information of the fruit. At the same time, combined with corresponding signal processing methods and deep learning algorithms, a fruit hardness prediction model is established to realize the rapid non-destructive detection of fruit hardness.

[0006] The technical solution adopted by the present invention is as follows:

[0007] I. A fruit hardness detection device based on machine vision and acoustic vibration technology

[0008] A conveying mechanism is arranged on the detection site, the fruit is placed on the conveying mechanism, and the conveying mechanism drives the fruit to move.

[0009] An image acquisition mechanism is installed on the conveying mechanism. The fruit moves through the image acquisition mechanism driven by the conveying mechanism, and the image acquisition mechanism takes and acquires the surface image of the fruit.

[0010] The vibration signal acquisition mechanism is arranged on the detection site and at the end of the conveying direction of the conveying mechanism. Fruits are conveyed to the vibration signal acquisition mechanism by the conveying mechanism, and the vibration signal acquisition mechanism acquires the vibration response signals of the fruits.

[0011] The data processing module is electrically connected to the image acquisition mechanism and the vibration signal acquisition mechanism respectively. The data processing module is used to process the acquired fruit surface images and vibration response signals to obtain the hardness of the fruits.

[0012] The conveying mechanism includes a conveying frame, two conveying wheels, a conveyor belt and a motor; the conveying frame is arranged on the detection site, the image acquisition mechanism is installed on the conveying frame, the two conveying wheels are respectively installed at both ends of the conveying direction of the conveying frame, the conveyor belt is respectively in rolling connection with the two conveying wheels, the motor is installed on the conveying frame, and the motor is in transmission connection with one of the conveying wheels.

[0013] The conveying speed of the conveyor belt is 0.05 - 0.10 m / s.

[0014] The image acquisition mechanism includes a detection dark box, a camera, a PLC module, a photoelectric sensor and an image acquisition module; the detection dark box is installed on the conveying frame, the two surfaces of the detection dark box along the conveying direction are respectively provided with an inlet and an outlet for fruit conveyance, the camera is installed above the interior of the detection dark box, the photoelectric sensor is installed on one of the two sides of the interior of the detection dark box along the conveying direction, the height of the photoelectric sensor is flush with the height of the fruits on the conveyor belt, the input end and the output end of the PLC module are respectively electrically connected to the photoelectric sensor and the camera, and the input end and the output end of the image acquisition module are respectively electrically connected to the camera and the data processing module.

[0015] The vibration signal acquisition mechanism includes a data acquisition module, a vibration signal acquisition sensor, a buffer plate and a corrugated plate.

[0016] The corrugated plate is provided with a slope. The end with a higher slope is flush with the end of the conveying direction of the conveyor belt, and the end with a lower slope is arranged on the ground. One end of the buffer plate is connected to the end with a lower slope of the corrugated plate, and the other end of the buffer plate extends in the direction where the fruits roll, and the vibration signal acquisition sensor is installed on the back surface of the surface of the corrugated plate in contact with the fruits. The vibration signal acquisition sensor is connected to the input end of the data acquisition module, and the output end of the data acquisition module is connected to the data processing module.

[0017] The angle of the slope set for the corrugated plate is 15 - 30 degrees. The set angle of the slope can well limit the rolling speed of the fruits on the corrugated plate 11.

[0018] The corrugated plate is an aluminum alloy plate, and the corrugated plate is composed of a plurality of arc surfaces spliced together.

[0019] The data acquisition module samples in a signal-triggered mode. When the signal transmitted by the vibration signal acquisition sensor to the data acquisition module is greater than 0.05g, the data acquisition module acquires the vibration signal.

[0020] II. Fruit hardness detection method of a fruit hardness detection device based on machine vision and acoustic vibration technology

[0021] S1. The image acquisition mechanism takes pictures of the fruit to acquire the fruit surface image. Then, the data processing module comprehensively processes the contour points of the fruit surface image to obtain all pixel coordinates of the fruit contour points, and comprehensively processes all pixel coordinates to obtain the contour feature parameters of the fruit.

[0022] S2. The vibration signal acquisition mechanism acquires the vibration signal generated by the rolling of the fruit. Then, the data processing module uses a filtering algorithm to perform noise reduction processing on the vibration signal, and successively performs fast Fourier transform processing and feature extraction on the noise-reduced vibration signal to obtain vibration feature parameters.

[0023] S3. A number of measurement points to be measured are evenly selected on the fruit, the hardness values of each measurement point to be measured are measured, and the average value of the hardness values of all measurement points to be measured is used as the fruit hardness label.

[0024] S4. Both the contour feature parameters and the vibration feature parameters are used as input data, and the hardness label is used as output data. The input data and output data are combined to obtain the hardness data of a single fruit.

[0025] S5. The hardness data of several fruits are obtained by using the same method as in steps S1 - S4, and the hardness data of all fruits are summarized to obtain the fruit hardness data set.

[0026] S6. In the data processing module, a fruit hardness prediction model is constructed using a deep learning model. The fruit hardness data set is input into the fruit hardness prediction model for training to obtain a trained fruit hardness prediction model.

[0027] S7. The fruit surface image and the vibration signal of the fruit to be measured are respectively acquired, and the fruit surface image and the vibration signal are simultaneously input into the data processing module for processing and detection by the trained fruit hardness prediction model to obtain the hardness of the fruit to be measured.

[0028] The specific content of step S7 is as follows: The fruit surface image and the vibration signal of the fruit to be measured are respectively acquired by the image acquisition mechanism and the vibration signal acquisition mechanism. The fruit surface image and the vibration signal are input into the data processing module for processing to respectively obtain the contour feature parameters and the vibration feature parameters. Both the contour feature parameters and the vibration feature parameters are used as input data, and then the input data is input into the fruit hardness prediction model in the data processing module for detection to obtain the hardness of the fruit to be measured.

[0029] The specific steps of step S1 are as follows:

[0030] S11. Use an image acquisition mechanism to capture an image of the surface of the fruit.

[0031] S12. The data processing module sequentially performs gray-scale transformation, median filtering, and threshold segmentation on the captured fruit surface image to obtain a binary image of the fruit. Then, use an edge extraction function to perform contour extraction on the binary image of the fruit to obtain a number of fruit contour points and the pixel coordinates of each fruit contour point.

[0032] S13. Calculate the centroid coordinates of the fruit contour points based on all the fruit contour points and the pixel coordinates of each fruit contour point. Then, perform polar coordinate transformation and normalization processing on all the pixel coordinates of the fruit contour points in sequence according to the centroid coordinates to obtain normalized polar coordinates. Perform periodic function processing on the normalized polar coordinates to obtain the periodic function of the fruit contour.

[0033] S14. Expand the periodic function according to the Fourier series and combine it with Euler's formula to obtain the harmonic components at all levels. Use the harmonic components at all levels as the contour feature parameters of the fruit.

[0034] The periodic function of the fruit contour in step S13 is set according to the following formula:

[0035] r g (θ + 2π) = r g (θ)

[0036] where θ represents the angular parameter in the conversion of the centroid coordinates to polar coordinates, r g (θ) represents the normalized polar radius with respect to θ, r g (θ + 2π) represents the polar radius after a 2π period with respect to the polar radius r g (θ), and π is a constant.

[0037] After expanding the periodic function according to the Fourier series and combining it with Euler's formula in step S14, the harmonic components at all levels are obtained according to the following formula:

[0038]

[0039] where F(h) represents the harmonic component at index h, r g () represents the normalized polar radius, n represents the number of pixel coordinates of the fruit contour points, 2π / n represents the radial angle between adjacent pixel coordinates, h represents the index value, j represents the imaginary unit, k represents the summation variable, and π is a constant.

[0040] The specific steps of step S2 are as follows:

[0041] S21. Collect the vibration signals generated by the fruit rolling on the corrugated board through the vibration signal acquisition mechanism.

[0042] S22. The data processing module uses a filtering method to denoise the vibration signals and obtains the vibration signals with noise removed.

[0043] S23. Perform fast Fourier transform processing on the vibration signals with noise removed to obtain time-domain data and frequency-domain data respectively. Feature extraction is performed on the time-domain data and frequency-domain data respectively to obtain time-domain feature parameters and frequency-domain feature parameters. The time-domain feature parameters and frequency-domain feature parameters are summarized to obtain vibration feature parameters.

[0044] The filtering method in step S22 uses the wavelet threshold denoising method or the Kalman filtering method.

[0045] The feature extraction of the time-domain data in step S23 is to extract the peak value, mean value, variance and kurtosis of the time-domain data.

[0046] The feature extraction of the frequency-domain data in step S23 is to extract the peak frequency, spectral bandwidth and power spectral density of the frequency-domain data.

[0047] The deep learning model in step S6 includes a convolutional neural network, a deep belief network and a generative adversarial network.

[0048] Compared with the background technology, the beneficial effects of the present invention are:

[0049] 1. The present invention combines machine vision and acoustic vibration technology. It can first obtain the fruit shape characteristics and then perform acoustic vibration characteristic detection, realizing accurate and fast detection for fruit detection.

[0050] 2. The acoustic vibration characteristics of fruits are not only affected by their tissue structures but also related to their shapes. The present invention solves the problem that the influence of fruit shapes on acoustic vibration characteristics cannot be eliminated when solely using acoustic vibration technology for non-destructive hardness detection, providing a new idea for on-line fruit detection.

[0051] 3. The present invention uses deep learning algorithms to process multi-parameter data of images and vibration information, improving the prediction accuracy and adaptability of the model. Description of the Drawings

[0052] Figure 1 It is a schematic structural diagram of the device of the present invention.

[0053] Figure 2 It is a flowchart of the method of the present invention.

[0054] Figure 3 It is the time-domain data of the vibration signal of a kiwifruit in an embodiment of the present invention.

[0055] Figure 4 This is the frequency-domain data of a kiwifruit vibration signal in an embodiment of the present invention.

[0056] In the figure: 1, conveyor belt; 2, motor; 3, light source; 4, detection dark box; 5, camera; 6, image acquisition module; 7, data processing module; 8, data acquisition module; 9, vibration signal acquisition sensor; 10, buffer plate; 11, corrugated plate; 12, PLC module; 13, photoelectric sensor. Detailed implementation manners

[0057] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0058] As Figure 1 shown, on the one hand, the present invention provides a fruit hardness detection device based on machine vision and acoustic vibration technology, including:

[0059] A conveying mechanism, arranged on the detection site, with fruits placed on the conveying mechanism, and the conveying mechanism drives the fruits to move.

[0060] The conveying mechanism includes a conveying frame, two conveyor wheels, a conveyor belt 1 and a motor 2; the conveying frame is arranged on the detection site, an image acquisition mechanism is installed on the conveying frame, the two conveyor wheels are respectively installed at both ends of the conveying direction of the conveying frame, the conveyor belt is respectively in rolling connection with the two conveyor wheels, the motor 2 is installed on the conveying frame, the motor 2 is in transmission connection with one of the conveyor wheels, the motor 2 drives one of the conveyor wheels to rotate, and through the rolling friction of the conveyor belt 1, the other conveyor wheel also rotates, and the rotation of the conveyor belt 1 drives the fruits placed on the conveyor belt 1 to move.

[0061] The conveying speed of the conveyor belt 1 is 0.05 - 0.10 m / s.

[0062] An image acquisition mechanism, installed on the conveying mechanism, with the fruits moving through the image acquisition mechanism driven by the conveying mechanism, and the image acquisition mechanism captures the surface images of the fruits.

[0063] The image acquisition mechanism includes a detection dark box 4, a camera 5, a PLC module 12, a photoelectric sensor 13, and an image acquisition module 6; the detection dark box 4 is installed on the conveyor frame. On two surfaces of the detection dark box 4 along the conveying direction, there are respectively an inlet and an outlet for fruit conveyance. The bottom ends of two surfaces on both sides of the detection dark box 4 along the conveying direction are respectively installed on the conveyor frames on both sides along the conveying direction. The detection dark box 4 does not contact the conveyor belt 1 and is located above the conveyor belt 1. The camera 5 is installed above the interior of the detection dark box 4, and the photoelectric sensor 13 is installed on one of the two surfaces along both sides of the interior of the detection dark box 4 along the conveying direction. The height of the photoelectric sensor 13 is flush with the height of the fruit on the conveyor belt 1. The input end and the output end of the PLC module 12 are respectively electrically connected to the photoelectric sensor 13 and the camera 5, and the input end and the output end of the image acquisition module 6 are respectively electrically connected to the camera 5 and the data processing module 7. The fruit is conveyed to the photoelectric sensor 13 by the conveyor belt 1 to trigger a signal, and the trigger signal controls the camera 5 to take a picture through the PLC module 12. The taken picture is transmitted to the data processing module 7 through the image acquisition module 6 for processing.

[0064] In specific implementation, the height of the fruit on the conveyor belt 1 is the height of the midpoint of the fruit. The camera 5 uses a CCD camera of model DVP-30GC03E, and the photoelectric sensor 13 selects the RK-G2 model.

[0065] The vibration signal acquisition mechanism is arranged on the detection site and at the end of the conveying direction of the conveying mechanism. The fruit is conveyed to the vibration signal acquisition mechanism by the conveying mechanism, and the vibration signal acquisition mechanism acquires the vibration response signal of the fruit.

[0066] The vibration signal acquisition mechanism includes a data acquisition module 8, a vibration signal acquisition sensor 9, a buffer plate 10, and a corrugated plate 11; the corrugated plate 11 is provided with a slope through support columns. One end with a higher slope is flush with the end of the conveyor belt 1 in the conveying direction, and the end with a lower slope is arranged on the ground. The setting of the high and low slopes of the corrugated plate 11 is to allow the fruits to roll freely on the corrugated plate 11. One end of the buffer plate 1 is connected to the end with a lower slope of the corrugated plate 11, and the other end of the buffer plate 1 extends in the direction where the fruits roll. The buffer plate 1 is for the fruits to have a buffer distance after rolling down from the corrugated plate 11, so as not to let the fruits directly fall on the ground and be damaged. On the back surface of the surface of the corrugated plate 11 in contact with the fruits, a vibration signal acquisition sensor 9 is fixedly installed through double-headed bolts. The vibration signal acquisition sensor 9 is connected to the input end of the data acquisition module 8, and the output end of the data acquisition module 8 is connected to the data processing module 7. When the fruits roll on the corrugated plate 11 and pass by the vibration signal acquisition sensor 9, the vibration signal acquisition sensor 9 collects the vibration signal and sends it to the data acquisition module in real time. When the vibration signal received by the data acquisition module is greater than 0.05g, the data acquisition module continuously collects the vibration signals greater than 0.05g. After the collection is completed, the data is transmitted to the data processing module 7 for processing. The fruits roll back from the corrugated plate 11 and enter the buffer plate 1 for buffering and finally stop.

[0067] In specific implementation, the vibration signal acquisition mechanism further includes a number of support columns with the same height. All the support columns are installed on the detection site near the end of the conveyor belt 1 in the conveying direction. One end of the corrugated plate 11 is installed on the support columns, and the other end is fixed on the ground to form a slope.

[0068] The angle of the slope set for the corrugated plate 11 is 15 - 30 degrees.

[0069] The material of the corrugated plate 11 is aluminum alloy, and the corrugated plate 11 is composed of multiple arc surfaces spliced together.

[0070] In specific implementation, the data acquisition module 8 adopts the NI USB - 4431 model and uses the signal trigger mode for sampling. When the signal transmitted by the vibration signal acquisition sensor 9 to the data acquisition module 8 is greater than 0.05g, the data acquisition module 8 performs vibration signal acquisition.

[0071] In specific implementation, the length of the corrugated plate 11 is 240mm, the height is 16mm. The surface in contact with the fruits is the rolling surface. The rolling surface is divided into 8 segments at intervals of 30mm. Each segment is an arc surface with a radius of curvature of 625mm, and the height of each segment decreases by 2mm. The vibration signal acquisition sensor 9 selects an IEPE acceleration sensor.

[0072] The data processing module 7 is electrically connected to the image acquisition mechanism and the vibration signal acquisition mechanism respectively. The data processing module 7 is used to process the collected fruit surface image and vibration response signal to obtain the hardness of the fruit.

[0073] The working process of the device of the present invention is as follows:

[0074] Place the fruit on the conveyor belt 1. The fruit is conveyed by the conveyor belt 1 to the photoelectric sensor 13 to trigger a signal. The trigger signal controls the camera 5 to take a picture through the PLC module 12. The taken picture is transmitted to the data processing module 7 through the image acquisition module 6 for processing.

[0075] After the fruit is conveyed to the end of the conveyor belt 1, it enters the corrugated plate 11. When it rolls on the corrugated plate 11 and passes through the vibration signal acquisition sensor 9, the vibration signal acquisition sensor 9 collects the vibration signal and sends it to the data acquisition module in real time. When the data acquisition module transmits the collected data to the data processing module 7 for processing.

[0076] The data processing module 7 processes the data transmitted by the image acquisition module 6 and the data transmitted by the data acquisition module together to obtain the hardness of the fruit.

[0077] As Figure 2 shown, on the other hand, the present invention provides a fruit hardness detection method for a fruit hardness detection device based on machine vision and acoustic vibration technology, including the following steps:

[0078] S1. Use the image acquisition mechanism to take pictures of the fruit to collect the fruit surface image. Then, the data processing module 7 performs comprehensive processing on the contour points of the fruit surface image to obtain all pixel coordinates of the fruit contour points, and performs comprehensive processing on the contour feature parameters of all pixel coordinates of the fruit contour points to obtain the contour feature parameters of the fruit.

[0079] S11. Use the image acquisition mechanism to take pictures of the fruit to collect the fruit surface image.

[0080] S12. The data processing module 7 sequentially performs gray-scale transformation, median filtering, and threshold segmentation processing on the collected fruit surface image to obtain a binary image of the fruit. The binary image of the fruit is subjected to contour extraction processing using an edge extraction function to obtain several fruit contour points and the pixel coordinates of each fruit contour point in the fruit surface image.

[0081] In specific implementation, the edge extraction function can adopt Sobel operator, Prewitt operator, Laplacian operator, etc.

[0082] S13. Calculate the centroid coordinates of all fruit contour points based on all fruit contour points and the pixel coordinates of each fruit contour point. After performing polar coordinate transformation and normalization processing on all pixel coordinates of the fruit contour points in sequence according to the centroid coordinates, obtain the normalized polar coordinates, and perform periodic function processing on the normalized polar coordinates to obtain the periodic function of the fruit contour.

[0083] The periodic function of the fruit contour is set according to the following formula:

[0084] r g r(θ + 2π) = r(θ) g (θ)

[0085] where θ represents the angular parameter in the polar coordinates obtained by converting the centroid coordinates, r(θ) represents the normalized polar radius with respect to θ, r(θ + 2π) represents the polar radius after a 2π period with respect to the polar radius r(θ), and π is a constant. g (θ) represents the normalized polar radius with respect to θ, r g (θ + 2π) represents the polar radius after a 2π period with respect to the polar radius r(θ), g (θ), and π is a constant.

[0086] S14. Expand the periodic function according to the Fourier series and combine it with Euler's formula to obtain the harmonic components at all levels, and use the harmonic components at all levels as the contour feature parameters of the fruit.

[0087] After expanding the periodic function according to the Fourier series and combining it with Euler's formula, the harmonic components at all levels are obtained according to the following formula:

[0088]

[0089] where F(h) represents the harmonic component at index h, r(θ) represents the normalized polar radius, n represents the number of pixel coordinates of the fruit contour points, 2π / n represents the radial angle between every two adjacent pixel coordinates, h represents the index value, j represents the imaginary unit, k represents the summation variable, and π is a constant. g () represents the normalized polar radius, n represents the number of pixel coordinates of the fruit contour points, 2π / n represents the radial angle between every two adjacent pixel coordinates, h represents the index value, j represents the imaginary unit, k represents the summation variable, and π is a constant.

[0090] S2. Collect the vibration signals generated by the rolling of the fruit through the vibration signal acquisition mechanism, and then the data processing module 7 uses a filtering algorithm to perform noise reduction processing on the vibration signals, and performs fast Fourier transform processing and feature extraction on the vibration signals after noise reduction processing in sequence to obtain vibration feature parameters.

[0091] S21. Collect the vibration signals generated by the fruit rolling on the corrugated board through the vibration signal acquisition mechanism.

[0092] S22. The data processing module 7 uses a filtering method to perform noise reduction processing on the vibration signals to obtain the vibration signals with noise removed.

[0093] The filtering method uses the wavelet threshold denoising method or the Kalman filtering method.

[0094] S23. Perform fast Fourier transform processing on the vibration signal after noise removal to obtain time-domain data and frequency-domain data respectively. Feature extraction is performed on the time-domain data and the frequency-domain data respectively to obtain time-domain feature parameters and frequency-domain feature parameters. The time-domain feature parameters and the frequency-domain feature parameters are summarized to obtain vibration feature parameters.

[0095] Performing feature extraction on the time-domain data is to extract the peak value, mean value, variance and kurtosis of the time-domain data.

[0096] Performing feature extraction on the frequency-domain data is to extract the peak frequency, spectral bandwidth and power spectral density of the frequency-domain data.

[0097] S3. Uniformly select several measurement points on the fruit, and use a texture analyzer to measure the hardness values of each measurement point. The average value of the hardness values of all measurement points is used as the hardness label of the fruit.

[0098] S4. Both the contour feature parameters and the vibration feature parameters are used as input data, and the hardness label is used as output data. The input data and the output data are combined to obtain the hardness data of a single fruit.

[0099] S5. Use the same method as in steps S1 - S4 to obtain the hardness data of several fruits. The hardness data of all fruits are summarized to obtain a fruit hardness data set.

[0100] S6. In the data processing module 7, use a deep learning model to construct a fruit hardness prediction model. Input the fruit hardness data set into the fruit hardness prediction model for training to obtain a trained fruit hardness prediction model.

[0101] The deep learning model includes a convolutional neural network CNN, a deep belief network DBN and a generative adversarial network GAN.

[0102] In specific implementation, the input data is used as the input of the fruit hardness prediction model, and the output data is used as the output of the fruit hardness prediction model, so as to train the fruit hardness prediction model.

[0103] S7. Collect the fruit surface image and vibration signal of the fruit to be measured respectively. The fruit surface image and the vibration signal are simultaneously input into the data processing module 7 for processing in sequence and detection by the trained fruit hardness prediction model to obtain the hardness of the fruit to be measured.

[0104] Step S7 specifically includes: respectively collecting the fruit surface image and vibration signal of the fruit to be measured through the image acquisition mechanism and the vibration signal acquisition mechanism, inputting the fruit surface image and the vibration signal into the data processing module 7 for processing to obtain the contour feature parameters and vibration feature parameters respectively, using both the contour feature parameters and the vibration feature parameters as input data, and then inputting the input data into the fruit hardness prediction model in the data processing module 7 for detection to obtain the hardness of the fruit to be measured.

[0105] Embodiment

[0106] The detection of the fruit hardness by the present invention is universal. Taking kiwifruit as an example, the implementation process of the non-destructive detection of the hardness of kiwifruit by the present invention is introduced. For other fruits, the corresponding fruit hardness prediction model can be established by referring to the method of this embodiment, and the hardness of different fruits can be detected non-destructively.

[0107] 1. Collect the fruit surface image

[0108] Place the kiwifruit horizontally on the tray along the "fruit stalk - calyx" direction and convey it to directly below the CCD camera 5 in the detection dark box 4 through the conveyor belt. The CCD camera 5 takes the fruit surface image, and the fruit external image is transmitted to the data processing module 7 through the data acquisition module 8.

[0109] 2. Extract the pixel coordinates of the fruit contour points

[0110] After steps such as gray-scale transformation, median filtering, and threshold segmentation on the collected fruit surface image, a binary image is obtained; then the edge extraction function is used for contour extraction, and the pixel coordinates of the contour points are obtained after extraction.

[0111] 3. Extract the contour feature parameters

[0112] Regarding the extracted pixel coordinates of the contour points as the curve formed by the point set in the coordinate system, the fruit contour point curve is obtained. Calculate the centroid coordinates of the curve according to the fruit contour point curve; according to the centroid coordinates, convert the pixel coordinates of all contour points into polar coordinate form r(θ), and perform normalization processing to obtain the normalized polar radius r g (θ), so that the contour of the fruit can be described as a periodic function:

[0113] r g (θ + 2π) = r g (θ)

[0114] where θ represents the angle parameter in the conversion of the centroid coordinates to polar coordinates, r g (θ) represents the normalized polar radius with respect to θ, and r g (θ + 2π) represents the polar radius after a 2π period with respect to the polar radius r g (θ), and π is a constant.

[0115] By expanding the periodic function according to the Fourier series and combining with Euler's formula, the harmonic components F(h) of the fruit at all levels can be obtained as the contour characteristic parameters of the fruit:

[0116]

[0117] Among them, F(h) represents the harmonic component at index h, and r g () represents the normalized polar radius, n represents the number of pixel coordinates of the fruit contour points, 2π / n represents the radial angle between every two adjacent pixel coordinates, h represents the index value, j represents the imaginary unit, k represents the summation variable, and π is a constant.

[0118] 4. Collect vibration signals

[0119] The kiwifruit is conveyed by the conveyor belt 1 to the front section of an inclined corrugated plate 11, and then rolls down from the corrugated plate 11. A vibration signal acquisition sensor 9 is installed on the back of the corrugated plate 11. The vibration signal generated by rolling is obtained through the vibration signal acquisition sensor 9 and collected by the data acquisition module 8 and sent to the data processing module 7 for storage and processing. The sampling frequency of the vibration signal acquisition sensor 9 is 2560 Hz.

[0120] 5. Preprocess the vibration signals

[0121] Wavelet threshold denoising is used to remove the noise interference in the vibration signals and improve the signal-to-noise ratio.

[0122] 6. Extract vibration characteristic parameters

[0123] As Figure 3 and Figure 4 shown, the vibration signals after denoising are processed by fast Fourier transform to obtain time-domain data and frequency-domain data respectively, and vibration characteristic parameters are extracted from the time-domain data and frequency-domain data respectively. Among them, the time-domain characteristic parameters include peak value, mean value, variance and kurtosis, and the frequency-domain characteristic parameters include peak frequency, spectral bandwidth and power spectral density.

[0124] 7. Determine the hardness value

[0125] A puncture test is carried out. Three measurement points to be measured are evenly selected at the equator of the fruit, with an interval of 120 degrees between points. The average value of the hardness values of the three points is used as the hardness value of the fruit. At the selected measurement points to be measured, a cylindrical probe with a diameter of 5 mm is pressed into the pulp at a speed of 1 mm / s for 10 mm to obtain the "force-displacement" curve at this point. The hardness value is selected as the maximum force during the puncture process, and the hardness value is used as the hardness label.

[0126] 8. Obtain the hardness data of a single fruit

[0127] Both the contour feature parameters and the vibration feature parameters are used as input data, and the hardness label is used as output data. The input data and the output data are combined to obtain the hardness data of a single fruit.

[0128] 9. Establish a fruit hardness prediction model and measure the hardness value

[0129] Select 200 kiwifruits of the same batch, including test samples and samples to be measured. The test samples are used to establish a fruit hardness prediction model, and the samples to be measured are used to verify the fruit hardness prediction model. Measure the hardness value of all selected kiwifruits to obtain the hardness values of all kiwifruits, and the hardness values are used as hardness labels.

[0130] 10. Construct and train a fruit hardness prediction model.

[0131] Sort the 200 kiwifruit samples according to the hardness value. The samples with the maximum and minimum hardness values are used as calibration set samples (test samples), and the remaining samples are selected one every 4 in order as validation set samples (samples to be measured), and the others are used as calibration set samples. For all calibration set samples, obtain the contour feature parameters and vibration feature parameters of each fruit in the calibration set samples according to the same method in steps 1 - 6. The contour feature parameters and vibration feature parameters of each fruit are used as input data, and the hardness labels of the fruits in the calibration set samples collected in step 10 are used as output data. The input data and the output data are combined to obtain the hardness data of a single fruit, and the hardness data of all fruits are summarized to obtain the fruit hardness data set of kiwifruit. The hardness statistical values of the calibration set and the validation set are shown in Table 1.

[0132] Table 1 Hardness value statistics of calibration set and validation set samples

[0133] Sample set Sample quantity Hardness value range N Average hardness value N Standard deviation of hardness value N Calibration set 150 6.46-18.12 12.51 1.86 Validation set 50 7.10-17.45 12.36 1.70

[0134] Use a convolutional neural network (CNN) to establish a fruit hardness prediction model, which includes an input layer, a convolutional layer, a pooling layer, a fully connected layer, and an output layer. Input the fruit hardness data set into the fruit hardness prediction model for training.

[0135] The input layer nodes of the fruit hardness prediction model are the contour feature parameters and the vibration feature parameters. Set 3 - 5 convolutional layers, use the ReLU activation function, and adjust the convolutional kernels to 32, 64, or 128 according to the data complexity. The convolutional kernel size is 3×3 or 5×5, and the stride is 1 or 2. Add a pooling layer (Max Pooling) after each convolutional layer. Add 1 - 2 fully connected layers after the convolutional layer and the pooling layer. The output layer nodes are hardness, use the softmax activation function or the linear activation function, set the learning rate to 0.001, use the mean square error as the loss function, and select Adam or SGD as the optimizer.

[0136] The samples to be tested in the validation set of kiwifruit are sequentially input into the data processing module 7 to obtain contour feature parameters and vibration feature parameters. Then, the contour feature parameters and vibration feature parameters are input into the fruit hardness prediction model in the data processing module 7 for detection to obtain the hardness value corresponding to the fruit to be tested. The predicted hardness value is compared with the hardness index value measured in step 9, so as to know the accuracy of the prediction.

[0137] The results of CNN modeling are shown in Table 2, indicating that this method can be used for non-destructive detection of kiwifruit hardness. By adding the contour feature parameter F(h) to the input layer nodes, compared with only selecting the vibration feature parameter as the input layer node, the model prediction ability has been significantly improved: the correlation coefficient r between the predicted value and the actual value in the calibration set has increased from 0.833 to 0.855, and the root mean square error of calibration RMSEC has decreased from 0.976N to 0.919N; the correlation coefficient r in the validation set has increased from 0.794 to 0.840, and the root mean square error of prediction RMSEP has decreased from 1.253N to 0.959N.

[0138] The modeling results of the BP neural network for kiwifruit hardness are shown in Table 2. The modeling results show that this method can be used for non-destructive detection of kiwifruit hardness. At the same time, after adding the contour feature parameter and vibration feature parameter to the input layer nodes, the prediction results have been significantly improved: the correlation coefficient r between the predicted value and the actual value of hardness in the calibration set has increased from 0.866 to 0.953, and the root mean square error of calibration RMSEC has decreased from 0.864N to 0.786N; the correlation coefficient r in the validation set has increased from 0.835 to 0.931, and the root mean square error of prediction RMSEP has decreased from 1.156N to 0.846N.

[0139] Table 2 Modeling results of BP convolutional neural network CNN for kiwifruit hardness

[0140]

[0141] Therefore, the present invention solves the problem that the non-destructive detection of fruit acoustic vibration ignores the influence of fruit shape by adopting the method of combining machine vision and acoustic vibration technology; at the same time, by using deep learning algorithms to process multi-parameter data of images and vibration information, the prediction accuracy and adaptability of the model are improved, and remarkable technical effects are achieved.

[0142] The present invention constructs a multi-modal detection system. First, machine vision is used to quantify the shape characteristics of fruits, and then acoustic vibration technology is combined to further analyze their internal quality. This combination not only improves the accuracy and efficiency of detection, but also reduces the labor cost, and has broad application prospects.

[0143] In summary, the on-line fruit hardness detection device and method based on machine vision and acoustic vibration technology not only have technological innovation but also have good market application prospects. By timely and accurately detecting the fruit hardness, it helps to improve the quality and safety standards of fruits and promotes the sustainable and healthy development of the fruit industry.

[0144] The above specific embodiments are used to explain and illustrate the present invention rather than limit the present invention. Any modification and change made to the present invention within the spirit and scope of the claims of the present invention fall within the protection scope of the present invention.

Claims

1. A fruit hardness detection device based on machine vision and acoustic vibration technology, characterized in that: include: The conveying mechanism is arranged on the testing site, the fruit is placed on the conveying mechanism, and the conveying mechanism drives the fruit to move; An image acquisition mechanism is installed on the conveying mechanism. The fruit moves through the image acquisition mechanism under the drive of the conveying mechanism, and the image acquisition mechanism captures the surface image of the fruit. A vibration signal collection mechanism is arranged on the detection site and at the end of the transmission direction of the transmission mechanism. The fruit is transmitted to the vibration signal collection mechanism through the transmission mechanism, and the vibration signal collection mechanism collects the vibration response signal of the fruit; The data processing module (7) is electrically connected to the image acquisition mechanism and the vibration signal acquisition mechanism respectively. The data processing module (7) is used to process the collected fruit surface image and vibration response signal to obtain the hardness of the fruit.

2. The fruit hardness detection device based on machine vision and acoustic vibration technology according to claim 1, characterized in that: The conveying mechanism comprises a conveying frame, two conveying wheels, a conveying belt (1) and a motor (2); the conveying frame is arranged on a detection site, an image acquisition mechanism is installed on the conveying frame, the two conveying wheels are respectively installed at two ends of the conveying direction of the conveying frame, the transmission belt is respectively connected to the two conveying wheels in a rolling manner, the motor (2) is installed on the conveying frame, and the motor (2) is connected to one of the conveying wheels in a driving manner.

3. The fruit hardness detection device based on machine vision and acoustic vibration technology according to claim 2, characterized in that: The image acquisition mechanism comprises a detection dark box (4), a camera (5), a PLC module (12), a photoelectric sensor (13) and an image acquisition module (6); the detection dark box (4) is installed on a conveying frame, and the two surfaces of the detection dark box (4) along the conveying direction are respectively provided with an inlet and an outlet for fruit conveying, the camera (5) is installed at the top of the inside of the detection dark box (4), the photoelectric sensor (13) is installed on one of the two surfaces along the conveying direction inside the detection dark box (4), the height of the photoelectric sensor (13) is flush with the height of the fruit on the conveyor belt (1), the input end and the output end of the PLC module (12) are respectively electrically connected to the photoelectric sensor (13) and the camera (5), and the input end and the output end of the image acquisition module (6) are respectively electrically connected to the camera (5) and the data processing module (7).

4. The fruit hardness detection device based on machine vision and acoustic vibration technology according to claim 3 is characterized in that: The vibration signal acquisition mechanism comprises a data acquisition module (8), a vibration signal acquisition sensor (9), a buffer plate (10) and a corrugated plate (11); The corrugated plate (11) is provided with a slope, wherein the end with a higher slope is flush with the end of the conveying direction of the conveyor belt (1), and the end with a lower slope is arranged on the ground; one end of the buffer plate (1) is connected to the end with a lower slope of the corrugated plate (11), and the other end of the buffer plate (1) extends in the direction in which the fruit rolls down; a vibration signal acquisition sensor (9) is installed on the back side of the surface of the corrugated plate (11) in contact with the fruit; the vibration signal acquisition sensor (9) is connected to the input end of the data acquisition module (8), and the output end of the data acquisition module (8) is connected to the data processing module (7).

5. A method for detecting fruit hardness using the fruit hardness detection device based on machine vision and acoustic vibration technology according to any one of claims 1 to 4, characterized in that: The method comprises the following steps: S1, capturing a fruit surface image by an image acquisition mechanism, then performing contour point comprehensive processing on the fruit surface image to obtain all pixel coordinates of the fruit contour points, and performing contour feature parameter comprehensive processing on all pixel coordinates to obtain contour feature parameters of the fruit; S2, collecting the vibration signal generated by the rolling of the fruit through the vibration signal collection mechanism, and then the data processing module (7) uses a filtering algorithm to perform noise reduction processing on the vibration signal, and performs fast Fourier transform processing and feature extraction on the vibration signal after the noise reduction processing to obtain vibration feature parameters; S3, evenly selecting a number of test points on the fruit, measuring the hardness value of each test point, and taking the average value of the hardness values ​​of all the test points as the hardness label of the fruit; S4, using the contour feature parameters and the vibration feature parameters as input data, the hardness label as output data, and combining the input data and the output data to obtain the hardness data of a single fruit; S5, using the same method as step S1 to step S4 to obtain hardness data of several fruits, and summarizing the hardness data of all fruits to obtain a fruit hardness data set; S6. In the data processing module (7), a deep learning model is used to construct a fruit hardness prediction model, and the fruit hardness data set is input into the fruit hardness prediction model for training to obtain a trained fruit hardness prediction model; S7, respectively collecting the fruit surface image and vibration signal of the fruit to be tested, and simultaneously inputting the fruit surface image and vibration signal into the data processing module (7) for sequential processing and detection by the trained fruit hardness prediction model to obtain the hardness of the fruit to be tested.

6. The method for detecting fruit hardness of a fruit hardness detection device based on machine vision and acoustic vibration technology according to claim 5, characterized in that: The step S1 is specifically as follows: S11, photographing the fruit and collecting the surface image of the fruit by an image acquisition mechanism; S12, the data processing module (7) sequentially performs grayscale transformation, median filtering and threshold segmentation processing on the collected fruit surface image to obtain a fruit binary image, and uses an edge extraction function to perform contour extraction processing on the fruit binary image to obtain a plurality of fruit contour points and the pixel coordinates of each fruit contour point; S13, according to the pixel coordinates of all fruit outline points and each fruit outline point, the centroid coordinates of the fruit outline point are calculated, and all pixel coordinates of the fruit outline point are subjected to polar coordinate conversion and normalization processing in sequence according to the centroid coordinates to obtain normalized polar coordinates, and the normalized polar coordinates are subjected to periodic function processing to obtain the periodic function of the fruit outline; S14. Expand the periodic function according to the Fourier series and combine it with the Euler formula to obtain harmonic components at various levels, and use the harmonic components at various levels as contour feature parameters of the fruit.

7. The method for detecting fruit hardness of a fruit hardness detection device based on machine vision and acoustic vibration technology according to claim 6, characterized in that: The periodic function of the fruit contour in step S13 is set according to the following formula: r g (θ+2π)=r g (i) Among them, θ represents the angle parameter for converting centroid coordinates into polar coordinates, r g (θ) represents the normalized polar radius with respect to θ, r g (θ+2π) represents the polar radius r g The polar radius after 2π periods of (θ), where π is a constant; In step S14, the periodic function is expanded according to the Fourier series and combined with the Euler formula to obtain the harmonic components of each level according to the following formula: Where F(h) represents the harmonic component at index h, r g () represents the normalized polar radius, n represents the number of pixel coordinates of the fruit contour points, 2π / n represents the radial angle between each adjacent pixel coordinate, h represents the index value, j represents the imaginary unit, k represents the summation variable, and π is a constant.

8. The method for detecting fruit hardness of a fruit hardness detection device based on machine vision and acoustic vibration technology according to claim 5, characterized in that: The step S2 is specifically as follows: S21, collecting vibration signals generated by the fruit rolling on the corrugated plate through a vibration signal collecting mechanism; S22, the data processing module (7) uses a filtering method to perform noise reduction processing on the vibration signal to obtain a vibration signal with noise removed; S23. Perform fast Fourier transform processing on the vibration signal from which noise is removed to obtain time domain data and frequency domain data respectively, perform feature extraction on the time domain data and frequency domain data respectively to obtain time domain feature parameters and frequency domain feature parameters respectively, and summarize the time domain feature parameters and frequency domain feature parameters to obtain vibration feature parameters.

9. The method for detecting fruit hardness of a fruit hardness detection device based on machine vision and acoustic vibration technology according to claim 8, characterized in that: The filtering method in step S22 adopts a wavelet threshold denoising method or a Kalman filtering method; The feature extraction of the time domain data in step S23 is to extract the peak value, mean value, variance and kurtosis of the time domain data; The feature extraction of the frequency domain data in step S23 is to extract the peak frequency, spectrum bandwidth and power spectrum density of the frequency domain data.

10. The method for detecting fruit hardness of a fruit hardness detection device based on machine vision and acoustic vibration technology according to claim 5, characterized in that: The deep learning model in step S6 includes a convolutional neural network, a deep belief network and a generative adversarial network.

Images