Fruit hardness detection method, system, terminal and storage medium

Abstract

The present invention provides a fruit hardness detection method, system, terminal and storage medium. The method includes: driving a clamping device to perform a grasping test on the fruit to be detected to obtain strain data, and performing signal preprocessing on the strain data to obtain preprocessed strain data, where the strain data includes the dynamic strain signal of the clamping device during the clamping process; extracting features from the preprocessed strain data to obtain a three-dimensional feature tensor, and extracting the characteristic frequency band of the three-dimensional feature tensor; constructing a time-frequency fusion image according to the characteristic frequency band, and inputting the time-frequency fusion image into a pre-trained fruit hardness detection model for hardness classification to obtain the target fruit hardness. In the embodiments of the present invention, the target fruit hardness of the fruit to be detected can be automatically analyzed, and there is no need to strike the fruit to be detected, preventing damage to the fruit.

Description

Technical Field

[0001] The present invention relates to the technical field of fruit detection, and particularly to a method, a system, a terminal and a storage medium for detecting the hardness of fruits. Background Art

[0002] Due to the influence of comprehensive factors such as fruit production areas, planting techniques, and climate, the post-harvest quality of fruits is difficult to be consistent. Even for the same variety in the same production area, due to different harvesting and maturity periods, tree body differences, etc., the fruit quality is also difficult to be the same. However, with the improvement of living standards, people need to be differentially satisfied in terms of nutrition, health care, habits, etc. This requires fruits to be graded according to sugar content, hardness, flavor, color, etc. after harvesting to meet people's needs. Therefore, the problem of fruit hardness detection has attracted more and more attention.

[0003] In the existing fruit hardness detection process, generally, acoustic detection by knocking is used to detect the hardness of fruits, which easily causes damage to the fruits and reduces the user experience. Summary of the Invention

[0004] The purpose of the embodiments of the present invention is to provide a method, a system, a terminal and a storage medium for detecting the hardness of fruits, so as to solve the problem that fruits are easily damaged in the existing fruit hardness detection process.

[0005] The embodiments of the present invention are implemented as follows. A method for detecting the hardness of fruits, the method includes:

[0006] Driving a gripper device to perform a grasping test on a fruit to be detected, obtaining strain data, and performing signal preprocessing on the strain data to obtain preprocessed strain data, where the strain data includes the dynamic strain signal of the gripper device during the grasping process;

[0007] Performing feature extraction on the preprocessed strain data to obtain a three-dimensional feature tensor, and extracting the characteristic frequency band of the three-dimensional feature tensor;

[0008] Constructing a time-frequency fusion image according to the characteristic frequency band, and inputting the time-frequency fusion image into a pre-trained fruit hardness detection model for hardness classification to obtain the target fruit hardness.

[0009] Preferably, before inputting the time-frequency fusion image into the pre-trained fruit hardness detection model for hardness classification, it further includes:

[0010] Inputting a time-frequency sample image into a first feature extraction module in the fruit hardness detection model for feature extraction to obtain sample extraction features, and performing global average pooling processing on the sample extraction features to obtain a channel description vector;

[0011] Input the channel description vector into the first fully connected layer for fully connected processing to obtain a first fully connected vector, and compress the number of channels of the first fully connected layer to obtain an activation value;

[0012] Input the first fully connected vector into the second fully connected layer for fully connected processing to obtain a second fully connected vector, and restore the number of channels of the second fully connected layer to obtain an attention weight;

[0013] Multiply the attention weight and the sample extraction feature channel by channel to obtain an optimized feature, and perform batch normalization on the optimized feature to obtain a normalized feature;

[0014] Calculate the model loss based on the normalized feature, and update the parameters of the fruit hardness detection model according to the model loss until the fruit hardness detection model converges.

[0015] Preferably, perform signal preprocessing on the strain data to obtain preprocessed strain data, including:

[0016] Locate the null values in the strain data to obtain the null value positions, and perform adaptive median filtering on the strain data according to the null value positions to obtain filtered data;

[0017] Locate the zero values in the filtered data to obtain the zero value positions, and perform linear interpolation on the zero value positions in the filtered data to obtain interpolated data;

[0018] Perform non-linear signal demodulation on the interpolated data to obtain instantaneous amplitude envelope features, and perform normalization processing on the instantaneous amplitude envelope features to obtain the preprocessed strain data.

[0019] Preferably, construct a time-frequency fusion image according to the characteristic frequency band, including:

[0020] Perform wavelet transform on the characteristic frequency band, and calculate the energy density of the characteristic frequency band after wavelet transform;

[0021] Determine the frequency axis according to the energy density, and perform visualization processing on the characteristic frequency band according to the frequency axis and the energy density to obtain a time-frequency image;

[0022] Perform bicubic interpolation and adaptive grid resampling on the time-frequency image to obtain a sampled image, and perform three-channel time-frequency mapping on the sampled image to obtain a three-channel mapped image;

[0023] Fuse the three-channel mapped image to obtain the time-frequency fusion image.

[0024] Preferably, non-linear signal demodulation is performed on the interpolation data to obtain the instantaneous amplitude envelope feature, including:

[0025] Sample the interpolation data to obtain sampling information, and obtain the reference frequency in the sampling signal according to the fast Fourier transform;

[0026] Construct a mixed objective function between energy entropy and signal-to-noise ratio according to the reference frequency, and determine the objective function value of the band signal in the interpolation data according to the mixed objective function;

[0027] Extract the band signals in the interpolation data whose objective function values are greater than the function threshold to obtain a set of target bands, and merge the band signals with a distance less than the preset distance in the set of target bands to obtain an optimized band signal;

[0028] Perform signal decomposition on the optimized band signal to obtain intrinsic mode function components, and perform Hilbert transform on the intrinsic mode function components to generate an analytic signal;

[0029] Filter the analytic signal to obtain the instantaneous amplitude envelope feature.

[0030] Preferably, drive the gripper device to perform a grasping test on the fruit to be detected to obtain strain data, including:

[0031] Control the stepping motor to drive the gripper device to close, and continuously apply a closing force to the gripper device;

[0032] Control the working surface of the gripper device to fit the surface of the fruit to be detected to generate a bending deformation, and control the strain array on the gripper device to record the continuous strain data of the gripper device in real time;

[0033] Calibrate the strain array according to the temperature compensation model and the strain compensation matrix, and stop data recording when the stepping motor reaches the predetermined stroke to obtain the strain data;

[0034] The temperature compensation model is:

[0035]

[0036] where Δ R raw is the original resistance change amount, T is the real-time temperature detection value, T 0 is the preset reference temperature, α is the preset temperature coefficient, β is the dynamic response correction factor, F is the contact force change rate, F max= 8.5 N is the maximum design load of the gripper device, Δ R cal is the change amount of the compensation resistance for the gripper device;

[0037] The strain compensation matrix is:

[0038]

[0039] wherein, V1, V2, and V3 are the original voltage signals of the three strain arrays, V1 cal , V2 cal , V3 cal are the voltage signals after calibration corresponding to the strain arrays.

[0040] Another object of the embodiments of the present invention is to provide a fruit hardness detection system, and the system includes:

[0041] A gripper device for performing a grasping test on the fruit to be detected;

[0042] A signal processing module for driving the gripper device to perform a grasping test on the fruit to be detected, obtaining strain data, and performing signal preprocessing on the strain data to obtain strain preprocessing data, where the strain data includes the dynamic strain signal of the gripper device during the grasping process;

[0043] A feature extraction module for extracting features from the strain preprocessing data to obtain a three-dimensional feature tensor and extracting the feature frequency band of the three-dimensional feature tensor;

[0044] A hardness classification module for constructing a time-frequency fusion image according to the feature frequency band and inputting the time-frequency fusion image into a pre-trained fruit hardness detection model for hardness classification to obtain the target fruit hardness.

[0045] Preferably, the gripper device includes three flexible grippers, a strain array connected to the flexible grippers, a slide bar, and a gripper drive module. The gripper drive module includes a motor fixing platform, a stepping motor, a threaded flange, and a bottom plate. The three flexible grippers are arranged on the gripper drive module at equal intervals along the circumference. One end of each flexible gripper is hinged to the stepping motor through the slide bar. The side surface of the flexible gripper close to the gripper drive module is used as the working surface, and a silica gel protective sleeve is arranged on the working surface along the direction pointing to the gripper drive module.

[0046] In the embodiments of the present invention, by performing signal preprocessing on strain data, the extraction of three-dimensional feature tensors is effectively facilitated. By extracting the characteristic frequency bands of the three-dimensional feature tensors, a time-frequency fusion image can be effectively constructed. By inputting the time-frequency fusion image into a pre-trained fruit hardness detection model for hardness classification, the target fruit hardness of the fruit to be detected can be automatically analyzed, without the need to strike the fruit to be detected, preventing damage to the fruit and improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 is a flowchart of a fruit hardness detection method provided by the first embodiment of the present invention;

[0048] Figure 2 is a schematic diagram of three-channel original signals in strain data provided by the first embodiment of the present invention;

[0049] Figure 3 is a schematic diagram of strain preprocessing data provided by the first embodiment of the present invention;

[0050] Figure 4 is a schematic diagram of a time-frequency image provided by the first embodiment of the present invention;

[0051] Figure 5 is a schematic diagram of a time-frequency fusion image provided by the first embodiment of the present invention;

[0052] Figure 6 is a schematic diagram of the structure of a fruit hardness detection system provided by the second embodiment of the present invention;

[0053] Figure 7 is a schematic diagram of the structure of a clamping device provided by the second embodiment of the present invention;

[0054] Figure 8 is a schematic diagram of a prediction result provided by the second embodiment of the present invention;

[0055] Figure 9 is a schematic diagram of the structure of a terminal device provided by the third embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0056] In order to make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0057] In order to illustrate the technical solutions described in the present invention, the following will be described through specific embodiments.

[0058] Embodiment 1

[0059] Please refer to Figures 1 to 5, is a flowchart of the fruit hardness detection method provided by the first embodiment of the present invention. This fruit hardness detection method can be applied to any device or system. The fruit hardness detection method includes the steps:

[0060] Step S10, drive the gripper device to perform a grasping test on the fruit to be detected, obtain strain data, and perform signal preprocessing on the strain data to obtain preprocessed strain data;

[0061] Among them, the strain data includes the dynamic strain signal of the gripper device during the grasping process;

[0062] Specifically, in this step, the fruit to be detected is placed in the flexible gripper of the gripper device. The flexible gripper is driven to close by the forward rotation of the stepper motor, and a closing force is continuously applied to the flexible gripper to make the working surfaces of the flexible gripper fit the fruit surface and generate bending deformation; in all grasping tests, the working stroke of the stepper motor remains the same. When the stepper motor starts, the strain array on the gripper device starts to record the dynamic strain signal of the gripper. When the stepper motor reaches the predetermined stroke, the gripper no longer deforms further, and at the same time, the data recording stops to obtain the strain data.

[0063] Optionally, the strain signal acquisition process includes: controlling the stepper motor to close the flexible gripper at a constant rate, with the motor stroke fixed at 15 mm; continuously applying a closing force after the flexible gripper contacts the fruit surface until the preset stroke end point is reached; immediately stopping data acquisition when the deformation amount of any strain array exceeds 80% of the range.

[0064] Optionally, in this step, multi-dimensional signal reconstruction can be implemented based on the three-axis sensing data in the strain array. First, perform abnormal data purification on the strain data, use the null value identification algorithm to locate and compensate for non-numeric (Not a Number) and infinity values in the strain data, and use the adaptive median filter combined with the zero-value filling strategy to eliminate signal distortion in the strain data. Use a fourth-order Butterworth filter bank to construct a band-pass filter array of 0.54 - 4.99 Hz, and accurately match the vibration response characteristics of the pulp tissue in the strain data through the frequency band optimization algorithm. For the matched vibration response characteristics of the pulp tissue, apply the Hilbert-Huang transform (HHT) for non-linear signal demodulation, extract the instantaneous amplitude envelope characteristics with physical interpretability, use the dynamic range compression technology (DRC), establish an adaptive normalization model based on a sliding window, and dynamically map the instantaneous amplitude envelope characteristics to the non-saturated interval of [0.1, 0.9] based on the adaptive normalization model, retaining the non-stationary characteristics of the signal to obtain the preprocessed strain data.

[0065] Optionally, performing signal preprocessing on the strain data to obtain preprocessed strain data includes:

[0066] Locate the null values in the strain data to obtain the null positions, and perform adaptive median filtering on the strain data according to the null positions to obtain filtered data;

[0067] Locate the zero values in the filtered data to obtain the zero positions, and perform linear interpolation on the zero positions in the filtered data to obtain interpolated data;

[0068] Perform non-linear signal demodulation on the interpolated data to obtain the instantaneous amplitude envelope feature, and perform normalization processing on the instantaneous amplitude envelope feature to obtain the preprocessed strain data;

[0069] Further, performing non-linear signal demodulation on the interpolated data to obtain the instantaneous amplitude envelope feature includes:

[0070] Sample the interpolated data to obtain sampling information, and obtain the reference frequency in the sampling signal according to the fast Fourier transform; among them, the interpolated data is sampled at a sampling rate of ≥5Hz, and the first 3 significant resonance peaks are extracted by fast Fourier transform analysis as the reference frequency;

[0071] Construct a mixed objective function between energy entropy and signal-to-noise ratio according to the reference frequency, and determine the objective function value of the band signal in the interpolated data according to the mixed objective function; among them, a mixed objective function including energy entropy and signal-to-noise ratio is established (objective function value F = 0.6*(1 / H_e)+0.4*signal-to-noise ratio, H_e is energy entropy);

[0072] Extract the band signals in the interpolated data whose objective function values are greater than the function threshold to obtain a set of target bands, and merge the band signals with a spacing less than the preset spacing in the set of target bands to obtain optimized band signals;

[0073] Perform signal decomposition on the optimized band signals to obtain intrinsic mode function components, perform Hilbert transform on the intrinsic mode function components to generate analytic signals, and filter the analytic signals to obtain the instantaneous amplitude envelope feature;

[0074] Among them, the particle swarm optimization algorithm is used to dynamically adjust the filter parameters (center frequency, bandwidth, roll-off rate), the bandwidth is constrained to be 0.54 - 4.99Hz and the overlap degree <20%, the bands with F value >0.8 are retained, and the adjacent bands with a spacing <50Hz are merged to obtain optimized band signals to ensure that more than 90% of the vibration energy is covered.

[0075] In this step, the optimized band signal is decomposed into intrinsic mode function components, a cubic spline interpolation is used to construct an envelope line, a termination condition with a mean envelope energy ratio < 0.05 is set, and a Hilbert transform is performed on each intrinsic mode function component to generate an analytic signal, and the smooth instantaneous amplitude envelope feature is extracted in combination with a Savitzky-Golay filter.

[0076] Furthermore, the driving gripper device performs a grasping test on the fruit to be detected to obtain strain data, including:

[0077] Control the stepping motor to drive the gripper device to close, and continuously apply a closing force to the gripper device;

[0078] Control the working surface of the gripper device to fit the surface of the fruit to be detected to generate a bending deformation, and control the strain array on the gripper device to record the continuous strain data of the gripper device in real time;

[0079] Calibrate the strain array according to the temperature compensation model and the strain compensation matrix, and when the stepping motor reaches a predetermined stroke, stop data recording to obtain the strain data; among them, by calibrating the strain array according to the temperature compensation model and the strain compensation matrix, the accuracy of the strain data is effectively improved;

[0080] The temperature compensation model is:

[0081]

[0082] where, Δ R raw is the original resistance change, T is the real-time temperature detection value, T 0 is the preset reference temperature, α is the preset temperature coefficient, β is the dynamic response correction factor, F is the contact force change rate, F max = 8.5N is the maximum design load of the gripper device, Δ R cal is the compensated resistance change of the gripper device;

[0083] The strain compensation matrix is:

[0084]

[0085] where, V1, V2, V3 are the original voltage signals of the three strain arrays, V1 cal , V2 cal , V3 cal are the voltage signals after calibrating the corresponding strain arrays.

[0086] Step S20: Extract features from the strain preprocessed data to obtain a three-dimensional feature tensor, and extract the characteristic frequency band of the three-dimensional feature tensor;

[0087] Among them, a time-frequency analysis matrix is constructed using the Morlet complex wavelet basis function, and the dynamic balance of time-frequency resolution is achieved through a parametric wavelet basis optimization algorithm (center frequency fc = 1 Hz, bandwidth parameter σ = 5), generating a three-dimensional feature tensor containing time-frequency-energy.

[0088] Optionally, the steps for generating the three-dimensional feature tensor are: Set the initial parameters of the Morlet wavelet basis to the center frequency fc = 1 Hz and the bandwidth σ = 5, and construct the Morlet wavelet basis function in complex form ψ :

[0089]

[0090] where t is time, σ is the bandwidth, π is the pi, j is the imaginary unit, satisfying j2 = -1, f c is the carrier frequency, and e j2πfct represents the complex exponential modulation term with f c as the center frequency.

[0091] Adopt a particle swarm optimization strategy to dynamically adjust the (fc, σ) parameter pair:

[0092] Define the objective function F = α * (frequency resolution) + β * (time resolution) (α = 0.6, β = 0.4), and the constraint conditions are: f c ∈[0.54, 4.99] Hz, σ ∈ [3, 8];

[0093] Achieve dynamic balance through the time-frequency focusing degree criterion:

[0094] Calculate the time-frequency plane energy entropy H = -Σ(pij log pij), where pij is the proportion of the time-frequency point energy, and stop the iteration optimization when H ≤ 0.35. Construct the time-frequency analysis matrix and perform the complex Morlet wavelet transform:

[0095]

[0096] where W(t, f) represents the time-frequency analysis matrix, x(τ) is the input signal function, τ is the time variable, representing the change of the signal over time, and ψ * represents the complex conjugate of ψ, t is the time parameter, used to control the translation of ψ * on the time axis, s is the scale parameter, used to control the stretching of ψ * and dτ represents the integral operation on τ.

[0097] The two-dimensional time-frequency energy matrix E(t, f) = |W(t, f)|, expands and tensors the feature dimension, extracts multi-scale features by slicing along the time axis, uses Haar wavelet decomposition to obtain four time-frequency sub-bands (ALL, HLR, VRL, DRR), and constructs a three-dimensional feature tensor according to the obtained four time-frequency sub-bands;

[0098] Among them, ALL is the approximate sub-band, retaining the main low-frequency features of the signal, HLR represents the detail sub-band in the horizontal direction, VRL represents the detail sub-band in the vertical direction, and DRR represents the detail sub-band in the diagonal direction;

[0099] Respectively construct the two-dimensional time-frequency energy matrix corresponding to each layer of time-frequency sub-band, stack the four two-dimensional time-frequency energy matrices in sequence along the third dimension to form a three-dimensional feature tensor of t×f×4, where the first dimension t is time, the second dimension f is frequency, and the third dimension is the sub-band index.

[0100] Step S30, construct a time-frequency fusion image according to the feature frequency band, and input the time-frequency fusion image into a pre-trained fruit hardness detection model for hardness classification to obtain the target fruit hardness;

[0101] Optionally, constructing a time-frequency fusion image according to the feature frequency band includes:

[0102] Perform wavelet transform on the feature frequency band, and calculate the energy density of the feature frequency band after wavelet transform;

[0103] Determine the frequency axis according to the energy density, and perform visualization processing on the feature frequency band according to the frequency axis and the energy density to obtain a time-frequency image;

[0104] Perform bicubic interpolation and adaptive grid resampling on the time-frequency image to obtain a sampled image, and perform three-channel time-frequency mapping on the sampled image to obtain a three-channel mapped image;

[0105] Fuse the three-channel mapped image to obtain the time-frequency fusion image.

[0106] Optionally, in this embodiment, the steps of generating a time-frequency fusion image further include:

[0107] Segment the spectral energy of the feature frequency band:

[0108] R channel: Integral operation in the 0.45 - 2Hz frequency band:

[0109]

[0110] where f(t): is the amplitude of the signal in the frequency domain, df is the frequency differential variable, ∣f(t)∣2 is the energy density of the signal at frequency f, E R (t) is the total energy of the 0.45 - 2 Hz frequency band corresponding to time t.

[0111] G channel: Extraction of complex analytic signal in the 2 - 3.5 Hz frequency band:

[0112]

[0113] where x(t) is the original time - domain signal, Hilbert(x(t)) is the complex analytic signal generated by Hilbert transform, and E G (t) is the total energy of the 2 - 3.5 Hz frequency band corresponding to time t.

[0114] B channel: Wavelet energy coefficient in the 3.5 - 5 Hz frequency band:

[0115]

[0116] where CWT is the continuous wavelet transform coefficient, corresponding to the 3.5 - 5 Hz frequency band, and E B (t) is the total energy of the 3.5 - 5 Hz frequency band corresponding to time t.

[0117] After segmentation, energy normalization and gamma correction enhancement processing are performed on the characteristic frequency bands:

[0118]

[0119] where E is the original energy value, max(E) is the maximum value of the channel energy, E / max(E) is the normalized energy value, 1 / 2.2 is the gamma correction coefficient for enhancing dark - part details, E′ is the output energy value after normalization and gamma correction. After obtaining the output energy value, the time axis is expanded into the spatial dimension and then vertical pixel columns are generated according to the sampling points to form a time - frequency fusion image.

[0120] Optionally, before inputting the time - frequency fusion image into the pre - trained fruit hardness detection model for hardness classification, it further includes:

[0121] Inputting the time - frequency sample image into the first feature extraction module in the fruit hardness detection model for feature extraction to obtain sample extraction features, and performing global average pooling processing on the sample extraction features to obtain a channel description vector;

[0122] Inputting the channel description vector into the first fully - connected layer for fully - connected processing to obtain a first fully - connected vector, and compressing the number of channels of the first fully - connected layer to obtain an activation value;

[0123] Input the first fully-connected vector into the second fully-connected layer for fully-connected processing to obtain a second fully-connected vector, and restore the number of channels of the second fully-connected layer to obtain attention weights;

[0124] Multiply the attention weights and the sample-extracted features channel by channel to obtain optimized features, and perform normalization processing on the optimized features using batch normalization to obtain normalized features;

[0125] Calculate the loss based on the normalized features to obtain a model loss, and update the parameters of the fruit hardness detection model according to the model loss until the fruit hardness detection model converges;

[0126] Among them, the fruit hardness detection model uses an attention-based CNN model, and based on the feature enhancement mechanism of the channel attention module, dynamically adjusts the weights of the feature channels to strengthen the key frequency band information related to the fruit hardness. For the input multi-channel time-frequency sample image (dimension: channel × height × width), perform global average pooling operation along the spatial dimension, compress the two-dimensional features of each channel into a scalar value, and generate a channel-level channel description vector to represent the overall activation intensity of each channel. Learn the non-linear relationship between channels through a two-layer fully-connected network, input the channel description vector into the first fully-connected layer, compress the number of channels to 1 / 16 of the original dimension, and reduce the computational complexity. Input the dimension-reduced features into the second fully-connected layer to restore to the original number of channels. Normalize the output value to the [0,1] interval through the Sigmoid function to generate attention weights. The larger the weight value, the more significant the contribution of the corresponding channel to the hardness prediction. Multiply the learned attention weights and the sample-extracted features channel by channel to enhance the response intensity of important channels and suppress irrelevant or noisy channels. This process enables the network to autonomously focus on the vibration frequency band features related to the fruit elastic modulus. Normalize the optimized features through batch normalization to accelerate training convergence and reduce internal covariate shift.

[0127] The fruit hardness detection model includes the following sequentially connected structures:

[0128] Input layer: Receive an RGB time-frequency image with a size of 224×224×3;

[0129] First feature extraction module: 3×3 convolutional layer (32 channels) → batch normalization → ReLU activation → 2×2 max pooling;

[0130] Squeeze-and-Excitation attention module: Generate a channel description vector through global average pooling, and generate attention weights after performing channel correlation modeling through two fully-connected layers;

[0131] Second feature extraction module: 3×3 convolutional layer (64 channels) → batch normalization → ReLU activation → 2×2 max pooling;

[0132] Classification module: 128-unit fully connected layer → ReLU activation → Dropout layer → Softmax classification layer.

[0133] Optionally, the working process of the channel attention module includes:

[0134] Performing global average pooling on the input feature map to generate a channel description vector z ∈ R C ,R C represents the space composed of all C-dimensional real vectors;

[0135] Reducing the dimension through a fully connected layer with a compression ratio of 16: activation value s' = δ(W1z), W1 (the first weight matrix) ∈ R (C / 16×C) for linearly transforming the input z, and δ is the ReLU activation function;

[0136] Restoring the dimension through a fully connected layer and generating attention weights s = σ(W2s'), where W2 (the second weight matrix) ∈ R (C×C / 16) and σ is the Sigmoid function;

[0137] Multiplying the attention weights s with the original feature map channel by channel to achieve feature recalibration.

[0138] The classification module outputs five hardness levels, including:

[0139] Class I: Too soft (hardness value 3.7 N / cm² < 4.4 N / cm²);

[0140] Class II: Soft (4.5 N / cm² ≤ hardness value < 5.3 N / cm²);

[0141] Class III: Moderate (5.4 N / cm² ≤ hardness value < 6.7 N / cm²);

[0142] Class IV: Hard (6.8 N / cm² ≤ hardness value < 7.7 N / cm²);

[0143] Class V: Too hard (7.8 N / cm² ≤ hardness value < 8.4 N / cm²).

[0144] In this embodiment, by performing signal preprocessing on the strain data, it effectively facilitates the extraction of the three-dimensional feature tensor. By extracting the characteristic frequency band of the three-dimensional feature tensor, a time-frequency fusion image can be effectively constructed. By inputting the time-frequency fusion image into the pre-trained fruit hardness detection model for hardness classification, the target fruit hardness of the fruit to be detected can be automatically analyzed. There is no need to strike the fruit to be detected, which prevents damage to the fruit and improves the user experience. Expanding the one-dimensional signal into two-dimensional spatio-temporal features has high precision and can simultaneously achieve stable grasping of the fruit and detection of the fruit hardness.

[0145] Embodiment 2

[0146] Please refer to Figures 6 to 8 , which is a schematic structural diagram of the fruit hardness detection system 100 provided by the second embodiment of the present invention, including:

[0147] The gripper device 10 is used to perform a grasping test on the fruit to be detected;

[0148] The signal processing module 11 is used to drive the gripper device 10 to perform a grasping test on the fruit to be detected, obtain strain data, and perform signal preprocessing on the strain data to obtain strain preprocessed data. The strain data includes the dynamic strain signal of the gripper device 10 during the grasping process.

[0149] Optionally, the signal processing module 11 is further used to: locate null values in the strain data to obtain the null value positions, and perform adaptive median filtering on the strain data according to the null value positions to obtain filtered data;

[0150] Locate zero values in the filtered data to obtain the zero value positions, and perform linear interpolation on the zero value positions in the filtered data to obtain interpolated data;

[0151] Perform non-linear signal demodulation on the interpolated data to obtain the instantaneous amplitude envelope feature, and perform normalization processing on the instantaneous amplitude envelope feature to obtain the strain preprocessed data.

[0152] Furthermore, the signal processing module 11 is further used to: perform non-linear signal demodulation on the interpolated data to obtain the instantaneous amplitude envelope feature, including:

[0153] Sample the interpolated data to obtain sampling information, and obtain the reference frequency in the sampling signal according to the fast Fourier transform;

[0154] Construct a mixed objective function between energy entropy and signal-to-noise ratio according to the reference frequency, and determine the objective function value of the band signal in the interpolated data according to the mixed objective function;

[0155] Extract the band signals in the interpolation data where the objective function value is greater than the function threshold to obtain an objective band set, and merge the band signals in the objective band set with a spacing less than a preset spacing to obtain an optimized band signal;

[0156] Perform signal decomposition on the optimized band signal to obtain intrinsic mode function components, and perform Hilbert transform on the intrinsic mode function components to generate an analytic signal;

[0157] Filter the analytic signal to obtain the instantaneous amplitude envelope feature.

[0158] Furthermore, the signal processing module 11 is further configured to: control the stepping motor 4 to drive the gripper device 10 to close, and continuously apply a closing force to the gripper device 10;

[0159] Control the working surface of the gripper device 10 to fit the surface of the fruit to be detected to generate a bending deformation, and control the strain array 2 on the gripper device 10 to continuously record the continuous strain data of the gripper device 10;

[0160] Calibrate the strain array 2 according to the temperature compensation model and the strain compensation matrix, and stop data recording when the stepping motor 4 reaches a predetermined stroke to obtain the strain data;

[0161] The temperature compensation model is:

[0162]

[0163] where, Δ R raw is the original resistance change amount, T is the real-time temperature detection value, T 0 is the preset reference temperature, α is the preset temperature coefficient, β is the dynamic response correction factor, F is the contact force change rate, F max = 8.5N is the maximum design load of the gripper device 10, Δ R cal is the compensated resistance change amount of the gripper device 10;

[0164] The strain compensation matrix is:

[0165]

[0166] where, V1, V2, V3 are the original voltage signals of the strain array 2, V1 cal , V2 cal , V3 calIt is the voltage signal after calibrating the strain array 2.

[0167] The feature extraction module 12 is used to extract features from the strain preprocessed data to obtain a three-dimensional feature tensor and extract the feature frequency band of the three-dimensional feature tensor.

[0168] The hardness classification module 13 is used to construct a time-frequency fusion image according to the feature frequency band and input the time-frequency fusion image into a pre-trained fruit hardness detection model for hardness classification to obtain the target fruit hardness.

[0169] Optionally, the hardness classification module 13 is further used for:

[0170] Performing wavelet transform on the feature frequency band and calculating the energy density of the feature frequency band after wavelet transform;

[0171] Determining the frequency axis according to the energy density and performing visualization processing on the feature frequency band according to the frequency axis and the energy density to obtain a time-frequency image;

[0172] Performing bicubic interpolation and adaptive grid resampling on the time-frequency image to obtain a sampled image, and performing three-channel time-frequency mapping on the sampled image to obtain a three-channel mapped image;

[0173] Fusing the three-channel mapped images to obtain the time-frequency fusion image.

[0174] Furthermore, the hardness classification module 13 is further used for: inputting the time-frequency sample image into the first feature extraction module in the fruit hardness detection model for feature extraction to obtain sample extraction features, and performing global average pooling processing on the sample extraction features to obtain a channel description vector;

[0175] Inputting the channel description vector into the first fully connected layer for fully connected processing to obtain a first fully connected vector, and compressing the number of channels of the first fully connected layer to obtain an activation value;

[0176] Inputting the first fully connected vector into the second fully connected layer for fully connected processing to obtain a second fully connected vector, and restoring the number of channels of the second fully connected layer to obtain an attention weight;

[0177] Multiplying the attention weight and the sample extraction features channel by channel to obtain optimized features, and performing normalization processing on the optimized features by using batch normalization to obtain normalized features;

[0178] Calculating a model loss according to the normalized features, and updating the parameters of the fruit hardness detection model according to the model loss until the fruit hardness detection model converges.

[0179] The gripper device 10 includes three flexible grippers 1, a strain array 2 connected to the flexible grippers 1, a slide bar 3, and a gripper driving module 5. The gripper driving module 5 includes a motor fixing platform, a stepping motor 4, a threaded flange, and a bottom plate. The three flexible grippers 1 are arranged on the gripper driving module 5 at equal intervals along the circumference. One end of each flexible gripper 1 is hinged to the stepping motor 4 through the slide bar 3. The side surface of the flexible gripper 1 close to the gripper driving module 5 serves as the working surface, and a silica gel protective sleeve is arranged on the working surface along the direction pointing to the gripper driving module 5.

[0180] The bottom plate and the motor fixing platform are arranged at intervals from bottom to top. A plurality of card slots are opened on the motor fixing platform. The lower end of the slide bar 3 is fixedly installed on the motor fixing platform, and the upper end of the slide bar 3 passes through the strip-shaped through groove and is hinged to the bottom surface side of the corresponding flexible gripper 1 close to the center of the gripper driving module 5.

[0181] A bottom plate counterbore is opened in the upper middle part of the bottom plate. The stepping motor 4 is fixedly installed on the lower surface of the motor fixing platform. The output shaft screw of the stepping motor 4 is coaxially connected to the threaded flange through a thread, and the end of the output shaft screw passes through the threaded flange and is movably installed in the bottom plate counterbore.

[0182] Place fruits between the flexible grippers 1. Start the stepping motor 4 to drive the flexible grippers 1 to move closer inward to grasp the fruits. The strain array 2 records the deformation during the grasping process of the grippers to obtain strain data. The strain data is converted into a time-frequency image through wavelet transform, and the time-frequency image is processed using an attention-based CNN algorithm to predict the fruit hardness information and obtain the target fruit hardness, realizing the detection of fruit hardness.

[0183] In this embodiment, due to its special soft structure, the flexible gripper 1 will change to fit the fruit surface according to the shape and size of the fruit when grasping the fruit, and a silica gel protective sleeve with a mesh structure is attached to the contact surface of the gripper, which can not only ensure the stable grasping of the fruit but also will not cause extrusion damage to the fruit.

[0184] The bottom end of the working surface of the flexible gripper 1 is hinged to the center of the gripper driving module 5 through a connecting rod. When grasping the fruit, the working stroke of the stepping motor 4 remains the same, and the gripper can no longer deform further, which can ensure that the power of the motor will not be overloaded due to different fruit shapes and sizes during grasping, and can well protect the fruit.

[0185] In this embodiment, the test samples of the fruit hardness detection model are 300 Hayward kiwifruits, with a hardness range of 3.7 - 8.4 N / cm. The overall accuracy of the pre-trained fruit hardness detection model is shown as 84.3% (253 / 300), and the weighted average of F1-score is 84.1%.

[0186] In this embodiment, the flexible gripper 1 is driven by the rotation of the stepper motor 4 to drive the slide rod 3 to open and close the flexible gripper 1. When the flexible gripper 1 is closed, the flexible gripper 1 will bend and deform according to the fruit shape curve. The silicone protective sleeve on the flexible gripper 1 will closely fit the fruit, and the fruit is not easy to slip. The strain array 2 records the deformation during the gripping process of the gripper and uploads it to the signal processing module 11. The strain data is converted into a time-frequency image through wavelet transform, and the image is processed using an attention-based CNN algorithm to predict the fruit hardness information. This flexible gripper 1 can be used to grasp fruits during the picking, transferring, and loading process. The flexible gripper can reduce fruit damage. At the same time, the fruit hardness information obtained through the algorithm can obtain different maturity levels of fruits, realize graded conveying, achieve non-destructive detection, low cost, expand one-dimensional signals into two-dimensional spatio-temporal features, high accuracy, and can simultaneously achieve stable grasping of fruits and detection of fruit hardness.

[0187] In this embodiment, by performing signal preprocessing on the strain data, it effectively facilitates the extraction of three-dimensional feature tensors. By extracting the characteristic frequency bands of the three-dimensional feature tensors, a time-frequency fusion image can be effectively constructed. By inputting the time-frequency fusion image into a pre-trained fruit hardness detection model for hardness classification, the target fruit hardness of the fruit to be detected can be automatically analyzed, without the need to strike the fruit to be detected, preventing damage to the fruit and improving the user experience.

[0188] Embodiment III

[0189] Figure 9 is a structural block diagram of a terminal device provided in the third embodiment of the present application. As Figure 9 shown, the terminal device of this embodiment includes: a processor 20, a memory 21, and a computer program 22 stored in the memory 21 and executable on the processor 20, such as a program for the fruit hardness detection method. When the processor 20 executes the computer program 22, the steps in each embodiment of the above various fruit hardness detection methods are implemented.

[0190] Exemplarily, the computer program 22 can be divided into one or more modules. The one or more modules are stored in the memory 21 and executed by the processor 20 to complete the present application. The one or more modules can be a series of computer program instruction segments capable of performing specific functions, and this instruction segment is used to describe the execution process of the computer program 22 in the terminal device. The terminal device may include, but is not limited to, a processor 20 and a memory 21.

[0191] The so-called processor 20 may be a Central Processing Unit (CPU), or may also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor, or the processor may also be any conventional processor, etc.

[0192] The memory 21 may be an internal storage unit of the terminal device, such as the hard disk or memory of the terminal device. The memory 21 may also be an external storage device of the terminal device, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. equipped on the terminal device. Further, the memory 21 may also include both the internal storage unit and the external storage device of the terminal device. The memory 21 is used to store the computer program and other programs and data required by the terminal device. The memory 21 may also be used to temporarily store the data that has been output or is to be output.

[0193] In addition, in each embodiment of the present application, the various functional modules may be integrated in a processing unit, or each unit may exist physically alone, or two or more units may be integrated in one unit. The above integrated unit may be implemented in the form of hardware or in the form of a software functional unit.

[0194] When an integrated module is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Among them, the computer-readable storage medium can be non-volatile or volatile. Based on such an understanding, to implement all or part of the processes in the above-mentioned embodiment methods of the present application, it can also be completed by instructing relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file or some intermediate form, etc. The computer-readable storage medium can include: any entity or device capable of carrying computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content included in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice within the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable storage medium does not include electrical carrier signals and telecommunication signals.

[0195] The above-mentioned embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for detecting the hardness of fruits, characterized in that, The method includes: Driving a gripper device to perform a grasping test on a fruit to be detected, obtaining strain data, and performing signal preprocessing on the strain data to obtain preprocessed strain data, where the strain data includes the dynamic strain signal of the gripper device during the grasping process; Performing feature extraction on the preprocessed strain data to obtain a three-dimensional feature tensor, and extracting the characteristic frequency band of the three-dimensional feature tensor; Constructing a time-frequency fusion image according to the characteristic frequency band, and inputting the time-frequency fusion image into a pre-trained fruit hardness detection model for hardness classification to obtain the target fruit hardness; Constructing a time-frequency fusion image according to the characteristic frequency band, including: Performing wavelet transform on the characteristic frequency band, and calculating the energy density of the characteristic frequency band after wavelet transform; Determining the frequency axis according to the energy density, and performing visualization processing on the characteristic frequency band according to the frequency axis and the energy density to obtain a time-frequency image; Performing bicubic interpolation and adaptive grid resampling on the time-frequency image to obtain a sampled image, and performing three-channel time-frequency mapping on the sampled image to obtain a three-channel mapped image; Fusing the three-channel mapped images to obtain the time-frequency fusion image.

2. The fruit hardness detection method according to claim 1, wherein, Before inputting the time-frequency fusion image into a pre-trained fruit hardness detection model for hardness classification, it further includes: Inputting a time-frequency sample image into the first feature extraction module in the fruit hardness detection model for feature extraction to obtain sample extraction features, and performing global average pooling on the sample extraction features to obtain a channel description vector; Inputting the channel description vector into a first fully connected layer for fully connected processing to obtain a first fully connected vector, and compressing the number of channels of the first fully connected layer to obtain an activation value; Inputting the first fully connected vector into a second fully connected layer for fully connected processing to obtain a second fully connected vector, and restoring the number of channels of the second fully connected layer to obtain an attention weight; Multiplying the attention weight and the sample extraction features channel by channel to obtain optimized features, and performing normalization processing on the optimized features using batch normalization to obtain normalized features; Calculating a model loss according to the normalized features, and updating the parameters of the fruit hardness detection model according to the model loss until the fruit hardness detection model converges.

3. The fruit hardness detection method according to claim 1, wherein Performing signal preprocessing on the strain data to obtain preprocessed strain data, including: Locating null values in the strain data to obtain null positions, and performing adaptive median filtering on the strain data according to the null positions to obtain filtered data; Locating zero values in the filtered data to obtain zero positions, and performing linear interpolation on the zero positions in the filtered data to obtain interpolated data; Performing non-linear signal demodulation on the interpolated data to obtain an instantaneous amplitude envelope feature, and performing normalization processing on the instantaneous amplitude envelope feature to obtain the preprocessed strain data.

4. The fruit hardness detection method according to claim 3, wherein, Performing non-linear signal demodulation on the interpolated data to obtain an instantaneous amplitude envelope feature, including: Sample the interpolation data to obtain sampling information, and acquire the reference frequency in the sampling signal according to the fast Fourier transform; Construct a mixed objective function between energy entropy and signal-to-noise ratio according to the reference frequency, and determine the objective function value of the band signal in the interpolation data according to the mixed objective function; Extract the band signals in the interpolation data whose objective function values are greater than the function threshold to obtain a target band set, and merge the band signals in the target band set with a spacing less than the preset spacing to obtain an optimized band signal; Perform signal decomposition on the optimized band signal to obtain intrinsic mode function components, and perform Hilbert transform on the intrinsic mode function components to generate an analytic signal; Filter the analytic signal to obtain the instantaneous amplitude envelope feature.

5. The fruit hardness detection method according to claim 1, wherein Drive the gripper device to perform a grasping test on the fruit to be detected to obtain strain data, including: Control the stepping motor to drive the gripper device to close, and continuously apply a closing force to the gripper device; Control the working surface of the gripper device to fit the surface of the fruit to be detected to generate a bending deformation, and control the strain array on the gripper device to record the continuous strain data of the gripper device in real time; Calibrate the strain array according to the temperature compensation model and the strain compensation matrix, and stop data recording when the stepping motor reaches a predetermined stroke to obtain the strain data; The temperature compensation model is: Among them, Δ R raw is the original resistance change amount, T is the real-time temperature detection value, T 0 is the preset reference temperature, α is the preset temperature coefficient, β is the dynamic response correction factor, F is the contact force change rate, F max = 8.5N is the maximum design load of the gripper device, Δ R cal is the compensated resistance change amount for the gripper device; The strain compensation matrix is: Among them, V1, V2, and V3 are the original voltage signals of the three strain arrays, and V1 cal , V2 cal , V3 cal are the voltage signals after calibration corresponding to the strain arrays.

6. A fruit hardness detection system, characterized in that, The system includes: A gripper device for performing a grasping test on the fruit to be detected; A signal processing module for driving the gripper device to perform a grasping test on the fruit to be detected to obtain strain data, and performing signal preprocessing on the strain data to obtain preprocessed strain data, where the strain data includes the dynamic strain signal of the gripper device during the grasping process; A feature extraction module for extracting features from the preprocessed strain data to obtain a three-dimensional feature tensor, and extracting the characteristic frequency band of the three-dimensional feature tensor; A hardness classification module for constructing a time-frequency fusion image according to the characteristic frequency band, and inputting the time-frequency fusion image into a pre-trained fruit hardness detection model for hardness classification to obtain the target fruit hardness; The hardness classification module is further used for: performing wavelet transform on the characteristic frequency band, and calculating the energy density of the characteristic frequency band after wavelet transform; Determine the frequency axis according to the energy density, and perform visualization processing on the characteristic frequency band according to the frequency axis and the energy density to obtain a time-frequency image; Perform bicubic interpolation and adaptive grid resampling on the time-frequency image to obtain a sampled image, and perform three-channel time-frequency mapping on the sampled image to obtain a three-channel mapped image; Fuse the three-channel mapped image to obtain the time-frequency fusion image.

7. The fruit hardness detection system according to claim 6, wherein The gripper device includes three flexible grippers, a strain array connected to the flexible grippers, a slide bar, and a gripper driving module. The gripper driving module includes a motor fixing platform, a stepping motor, a threaded flange, and a bottom plate. The three flexible grippers are arranged on the gripper driving module at equal intervals along the circumference. One end of each flexible gripper is hinged to the stepping motor through the slide bar. The side surface of the flexible gripper close to the gripper driving module serves as a working surface, and a silica gel protective sleeve is arranged on the working surface in the direction pointing to the gripper driving module.

8. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images