Fruit nondestructive testing method and system

Abstract

The invention provides a fruit nondestructive testing method and system, and belongs to the technical field of data processing, the method comprises the following steps: obtaining a first feature according to first data of a fruit to be detected, the first data comprising a hyperspectral image, a grayscale image and an ultrasonic echo, and the first feature comprising an internal feature, an epidermis feature and a structural feature; constructing a multi-modal fusion feature vector according to the first feature; and determining an evaluation result of the to-be-detected fruit according to the multi-modal fusion feature vector. According to the fruit nondestructive testing method and system provided by the invention, the accuracy of fruit nondestructive testing can be improved.

Description

Technical Field

[0001] The present application belongs to the field of data processing technology, and more specifically, relates to a fruit non-destructive testing method and system. Background Art

[0002] In the field of fruit production, non-destructive testing of fruit quality is crucial.

[0003] Traditional nondestructive fruit testing methods, such as manual visual inspection, rely heavily on the experience and subjective judgment of the tester, resulting in low efficiency and large errors. Near-infrared spectroscopy can obtain information about the internal composition of the fruit, but its ability to identify external defects is limited. Machine vision inspection is susceptible to accuracy issues when processing fruit images under complex lighting conditions. As the fruit industry continues to improve the accuracy of quality inspections, existing single-use inspection technologies are no longer able to meet these demands. Improving the accuracy of fruit inspections is a pressing technical issue to be addressed in this application. Summary of the Invention

[0004] The purpose of this application is to provide a fruit non-destructive testing method and system to achieve non-destructive testing of fruits and improve the accuracy of fruit quality testing.

[0005] A first aspect of the embodiments of the present application provides a method for non-destructive testing of fruit, comprising: obtaining first features according to first data of the fruit to be inspected, wherein the first data includes a hyperspectral image, a grayscale image, and an ultrasonic echo, and the first features include internal features, epidermal features, and structural features; Constructing a multimodal fusion feature vector based on the first feature; An evaluation result of the fruit to be detected is determined according to the multimodal fusion feature vector, wherein the evaluation result includes a quality grade of the fruit to be detected.

[0006] A second aspect of the embodiments of the present application provides a fruit non-destructive testing system, comprising: an acquisition module, configured to acquire a first feature based on first data of the fruit to be detected, wherein the first data includes a hyperspectral image, a grayscale image, and an ultrasonic echo, and the first feature includes an internal feature, a skin feature, and a structural feature; A construction module, configured to construct a multimodal fusion feature vector based on the first feature; An evaluation module is used to determine an evaluation result of the fruit to be detected based on the multimodal fusion feature vector, wherein the evaluation result includes a quality grade of the fruit to be detected.

[0007] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements the steps of the above-mentioned fruit non-destructive testing method when executing the computer program.

[0008] According to a fourth aspect of the embodiments of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned fruit non-destructive testing method are implemented.

[0009] The beneficial effects of the fruit non-destructive testing method, system, equipment, and medium provided by the embodiments of the present application are as follows: by obtaining hyperspectral images, grayscale images, and ultrasonic images of the fruit to be tested, the internal characteristics, epidermal characteristics, and structural characteristics of the fruit to be tested are determined, and information on the fruit to be tested can be comprehensively collected from multiple dimensions, avoiding the limitations of a single data source; by using different types of first features to construct a multimodal fusion feature vector, the multimodal fusion feature vector can be made to more comprehensively and accurately represent the overall condition of the fruit, thereby more accurately evaluating its quality, and further taking corresponding measures for the fruit to be tested based on the evaluation results. The entire process realizes automated processing, improves the precision of fruit detection, and ensures the accuracy and reliability of fruit detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0011] Figure 1 A schematic diagram of a flow chart of a fruit non-destructive testing method provided in one embodiment of the present application; Figure 2 A schematic diagram of the structure of a fruit non-destructive testing system provided in one embodiment of the present application; Figure 3 A schematic block diagram of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0012] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0013] In order to make the purpose, technical solutions and advantages of this application clearer, specific embodiments will be described below with reference to the accompanying drawings.

[0014] Please refer to Figure 1 , Figure 1 This is a flow chart of a non-destructive testing method for fruit provided in one embodiment of the present application, the method comprising: S101: Acquire first features based on first data of a fruit to be detected, where the first data includes a hyperspectral image, a grayscale image, and an ultrasonic echo, and the first features include internal features, epidermal features, and structural features.

[0015] In this embodiment, hyperspectral imaging equipment can be used to capture hyperspectral images of the fruit, machine vision equipment can be used to capture color images of the fruit, and ultrasonic testing equipment can be used to capture ultrasonic echoes of the fruit. The hyperspectral images can reveal information about the internal composition of the fruit, the color images can reveal information about the fruit's epidermal morphology and surface defects, and the ultrasonic echoes can reveal information about the fruit's internal structure.

[0016] In this embodiment, the various types of collected data can be preprocessed. The preprocessing may include spectral correction and denoising of the collected hyperspectral images; grayscale conversion and image enhancement of color images; and filtering and feature extraction of ultrasonic echoes to improve data quality and the accuracy of feature extraction. Features can be extracted from the preprocessed first data. For example, internal features of the fruit to be detected can be obtained from the preprocessed hyperspectral image, and the internal features can represent internal information of the fruit to be detected; epidermal features of the fruit to be detected can be obtained from the preprocessed grayscale image, and the epidermal features can represent epidermal information of the fruit to be detected; and structural features of the fruit to be detected can be obtained from the preprocessed ultrasonic data, and the structural features can represent structural information of the fruit to be detected.

[0017] S102: Construct a multimodal fusion feature vector based on the first feature.

[0018] In this embodiment, the first feature may be processed to convert all features of different dimensions in the first feature into features of the same dimension. For example, each feature in the first feature may be mapped to a feature space of the same dimension to obtain features of the same dimension after mapping.

[0019] After obtaining features of the same dimension, the weight corresponding to each feature in the first feature in different dimensions can be determined. Based on the weight of each dimension, a weighted calculation is performed on each dimension of each feature in the first feature to obtain a multimodal fusion feature vector.

[0020] By constructing a multimodal fusion feature vector, the description of the fruit can be comprehensive and accurate compared to single modal features, which can provide more accurate information for fruit quality assessment and improve the accuracy of quality assessment.

[0021] S103: Determine an evaluation result of the fruit to be detected based on the multimodal fusion feature vector, where the evaluation result includes a quality grade of the fruit to be detected.

[0022] In this embodiment, a first network model can be constructed, and the first neural network model can include a multi-layer perceptron module composed of multiple fully connected layers, an attention mechanism layer, and an output layer. Among them, the multi-layer perceptron module can learn the feature information of the input multimodal fusion feature vector, the attention mechanism layer can determine the attention weight corresponding to each feature dimension in the feature vector output by the multi-layer perceptron module, and multiply the feature vector output by the multi-layer perceptron module element by element according to the attention weight to obtain a quality feature vector; the output layer can convert the quality feature vector output by the attention mechanism layer into a probability distribution corresponding to each quality level, determine the quality level corresponding to the category with the highest probability as the evaluation result of the fruit to be tested, and output it.

[0023] A large amount of sample data from inspected fruits can be collected, including hyperspectral images, grayscale images, and ultrasonic echoes of each fruit. Based on the above method, the internal, epidermal, and structural characteristics of each fruit can be obtained. A multimodal fusion feature vector is determined based on the internal, epidermal, and structural characteristics of each fruit, and a first network model is trained based on the multimodal fusion feature vector and quality grade corresponding to each fruit. The lower the quality grade of the fruit, the more defects the fruit has. If the quality grade is below a preset first-level threshold, the fruit is judged to be unqualified. If the quality grade is above a preset second-level threshold, the fruit is judged to be high-quality.

[0024] In this embodiment, by adding an attention mechanism module to the traditional convolutional neural network, fruit features can be extracted more effectively, and the internal quality of the fruit can be accurately evaluated to obtain an evaluation result, which includes a quality grade.

[0025] The fruit quality assessment model based on deep learning, combined with the improved CNN structure and attention mechanism module, can more accurately extract fruit quality information, improving the accuracy and efficiency of fruit quality assessment.

[0026] In one embodiment of the present application, obtaining the internal features of the first feature according to the first data of the fruit to be detected includes: Obtaining a target wavelength corresponding to minimum redundant information, performing dimensionality reduction processing on the hyperspectral image according to the target wavelength, and obtaining first image data after dimensionality reduction; determining a contribution rate of a principal component in the first image data, where the principal component is used to represent a key feature in the first image data; The principal component whose contribution rate is greater than a preset first threshold is determined as the internal feature, and the internal feature is used to characterize the internal components in the fruit to be detected.

[0027] In this embodiment, the redundant information corresponding to data information at different wavelengths in a hyperspectral image can be determined, and the minimum redundant information within this redundant information can be obtained. Specifically, data corresponding to each wavelength in the hyperspectral image can be obtained. There is correlation between data corresponding to different wavelengths. If the angle between the vectors corresponding to data at two wavelengths in space is less than a preset angle threshold, then the trend of change is similar, indicating that the data corresponding to the two wavelengths contain duplicate content, i.e., redundant information. Conversely, a larger angle indicates a larger data gap and less redundant information.

[0028] In this embodiment, a vector corresponding to any data can be selected from all data information as a reference vector, and the projection length of the vectors corresponding to the remaining data in the direction of the reference vector can be calculated. The projection length reflects the degree of similarity between the wavelength variable and the initial wavelength variable. The shorter the length, the greater the information difference between the two, the more new information it contains, and the less redundant information. The data corresponding to the shortest projection length can be added to the set of minimum redundant information. Then, the data corresponding to the shortest projection length and the data corresponding to the reference vector are used together as the new reference vectors for the next iteration. The projection length of the remaining data in the space corresponding to the new reference vector is continued to be calculated, and the data corresponding to the shortest projection length is then selected and added to the set of minimum redundant information, and then a new reference vector is re-determined. This iteration is repeated until a preset stopping condition is reached. The preset stopping condition can be determined based on actual conditions, such as selecting a set of specified size, or the data newly added to the set has an overall information gain less than a specific threshold.

[0029] After the iteration, the minimum redundant information set obtained is determined based on the wavelength corresponding to each data point in the set. The data corresponding to each wavelength in the wavelength combination has a low degree of information overlap. Therefore, the hyperspectral image can be subjected to dimensionality reduction based on the wavelength combination, retaining the data corresponding to each wavelength in the wavelength combination to obtain the first image data after dimensionality reduction.

[0030] After obtaining the first image data, the principal components in the first image data can be further determined, and the contribution rate of each principal component can be determined. The data corresponding to different wavelengths in the first image data can be normalized, and then the normalized data can be formed into a matrix. The covariance matrix of the matrix is calculated, and the eigenvalue decomposition of the covariance matrix is performed to solve its eigenvalues and eigenvectors. The solved eigenvalues are sorted from large to small, and the eigenvectors corresponding to the eigenvalues are also arranged in the same order as the eigenvalues. These eigenvectors are determined as principal components, i.e., key features in the first image data. The contribution rate corresponding to each principal component can be determined by the ratio between the eigenvalue corresponding to the principal component and the sum of all eigenvalues.

[0031] In this embodiment, the principal component whose contribution rate is greater than a preset first threshold value can be determined as the internal feature, wherein the internal feature is used to characterize the internal components in the fruit to be detected.

[0032] The principal components finally obtained in this embodiment can retain most of the information in the original hyperspectral image, while achieving data dimensionality reduction, removing redundant information, and improving the efficiency of subsequent detection.

[0033] In one embodiment of the present application, obtaining the first characteristic skin feature according to the first data of the fruit to be detected may include: Divide the grayscale image into multiple sub-regions; For each sub-region, determine the grayscale variance of the pixels in the sub-region, and determine the target neighborhood radius corresponding to the sub-region based on the grayscale variance: Traverse the pixels from the starting position in the sub-area based on a preset order, and for each traversal, construct a neighborhood range of the current pixel based on the target neighborhood radius with the current pixel as the center; if the grayscale value of the first pixel in the neighborhood is greater than the grayscale value of the current pixel, mark the first pixel as a first mark value; if the grayscale value of the first pixel in the neighborhood is less than or equal to the grayscale value of the current pixel, mark the first pixel as a second mark value; determine a first code of the current pixel based on the mark values corresponding to all pixels in the neighborhood; The epidermal features are determined according to the first codes corresponding to all the sub-regions, and the epidermal features are used to characterize epidermal information of the fruit to be detected.

[0034] In this embodiment, the grayscale image can be divided into multiple sub-regions, and the grayscale variance of each sub-region is calculated. If the grayscale variance of a sub-region is greater than a preset variance threshold, it means that the grayscale change of the sub-region is more drastic, and the sub-region is determined to be a complex texture region; if the grayscale variance is smaller, it means that the texture of the sub-region is simpler, and the sub-region is determined to be a simple texture region.

[0035] In this embodiment, the target neighborhood radius may be determined based on the variance of the sub-regions, and may be determined based on formula (1).

[0036] R=max(M, ))(1) In formula (1), R represents the target neighborhood radius, M represents the minimum neighborhood radius, k represents the coefficient, and b represents the offset. Represents grayscale variance. M can be 1, M is the minimum neighborhood radius in the first texture area, such as M can be 1, k is a coefficient, b is an offset, and the values of k and b can be determined through a large number of experiments. For example, you can first collect appropriate neighborhood radius data corresponding to sub-areas of different texture complexities, that is, different grayscale variances, and then use a curve fitting method such as the least squares method to find the k and b values that minimize the error between the calculated radius and the actual appropriate radius. The value does not exceed the maximum neighborhood radius preset in the second texture area.

[0037] In this embodiment, for each sub-region, after determining the target neighborhood radius corresponding to the sub-region, local binary pattern encoding may be performed on each pixel point in the sub-region.

[0038] For each sub-region, the pixels can be traversed based on a preset order from the starting position in the sub-region. For each traversal, the neighborhood range of the current pixel is constructed based on the target neighborhood radius with the current pixel as the center; if the grayscale value of the first pixel in the neighborhood is greater than the grayscale value of the current pixel, the first pixel is marked as the first mark value; if the grayscale value of the first pixel in the neighborhood is less than or equal to the grayscale value of the current pixel, the first pixel is marked as the second mark value; the first code of the current pixel is determined based on the mark values corresponding to all pixels in the neighborhood. As an example, after determining the pixels in the neighborhood, the grayscale value g of the center pixel is set. c As the threshold, it is sequentially compared with the gray value g of each pixel in the neighborhood. i For example, within the neighborhood, there is a pixel with coordinates ((x+1,y) and grayscale value (g x+1,y ), if g x+1,y >g c , then mark the pixel as 1; if g x+1,y g c, then it is marked as 0. In this way, all pixels in the neighborhood are compared and marked one by one. After completing the marking of all pixels in the neighborhood, these marked values can be arranged in a pre-set order, such as in a clockwise direction. Assuming there are 8 pixels in the neighborhood, the sequence obtained after comparison and marking is 1, 0, 1, 1, 0, 0, 1, 1. This sequence is composed into an 8-bit binary number 10110011, which is the local binary pattern encoding of the current center pixel. Through this encoding, the relative relationship between the grayscale values of the center pixel and its neighboring pixels can be reflected, thereby describing the texture characteristics of the area.

[0039] In this embodiment, the grayscale co-occurrence matrix of the grayscale image can be calculated based on the first encoding of all sub-regions. The number of grayscale levels of the grayscale image can be determined first, such as the grayscale value of the image can be quantized to a certain number of levels, such as a 256-level grayscale image can be quantized to 16 levels), and then in the image, according to a specified direction, such as 0°, 45°, 90°, 135° and a distance d, the number of times a pixel with a grayscale value of i and a pixel with a grayscale value of j appear at the same time is counted, and it is used as the element of the i-th row and j-th column in the matrix, thereby constructing a grayscale co-occurrence matrix. For example, in the case of a 0° direction and a distance d=1, the number of times the grayscale value combination of horizontally adjacent pixels appears is counted; based on the statistical results, a grayscale co-occurrence matrix is constructed, and its texture feature parameters such as energy, entropy, and contrast are calculated, and the texture feature parameters are determined as the second texture feature.

[0040] In this embodiment, the first texture feature and the second texture feature may be combined to form a new feature vector, namely, a skin feature. The skin feature may be used to characterize the external texture feature of the fruit to be detected.

[0041] This embodiment can effectively extract local features around each pixel in each sub-region, and can integrate the local features of each sub-region to form a feature vector that can fully describe the fruit skin information.

[0042] In one embodiment of the present application, obtaining the structural feature in the first feature according to the first data of the fruit to be detected may include: Perform wavelet decomposition on the signal corresponding to the ultrasonic echo to obtain the first component feature; Performing empirical mode decomposition on the signal corresponding to the ultrasonic echo to obtain the second component feature; Determine a first information entropy corresponding to each frequency component in the first component feature, and determine a second information entropy corresponding to each frequency component in the second component feature; Determine a first weight of the first component feature based on the first information entropy and the second information entropy, and determine a second weight of the second component feature based on the first information entropy and the second information entropy; The first component feature and the second component feature are fused according to the first weight and the second weight to obtain a structural feature, which is used to characterize the structural information of the fruit to be detected.

[0043] In this embodiment, the wavelet transform can be used to analyze the echo signal at different scales, decompose the high-frequency components and low-frequency components in the signal, and extract the first component feature that reflects the changes in the internal structure of the fruit. The first feature includes the features of the high-frequency components and the different frequency components corresponding to the low-frequency components; the echo signal can be decomposed into multiple intrinsic mode functions by empirical mode decomposition, and then each intrinsic mode function is subjected to Hilbert transform to obtain the second component feature of the signal. The second component feature includes the features of different frequencies corresponding to different intrinsic mode functions. The first component features and the second component features extracted by the two transforms are combined, and the first component features and the second component features are fused to construct the characteristic vector of the ultrasonic echo and obtain the structural features, thereby realizing the effective extraction of the internal structure information of the fruit. Among them, the fusion processing method can be: the first information entropy corresponding to each frequency component in the first component feature can be determined, and the second information entropy corresponding to each frequency component in the second component feature can be determined. For example, the first information entropy corresponding to the i-th frequency component in the first component feature can be expressed as , the second information entropy corresponding to the jth frequency component in the second component feature can be expressed as The sum of the information entropies corresponding to all frequency components of the first component feature can be determined based on each first information entropy of the first component feature, which can be expressed as , n1 is the number of frequency components of the first feature, and the sum of the information entropy of all frequency components of the second component feature can be determined based on each second information entropy of the second component feature, which can be expressed as , n2 is the number of frequency components of the second feature. Then the first weight of the first feature , the second weight of the second feature This method assumes that the larger the total amount of information entropy, the richer the information contained in the feature, and the higher the weight should be in the fusion.

[0044] This embodiment can obtain richer signal detail information through multi-dimensional signal analysis; by quantizing the first component features and the second component features, and reasonably allocating their weights in the fusion process according to the amount of information protected by each component feature, it can obtain more accurate structural features.

[0045] In one embodiment of the present application, constructing a multimodal fusion feature vector based on the first feature may include: For each feature in the first feature of the fruit to be detected, mapping processing is performed on it to obtain a feature vector of the first dimension corresponding to each feature; Determine the weight matrix of each feature based on the feature vector; Each dimension of all features in the first feature is fused based on multiple weight matrices to obtain a multimodal fusion feature vector.

[0046] In this embodiment, each of the internal features, epidermal features, and structural features in the first feature can be mapped to a preset feature space of the first dimension to obtain a feature vector of the first dimension corresponding to each feature. Among them, each feature can be mapped based on a convolutional neural network, or mapped based on the encoding layer of the Transformer, which is not limited in this application. By mapping each feature to obtain a feature vector of the same first dimension, features of different types and dimensions are unified into the same vector space, which facilitates subsequent calculations and processing.

[0047] A weight matrix for the eigenvector corresponding to each feature can be determined. For each dimension of all eigenvectors in the first feature, the weight value corresponding to that dimension in the weight matrix can be obtained. Based on this weight value, the value corresponding to each eigenvector in that dimension is weighted to obtain a multimodal fusion feature vector. Determining the weight matrix for each feature based on the eigenvectors allows for adaptive weighting based on the importance of the feature. By fusing each dimension of all features in the first feature according to the weight, the advantages of each feature can be integrated, avoiding the dilution of key information that can result from simple average fusion.

[0048] In one embodiment of the present application, constructing a multimodal fusion feature vector based on the first feature may include: For each vector in the feature vector, take the vector being processed as the first vector and obtain the query vector and key vector of the first vector; determining a similarity matrix based on similarities between the query vector and each candidate vector, wherein the candidate vector represents a key vector of a feature vector other than the first vector in the feature vectors; The similarity matrix is normalized to obtain the weight matrix.

[0049] In this embodiment, for each vector in the feature vector, the vector being processed can be used as the first vector to obtain the query vector and key vector of the first vector. The query vector and key vector can be obtained through the encoding layer in the Transformer structure. The query vector can be used to represent the focus of the current first vector, and the key vector can be used to represent the feature representation of the current first vector for querying by other vectors.

[0050] For each first vector, the similarity between the query vector corresponding to the first vector and each candidate vector is determined to obtain a similarity matrix. Among them, the candidate vector can represent the key vector of the other feature vectors in the feature vector except the first vector. By calculating the similarity between the query vector of each first vector and other candidate vectors and constructing a similarity matrix based on this, the degree of correlation between the feature vectors can be accurately quantified. The higher the similarity, the more similar or correlated the two feature vectors are in characterizing the characteristics of the fruit to be detected, and vice versa, the less correlated they are, which can clearly reveal the intrinsic connection between the various features.

[0051] In this embodiment, the similarity matrix may be normalized to obtain a corresponding weight matrix. The normalization process can eliminate the influence of different dimensions or value ranges of different elements in the similarity matrix.

[0052] In this embodiment, the values of each dimension of all eigenvectors can be weighted based on the values corresponding to each dimension of the weight matrix to obtain a multimodal fusion eigenvector. By determining the multimodal feature fusion vector based on the weight matrix obtained by normalizing the similarity matrix, the information of different eigenvectors can be more accurately fused. The information of the eigenvectors with higher weights is more fully integrated into the multimodal fusion eigenvector, while the influence of the eigenvectors with lower weights is appropriately reduced, so that the multimodal fusion eigenvector can more accurately capture the characteristics of the fruit to be detected, thereby improving the accuracy and reliability of the fruit evaluation results.

[0053] In the present application, determining the evaluation result of the fruit to be detected based on the multimodal fusion feature vector may include: The trained first neural network is used to determine the evaluation result based on the multimodal fusion features.

[0054] In one embodiment of the present application, the fruit quality assessment results can also be visually output through a display screen, and the assessment results can be fed back to the fruit sorting system to achieve automatic grading and screening of fruits based on quality grades.

[0055] The present invention adopts multimodal data acquisition and comprehensively utilizes hyperspectral imaging, machine vision and ultrasonic detection technology to comprehensively obtain the internal quality and external defect information of the fruit, making up for the shortcomings of a single detection technology.

[0056] The fruit non-destructive testing system of the present invention realizes an automated process of fruit non-destructive testing and can effectively and accurately detect the quality of the fruit.

[0057] Corresponding to the fruit non-destructive testing method of the above embodiment, Figure 2 This is a schematic diagram of a fruit non-destructive testing system provided in one embodiment of the present application. For ease of illustration, only the parts related to the embodiment of the present application are shown. Figure 2 The fruit non-destructive testing system 20 includes: an acquisition module 21, a construction module 22 and an evaluation module 23. The acquisition module 21 is used to: acquire a first feature based on first data of the fruit to be detected, the first data including a hyperspectral image, a grayscale image and an ultrasonic echo, and the first feature including an internal feature, a skin feature and a structural feature; The construction module 22 is used to: construct a multimodal fusion feature vector according to the first feature; The evaluation module 23 is used to determine the evaluation result of the fruit to be detected based on the multimodal fusion feature vector, and the evaluation result includes the quality grade of the fruit to be detected.

[0058] In one embodiment of the present application, the acquisition module 21 is also used to: determine a minimum redundant information set based on the redundant information corresponding to data of different wavelengths in the hyperspectral image; obtain a wavelength combination in the minimum redundant information set, perform dimensionality reduction processing on the hyperspectral image based on the wavelength combination, and obtain the first image data after dimensionality reduction; determine the contribution rate of the principal component in the first image data; determine the principal component whose contribution rate is greater than a preset first threshold as an internal feature, and the internal feature is used to characterize the internal information of the fruit to be detected.

[0059] In one embodiment of the present application, the acquisition module 21 is also used to: divide the grayscale image into multiple sub-regions; for each sub-region, determine the grayscale variance of the pixels in the sub-region, and determine the target neighborhood radius corresponding to the sub-region based on the grayscale variance: traverse the pixels from the starting position in the sub-region based on a preset order, and for each traversal, construct the neighborhood range of the current pixel based on the target neighborhood radius with the current pixel as the center; if the grayscale value of the first pixel in the neighborhood is greater than the grayscale value of the current pixel, then mark the first pixel as a first label value; if the grayscale value of the first pixel in the neighborhood is less than or equal to the grayscale value of the current pixel, then mark the first pixel as a second label value; determine the first code of the current pixel based on the label values corresponding to all pixels in the neighborhood; determine the first epidermal feature of the grayscale image based on the first codes of all sub-regions; determine the second epidermal feature of the grayscale image based on the grayscale co-occurrence matrix corresponding to the grayscale image; splice the first epidermal feature and the second epidermal feature to obtain the epidermal feature of the grayscale image, and the epidermal feature is used to characterize the epidermal information of the fruit to be detected.

[0060] In one embodiment of the present application, the acquisition module 21 is also used to: perform wavelet decomposition processing on the signal corresponding to the ultrasonic echo to obtain a first component feature; perform empirical mode decomposition processing on the signal corresponding to the ultrasonic echo to obtain a second component feature; determine the first information entropy corresponding to each frequency component in the first component feature, and determine the second information entropy corresponding to each frequency component in the second component feature; determine the first weight of the first component feature based on the first information entropy and the second information entropy, and determine the second weight of the second component feature based on the first information entropy and the second information entropy; fuse the first component feature and the second component feature according to the first weight and the second weight to obtain a structural feature, which is used to characterize the structural information of the fruit to be detected.

[0061] In one embodiment of the present application, the construction module 22 is also used to: perform mapping processing on each feature of the first feature of the fruit to be detected to obtain a feature vector of the first dimension corresponding to the feature; determine the weight matrix of each feature based on the feature vector; and fuse all feature vectors based on the weight matrix to obtain a multimodal fusion feature vector.

[0062] In one embodiment of the present application, the construction module 22 is further used to: for each vector in the feature vector, take the vector being processed as the first vector, and obtain the query vector and key vector of the first vector; determine the similarity matrix based on the similarity between the query vector and each candidate vector, wherein the candidate vector represents the key vector of the feature vector other than the first vector in the feature vector; and normalize the similarity matrix to obtain a weight matrix.

[0063] In one embodiment of the present application, the evaluation module 23 is further configured to: determine an evaluation result based on the multimodal fusion feature vector using the trained first neural network.

[0064] See also Figure 3 , Figure 3 This is a schematic block diagram of an electronic device provided in one embodiment of the present application. Figure 3 The electronic device 300 in the embodiment shown may include: one or more processors 301, one or more input devices 302, one or more output devices 303, and one or more memories 304. The processors 301, input devices 302, output devices 303, and memories 304 communicate with each other via a communication bus 305. The memory 304 is used to store computer programs, which include program instructions. The processor 301 is used to execute the program instructions stored in the memory 304. The processor 301 is configured to call the program instructions to execute the functions of the modules / units in the above-mentioned system embodiments, such as Figure 2The functions of the acquisition module 21, the construction module 22 and the evaluation module 23 are shown.

[0065] It should be understood that in the embodiment of the present application, the processor 301 may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0066] The input device 302 may include a touchpad, a fingerprint collection sensor (for collecting user fingerprint information and fingerprint direction information), a microphone, etc. The output device 303 may include a display (LCD, etc.), a speaker, etc.

[0067] The memory 304 may include a read-only memory and a random access memory, and provides instructions and data to the processor 301. A portion of the memory 304 may also include a non-volatile random access memory. For example, the memory 304 may also store device type information.

[0068] In a specific implementation, the processor 301, input device 302, and output device 303 described in the embodiments of the present application can execute the implementation methods described in the first and second embodiments of the fruit non-destructive testing method provided in the embodiments of the present application, and can also execute the implementation methods of the electronic device described in the embodiments of the present application, which will not be repeated here.

[0069] In another embodiment of the present application, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program. The computer program includes program instructions. When the program instructions are executed by a processor, all or part of the process of the method in the above embodiment is implemented. The computer program can also be used to instruct related hardware to complete the process. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of each of the above method embodiments are implemented. The computer program includes computer program code, which can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can include: any entity or device capable of carrying computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium.

[0070] The computer-readable storage medium can be an internal storage unit of the electronic device in any of the aforementioned embodiments, such as a hard disk or memory of the electronic device. The computer-readable storage medium can also be an external storage device of the electronic device, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash memory card, etc. Furthermore, the computer-readable storage medium can include both an internal storage unit of the electronic device and an external storage device. The computer-readable storage medium is used to store computer programs and other programs and data required by the electronic device. The computer-readable storage medium can also be used to temporarily store data that has been output or is about to be output.

[0071] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0072] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the electronic devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0073] In the several embodiments provided in this application, it should be understood that the disclosed electronic devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces or units, or can be an electrical, mechanical or other form of connection.

[0074] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0075] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0076] The above are only specific embodiments of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and such modifications or substitutions should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A method for non-destructive testing of fruit, characterized in that: include: obtaining first features according to first data of the fruit to be inspected, wherein the first data includes a hyperspectral image, a grayscale image, and an ultrasonic echo, and the first features include internal features, epidermal features, and structural features; Constructing a multimodal fusion feature vector based on the first feature; An evaluation result of the fruit to be detected is determined according to the multimodal fusion feature vector, wherein the evaluation result includes a quality grade of the fruit to be detected.

2. The fruit non-destructive testing method according to claim 1, wherein: The obtaining of the first feature according to the first data of the fruit to be detected includes: Determining a minimum redundant information set based on redundant information corresponding to data of different wavelengths in the hyperspectral image; Acquire a wavelength combination in the minimum redundant information set, and perform dimensionality reduction processing on the hyperspectral image based on the wavelength combination to obtain first image data after dimensionality reduction; determining a contribution rate of a principal component in the first image data, where the principal component is used to represent a key feature in the first image data; The principal component whose contribution rate is greater than a preset first threshold is determined as an internal feature, and the internal feature is used to characterize the internal information of the fruit to be detected.

3. The fruit non-destructive testing method according to claim 1, wherein: The obtaining of the first feature according to the first data of the fruit to be detected includes: Dividing the grayscale image into a plurality of sub-regions; For each sub-region, determine the grayscale variance of the pixels in the sub-region, and determine the target neighborhood radius corresponding to the sub-region based on the grayscale variance: Traversing the pixels from the starting position in the sub-area based on a preset order, for each traversal, constructing a neighborhood range of the current pixel based on the target neighborhood radius with the current pixel as the center; if the grayscale value of the first pixel in the neighborhood is greater than the grayscale value of the current pixel, marking the first pixel as a first marking value; if the grayscale value of the first pixel in the neighborhood is less than or equal to the grayscale value of the current pixel, marking the first pixel as a second marking value; determining a first code of the current pixel based on the marking values corresponding to all pixels in the neighborhood; determining a first epidermal feature of the grayscale image according to the first codes of all sub-regions; determining a second epidermal feature of the grayscale image according to a gray-level co-occurrence matrix corresponding to the grayscale image; The first epidermal feature and the second epidermal feature are spliced together to obtain the epidermal feature of the grayscale image, and the epidermal feature is used to characterize the epidermal information of the fruit to be detected.

4. The fruit non-destructive testing method according to claim 1, wherein: The obtaining of the first feature according to the first data of the fruit to be detected includes: Performing wavelet decomposition processing on the signal corresponding to the ultrasonic echo to obtain a first component feature; Performing empirical mode decomposition on the signal corresponding to the ultrasonic echo to obtain a second component feature; Determine a first information entropy corresponding to each frequency component in the first component feature, and determine a second information entropy corresponding to each frequency component in the second component feature; Determine a first weight of the first component feature based on the first information entropy and the second information entropy, and determine a second weight of the second component feature based on the first information entropy and the second information entropy; The first component feature and the second component feature are fused according to the first weight and the second weight to obtain the structural feature, which is used to characterize the structural information of the fruit to be detected.

5. The fruit non-destructive testing method according to claim 1, wherein: The constructing a multimodal fusion feature vector according to the first feature includes: For each feature of the first feature of the fruit to be detected, mapping processing is performed to obtain a feature vector of the first dimension corresponding to each feature; Determine a weight matrix for each feature based on the feature vector; All feature vectors are fused based on multiple weight matrices to obtain the multimodal fused feature vector.

6. The fruit non-destructive testing method according to claim 5, wherein: Determining a weight matrix for each feature according to the feature vector includes: For each vector in the feature vector, take the vector being processed as the first vector and obtain the query vector and key vector of the first vector; determining a similarity matrix based on similarities between the query vector and each candidate vector, wherein the candidate vector represents a key vector of a feature vector other than the first vector in the feature vectors; The similarity matrix is normalized to obtain the weight matrix.

7. The fruit non-destructive testing method according to claim 1, wherein: Determining the evaluation result of the fruit to be detected according to the multimodal fusion feature vector includes: An evaluation result is determined based on the multimodal fusion feature vector using the trained first neural network.

8. A fruit non-destructive testing system, characterized in that: include: an acquisition module, configured to acquire a first feature based on first data of the fruit to be detected, wherein the first data includes a hyperspectral image, a grayscale image, and an ultrasonic echo, and the first feature includes an internal feature, a skin feature, and a structural feature; A construction module, configured to construct a multimodal fusion feature vector based on the first feature; An evaluation module is used to determine an evaluation result of the fruit to be detected based on the multimodal fusion feature vector, wherein the evaluation result includes a quality grade of the fruit to be detected.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images