Closed-shell oyster fatness degree nondestructive testing method and system based on multi-sensor heterogeneous data fusion

Abstract

The invention belongs to the field of closed-shell oyster fatness degree nondestructive testing, and provides a closed-shell oyster fatness degree nondestructive testing method and a closed-shell oyster fatness degree nondestructive testing system based on multi-sensor heterogeneous data fusion, and the method comprises the following steps: collecting heterogeneous data of a sensor to be detected and detecting a fatness degree classification model; wherein the training process of the fatness degree classification model comprises the steps of multi-type data acquisition, data preprocessing, data set division, data fusion, fatness degree classification model training and model verification. According to the method, more sufficient and complementary information is provided for detection of the fatness degree of the closed-shell oysters through fusion of multi-source heterogeneous data, the fatness degree of the closed-shell oysters can be accurately identified, the method has the advantages of real-time performance, non-invasiveness, high accuracy and the like, nondestructive detection of the freshness of the closed-shell oysters is achieved, and the detection accuracy of the freshness of the closed-shell oysters is improved. And a more powerful nondestructive testing technical support is provided for sustainable development of the oyster industry.

Description

Technical Field

[0001] The invention relates to the field of nondestructive detection of fat fullness of closed shell oysters, and in particular to a method and system for nondestructive detection of fat fullness of closed shell oysters based on multi-sensor heterogeneous data fusion. Background Art

[0002] Oyster farming is one of the oldest forms of aquaculture and is practiced extensively around the world. Similar to other aquaculture industries, the production of oyster farming has increased rapidly in recent decades. Oysters need to be sorted during the process from catching to selling. At present, oysters are mainly divided into multiple grades according to weight ranges on the market, and oysters of the same weight range are packed and sold. Generally, heavier oysters are sold at higher prices. Oyster shells not only protect their soft tissues from external environmental contamination, but also slow down water loss. In the actual supply chain process, oysters in shell can maintain their freshness better than oysters without shells. PVC heat shrink label film is usually used to fasten oysters in shell to reduce their mouth opening movement, thereby reducing energy consumption and extending the shelf life. Therefore, due to the shielding effect of the oyster shell, it is difficult to visually detect the fatness of oysters in actual operation.

[0003] However, because the degree of fullness of the soft tissue of oysters in the shell, i.e., the fatness, varies significantly depending on the degree of gonadal development, this weight-based sorting method often results in a large number of oysters with lower fatness being mixed in with oysters of the same weight range, which greatly reduces the sales quality of oysters. Acoustic and machine vision technology, due to its non-invasive, penetrating and non-radiative advantages, has become a potential means to solve this problem, but there is currently no technology for non-destructive detection of the fatness of closed-shell oysters using acoustic and machine vision technology. Summary of the invention

[0004] In order to overcome the shortcomings of the prior art, the purpose of the present invention is to provide a method and system for non-destructive detection of fatness and fullness of closed shell oysters based on multi-sensor heterogeneous data fusion, so as to realize non-destructive detection of freshness of closed shell oysters using acoustic and machine vision technology.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] A non-destructive detection method for fatness of closed-shell oysters based on multi-sensor heterogeneous data fusion, comprising:

[0007] Collect sensor data of the target oyster for detection, and obtain heterogeneous sensor data to be detected;

[0008] The heterogeneous data of the sensor to be detected is input into a pre-trained fatness classification model for detection to obtain a target fatness non-destructive detection result; the training process of the fatness classification model includes:

[0009] Collecting shell images of oysters ready for closing at multiple angles, vibration acoustic signals after knocking on the shells of oysters, and measured indicators of oyster fatness level; the shell images include: front view images and side view images;

[0010] Preprocessing the shell image and the vibroacoustic signal to obtain multi-sensor heterogeneous data;

[0011] Constructing a closed-shell oyster fat fullness data set according to the measured index of the oyster fat fullness level and the multi-sensor heterogeneous data, and dividing the closed-shell oyster fat fullness data set into a training set and a test set;

[0012] A hybrid data fusion model integrating multiple feature extraction networks is constructed, the hybrid data fusion model is trained using a distributed training strategy, and the multi-sensor heterogeneous data in the training set is input into the hybrid data fusion model to fuse the multi-source heterogeneous data to obtain a fused feature vector; the hybrid data fusion model integrates an OEMI-CNN feature extraction network, an MTFI-VGG feature extraction network, and a feature fusion network; the OEMI-CNN feature extraction network is used to extract deep features of a first type of visual sensor data; the MTFI-VGG feature extraction network is used to extract deep features of a second type of acoustic sensor data; the feature fusion network is used to fuse features output by the OEMI-CNN feature extraction network and the MTFI-VGG feature extraction network;

[0013] Constructing a fatness classification model based on a neural network, taking the fused feature vector corresponding to the training set as input and the measured index of the oyster fatness level in the training set as target output to train the fatness classification model;

[0014] The trained fatness classification model is verified using the test set to obtain a verification result. If the verification result meets the target requirement, the fatness classification model is saved.

[0015] Preferably, the shell images of the oysters prepared for closing the shell at multiple angles, the vibration acoustic signals after knocking the oyster shells, and the measured indicators of the oyster fatness level are collected, including:

[0016] Using two industrial cameras with mutually perpendicular lens angles to obtain the front view image and the side view image of the prepared closed shell oyster from two angles of top view and side view;

[0017] Using a steel rod released at a fixed height to strike the center of the shell of the prepared closed-shell oyster, and using a microphone sensor to collect the vibration acoustic signal;

[0018] The whole weight of each of the prepared closed shell oysters is weighed using a balance, and after the prepared closed shell oysters are shelled, the soft tissue of the prepared closed shell oysters is weighed to obtain the weight of the soft tissue, and the measured index of the oyster fat fullness level is obtained according to the whole weight and the weight of the soft tissue; the calculation formula of the measured index of the oyster fat fullness level is: Among them, OFI is the measured index of the oyster fatness level; R is the weight of the soft tissue; D is the overall weight;

[0019] The ready-to-close oysters corresponding to the oyster fatness level measured by the actual measurement index is less than or equal to the preset fatness threshold value are classified as low fatness, and the ready-to-close oysters corresponding to the oyster fatness level measured by the actual measurement index is greater than the preset fatness threshold value are classified as high fatness.

[0020] Preferably, the housing image and the vibroacoustic signal are preprocessed to obtain multi-sensor heterogeneous data, including:

[0021] Enhance the contrast and color saturation of the front view image and the side view image by histogram equalization processing;

[0022] Repairing the noise in the enhanced front view image and the side view image by using a wavelet threshold filtering algorithm;

[0023] Using a K-means clustering algorithm to extract the oyster shell portion in the noise-repaired front view image and the side view image;

[0024] The extracted images of the front view image and the side view image are horizontally spliced ​​and fused to obtain an external morphological image of the closed shell oyster, and the external morphological image of the closed shell oyster is converted into an RGB three-channel image of 256×256 pixels;

[0025] The vibroacoustic signal is converted into a two-dimensional Markov transfer field spectrum image with a size of 256×256 pixels.

[0026] Preferably, a hybrid data fusion model integrating multiple feature extraction networks is constructed, the hybrid data fusion model is trained using a distributed training strategy, and the multi-sensor heterogeneous data in the training set is input into the hybrid data fusion model to fuse the multi-source heterogeneous data to obtain a fused feature vector, including:

[0027] The first feature with a dimension of 1×1024 obtained from the last fully connected layer in the OEMI-CNN feature extraction network and the second feature with a dimension of 1×4096 obtained from the last fully connected layer in the MTFI-VGG feature extraction network are horizontally spliced ​​and fused using the Concat method to obtain the fused feature vector; the acquisition expression of the fused feature vector is: T=concat(m a ,m b ), wherein T is the fused feature vector of 1×5120 dimensions; m a is the first feature; m b is the second feature; concat() is the fusion equation.

[0028] Preferably, the OEMI-CNN feature extraction network comprises: a first input layer, a first output layer, first to fourth convolutional layers with a stride of 1, and first to third fully connected layers with a stride of 2; all convolutional layers and connection layers in the OEMI-CNN feature extraction network adopt batch normalization and rectified linear unit activation functions; the first fully connected layer, the second fully connected layer, and the third fully connected layer are all embedded with a discard algorithm; the first to fourth convolutional layers each contain 100 convolutional kernels of size 3×3; the first to fourth convolutional layers are each followed by a maximum pooling layer of size 2×2; the number of neuron units in the first to third fully connected layers is 2048, 1024, and 1024, respectively; the first output layer is a softmax layer containing 2 nodes;

[0029] The first input layer is used to receive the external morphological image of the closed-shell oyster; the first fully connected layer is used to receive the first flattened feature vector output by the maximum pooling layer from the first convolutional layer to the fourth convolutional layer; the second fully connected layer is used to receive the output of the first fully connected layer; the third fully connected layer is used to receive the output of the second fully connected layer; the first output layer is used to receive the output of the third fully connected layer and output the first feature.

[0030] Preferably, the MTFI-VGG feature extraction network includes: a second input layer, a second output layer, fifth to twenty-first convolutional layers with a stride of 1 and a ReLU activation function, and fourth to sixth fully connected layers with a stride of 2 and a ReLU activation function; the numbers of 3×3 convolutional kernels contained in the fifth to seventeenth convolutional layers are: 64, 64, 128, 128, 256, 256, 512, 512 respectively; the sixth convolutional layer, the eighth convolutional layer, the eleventh convolutional layer, the fourteenth convolutional layer, and the seventeenth convolutional layer are all followed by a maximum pooling layer with a size of 2×2 and a stride of 2; the fourth to sixth fully connected layers each include 4096 neurons;

[0031] The second input layer is used to receive the Markov transfer field spectrum image; the output layer is used to output the second feature.

[0032] Preferably, a hybrid data fusion model integrating multiple feature extraction networks is constructed, the hybrid data fusion model is trained using a distributed training strategy, and the multi-sensor heterogeneous data in the training set is input into the hybrid data fusion model to fuse the multi-source heterogeneous data to obtain a fused feature vector, further comprising:

[0033] Using the ILSVRC2012 dataset in the image database ImageNet to perform initial pre-training on the OEMI-CNN feature extraction network;

[0034] Taking the actual image data in the training set as input and the measured index of the oyster fat fullness level as output, the pre-trained OEMI-CNN feature extraction network is subjected to secondary training; the secondary training includes: fine-tuning and optimization;

[0035] Using the AudioSet dataset to initially pre-train the MTFI-VGG feature extraction network;

[0036] Taking the Markov transfer field spectrum image in the training set as input and the measured index of oyster fatness level as output, the pre-trained MTFI-VGG feature extraction network is subjected to secondary training; the secondary training includes: fine-tuning and correction.

[0037] Preferably, the fatness classification model includes: the seventh to ninth fully connected layers and the third output layer using batch normalization and ReLU activation function; the third output layer is a softmax layer containing 2 nodes; the seventh to ninth fully connected layers are all embedded with a dropout algorithm; the number of neurons in the seventh to ninth fully connected layers are 2048, 2048 and 1024 respectively;

[0038] The third output layer is used to output the target fatness nondestructive detection result.

[0039] Preferably, a closed-shell oyster fatness nondestructive detection system based on multi-sensor heterogeneous data fusion includes: a data acquisition module, a data preprocessing module, a data fusion model construction module and a fatness detection and grading module;

[0040] A data acquisition module, used to acquire multiple sets of multi-angle images of closed-shell oyster shells, and acoustic signals after knocking on the oyster shells;

[0041] A data preprocessing module, used for preprocessing the multiple groups of multi-angle closed-shell oyster shell images and the acoustic signal;

[0042] A data fusion model building module is used to establish and save a hybrid data fusion model, and use the hybrid data fusion model to extract and merge the processed multiple groups of multi-angle closed-shell oyster shell images and the acoustic signal to obtain a fusion feature vector;

[0043] The fatness detection and grading module is used to establish a fatness classification model based on a neural network according to the fused feature vector, and use the fatness classification model to classify the fatness of the closed-shell oysters to be detected, and output a fatness grade index.

[0044] Preferably, a readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the above-mentioned method for non-destructive detection of fat fullness of closed-shell oysters based on multi-sensor heterogeneous data fusion are implemented.

[0045] The present invention discloses the following technical effects:

[0046] The present invention provides a method and system for nondestructive detection of fatness and fullness of closed-shell oysters based on multi-sensor heterogeneous data fusion. Through the fusion of multi-source heterogeneous data, the defects of conventional detection methods such as invasiveness, inefficiency and poor accuracy are solved, and nondestructive detection of the freshness of closed-shell oysters using acoustic and machine vision technology is realized. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0048] Figure 1 A schematic diagram of a non-destructive detection process for fatness of closed-shell oysters provided in an embodiment of the present invention;

[0049] Figure 2 A 256×256 OEMI image generated after preprocessing provided by an embodiment of the present invention;

[0050] Figure 3 A 256×256 MTFI image generated after preprocessing provided by an embodiment of the present invention;

[0051] Figure 4 A schematic diagram of the structure of a hybrid data fusion model (FCM) provided in an embodiment of the present invention;

[0052] Figure 5 A schematic diagram of the structure of a fatness classification model provided in an embodiment of the present invention;

[0053] Figure 6 A schematic diagram of the structure of a nondestructive detection system for fat fullness of closed-shell oysters based on multi-sensor heterogeneous data fusion provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] The purpose of the present invention is to provide a method and system for nondestructive detection of fatness of closed-shell oysters based on multi-sensor heterogeneous data fusion, so as to realize nondestructive detection of freshness of closed-shell oysters by using acoustic and machine vision technology.

[0056] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0057] Figure 1 A schematic diagram of a nondestructive detection process for fatness of closed shell oysters provided in an embodiment of the present invention is shown in FIG. Figure 1 As shown, the present invention provides a non-destructive detection method for fatness of closed-shell oysters based on multi-sensor heterogeneous data fusion, comprising:

[0058] Step 100: collecting sensor data of the target oyster to be detected, and obtaining heterogeneous sensor data to be detected;

[0059] Step 200: Input the heterogeneous sensor data to be detected into a pre-trained fatness classification model for detection to obtain a target fatness non-destructive detection result; the training process of the fatness classification model includes:

[0060] Step 201: collecting images of the shells of oysters ready for closing their shells at multiple angles, vibration acoustic signals after knocking on the shells of oysters, and measured indicators of the fatness level of oysters; the shell images include: front view images and side view images;

[0061] Step 202: pre-processing the housing image and the vibroacoustic signal to obtain multi-sensor heterogeneous data;

[0062] Step 203: constructing a closed-shell oyster fat fullness data set according to the measured index of the oyster fat fullness level and the multi-sensor heterogeneous data, and dividing the closed-shell oyster fat fullness data set into a training set and a test set;

[0063] Step 204: construct a hybrid data fusion model integrating multiple feature extraction networks, train the hybrid data fusion model using a distributed training strategy, and input the multi-sensor heterogeneous data in the training set into the hybrid data fusion model to fuse the multi-source heterogeneous data to obtain a fused feature vector; the hybrid data fusion model integrates the OEMI-CNN feature extraction network, the MTFI-VGG feature extraction network and the feature fusion network; the OEMI-CNN feature extraction network is used to extract deep features of the first type of visual sensor data; the MTFI-VGG feature extraction network is used to extract deep features of the second type of acoustic sensor data; the feature fusion network is used to fuse the features output by the OEMI-CNN feature extraction network and the MTFI-VGG feature extraction network;

[0064] Step 205: constructing a fatness classification model based on a neural network, taking the fused feature vector corresponding to the training set as input and the measured index of the oyster fatness level in the training set as target output to train the fatness classification model;

[0065] Step 206: Use the test set to verify the trained fatness classification model to obtain a verification result. If the verification result meets the target requirement, save the fatness classification model.

[0066] Preferably, the shell images of the oysters prepared for closing the shell at multiple angles, the vibration acoustic signals after knocking the oyster shells, and the measured indicators of the oyster fatness level are collected, including:

[0067] Using two industrial cameras with mutually perpendicular lens angles to obtain the front view image and the side view image of the prepared closed shell oyster from two angles of top view and side view;

[0068] Using a steel rod released at a fixed height to strike the center of the shell of the prepared closed-shell oyster, and using a microphone sensor to collect the vibration acoustic signal;

[0069] The whole weight of each of the prepared closed shell oysters is weighed using a balance, and after the prepared closed shell oysters are shelled, the soft tissue of the prepared closed shell oysters is weighed to obtain the weight of the soft tissue, and the measured index of the oyster fat fullness level is obtained according to the whole weight and the weight of the soft tissue; the calculation formula of the measured index of the oyster fat fullness level is: Among them, OFI is the measured index of the oyster fatness level; R is the weight of the soft tissue; D is the overall weight;

[0070] The ready-to-close oysters corresponding to the oyster fatness level measured by the actual measurement index is less than or equal to the preset fatness threshold value are classified as low fatness, and the ready-to-close oysters corresponding to the oyster fatness level measured by the actual measurement index is greater than the preset fatness threshold value are classified as high fatness.

[0071] Preferably, the housing image and the vibroacoustic signal are preprocessed to obtain multi-sensor heterogeneous data, including:

[0072] Enhance the contrast and color saturation of the front view image and the side view image by histogram equalization processing;

[0073] Repairing the noise in the enhanced front view image and the side view image by using a wavelet threshold filtering algorithm;

[0074] Using a K-means clustering algorithm to extract the oyster shell portion in the noise-repaired front view image and the side view image;

[0075] The extracted images of the front view image and the side view image are horizontally spliced ​​and fused to obtain an external morphological image of the closed shell oyster, and the external morphological image of the closed shell oyster is converted into an RGB three-channel image of 256×256 pixels;

[0076] The vibroacoustic signal is converted into a two-dimensional Markov transfer field spectrum image with a size of 256×256 pixels.

[0077] Specifically, a hybrid data fusion model integrating multiple feature extraction networks is constructed, the hybrid data fusion model is trained using a distributed training strategy, and the multi-sensor heterogeneous data in the training set is input into the hybrid data fusion model to fuse the multi-source heterogeneous data to obtain a fused feature vector, including:

[0078] The first feature with a dimension of 1×1024 obtained from the last fully connected layer in the OEMI-CNN feature extraction network and the second feature with a dimension of 1×4096 obtained from the last fully connected layer in the MTFI-VGG feature extraction network are horizontally spliced ​​and fused using the Concat method to obtain the fused feature vector; the acquisition expression of the fused feature vector is: T=concat(m a ,m b ), wherein T is the fused feature vector of 1×5120 dimensions; m a is the first feature; m b is the second feature; concat() is the fusion equation.

[0079] Furthermore, the OEMI-CNN feature extraction network includes: a first input layer, a first output layer, first to fourth convolutional layers with a stride of 1, and first to third fully connected layers with a stride of 2; all convolutional layers and connection layers in the OEMI-CNN feature extraction network use batch normalization and rectified linear unit activation functions; the first fully connected layer, the second fully connected layer, and the third fully connected layer are all embedded with a discard algorithm; the first convolutional layer to the fourth convolutional layer each contain 100 convolutional kernels of size 3×3; the first convolutional layer to the fourth convolutional layer are each followed by a maximum pooling layer of size 2×2; the number of neuron units in the first fully connected layer to the third fully connected layer are 2048, 1024, and 1024, respectively; the first output layer is a softmax layer containing 2 nodes;

[0080] The first input layer is used to receive the external morphological image of the closed-shell oyster; the first fully connected layer is used to receive the first flattened feature vector output by the maximum pooling layer from the first convolutional layer to the fourth convolutional layer; the second fully connected layer is used to receive the output of the first fully connected layer; the third fully connected layer is used to receive the output of the second fully connected layer; the first output layer is used to receive the output of the third fully connected layer and output the first feature.

[0081] Specifically, the MTFI-VGG feature extraction network includes: a second input layer, a second output layer, the fifth to twenty-first convolutional layers with a stride of 1 and a ReLU activation function, and the fourth to sixth fully connected layers with a stride of 2 and a ReLU activation function; the fifth to seventeenth convolutional layers contain 3×3 convolution kernels of 64, 64, 128, 128, 256, 256, 512, 512, respectively; the sixth convolutional layer, the eighth convolutional layer, the eleventh convolutional layer, the fourteenth convolutional layer, and the seventeenth convolutional layer are all followed by a maximum pooling layer of 2×2 and a stride of 2; the fourth to sixth fully connected layers each include 4096 neurons;

[0082] The second input layer is used to receive the Markov transfer field spectrum image; the output layer is used to output the second feature.

[0083] Furthermore, a hybrid data fusion model integrating multiple feature extraction networks is constructed, the hybrid data fusion model is trained using a distributed training strategy, and the multi-sensor heterogeneous data in the training set is input into the hybrid data fusion model to fuse the multi-source heterogeneous data to obtain a fused feature vector, further comprising:

[0084] Using the ILSVRC2012 dataset in the image database ImageNet to perform initial pre-training on the OEMI-CNN feature extraction network;

[0085] Taking the actual image data in the training set as input and the measured index of the oyster fat fullness level as output, the pre-trained OEMI-CNN feature extraction network is subjected to secondary training; the secondary training includes: fine-tuning and optimization;

[0086] Using the AudioSet dataset to initially pre-train the MTFI-VGG feature extraction network;

[0087] Taking the Markov transfer field spectrum image in the training set as input and the measured index of oyster fatness level as output, the pre-trained MTFI-VGG feature extraction network is subjected to secondary training; the secondary training includes: fine-tuning and correction.

[0088] Specifically, the fatness classification model includes: the seventh to ninth fully connected layers and the third output layer using batch normalization and ReLU activation function; the third output layer is a softmax layer containing 2 nodes; the seventh to ninth fully connected layers are all embedded with a dropout algorithm; the number of neurons in the seventh to ninth fully connected layers are 2048, 2048 and 1024 respectively;

[0089] The third output layer is used to output the target fatness nondestructive detection result.

[0090] Furthermore, a closed-shell oyster fatness nondestructive detection system based on multi-sensor heterogeneous data fusion includes: a data acquisition module, a data preprocessing module, a data fusion model building module and a fatness detection and grading module;

[0091] A data acquisition module, used to acquire multiple sets of multi-angle images of closed-shell oyster shells, and acoustic signals after knocking on the oyster shells;

[0092] A data preprocessing module, used for preprocessing the multiple groups of multi-angle closed-shell oyster shell images and the acoustic signal;

[0093] A data fusion model building module is used to establish and save a hybrid data fusion model, and use the hybrid data fusion model to extract and merge the processed multiple groups of multi-angle closed-shell oyster shell images and the acoustic signal to obtain a fusion feature vector;

[0094] The fatness detection and grading module is used to establish a fatness classification model based on a neural network according to the fused feature vector, and use the fatness classification model to classify the fatness of the closed-shell oysters to be detected, and output a fatness grade index.

[0095] Optionally, a readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the above-mentioned method for non-destructive detection of fat fullness of closed-shell oysters based on multi-sensor heterogeneous data fusion are implemented.

[0096] Specifically, the specific steps of the non-destructive detection method of the fatness of closed-shell oysters based on multi-sensor heterogeneous data fusion are as follows:

[0097] Obtain multi-angle images of the closed shell of oysters, acoustic signals after knocking on the oyster shell, and oyster fatness level indicators, including:

[0098] S1: A front view image A (pixel value is A(x, y)) and a side view image B (pixel value is B(x, y)) of the shell of the closed shell oyster are obtained from the top view and side view through a group of industrial cameras with mutually perpendicular lens angles; in this embodiment, two Hikvision 6 million color industrial cameras are used, with 12mm lenses, ISO set to 160, shutter speed of 1 / 20, aperture of F5.6, and shooting resolution of 1024×1024, and two front view images A(1024,1024) and side view images B(1024,1024) of the shell of the closed shell oyster are obtained.

[0099] S2: The center of the oyster shell is struck by a steel rod released from a fixed height, and the resulting vibration acoustic signal is collected by a microphone sensor;

[0100] S3: Use a balance to weigh the entire body weight (D) of each oyster, record the data, shuck the oyster, weigh the soft tissue, and record the soft tissue weight (R). Calculate the fatness index (OFI) of each oyster according to the following formula:

[0101]

[0102] Taking OFI as the classification standard of fatness, the oyster fatness index is divided into low fatness (OFI≤Y) and high fatness (OFI>Y) according to the preset fatness threshold Y. In the specific implementation, the fatness threshold Y=0.15 is used for classification, as shown in Table 1:

[0103] Table 1

[0104] Occupational Fatness Index (OFI) Fatness level index OFI≤0.15 High fatness (level 1) OFI ≥ 0.15 Low fatness (level 2)

[0105] Preprocessing the multi-angle images of the shells of the closed-shell oysters and the knocking acoustic signals to form multi-sensor heterogeneous data specifically includes:

[0106] S1: The front view image A and the side view image B are subjected to nonlinear transformation by using histogram equalization to enhance the contrast and color saturation of the images, thereby obtaining enhanced front view image Ax and side view image Bx:

[0107] Convert the image from (R, G, B) space to (H, S, V), where the pixel value of the H channel represents the hue, ranging from 0 to 360, the pixel value of the S channel represents the saturation of the image, ranging from 0 to 1, and the pixel value of the V channel represents the brightness of the image, also ranging from 0 to 1. The conversion formula is as follows:

[0108] V=max(R,G,B)

[0109]

[0110]

[0111] Among them, RGB is an additive color model that generates various colors by superimposing red, green, and blue light with different intensities. R, G, and B are the three basic components that represent the color. They represent: R represents the intensity of red light or the value of the red component, G represents the intensity of green light or the value of the green component, and B represents the intensity of blue light or the value of the blue component. The value range is 0 to 255.

[0112] In order to ensure that the color of the image will not be distorted during the conversion process, the V component is selected for histogram equalization to enhance the contrast and color saturation of the image, while retaining the local detail information of the image, keeping the hue and saturation unchanged, and calculating the grayscale histogram of the brightness channel. The calculation formula is as follows:

[0113]

[0114] Among them, p V (v k ) is the brightness level v k The probability density of k is the brightness level v k The frequency of occurrence in the brightness channel, M is the number of rows in the image, and N is the number of columns in the image;

[0115] The calculation formula of the cumulative distribution function (CDF) of the brightness channel is as follows:

[0116]

[0117] Among them, s k is the brightness level v k The corresponding cumulative distribution function value;

[0118] The calculation formula of the brightness value after histogram equalization is as follows:

[0119]

[0120] Where V′ is the brightness value after histogram equalization, and L is the total number of brightness values;

[0121] The processed brightness channel and the original hue and saturation channels are recombined into a new HSV image; the new HSV image is converted back to an RGB image to obtain the final histogram-equalized color image. The conversion formula is as follows:

[0122]

[0123] p=V*(1-S)

[0124] q=V * (1-f * S)

[0125] t=V * (1-(1-f)*S)

[0126]

[0127] Where floor means rounding down;

[0128] S2: Applying an improved wavelet threshold filtering algorithm to the enhanced front view image Ax and the side view image Bx to repair the noise in the image, and obtaining the noise-repaired front view image A* and the side view image B*, including:

[0129] For a given image, wavelet transform is performed. Let the original signal be x(t) and the wavelet basis function be ψ j,k (t), the wavelet coefficient is w j,k , then the wavelet transform formula is as follows:

[0130]

[0131] Among them, ψ j,k (t) represents the wavelet basis function The scale transformation and translation at scale j and position k are shown in the following formula:

[0132]

[0133] For the wavelet coefficient w j,k Perform threshold processing and use the following threshold function:

[0134]

[0135] in represents the signal after wavelet threshold denoising, and λ represents the set threshold.

[0136] S3: The noise-repaired front view image A* and side view image B* are processed using the K-means clustering algorithm to segment the oyster shell area from the background to obtain the segmented front view image A 0 and side view image B 0 , including the following steps:

[0137] The closed-shell oyster shell image is converted from RGB color space to XYZ color space using the following matrix:

[0138]

[0139] Among them, the XYZ color space represents the human visual stimulation value of the RGB three primary colors, the X component represents the response value to red and green, the Y component is used to represent the brightness, and the Z component represents the response value to blue.

[0140] Use the following formula to convert the XYZ color space to Lab color space for further color balance adjustments:

[0141] First, select the reference point standard X n =95.047, Y n =100, Z n=108.883. Among them, X n , Y n and Z n It represents the tristimulus values ​​of a white object when the illuminant first shines on a completely reflecting diffuse reflector and then reflects through it to the human eye.

[0142] Secondly, normalize the XYZ values ​​by dividing the X, Y, and Z values ​​by the corresponding reference point standard X n , Y n and Z n :

[0143]

[0144] For the normalized f X 、f Y 、f Z Apply the following function f(t) for nonlinear transformation:

[0145]

[0146] Finally, calculate the L*, a*, and b* values ​​to achieve the conversion from XYZ color space to Lab color space:

[0147] L ＊ =116f(f X )-16

[0148] a ＊ =500[f(f X )-f(f Y )]

[0149] b ＊ =200[f(f Y )-f(f Z )]

[0150] The L* component reflects the brightness of the color, and its value range is [0, 100]. L* = 0: completely black. L* = 100: completely white. a* is the red-green color coordinate, a*>0: the color is biased towards red, a*<0: the color is biased towards green. b* is the yellow-blue color coordinate, b*>0: the color is biased towards yellow, b*<0: the color is biased towards blue. The range of a* and b* values ​​is [127, -128].

[0151] To ensure a more accurate segmentation process, the sensitivity of the b* channel in the Lab color space to blue-yellow tones was utilized, and the K-means clustering algorithm was used to segment the image of the b channel component into two clusters (k=2), and the cluster label to which each pixel belongs was obtained. One cluster corresponds to the oyster shell area, and the other cluster corresponds to the background area except the oyster shell; the cluster labels were converted into a binary mask image to determine which cluster corresponds to the oyster shell area, and the mask was used to retain the oyster shell area on the original image and cover the background;

[0152] S4: The segmented front view image A 0 and side view image B 0 The images are horizontally spliced ​​and fused into an external morphological image of a closed shell oyster (OEMI), and the OEMI is converted into a 256×256 pixel RGB three-channel image. In the specific implementation, the front view image A of the same oyster is 0 (1024,1024) and side view image B 0 (1024,1024) horizontal splicing, the fused pixel value is OEMI(x,y), and the following formula can be used for fusion:

[0153] OEMI(x,y)=α·A 0 (x,y)+(1-α)·B 0 (x,y)

[0154] Among them, α is a weight parameter, which controls the degree of fusion of the two images at the splicing point, so that the spliced ​​image looks more natural and continuous. The choice of the parameter depends on the width of the blank edges of the two images. The value range of α is between 0 and 1. In this embodiment, α is 0.75, and the output OEMI image is C(1536,1536);

[0155] S5: Scale the spliced ​​image C(1536,1536) and calculate the scaling ratio in the horizontal and vertical directions, i.e. and For each pixel (u, v) in the target image, calculate its corresponding position (x, y) in the original image, that is, x = u / scale x and y = u / scale y ;

[0156] S6: According to the values ​​of the four pixels around (x, y) in the original image, use interpolation to calculate the pixel value at (u, v) in the target image. The formula is as follows:

[0157] f(x,y)=(1-α)(1-β)f(x 1 ,y 1 )+α(1-β)f(x 2 ,y1 )+(1-α)βf(x 1 ,y 2 )+αβf(x 2 ,y 2 )

[0158] Where f(x,y) is the pixel value at (x,y) in the target image, and f(x 1 ,y 1 )、f(x 2 ,y 1 )、f(x 1 ,y 2 and f(x 2 ,y 2 ) is the pixel value of the four surrounding pixels in the original image, x 2 =x 1 +1,y 2 =y 1 +1, α is the sum of x and x 1 The weight between y and y 1 The weight between them, α = xx 1 , β=yy 1 ;

[0159] The scaled pixel values ​​are obtained by interpolation calculation to generate the final 256×256 OEMI image, such as Figure 2 shown.

[0160] S7: The acoustic signal after knocking the oyster shell is converted into a two-dimensional Markov transfer field spectrum image (MTFI), the size of which is 256×256 pixels, as shown in Figure 3 In this embodiment, the time length of the acoustic signal is 1 second; specifically, the acoustic signal after knocking the oyster shell is converted into a two-dimensional MTFI including:

[0161] First, calculate the acoustic signal spectrum Then use the min-max normalization method to obtain the normalized spectrum

[0162] Secondly, arrange the amplitudes of ψ from small to large along the amplitude axis to obtain an ascending set And constructed based on quartiles The quartile statistic Q γ :

[0163]

[0164] Where, l = 1, 2, ..., N represents the amplitude axis;

[0165] Distribute ψ(k) to Q along the amplitude axis γ, and calculate the first-order Markov transition field matrix T mn :

[0166]

[0167] Wherein, m and n in the range of 1,2,…,N represent the Markov transfer field coordinate matrix;

[0168] Finally, construct the spectral Markov transition field M ij :

[0169] M ij =T mn |{ψ(i)∈T mn ,ψ(j)∈T mn}

[0170] Where i, j = 1, 2, ..., N represents the element coordinates of the spectral Markov transformation field, and MTFI is obtained.

[0171] Furthermore, based on the preprocessed multi-sensor heterogeneous data and oyster fatness index, a closed-shell oyster fatness dataset was constructed and divided into a training set and a test set; the characteristics of each tested closed-shell oyster can be expressed as X i = {P OEMI ,P MTFI}, where P OEMI Indicates the OMRI obtained, P MTFI Represents the obtained MTFI, each tested oyster corresponds to a fatness level y j , where y j There are two fatness levels divided by the actual measured oyster fatness index. Then the oyster closed shell oyster data set D can be expressed as a matrix (X, Y), where X = (X 1 ,X 2 ,…,X C ),Y=(y 1 ,y 2 ,…y C ), 70% of the sample data is used as the training set, and the remaining 30% is used as the test set;

[0172] Furthermore, a hybrid data fusion model (FEM) integrating two unique neural networks is constructed, and a distributed training strategy is used for model training. The trained model takes the preprocessed multi-sensor heterogeneous data as input, outputs the fused feature vector, and saves the FEM, such as Figure 4 As shown;

[0173] Specifically, the hybrid data fusion model integrating two unique neural networks consists of three parts, including: an OEMI-CNN feature extraction network, an MTFI-VGG feature extraction network and a feature fusion network;

[0174] Preferably, the OEMI-CNN feature extraction network consists of four convolutional layers and three fully connected layers. All convolutional layers and fully connected layers use batch normalization (BatchNorm) and rectified linear unit activation function (ReLU), and use dropout after each fully connected layer with a dropout probability of P to reduce overfitting. The detailed network structure is described as follows:

[0175] Input layer: accepts an OEMI image of size 256×256 pixels, with three RGB channels;

[0176] Convolutional layers: The network consists of four convolutional layers, each of which contains 100 convolution kernels of size 3×3 and stride 1. Each convolutional layer is followed by a 2×2 max pooling layer with a pooling stride of 2 to reduce the size of the feature map and extract the main features;

[0177] The first convolutional layer: contains 100 3×3 convolution kernels, followed by a 2×2 maximum pooling layer;

[0178] The second convolutional layer: contains 100 3×3 convolution kernels, followed by a 2×2 maximum pooling layer;

[0179] The third convolutional layer: contains 100 3×3 convolution kernels, followed by a 2×2 maximum pooling layer;

[0180] The fourth convolutional layer: contains 100 3×3 convolution kernels, followed by a 2×2 maximum pooling layer;

[0181] Fully connected layer: The input data is flattened into a one-dimensional feature vector after being processed by the fourth pooling layer; these extracted feature vectors are passed to three fully connected layers in turn, each of which contains 2048, 1024, and 1024 neuron units respectively;

[0182] The first fully connected layer: contains 2048 neuron units and receives the flattened feature vector;

[0183] The second fully connected layer: contains 1024 neuron units and receives the output of the first fully connected layer;

[0184] The third fully connected layer: contains 1024 neuron units and receives the output of the second fully connected layer;

[0185] Output layer: After the fully connected layer, a final output layer is connected. The final output layer is a softmax layer containing 2 nodes. The softmax layer is used to output the probability distribution of the fatness level of the closed-shell oyster.

[0186] Preferably, the MTFI-VGG feature extraction network consists of 16 convolutional layers and 3 fully connected layers. Each convolutional layer and fully connected layer uses BatchNorm and ReLU activation functions. Dropout is used after each fully connected layer, and the probability of dropout is P. The detailed network structure is described as follows:

[0187] Input layer: accepts MTFI images of size 256×256 pixels, and the image has three RGB channels;

[0188] Convolutional layer: The first to second convolutional layers: Each convolutional layer contains 64 3×3 convolution kernels, with a stride of 1 and a ReLU activation function;

[0189] The 3rd to 4th convolutional layers: Each convolutional layer contains 128 3×3 convolution kernels, with a stride of 1 and a ReLU activation function;

[0190] The 5th to 7th convolutional layers: Each convolutional layer contains 256 3×3 convolution kernels, with a stride of 1 and a ReLU activation function;

[0191] The 8th to 10th convolutional layers: Each convolutional layer contains 512 3×3 convolution kernels, with a stride of 1 and a ReLU activation function;

[0192] 11th to 13th convolutional layers: Each convolutional layer contains 512 3×3 convolution kernels, with a stride of 1 and a ReLU activation function;

[0193] Pooling layer: A 2×2 maximum pooling layer is connected after the 2nd, 4th, 7th, 10th, and 13th convolutional layers, respectively, with a pooling stride of 2 to reduce the size of the feature map and improve the effectiveness of feature extraction. ;

[0194] Fully connected layer: After the convolutional layer, three fully connected layers are connected. Each fully connected layer contains 4096 neurons and uses the ReLU activation function.

[0195] The first fully connected layer: contains 4096 neurons and receives the flattened feature vector;

[0196] The second fully connected layer: contains 4096 neurons and receives the output of the first fully connected layer;

[0197] The third fully connected layer: contains 4096 neurons and receives the output of the second fully connected layer;

[0198] Output layer: A final output layer is connected after the fully connected layer. The final output layer is a softmax layer containing 2 nodes, which is used to output the probability distribution of the fatness level of closed-shell oysters.

[0199] Preferably, feature fusion network: This network is used to effectively fuse features from two different feature extraction networks to provide more comprehensive information for subsequent classification models. Specifically, the feature fusion network concatenates and fuses features from the OEMI-CNN network and the MTFI-VGG network to form a high-dimensional feature vector.

[0200] OEMI-CNN feature m a : Features obtained from the last fully connected layer of the OEMI-CNN feature extraction network, with a dimension of 1×1024.

[0201] MTFI-VGG feature m b : Features obtained from the last fully connected layer of the MTFI-VGG feature extraction network, with a dimension of 1×4096.

[0202] The feature fusion network uses the Concat method to concatenate and fuse the above two features horizontally. Through the Concat operation, the two feature vectors are merged in the feature dimension to generate a new higher-dimensional feature representation. This fused feature representation can better integrate the deep features extracted from visual data and acoustic data, thereby improving the classification performance and generalization ability of subsequent classification models. The fused feature representation obtained by this method is as follows:

[0203] Fusion feature T = concat(m a ,m b )

[0204] Finally, a fused feature vector with a dimension of 1×(1024+4096)=1×5120 is obtained, which contains rich information from two different feature extraction networks and realizes multi-sensor heterogeneous data fusion.

[0205] Preferably, the model training process adopts a distributed training strategy, first pre-training through a large-scale public data set to obtain preliminary network weights, and then fine-tuning through a data set of a specific task, making full use of the existing rich data resources to improve the accuracy and generalization ability of model feature extraction. The specific process includes:

[0206] The OEMI-CNN feature extraction network was pre-trained using the ILSVRC2012 dataset in ImageNet, the world's largest public image recognition database. Then, the OEMI images in the training set were used as input, and the oyster fatness level index was used as output to train the OEMI-CNN feature extraction network that had been trained with the ILSVRC2012 dataset, thereby fine-tuning and optimizing the network.

[0207] At the same time, the MTFI-VGG feature extraction network was pre-trained using the AudioSet dataset publicly released by Google. Then, the MTFI data in the training set was used as input and the oyster fat fullness grade index was used as output to train the MTFI-VGG feature extraction network that had been pre-trained with the AudioSet dataset to achieve fine-tuning and correction of the network.

[0208] refer to Figure 5 , a fatness classification model (FCM) based on a neural network is constructed, which takes the fused feature vector T as input and outputs the oyster fatness grade index. The model is saved after the FCM performance is verified by the test set. The fatness classification model based on a neural network consists of three fully connected layers and a softmax layer containing 2 nodes, and the input features of the model come from the fused features T obtained by the previous feature extraction network. The specific architecture is as follows:

[0209] Fully connected layers: The model contains three fully connected layers with 2048, 2048, and 1024 nodes respectively. Each fully connected layer uses batch normalization (BatchNorm) and ReLU activation function to ensure the stability of the network and accelerate convergence during training. Dropout is applied after each fully connected layer with a dropout probability set to P to prevent overfitting of the model and improve its generalization ability.

[0210] The first fully connected layer: has 2048 nodes, and its input is the high-dimensional fused feature data T received by the model.

[0211] The second fully connected layer: also has 2048 nodes and receives the output of the first fully connected layer.

[0212] The third fully connected layer: contains 1024 nodes and receives the output of the second fully connected layer.

[0213] Final output layer: A final output layer is connected after the fully connected layer. The final output layer is a softmax layer containing 2 nodes, which is used to output 2 levels of closed-shell oyster fat fullness indicators to complete the non-destructive detection and identification of closed-shell oyster fat fullness;

[0214] Optionally, the accuracy, precision, and recall are used to evaluate the effect of the fatness classification model, and the calculation formula is:

[0215]

[0216] Among them, TP represents the number of samples predicted to be positive among the samples that are actually positive, FP represents the number of samples predicted to be positive among the samples that are actually negative, FN represents the number of samples predicted to be negative among the samples that are actually positive, and TN represents the number of samples that are actually negative.

[0217] Furthermore, the fusion feature T obtained by FEM on the training set was used to train the closed-shell oyster fatness grading model constructed by the existing machine learning algorithms Lasso, SVM, Decision Tree, XGBoost, Catboost and FCM, and the classification performance was tested by the fusion feature T obtained by FEM on the test set, and the comparison results are shown in Table 2. The accuracy, precision, and recall of the FCM model were 95%, 94%, and 95%, respectively, which were higher than the closed-shell oyster fatness grading model constructed by asso, SVM, Decision Tree, XGBoost, and Catboost, indicating that the established FCM is suitable for classifying the fatness of closed-shell oysters, and shows obvious advantages, and the fatness classification model is preserved.

[0218] Table 2

[0219] Model Accuracy Precision Recall Lasso 0.82 0.84 0.83 SVM 0.86 0.88 0.86 RandomForest 0.89 0.90 0.89 XGBoost 0.93 0.91 0.93 Catboost 0.92 0.93 0.91 FCM 0.95 0.94 0.95

[0220] Preferably, this embodiment provides a non-destructive detection system for fatness of closed shell oysters based on multi-sensor heterogeneous data fusion, such as Figure 6 As shown, the closed-shell oyster fatness nondestructive detection system based on multi-sensor heterogeneous data fusion includes: a data acquisition module, a data preprocessing module, a data fusion model construction module and a fatness detection and grading module;

[0221] Data acquisition module: used to obtain multiple sets of images of closed oyster shells from multiple angles, as well as acoustic signals after knocking on the oyster shells;

[0222] Data preprocessing module: used to preprocess the multi-angle closed oyster shell images and the acoustic signals after knocking the oyster shell to improve data quality and consistency;

[0223] Data fusion model building module: used to build and save a hybrid data fusion model integrating two unique neural networks, which realizes the data fusion of the shell image of the closed-shell oyster and the acoustic signal, and generates a fusion feature vector through feature extraction and merging;

[0224] Fatness detection and grading module: Based on the fused feature vector, a neural network-based fatness classification model is established. The model outputs the fatness grade index of the closed-shell oyster to achieve accurate grading of the fatness of the oyster.

[0225] The beneficial effects of the present invention are as follows:

[0226] The present invention provides more sufficient and complementary information for the fatness fullness detection of closed-shell oysters through the fusion of multi-source heterogeneous data, can accurately identify the fatness fullness of closed-shell oysters, has the advantages of real-time, non-invasive, high accuracy, etc., realizes non-destructive detection of the freshness of closed-shell oysters, and provides more powerful non-destructive detection technical support for the sustainable development of the oyster industry.

[0227] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0228] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A nondestructive detection method for the fatness of closed-shell oysters based on multi-sensor heterogeneous data fusion, characterized in that: include: Collect sensor data of the target oyster for detection, and obtain heterogeneous sensor data to be detected; Inputting the heterogeneous data of the sensor to be detected into a pre-trained fatness classification model for detection to obtain a non-destructive detection result of the target fatness; The training process of the fatness classification model includes: Collecting shell images of oysters ready for closing at multiple angles, vibration acoustic signals after knocking on the shells of oysters, and measured indicators of oyster fatness level; the shell images include: front view images and side view images; Preprocessing the shell image and the vibroacoustic signal to obtain multi-sensor heterogeneous data; Constructing a closed-shell oyster fat fullness data set according to the measured index of the oyster fat fullness level and the multi-sensor heterogeneous data, and dividing the closed-shell oyster fat fullness data set into a training set and a test set; A hybrid data fusion model integrating multiple feature extraction networks is constructed, the hybrid data fusion model is trained using a distributed training strategy, and the multi-sensor heterogeneous data in the training set is input into the hybrid data fusion model to fuse the multi-source heterogeneous data to obtain a fused feature vector; the hybrid data fusion model integrates an OEMI-CNN feature extraction network, an MTFI-VGG feature extraction network, and a feature fusion network; the OEMI-CNN feature extraction network is used to extract deep features of a first type of visual sensor data; the MTFI-VGG feature extraction network is used to extract deep features of a second type of acoustic sensor data; the feature fusion network is used to fuse features output by the OEMI-CNN feature extraction network and the MTFI-VGG feature extraction network; Constructing a fatness classification model based on a neural network, taking the fused feature vector corresponding to the training set as input and the measured index of the oyster fatness level in the training set as target output to train the fatness classification model; The trained fatness classification model is verified using the test set to obtain a verification result. If the verification result meets the target requirement, the fatness classification model is saved.

2. The nondestructive detection method of the fatness of closed-shell oysters based on multi-sensor heterogeneous data fusion according to claim 1 is characterized in that: Collect shell images of oysters ready for closing at multiple angles, vibration acoustic signals after knocking on the shells of oysters, and actual measurement indicators of oyster fatness, including: Using two industrial cameras with mutually perpendicular lens angles to obtain the front view image and the side view image of the prepared closed shell oyster from two angles of top view and side view; Using a steel rod released at a fixed height to strike the center of the shell of the prepared closed-shell oyster, and using a microphone sensor to collect the vibration acoustic signal; The whole weight of each of the prepared closed shell oysters is weighed using a balance, and after the prepared closed shell oysters are shelled, the soft tissue of the prepared closed shell oysters is weighed to obtain the weight of the soft tissue, and the measured index of the oyster fat fullness level is obtained according to the whole weight and the weight of the soft tissue; the calculation formula of the measured index of the oyster fat fullness level is: Among them, OFI is the measured index of the oyster fatness level; R is the weight of the soft tissue; D is the overall weight; The ready-to-close oysters corresponding to the oyster fatness level measured by the actual measurement index is less than or equal to the preset fatness threshold value are classified as low fatness, and the ready-to-close oysters corresponding to the oyster fatness level measured by the actual measurement index is greater than the preset fatness threshold value are classified as high fatness.

3. The nondestructive detection method of oyster fatness based on multi-sensor heterogeneous data fusion according to claim 1 is characterized in that: Preprocessing the shell image and the vibroacoustic signal to obtain multi-sensor heterogeneous data includes: Enhance the contrast and color saturation of the front view image and the side view image by histogram equalization processing; Repairing the noise in the enhanced front view image and the side view image by using a wavelet threshold filtering algorithm; Using a K-means clustering algorithm to extract the oyster shell portion in the noise-repaired front view image and the side view image; The extracted images of the front view image and the side view image are horizontally spliced ​​and fused to obtain an external morphological image of the closed shell oyster, and the external morphological image of the closed shell oyster is converted into an RGB three-channel image of 256×256 pixels; The vibroacoustic signal is converted into a two-dimensional Markov transfer field spectrum image with a size of 256×256 pixels.

4. The nondestructive detection method of oyster fatness based on multi-sensor heterogeneous data fusion according to claim 3 is characterized in that: Constructing a hybrid data fusion model integrating multiple feature extraction networks, training the hybrid data fusion model using a distributed training strategy, and inputting the multi-sensor heterogeneous data in the training set into the hybrid data fusion model to fuse the multi-source heterogeneous data to obtain a fused feature vector, including: The first feature with a dimension of 1×1024 obtained from the last fully connected layer in the OEMI-CNN feature extraction network and the second feature with a dimension of 1×4096 obtained from the last fully connected layer in the MTFI-VGG feature extraction network are horizontally spliced ​​and fused using the Concat method to obtain the fused feature vector; the acquisition expression of the fused feature vector is: T=concat ( m a ,m b) ; Wherein, T is the fused feature vector of 1×5120 dimensions; m a is the first feature; m b is the second feature; concat() is the fusion equation.

5. The nondestructive detection method of the fatness of closed-shell oysters based on multi-sensor heterogeneous data fusion according to claim 4 is characterized in that: The OEMI-CNN feature extraction network includes: a first input layer, a first output layer, first to fourth convolutional layers with a stride of 1, and first to third fully connected layers with a stride of 2; all convolutional layers and connection layers in the OEMI-CNN feature extraction network adopt batch normalization and rectified linear unit activation functions; the first fully connected layer, the second fully connected layer, and the third fully connected layer are all embedded with a dropout algorithm; the first to fourth convolutional layers each contain 100 convolution kernels of a size of 3×3; the first to fourth convolutional layers are each followed by a maximum pooling layer of a size of 2×2; the number of neuron units in the first to third fully connected layers is 2048, 1024, and 1024, respectively; the first output layer is a softmax layer containing 2 nodes; The first input layer is used to receive the external morphological image of the closed-shell oyster; the first fully connected layer is used to receive the first flattened feature vector output by the maximum pooling layer from the first convolutional layer to the fourth convolutional layer; the second fully connected layer is used to receive the output of the first fully connected layer; the third fully connected layer is used to receive the output of the second fully connected layer; the first output layer is used to receive the output of the third fully connected layer and output the first feature.

6. The nondestructive detection method of the fatness of closed-shell oysters based on multi-sensor heterogeneous data fusion according to claim 4 is characterized in that: The MTFI-VGG feature extraction network includes: a second input layer, a second output layer, the fifth to twenty-first convolutional layers with a stride of 1 and a ReLU activation function, and the fourth to sixth fully connected layers with a stride of 2 and a ReLU activation function; the fifth to seventeenth convolutional layers contain 3×3 convolution kernels of 64, 64, 128, 128, 256, 256, 512, 512, respectively; the sixth convolutional layer, the eighth convolutional layer, the eleventh convolutional layer, the fourteenth convolutional layer, and the seventeenth convolutional layer are all followed by a maximum pooling layer of 2×2 and a stride of 2; the fourth to sixth fully connected layers each include 4096 neurons; The second input layer is used to receive the Markov transfer field spectrum image; the output layer is used to output the second feature.

7. The nondestructive detection method of the fatness of closed-shell oysters based on multi-sensor heterogeneous data fusion according to claim 4 is characterized in that: Constructing a hybrid data fusion model integrating multiple feature extraction networks, training the hybrid data fusion model using a distributed training strategy, and inputting the multi-sensor heterogeneous data in the training set into the hybrid data fusion model to fuse the multi-source heterogeneous data to obtain a fused feature vector, further comprising: Using the ILSVRC2012 dataset in the image database ImageNet to perform initial pre-training on the OEMI-CNN feature extraction network; Taking the actual image data in the training set as input and the measured index of the oyster fat fullness level as output, the pre-trained OEMI-CNN feature extraction network is subjected to secondary training; the secondary training includes: fine-tuning and optimization; Using the AudioSet dataset to initially pre-train the MTFI-VGG feature extraction network; Taking the Markov transfer field spectrum image in the training set as input and the measured index of oyster fatness level as output, the pre-trained MTFI-VGG feature extraction network is subjected to secondary training; the secondary training includes: fine-tuning and correction.

8. The method for nondestructive detection of fatness of closed-shell oysters based on multi-sensor heterogeneous data fusion according to claim 4 is characterized in that: The fatness classification model includes: the seventh to ninth fully connected layers and the third output layer using batch normalization and ReLU activation function; the third output layer is a softmax layer containing 2 nodes; the seventh to ninth fully connected layers are all embedded with a dropout algorithm; the number of neurons in the seventh to ninth fully connected layers are 2048, 2048 and 1024 respectively; The third output layer is used to output the target fatness nondestructive detection result.

9. A non-destructive detection system for fatness of closed-shell oysters based on multi-sensor heterogeneous data fusion, characterized in that: include: Data acquisition module, data preprocessing module, data fusion model building module and fatness detection and grading module; A data acquisition module, used to acquire multiple sets of multi-angle images of closed-shell oyster shells, and acoustic signals after knocking on the oyster shells; A data preprocessing module, used for preprocessing the multiple groups of multi-angle closed-shell oyster shell images and the acoustic signal; A data fusion model building module is used to establish and save a hybrid data fusion model, and use the hybrid data fusion model to extract and merge the processed multiple groups of multi-angle closed-shell oyster shell images and the acoustic signal to obtain a fusion feature vector; The fatness detection and grading module is used to establish a fatness classification model based on a neural network according to the fused feature vector, and use the fatness classification model to classify the fatness of the closed-shell oysters to be detected, and output a fatness grade index.

10. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the steps of a non-destructive detection method for fat fullness of closed-shell oysters based on multi-sensor heterogeneous data fusion as described in any one of claims 1 to 8 are implemented.

Images