A method and system for nondestructive detection of the plumpness of closed shell oysters based on multi-sensor heterogeneous data fusion

Abstract

The present application belongs to the field of nondestructive detection of the fullness of closed shell oysters, and provides a nondestructive detection method and system for the fullness of closed shell oysters based on multi-sensor heterogeneous data fusion, which comprises: collection of sensor heterogeneous data to be detected and fullness classification model detection; wherein the training process of the fullness classification model comprises: multi-type data acquisition, data preprocessing, data set division, data fusion, fullness classification model training and model verification. The present application provides more sufficient and complementary information for the fullness detection of closed shell oysters through the fusion of multi-source heterogeneous data, can accurately identify the fullness of closed shell oysters, has the advantages of real-time, non-invasiveness, high accuracy, etc., realizes the nondestructive detection of the freshness of closed shell oysters, and provides stronger nondestructive detection technical support for the sustainable development of the oyster industry.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of closed shell oyster fullness nondestructive testing, and particularly relates to a closed shell oyster fullness nondestructive testing method and system based on multi-sensor heterogeneous data fusion. BACKGROUND

[0002] Oyster farming is one of the oldest forms of aquaculture and is widely carried out worldwide. Similar to other forms of aquaculture, the production of oysters has rapidly increased in recent decades. During the process of oysters from being fished to being sold, oysters need to be sorted. At present, oysters on the market are mainly divided into multiple grades according to weight ranges, and oysters with the same weight range are packed and sold. Generally, the heavier the oyster, the higher the price. The oyster shell not only protects the soft tissue from external environmental pollution, but also slows down water loss. In the actual supply chain process, shelled oysters are more likely to maintain their freshness than unshelled oysters. PVC heat shrink label film is often used to tighten the shelled oysters to reduce their mouth opening movement, thereby reducing energy consumption and prolonging the preservation time. Therefore, due to the shielding effect of the oyster shell, it is difficult to directly detect the fullness of the oyster in actual operation.

[0003] However, due to the degree of filling of the soft tissue of the oyster in the shell, i.e. the fullness, it will vary significantly due to the different degrees of gonadal development. This weight-based sorting method often leads to a large number of oysters with low fullness being mixed in oysters with the same weight range, thereby greatly reducing the sales quality of oysters. Acoustic and machine vision technologies are potential means to solve this problem due to their non-invasive, penetrating and non-radiative advantages, but there is currently no technology for nondestructive testing of the fullness of closed shell oysters using acoustic and machine vision technologies. SUMMARY

[0004] In order to overcome the shortcomings of the prior art, the purpose of the present application is to provide a closed shell oyster fullness nondestructive testing method and system based on multi-sensor heterogeneous data fusion, which realizes the nondestructive testing of the freshness of closed shell oysters using acoustic and machine vision technologies.

[0005] To achieve the above purpose, the present application provides the following solutions:

[0006] A closed shell oyster fullness nondestructive testing method based on multi-sensor heterogeneous data fusion, comprising:

[0007] Collecting sensor data of a target detection closed shell oyster to obtain to-be-detected sensor heterogeneous data;

[0008] Inputting the to-be-detected sensor heterogeneous data into a pre-trained fullness classification model for detection to obtain a target fullness nondestructive testing result; the training process of the fullness classification model comprises:

[0009] collecting shell images of the pre-prepared closed-shell oysters from multiple angles, vibration acoustic signals after knocking the shell of the oysters, and measured indicators of the oyster fullness grades;

[0010] preprocessing the shell images and the vibration acoustic signals to obtain multi-sensor heterogeneous data;

[0011] constructing a closed-shell oyster fullness dataset according to the measured indicators of the oyster fullness grades and the multi-sensor heterogeneous data, and dividing the closed-shell oyster fullness dataset into a training set and a test set;

[0012] constructing a hybrid data fusion model integrating multiple feature extraction networks, training the hybrid data fusion model using a distributed training strategy, inputting the multi-sensor heterogeneous data in the training set into the hybrid data fusion model for fusion of multi-source heterogeneous data to obtain a fusion feature vector; the hybrid data fusion model integrates an OEMI-CNN feature extraction network, an MTFI-VGG feature extraction network, and a feature fusion network; the OEMI-CNN feature extraction network is used to extract deep features of first-type visual sensor data; the MTFI-VGG feature extraction network is used to extract deep features of second-type acoustic sensor data; the feature fusion network is used to fuse features output by the OEMI-CNN feature extraction network and the MTFI-VGG feature extraction network;

[0013] constructing a fullness classification model based on a neural network, training the fullness classification model using the fusion feature vector corresponding to the training set as input and the measured indicators of the oyster fullness grades in the training set as target output;

[0014] verifying the trained fullness classification model using the test set to obtain a verification result, and saving the fullness classification model if the verification result meets target requirements.

[0015] Preferably, the collection of shell images of the pre-prepared closed-shell oysters from multiple angles, vibration acoustic signals after knocking the shell of the oysters, and measured indicators of the oyster fullness grades comprises:

[0016] using two industrial cameras with mutually perpendicular lens angles to obtain the front view image and the side view image of the pre-prepared closed-shell oysters from top view and side view angles;

[0017] using a steel rod released at a fixed height to knock the center of the shell of the pre-prepared closed-shell oysters, and using a microphone sensor to collect the vibration acoustic signals;

[0018] The whole weight of each of the prepared closed-shell oysters is weighed by using a balance, the soft tissue of the prepared closed-shell oysters is weighed after the prepared closed-shell oysters are shelled, the soft tissue weight is obtained, and the measured index of the oyster fullness degree level is obtained according to the whole weight and the soft tissue weight; the calculation formula of the measured index of the oyster fullness degree level is: ; wherein, is the measured index of the oyster fullness degree level; is the soft tissue weight; is the whole weight;

[0019] The prepared closed-shell oysters with the measured index of the oyster fullness degree level less than or equal to the preset fullness degree threshold are divided into low fullness, and the prepared closed-shell oysters with the measured index of the oyster fullness degree level greater than the preset fullness degree threshold are divided into high fullness.

[0020] Preferably, the shell image and the vibration acoustic signal are preprocessed to obtain multi-sensor heterogeneous data, including:

[0021] The contrast and color saturation of the front view image and the side view image are enhanced by histogram equalization processing;

[0022] The noise in the enhanced front view image and side view image is repaired by using a wavelet threshold filtering algorithm;

[0023] The front view image and the side view image after noise repair are subjected to K-means clustering processing and oyster shell region segmentation to obtain the segmented front view image and side view image;

[0024] The segmented front view image and side view image are transversely spliced and fused to obtain a closed-shell oyster external morphology image, and the closed-shell oyster external morphology image is converted into a 256x256 pixel RGB three-channel image;

[0025] The vibration acoustic signal is converted into a two-dimensional Markov transition field spectrum image with a size of 256x256 pixels.

[0026] Preferably, a hybrid data fusion model integrating multiple feature extraction networks is constructed, the hybrid data fusion model is trained by using a distributed training strategy, and the multi-sensor heterogeneous data in the training set is input into the hybrid data fusion model for fusion of multi-source heterogeneous data to obtain a fusion feature vector, including:

[0027] The first feature of 1x1024 dimension obtained by the last fully connected layer in the OEMI-CNN feature extraction network and the second feature of 1x4096 dimension obtained by the last fully connected layer in the MTFI-VGG feature extraction network are horizontally spliced and fused by using the Concat method to obtain the fusion feature vector; the expression for obtaining the fusion feature vector is: ; wherein, is the fusion feature vector of 1x5120 dimension; is the first feature; is the second feature; is a fusion equation.

[0028] Preferably, the OEMI-CNN feature extraction network comprises a first input layer, a first output layer, first to fourth convolutional layers with a stride of 1, and first to third fully connected layers with a stride of 2; all the convolutional layers and connected layers in the OEMI-CNN feature extraction network use batch normalization and rectified linear unit activation function; the first fully connected layer, the second fully connected layer, and the third fully connected layer all have dropout algorithm embedded; the first convolutional layer to the fourth convolutional layer all contain 100 convolutional kernels with a size of 3x3; the first convolutional layer to the fourth convolutional layer are all followed by a maximum pooling layer with a size of 2x2; the number of neuron units of the first fully connected layer to the third fully connected layer is 2048, 1024, and 1024, respectively; the first output layer is a softmax layer containing 2 nodes;

[0029] The first input layer is used to receive the closed oyster external morphology image; the first fully connected layer is used to receive the first flattened feature vector output by the maximum pooling layer in the first convolutional layer to the fourth convolutional layer; the second fully connected layer is used to receive the output of the first fully connected layer; the third fully connected layer is used to receive the output of the second fully connected layer; and the first output layer is used to receive the output of the third fully connected layer and output the first feature.

[0030] Preferably, the MTFI-VGG feature extraction network comprises: a second input layer, a second output layer, a fifth to twenty-first convolutional layer with a stride of 1 and using a ReLU activation function, a fourth to sixth fully connected layer with a stride of 2 and using a ReLU activation function; the fifth convolutional layer to the seventeenth convolutional layer contain a number of 3*3 convolution kernels with a size of 64, 64, 128, 128, 256, 256, 256, 512, 512, 512, 512, 512, 512 respectively; the sixth convolutional layer, the eighth convolutional layer, the eleventh convolutional layer, the fourteenth convolutional layer and the seventeenth convolutional layer are all followed by a maximum pooling layer with a size of 2*2 and a stride of 2; the fourth fully connected layer to the sixth fully connected layer all include 4096 neurons.

[0031] The second input layer is used to receive the Markov transition field spectrum image; and the second output layer is used to output the second feature.

[0032] Preferably, a hybrid data fusion model integrating multiple feature extraction networks is constructed, the hybrid data fusion model is trained by using a distributed training strategy, the multi-sensor heterogeneous data in the training set is input into the hybrid data fusion model for fusion of multi-source heterogeneous data, and a fusion feature vector is obtained, and the method further comprises:

[0033] The OEMI-CNN feature extraction network is initially pre-trained by using an ILSVRC2012 data set in an image database ImageNet.

[0034] The OEMI-CNN feature extraction network is pre-trained by using an ILSVRC2012 data set in an image database ImageNet.

[0035] The MTFI-VGG feature extraction network is initially pre-trained by using an AudioSet data set.

[0036] The MTFI-VGG feature extraction network is pre-trained by using an AudioSet data set.

[0037] Preferably, the fatness classification model comprises: the seventh to ninth fully connected layers and a third output layer adopting batch normalization and ReLU activation function; the third output layer is a softmax layer containing 2 nodes; the seventh fully connected layer to the ninth fully connected layer are embedded with dropout algorithm; the number of neurons of the seventh fully connected layer to the ninth fully connected layer is: 2048, 2048 and 1024 respectively.

[0038] The third output layer is used for outputting the target fatness nondestructive detection result.

[0039] Preferably, a closed oyster fatness nondestructive detection system based on multi-sensor heterogeneous data fusion comprises: a data acquisition module, a data preprocessing module, a data fusion model construction module and a fatness detection grading module.

[0040] The data acquisition module is used for acquiring multiple groups of multi-angle closed oyster shell images and acoustic signals after knocking the oyster shell.

[0041] The data preprocessing module is used for preprocessing the multiple groups of multi-angle closed oyster shell images and the acoustic signals.

[0042] The data fusion model construction module is used for establishing and saving a hybrid data fusion model, and performing feature extraction and merging on the preprocessed multiple groups of multi-angle closed oyster shell images and the acoustic signals by using the hybrid data fusion model to obtain a fusion feature vector.

[0043] The fatness detection grading module is used for establishing a fatness classification model based on a neural network according to the fusion feature vector, and classifying the fatness of the closed oyster to be detected by using the fatness classification model to output a fatness grade index.

[0044] Preferably, a readable storage medium is provided, and the readable storage medium stores programs or instructions, and the programs or instructions are executed by a processor to realize the steps of the above-mentioned closed oyster fatness nondestructive detection method based on multi-sensor heterogeneous data fusion.

[0045] The present application discloses the following technical effects:

[0046] The present application provides a closed oyster fatness nondestructive detection method and system based on multi-sensor heterogeneous data fusion, which solves the defects of invasiveness, inefficiency and poor accuracy of conventional detection methods through the fusion of multi-source heterogeneous data, and realizes the nondestructive detection of the freshness of closed oysters by using acoustic and machine vision technology. BRIEF DESCRIPTION OF DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0048] Figure 1 The closed shell oyster fullness nondestructive detection process schematic diagram provided for the embodiment of the present application is as follows:

[0049] Figure 2 The 256x256 OEMI image generated after pretreatment provided for the embodiment of the present application is as follows:

[0050] Figure 3 The 256x256 MTFI image generated after pretreatment provided for the embodiment of the present application is as follows:

[0051] Figure 4 The hybrid data fusion model (FCM) structure schematic diagram provided for the embodiment of the present application is as follows:

[0052] Figure 5 The fullness classification model structure schematic diagram provided for the embodiment of the present application is as follows:

[0053] Figure 6 The closed shell oyster fullness nondestructive detection system structure schematic diagram provided for the embodiment of the present application based on multi-sensor heterogeneous data fusion is as follows. DETAILED DESCRIPTION

[0054] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments only constitute some embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0055] The purpose of the present application is to provide a closed shell oyster fullness nondestructive detection method and system based on multi-sensor heterogeneous data fusion, to realize nondestructive detection of the freshness of closed shell oysters by using acoustic and machine vision technologies.

[0056] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0057] Figure 1 The closed shell oyster fullness nondestructive detection process schematic diagram provided for the embodiment of the present application is as follows: Figure 1As shown, the present application provides a multi-sensor heterogeneous data fusion-based closed shell oyster plumpness nondestructive detection method, comprising:

[0058] Step 100: Collect sensor data of target detection closed shell oysters to obtain sensor heterogeneous data to be detected;

[0059] Step 200: input the sensor heterogeneous data to be detected into a pre-trained plumpness classification model for detection to obtain a target plumpness nondestructive detection result; the training process of the plumpness classification model comprises:

[0060] Step 201: Collect the shell images of the prepared closed shell oysters under multiple angles, the vibration acoustic signals after knocking the oyster shell, and the measured indicators of oyster plumpness grade;

[0061] Step 202: Preprocess the shell images and the vibration acoustic signals to obtain multi-sensor heterogeneous data;

[0062] Step 203: Construct a closed shell oyster plumpness data set according to the measured indicators of oyster plumpness grade and the multi-sensor heterogeneous data, and divide the closed shell oyster plumpness data set into a training set and a test set;

[0063] Step 204: Construct a hybrid data fusion model integrating multiple feature extraction networks, train the hybrid data fusion model using a distributed training strategy, input the multi-sensor heterogeneous data in the training set into the hybrid data fusion model for multi-source heterogeneous data fusion, and obtain a fusion feature vector; the hybrid data fusion model integrates an OEMI-CNN feature extraction network, an MTFI-VGG feature extraction network, and a feature fusion network; the OEMI-CNN feature extraction network is used to extract deep features of the first type of visual sensor data; the MTFI-VGG feature extraction network is used to extract deep features of the second type of acoustic sensor data; the feature fusion network is used to fuse the features output by the OEMI-CNN feature extraction network and the MTFI-VGG feature extraction network;

[0064] Step 205: Construct a plumpness classification model based on a neural network, and train the plumpness classification model with the fusion feature vector corresponding to the training set as input and the measured indicators of oyster plumpness grade in the training set as target output;

[0065] Step 206: Verify the trained plumpness classification model using the test set to obtain a verification result, and save the plumpness classification model if the verification result meets the target requirements.

[0066] Preferably, the shell images of the pre-closed oysters under multiple angles, the vibration acoustic signals after knocking the oyster shells, and the measured indicators of the oyster plumpness grades are collected, including:

[0067] The front view images and the side view images of the pre-closed oysters are obtained from the top view and the side view by using two industrial cameras with the lens angles perpendicular to each other;

[0068] The center of the shell of the pre-closed oyster is knocked by using a steel rod released at a fixed height, and the vibration acoustic signals are collected by using a microphone sensor;

[0069] The overall weight of each pre-closed oyster is weighed by using a balance, after the pre-closed oyster is shelled, the soft tissue of the pre-closed oyster is weighed to obtain the soft tissue weight, and the measured indicator of the oyster plumpness grade is obtained according to the overall weight and the soft tissue weight; the calculation formula of the measured indicator of the oyster plumpness grade is: ; wherein, is the measured indicator of the oyster plumpness grade; is the soft tissue weight; is the overall weight;

[0070] The pre-closed oysters with the measured indicator of the oyster plumpness grade less than or equal to the preset plumpness threshold are divided into low plumpness, and the pre-closed oysters with the measured indicator of the oyster plumpness grade greater than the preset plumpness threshold are divided into high plumpness.

[0071] Preferably, the shell images and the vibration acoustic signals are preprocessed to obtain multi-sensor heterogeneous data, including:

[0072] The contrast and color saturation of the front view images and the side view images are enhanced by histogram equalization processing;

[0073] The noise in the enhanced front view images and side view images is repaired by using a wavelet threshold filtering algorithm;

[0074] The front view images and the side view images after noise repair are subjected to K-means clustering processing and oyster shell region segmentation to obtain segmented front view images and side view images;

[0075] The segmented front view images and side view images are horizontally spliced and fused to obtain the closed oyster external morphology image, and the closed oyster external morphology image is converted into a 256x256 pixel RGB three-channel image;

[0076] The vibration acoustic signals are converted into a two-dimensional Markov transition field spectrum image with a size of 256x256 pixels.

[0077] Specifically, a hybrid data fusion model integrating multiple feature extraction networks is constructed, a distributed training strategy is used to train the hybrid data fusion model, and the multi-sensor heterogeneous data in the training set is input into the hybrid data fusion model for fusion of multi-source heterogeneous data to obtain a fusion feature vector, including:

[0078] The first feature of 1x1024 dimension obtained by the last fully connected layer of the OEMI-CNN feature extraction network and the second feature of 1x4096 dimension obtained by the last fully connected layer of the MTFI-VGG feature extraction network are horizontally spliced and fused by using a Concat method to obtain the fusion feature vector; the expression for obtaining the fusion feature vector is: ; wherein, is the fusion feature vector of 1x5120 dimension; is the first feature; is the second feature; is a fusion equation.

[0079] Further, the OEMI-CNN feature extraction network includes a first input layer, a first output layer, first to fourth convolutional layers with a stride of 1, and first to third fully connected layers with a stride of 2; all convolutional layers and connected layers in the OEMI-CNN feature extraction network use batch normalization and a rectified linear unit activation function; the first fully connected layer, the second fully connected layer, and the third fully connected layer all have a dropout algorithm embedded therein; the first convolutional layer to the fourth convolutional layer all contain 100 convolutional kernels with a size of 3x3; the first convolutional layer to the fourth convolutional layer are all followed by a maximum pooling layer with a size of 2x2; the number of neuron units of the first fully connected layer to the third fully connected layer is 2048, 1024, and 1024, respectively; the first output layer is a softmax layer containing 2 nodes;

[0080] The first input layer is used to receive the closed shell oyster external morphology image; the first fully connected layer is used to receive a first flattened feature vector output by the maximum pooling layer in the first convolutional layer to the fourth convolutional layer; the second fully connected layer is used to receive the output of the first fully connected layer; the third fully connected layer is used to receive the output of the second fully connected layer; and the first output layer is used to receive the output of the third fully connected layer and output the first feature.

[0081] Specifically, the MTFI-VGG feature extraction network comprises: a second input layer, a second output layer, a fifth to twenty-first convolutional layer with a stride of 1 and using a ReLU activation function, and a fourth to sixth fully connected layer with a stride of 2 and using a ReLU activation function; the fifth convolutional layer to the seventeenth convolutional layer contain a number of 3*3 convolution kernels with a size of 64, 64, 128, 128, 256, 256, 256, 512, 512, 512, 512, 512, 512 respectively; the sixth convolutional layer, the eighth convolutional layer, the eleventh convolutional layer, the fourteenth convolutional layer and the seventeenth convolutional layer are all followed by a maximum pooling layer with a size of 2*2 and a stride of 2; the fourth fully connected layer to the sixth fully connected layer all include 4096 neurons.

[0082] The second input layer is used for receiving the Markov transition field spectrum image; and the second output layer is used for outputting the second feature.

[0083] Further, a hybrid data fusion model integrating multiple feature extraction networks is constructed, the hybrid data fusion model is trained by using a distributed training strategy, the multi-sensor heterogeneous data in the training set is input into the hybrid data fusion model for fusion of multi-source heterogeneous data, and a fusion feature vector is obtained, and the method further comprises:

[0084] The OEMI-CNN feature extraction network is initially pre-trained by using an ILSVRC2012 data set in an image database ImageNet;

[0085] The OEMI-CNN feature extraction network is pre-trained by using an ILSVRC2012 data set in an image database ImageNet;

[0086] The MTFI-VGG feature extraction network is initially pre-trained by using an AudioSet data set;

[0087] The MTFI-VGG feature extraction network is pre-trained by using an ILSVRC2012 data set in an image database ImageNet;

[0088] Specifically, the fatness classification model comprises: a seventh to ninth full connection layer adopting batch normalization and a ReLU activation function, and a third output layer; the third output layer is a softmax layer comprising 2 nodes; the seventh full connection layer to the ninth full connection layer are all embedded with a dropout algorithm; the number of neurons of the seventh full connection layer to the ninth full connection layer is respectively: 2048, 2048 and 1024.

[0089] The third output layer is used for outputting the target fatness nondestructive detection result.

[0090] Further, a closed oyster fatness nondestructive detection system based on multi-sensor heterogeneous data fusion comprises: a data acquisition module, a data preprocessing module, a data fusion model construction module and a fatness detection grading module.

[0091] The data acquisition module is used for acquiring multiple sets of multi-angle closed oyster shell images and acoustic signals after knocking the oyster shell.

[0092] The data preprocessing module is used for preprocessing the multiple sets of multi-angle closed oyster shell images and the acoustic signals.

[0093] The data fusion model construction module is used for establishing and saving a hybrid data fusion model, and performing feature extraction and merging on the preprocessed multiple sets of multi-angle closed oyster shell images and the acoustic signals by using the hybrid data fusion model to obtain a fusion feature vector.

[0094] The fatness detection grading module is used for establishing a fatness classification model based on a neural network according to the fusion feature vector, and classifying the fatness of a closed oyster to be detected by using the fatness classification model to output a fatness grade index.

[0095] Optionally, a readable storage medium stores programs or instructions, and the programs or instructions are executed by a processor to implement the steps of the above-mentioned closed oyster fatness nondestructive detection method based on multi-sensor heterogeneous data fusion.

[0096] Specifically, the closed oyster fatness nondestructive detection method based on multi-sensor heterogeneous data fusion comprises the following specific steps:

[0097] The multiple-angle closed oyster shell images, the acoustic signals after knocking the oyster shell and the oyster fatness grade index are acquired, and specifically include:

[0098] S1: Obtain the front view image A (pixel value is A(x, y)) and the side view image B (pixel value is B(x, y)) of the closed shell oyster shell from the top and side view angles through a set of industrial cameras whose lens angles are perpendicular to each other; in this embodiment, two Hikvision 6 million color industrial cameras are used, 12 mm lens, ISO is set to 160, shutter speed is 1 / 20, aperture is F5.6, the shooting resolution is set to 1024*1024, and two front view images A(1024, 1024) and side view images B(1024, 1024) of the closed shell oyster shell are obtained.

[0099] S2: Strike the center of the oyster shell with a steel rod released at a fixed height, and collect the vibration acoustic signal generated by the microphone sensor;

[0100] S3: Weigh the overall weight (D) of each oyster using a balance, record the data, then shell the oyster, weigh the soft tissue, and record the soft tissue weight (R), and calculate the fullness index (OFI) of each oyster according to the following formula:

[0101]

[0102] Take OFI as the division standard of the fullness grade, and divide the oyster fullness grade index into low fullness (OFI≤Y) and high fullness (OFI>Y) according to the pre-set fullness threshold Y. In specific implementation, the fullness threshold Y=0.15 is used for division, as shown in Table 1:

[0103] Table 1

[0104] Overfill index (OFI) Overfill grade indicator OFI < 0.15 High overfill (grade 1) OFI > 0.15 Low overfill (grade 2)

[0105] The multi-angle closed shell oyster shell image and the knocking acoustic signal are preprocessed to form multi-sensor heterogeneous data, specifically including:

[0106] S1: The front view image A and the side view image B are subjected to nonlinear conversion by histogram equalization to enhance the contrast and color saturation of the image, and the enhanced front view image Ax and side view image Bx are obtained:

[0107] The image is converted from the ( ) space to the ( , , ) space, where The pixel value of the S channel represents the saturation of the image, and the value range is 0 to 1, and the pixel value of the V channel represents the brightness of the image, and the value range is also 0 to 1, and the conversion formula is as follows:

[0108]

[0109]

[0110]

[0111] RGB is an additive color model, which generates various colors by superimposing red, green and blue light with different intensities. R, G and B are three basic components representing the color, which respectively represent the intensity of red light or the value of the red component, the intensity of green light or the value of the green component, and the intensity of blue light or the value of the blue component, with a value range of 0 to 255.

[0112] In order to ensure that the color of the image does not distort during the conversion process, the V component is selected for histogram equalization processing to enhance the contrast and color saturation of the image while preserving the local detail information of the image, keeping the hue and saturation unchanged. The gray level histogram of the luminance channel is calculated, and the calculation formula is as follows:

[0113]

[0114] wherein, is the probability density of the luminance level , is the frequency of the luminance level appearing in the luminance channel, is the number of rows of the image, is the number of columns of the image; The calculation formula of the cumulative distribution function (CDF) of the luminance channel is as follows:

[0115]

[0116] wherein,

[0117] is the cumulative distribution function value corresponding to the luminance level

[0118] The calculation formula of the luminance value after histogram equalization is as follows:

[0119]

[0120] wherein, is the luminance value after histogram equalization, is the total number of luminance values;

[0121] The processed luminance channel and the original hue and saturation channels are recombined into a new HSV image. The new HSV image is converted back to an RGB image to obtain the final histogram equalized color image, and the conversion formula is as follows:

[0122] ​​

[0123]

[0124]

[0125]

[0126]

[0127]

[0128] wherein denotes a down-rounding operation;

[0129] S2: applying the improved wavelet threshold filtering algorithm to the enhanced front view image Ax and side view image Bx, repairing the noise in the image to obtain the noise-repaired front view image A* and side view image B*, comprising:

[0130] For a given image, let the original signal be , the wavelet base function be , and the wavelet coefficient be , then the formula of wavelet transform is as follows:

[0131]

[0132] wherein, denotes the wavelet base function The scale transformation and translation at scale j and position k are shown in the following formula:

[0133]

[0134] The threshold value is processed on the wavelet coefficient , and the following threshold function is adopted:

[0135]

[0136] wherein denotes the signal after wavelet threshold denoising, is a threshold value set.

[0137] S3: performing K-means clustering algorithm processing on the noise-repaired front view image A* and side view image B*, segmenting the oyster shell area from the background to obtain the segmented front view image A 0 and side view image B 0 , comprising the following steps:

[0138] The closed oyster shell image is converted from the RGB color space to the XYZ color space by the following matrix:

[0139]

[0140] Where XYZ color space represents the RGB tristimulus value of human visual response, X component represents the response value of red and green, Y component represents the brightness, and Z component represents the response value of blue.

[0141] The XYZ color space is converted to Lab color space using the following formula for further color balance adjustment:

[0142] First, select the reference point standard , , . Where , represents the tristimulus value of the white color object illuminated by the illuminant first on the perfectly reflecting diffuse reflector and then reflected to the human eye.

[0143] Second, normalize the XYZ value, divide the X, Y and Z values by the corresponding reference reference point standard , :

[0144]

[0145] Apply the following function , , to the normalized for nonlinear conversion:

[0146]

[0147] Finally, calculate the L*, a*, b* values to realize the conversion from XYZ color space to Lab color space:

[0148]

[0149] The L* component reflects the light and dark degree of the color, and its value range is [0, 100], L*=0: indicates complete black. L*=100: indicates complete white. a* is the red-green chroma coordinate, a*>0: color is biased to red, a*<0: color is biased to green. b* is the yellow-blue chroma coordinate, b*>0: color is biased to yellow, b*<0: color is biased to blue. The value range of a*, b* is [127, -128].

[0150] To ensure the segmentation process more accurate, the sensitivity of b* channel to blue-yellow hue in Lab color space is used, and the K-means clustering algorithm is used to divide the image of b channel component into two clusters (k=2), to obtain the cluster label to which each pixel belongs, one cluster corresponds to the oyster shell area, and the other cluster corresponds to the background area except the oyster shell; the clustering label is converted into a binary mask image to determine which cluster corresponds to the oyster shell area, and the mask is used to retain the oyster shell area on the original image and mask the background;

[0151] S4: The segmented front view image A 0 and the side view image B 0 are horizontally spliced and fused into one closed shell oyster external morphology image (OEMI), and the OEMI is converted into a 256x256 pixel RGB three-channel image; in specific implementation, the front view image A 0 (1024, 1024) and the side view image B 0 (1024, 1024) of the same oyster are horizontally spliced, and the pixel value of the fused image is OEMI(x, y), which can be fused using the following formula:

[0152]

[0153] wherein, is a weight parameter, which controls the fusion degree of the two images at the splicing place, so that the spliced image looks more natural and continuous, and the selection of the parameter depends on the width of the blank edge of the two images, and the value of alpha is between 0 and 1; in this embodiment, 0.75 is taken, and the output OEMI image is ;

[0154] S5: The spliced image is scaled, and the scaling ratios in horizontal and vertical directions are calculated, i.e. and ; for each pixel in the target image, the corresponding position in the original image is calculated, i.e. and ;

[0155] S6: According to the values of the four pixels around in the original image, the pixel value at in the target image is calculated by interpolation, and the formula is as follows:

[0156]

[0157] wherein is the pixel value at in the target image, , , and is the pixel value of the four surrounding pixels in the original image, = + 1, = + 1, is the weight between and , is the weight between and , = - , = - ;

[0158] The scaled pixel values are calculated by interpolation, and the final 256x256 OEMI image is generated, as shown in Figure 2 .

[0159] S7: The acoustic signal after knocking the oyster shell is converted into a two-dimensional Markov transition field spectrum image (MTFI), and the size of the spectrum image is 256x256 pixels, as shown in Figure 3 . In this embodiment, the time length of the acoustic signal is 1s; specifically, the conversion of the acoustic signal after knocking the oyster shell into a two-dimensional MTFI includes:

[0160] First, the acoustic signal spectrum is calculated, and then the normalized spectrum is obtained using the minimum-maximum normalization method.

[0161] Second, the amplitudes of are arranged in ascending order along the amplitude axis to obtain an ascending set , and the quartile statistics of are constructed according to the quartiles:

[0162]

[0163] wherein, N represents the amplitude axis;

[0164] assigns to along the amplitude axis, and calculates the first-order Markov transition field matrix :

[0165]

[0166] wherein,​​​​​​ N range and denotes the Markov transition field coordinate matrix;

[0167] Finally, the spectral Markov transition field is constructed :

[0168]

[0169] wherein N represents the element coordinate of the spectral Markov transition field, and the MTFI is obtained.

[0170] Further, according to the preprocessed multi-sensor heterogeneous data and the oyster fullness index, a closed oyster fullness data set is constructed and divided into a training set and a test set; the features of each measured closed oyster can be represented as , wherein represents the obtained OMRI, represents the obtained MTFI, and each measured closed oyster corresponds to an oyster fullness grade , wherein is the actual measured oyster fullness index divided into 2 oyster fullness grades, and then the measured closed oyster data set D of the oyster can be represented as a matrix , wherein 70% of the sample data is used as the training set, and the remaining 30% is used as the test set;

[0171] Further, a hybrid data fusion model (FEM) integrating two unique neural networks is constructed, a distributed training strategy is used for model training, the trained model takes the preprocessed multi-sensor heterogeneous data as input, outputs a fusion feature vector, and saves the FEM, as shown in Figure 4 ;

[0172] Specifically, the hybrid data fusion model integrating two unique neural networks consists of three parts, including an OEMI-CNN feature extraction network, an MTFI-VGG feature extraction network, and a feature fusion network.

[0173] Preferably, the OEMI-CNN feature extraction network consists of four convolutional layers and three fully connected layers. All convolutional layers and fully connected layers use batch normalization (BatchNorm) and rectified linear unit activation function (ReLU), and dropout is used after each fully connected layer with a dropout probability P to reduce overfitting. The detailed network structure is described as follows:

[0174] Input layer: accepts OEMI images with a size of 256x256 pixels, and the images have RGB three channels;

[0175] Convolutional layers: The network includes four convolutional layers, each containing 100 convolutional kernels with a size of 3x3 and a stride of 1. After each convolutional layer, a 2x2 max pooling layer is connected, with a pooling stride of 2, to reduce the size of the feature map and extract the main features;

[0176] First convolutional layer: contains 100 3x3 convolutional kernels followed by a 2x2 max pooling layer;

[0177] Second convolutional layer: contains 100 3x3 convolutional kernels followed by a 2x2 max pooling layer;

[0178] Third convolutional layer: contains 100 3x3 convolutional kernels followed by a 2x2 max pooling layer;

[0179] Fourth convolutional layer: contains 100 3x3 convolutional kernels followed by a 2x2 max pooling layer;

[0180] Fully connected layers: After the fourth pooling layer, the input data is flattened into a one-dimensional feature vector; these extracted feature vectors are sequentially passed to three fully connected layers, each containing 2048, 1024, and 1024 neuron units, respectively;

[0181] First fully connected layer: contains 2048 neuron units and receives the flattened feature vector;

[0182] Second fully connected layer: contains 1024 neuron units and receives the output of the first fully connected layer;

[0183] Third fully connected layer: contains 1024 neuron units and receives the output of the second fully connected layer;

[0184] Output layer: After the fully connected layers, a final output layer is connected, which is a softmax layer containing 2 nodes. This softmax layer is used to output the probability distribution of the closed oyster plumpness grade.

[0185] Preferably, the MTFI-VGG feature extraction network is composed of 16 convolutional layers and 3 fully connected layers, each convolutional layer and fully connected layer uses BatchNorm and ReLU activation function, and dropout is used after each fully connected layer, with a dropout probability of P. The detailed network structure is described as follows:

[0186] Input layer: accepts MTFI images with a size of 256x256 pixels, with RGB three channels;

[0187] Convolutional layers: The first and second convolutional layers each contain 64 3x3 convolutional kernels with a stride of 1 and use a ReLU activation function.

[0188] The third and fourth convolutional layers each contain 128 3x3 convolutional kernels with a stride of 1 and use a ReLU activation function.

[0189] The fifth, sixth, and seventh convolutional layers each contain 256 3x3 convolutional kernels with a stride of 1 and use a ReLU activation function.

[0190] The eighth, ninth, and tenth convolutional layers each contain 512 3x3 convolutional kernels with a stride of 1 and use a ReLU activation function.

[0191] The eleventh, twelfth, and thirteenth convolutional layers each contain 512 3x3 convolutional kernels with a stride of 1 and use a ReLU activation function.

[0192] Pooling layers: After the second, fourth, seventh, tenth, and thirteenth convolutional layers, a 2x2 max-pooling layer is connected with a pooling stride of 2 to reduce the size of the feature maps and improve the effectiveness of feature extraction.

[0193] Fully connected layers: After the convolutional layers, three fully connected layers are connected, each containing 4096 neurons and using a ReLU activation function.

[0194] First fully connected layer: Contains 4096 neurons and receives the feature vector after flattening.

[0195] Second fully connected layer: Contains 4096 neurons and receives the output of the first fully connected layer.

[0196] Third fully connected layer: Contains 4096 neurons and receives the output of the second fully connected layer.

[0197] Output layer: After the fully connected layers, a final output layer is connected, which is a softmax layer containing 2 nodes, used to output the probability distribution of the closed oyster plumpness grade.

[0198] Preferably, feature fusion network: This network is used to effectively fuse features from two different feature extraction networks to provide more comprehensive information for subsequent classification models. Specifically, the feature fusion network splices and fuses features from the OEMI-CNN network and the MTFI-VGG network to form a high-dimensional feature vector.

[0199] OEMI-CNN features : Features obtained from the last fully connected layer of the OEMI-CNN feature extraction network, with a dimension of 1x1024.

[0200] MTFI-VGG features : features obtained from the last fully connected layer of the MTFI-VGG feature extraction network, with a dimension of 1x4096.

[0201] The feature fusion network uses the Concat method to horizontally splice and fuse the above two features. Through the Concat operation, the two feature vectors are merged in the feature dimension to generate a new higher-dimensional feature representation. This fused feature representation can better integrate the deep features extracted from visual data and acoustic data, thereby improving the classification performance and generalization ability of the subsequent classification model. The fused feature representation obtained by this method is as follows:

[0202]

[0203] Finally, a fused feature vector with a dimension of 1x(1024+4096)=1x5120 is obtained, which contains rich information from two different feature extraction networks, realizing multi-sensor heterogeneous data fusion.

[0204] Preferably, the model training process adopts a distributed training strategy, first pre-trains through a large-scale public dataset to obtain preliminary network weights, and then fine-tunes through a specific task dataset, fully utilizing existing rich data resources to improve the accuracy and generalization ability of the model in extracting features. The specific process includes:

[0205] The OEMI-CNN feature extraction network is pre-trained using the ILSVRC2012 dataset in the ImageNet, the world's largest image recognition database. Then, the OEMI images in the training set are used as input, and the oyster plumpness grade index is used as output. The OEMI-CNN feature extraction network that has been trained on the ILSVRC2012 dataset is trained again to fine-tune and optimize the network.

[0206] At the same time, the MTFI-VGG feature extraction network is pre-trained using the AudioSet dataset published by Google. Then, the MTFI data in the training set is used as input, and the oyster plumpness grade index is used as output. The MTFI-VGG feature extraction network that has been pre-trained on the AudioSet dataset is trained again to fine-tune and correct the network.

[0207] Reference Figure 5 , a neural network-based plumpness classification model (FCM) is constructed based on the fused feature vector For input, output oyster plumpness grade index, save the model after verifying the FCM performance through the test set. The plumpness classification model based on neural network consists of three fully connected layers and a softmax layer containing 2 nodes, and the input features of the model come from the fusion features obtained by the previous feature extraction network . The specific architecture is as follows:

[0208] Fully connected layer: The model contains three fully connected layers, respectively provided with 2048, 2048 and 1024 nodes. Each fully connected layer adopts BatchNorm and ReLU activation function to ensure the stability and accelerate convergence of the network during the training process. Dropout is applied after each fully connected layer, and the dropout probability is set to P to prevent model overfitting and improve its generalization ability.

[0209] First fully connected layer: has 2048 nodes, and the input is the high-dimensional fusion feature data received by the model .

[0210] Second fully connected layer: also has 2048 nodes, and receives the output of the first fully connected layer.

[0211] Third fully connected layer: contains 1024 nodes, and receives the output of the second fully connected layer.

[0212] Final output layer: a final output layer is connected after the fully connected layer, and the final output layer is a softmax layer containing 2 nodes, which is used to output 2 kinds of closed oyster plumpness index to complete the nondestructive detection and recognition of closed oyster plumpness;

[0213] Optionally, the accuracy (Accuracy), precision (Precision) and recall (Recall) are used to evaluate the effect of the plumpness classification model, and the calculation formula is:

[0214]

[0215]

[0216]

[0217] Among them, TP represents the number of samples actually positive and predicted positive, FP represents the number of samples actually negative and predicted positive, FN represents the number of samples actually positive and predicted negative, and TN represents the number of samples actually negative.

[0218] Further, the fusion features obtained by FEM through the training set The existing machine learning algorithms Lasso, SVM, Decision Tree, XGBoost, Catboost and FCM are trained to build a closed oyster plumpness grading model, and the fusion features obtained by FEM are obtained through the test set The classification performance obtained by the test is obtained, and the comparison results are shown in Table 2. The accuracy, precision and recall of the FCM model are 95%, 94% and 95% respectively, which are higher than those of the closed oyster plumpness grading model constructed by asso, SVM, Decision Tree, XGBoost and Catboost, indicating that the FCM model is suitable for classifying the closed oyster plumpness, and has obvious advantages. The plumpness classification model is saved.

[0219] Table 2

[0220] Model Accuracy Precision Recall Lasso 0.82 0.84 0.83 SVM 0.86 0.88 0.86 Random Forest 0.89 0.90 0.89 XGBoost 0.93 0.91 0.93 Catboost 0.92 0.93 0.91 FCM 0.95 0.94 0.95

[0221] Preferably, the embodiment provides a closed oyster plumpness nondestructive testing system based on multi-sensor heterogeneous data fusion, as shown in Figure 6 The closed oyster plumpness nondestructive testing system based on multi-sensor heterogeneous data fusion comprises a data acquisition module, a data preprocessing module, a data fusion model construction module and a plumpness detection grading module.

[0222] The data acquisition module is used for acquiring multiple sets of multi-angle closed oyster shell images and acoustic signals after knocking the oyster shell.

[0223] The data preprocessing module is used for preprocessing the multi-angle closed oyster shell images and the acoustic signals after knocking the oyster shell, so as to improve the data quality and consistency.

[0224] The data fusion model construction module is used for establishing and saving a hybrid data fusion model integrating two unique neural networks, which realizes data fusion of the closed oyster shell images and the acoustic signals, generates a fusion feature vector through feature extraction and merging.

[0225] The plumpness detection grading module is based on the fusion feature vector, and establishes a plumpness classification model based on a neural network. The model outputs the plumpness grade index of the closed oyster, and realizes accurate grading of the oyster plumpness.

[0226] The beneficial effects of the present application are as follows:

[0227] The application provides more sufficient and complementary information for the closed shell oyster fullness detection through the fusion of multi-source heterogeneous data, can accurately identify the closed shell oyster fullness, has the advantages of real-time, non-invasive, high accuracy and the like, realizes the nondestructive testing of the closed shell oyster freshness, and provides stronger nondestructive testing technical support for the sustainable development of the oyster industry.

[0228] The various embodiments are described in a progressive manner in the specification, and each embodiment focuses on the difference from other embodiments, and the same or similar parts between the various embodiments can be mutually referred to.

[0229] The principles and implementation manners of the application are described by applying specific examples herein, and the above description of the examples is only used to help understand the method and the core idea of the application; meanwhile, according to the idea of the application, the specific implementation manners and application ranges can be changed by the person skilled in the art. In conclusion, the content of the specification should not be understood as the limitation of the application.

Claims

1. A non-destructive testing method for the condition of closed-shell oysters based on multi-sensor heterogeneous data fusion, characterized in that, include: Collect sensor data for target detection of oyster shells to obtain heterogeneous data of the sensor to be detected; The heterogeneous data of the sensor to be detected is input into a pre-trained fatness classification model for detection, and the target fatness non-destructive detection result is obtained. The training process of the fatness classification model includes: Collect multi-angle images of the shells of prepared oysters, vibration acoustic signals after tapping the oyster shells, and measured indicators of oyster plumpness levels; the shell images include: front view images and side view images; The shell image and the vibration acoustic signal are preprocessed to obtain multi-sensor heterogeneous data; A closed-shell oyster plumpness dataset is constructed based on the measured index of oyster plumpness level and the heterogeneous data from the multi-sensor, and the closed-shell oyster plumpness dataset is divided into a training set and a test set. A hybrid data fusion model integrating multiple feature extraction networks is constructed. This model is trained using a distributed training strategy, and the heterogeneous multi-sensor data from the training set is input into the hybrid data fusion model to fuse the multi-source heterogeneous data, obtaining a fused feature vector. The hybrid data fusion model integrates an OPI-CNN feature extraction network, an MTFI-VGG feature extraction network, and a feature fusion network. The OPI-CNN feature extraction network is used to extract depth features from first-type visual sensor data; the MTFI-VGG feature extraction network is used to extract depth features from second-type acoustic sensor data; and the feature fusion network is used to fuse the features output by the OPI-CNN and MTFI-VGG feature extraction networks. A plumpness classification model based on a neural network is constructed, and the plumpness classification model is trained by taking the fusion feature vector corresponding to the training set as input and the measured plumpness level index of the oyster in the training set as the target output. The trained fatness classification model is validated using the test set to obtain validation results. If the validation results meet the target requirements, the fatness classification model is saved.

2. The non-destructive testing method for the fullness of closed-shell oysters based on multi-sensor heterogeneous data fusion according to claim 1, characterized in that, Images of the shells of prepared oysters from multiple angles, acoustic vibration signals after tapping the oyster shells, and measured indicators of oyster plumpness were collected, including: The front view and side view of the prepared closed-shell oyster are obtained from top and side views using two industrial cameras with mutually perpendicular lens angles. The center of the shell of the prepared closed oyster is struck using a steel rod released from a fixed height, and the vibration acoustic signal is collected using a microphone sensor. The total weight of each of the prepared oyster shells was measured using a balance. After removing the shells from the prepared oyster shells, the soft tissue of the prepared oyster shells was weighed to obtain the soft tissue weight. The measured index of oyster plumpness grade was obtained based on the total weight and the soft tissue weight. The calculation formula for the measured index of oyster plumpness grade is as follows: ;in, The measured index for the oyster plumpness grade; The weight of the soft tissue; The total weight; Oysters with measured plumpness levels less than or equal to a preset plumpness threshold are classified as low plumpness, while oysters with measured plumpness levels greater than the preset plumpness threshold are classified as high plumpness.

3. The non-destructive testing method for the fullness of closed-shell oysters based on multi-sensor heterogeneous data fusion according to claim 1, characterized in that, The shell image and the vibration acoustic signal are preprocessed to obtain multi-sensor heterogeneous data, including: The contrast and color saturation of the front view image and the side view image are enhanced by histogram equalization. The noise in the enhanced front view image and the side view image is repaired using a wavelet threshold filtering algorithm; K-means clustering and oyster shell region segmentation are performed on the noise-restored front view image and side view image to obtain the segmented front view image and side view image; The segmented front view image and the side view image are horizontally stitched and fused to obtain an external morphological image of the oyster shell, and the external morphological image of the oyster shell is converted into a 256×256 pixel RGB three-channel image. The vibration acoustic signal is converted into a two-dimensional Markov transfer field spectrum image with a size of 256×256 pixels.

4. The non-destructive testing method for the fullness of closed-shell oysters based on multi-sensor heterogeneous data fusion according to claim 3, characterized in that, A hybrid data fusion model integrating multiple feature extraction networks is constructed. This model is trained using a distributed training strategy. The multi-sensor heterogeneous data from the training set is then input into the hybrid data fusion model to fuse the multi-source heterogeneous data, resulting in a fused feature vector, including: The Concat method is used to horizontally concatenate and fuse the first feature (1×1024 dimensions) obtained from the last fully connected layer of the OMI-CNN feature extraction network and the second feature (1×4096 dimensions) obtained from the last fully connected layer of the MTFI-VGG feature extraction network to obtain the fused feature vector. The expression for obtaining the fused feature vector is as follows: ;in, The fused feature vector is of dimension 1×5120; This is the first feature; This is the second feature; This is the fusion equation.

5. A non-destructive testing method for the fullness of closed-shell oysters based on multi-sensor heterogeneous data fusion according to claim 4, characterized in that, The OPI-CNN feature extraction network includes: a first input layer, a first output layer, first to fourth convolutional layers with a stride of 1, and first to third fully connected layers with a stride of 2; all convolutional and connected layers in the OPI-CNN feature extraction network employ batch normalization and modified linear unit activation functions; the first, second, and third fully connected layers all embed a dropout algorithm; the first to fourth convolutional layers each contain 100 3×3 convolutional kernels; each of the first to fourth convolutional layers is followed by a 2×2 max pooling layer; the number of neurons in the first, third, and fourth fully connected layers are 2048, 1024, and 1024, respectively; the first output layer is a softmax layer containing 2 nodes; The first input layer is used to receive the external morphological image of the closed-shell oyster; the first fully connected layer is used to receive the first flattened feature vector output by the max pooling layer from the first convolutional layer to the fourth convolutional layer; the second fully connected layer is used to receive the output of the first fully connected layer; the third fully connected layer is used to receive the output of the second fully connected layer; and the first output layer is used to receive the output of the third fully connected layer and output the first feature.

6. The non-destructive testing method for the fullness of closed-shell oysters based on multi-sensor heterogeneous data fusion according to claim 4, characterized in that, The MTFI-VGG feature extraction network includes: a second input layer, a second output layer, fifth to twenty-first convolutional layers with a stride of 1 and ReLU activation, and fourth to sixth fully connected layers with a stride of 2 and ReLU activation. The fifth to seventeenth convolutional layers contain 3×3 convolutional kernels of the following numbers: 64, 64, 128, 128, 256, 256, 256, 512, 512, 512, 512, 512, 512. The sixth, eighth, eleventh, fourteenth, and seventeenth convolutional layers are each followed by a 2×2 max-pooling layer with a stride of 2. The fourth to sixth fully connected layers each contain 4096 neurons. The second input layer is used to receive the Markov transfer field spectrum image; the second output layer is used to output the second feature.

7. A non-destructive testing method for the fullness of closed-shell oysters based on multi-sensor heterogeneous data fusion according to claim 4, characterized in that, A hybrid data fusion model integrating multiple feature extraction networks is constructed. This model is trained using a distributed training strategy. The multi-sensor heterogeneous data from the training set is input into the hybrid data fusion model to fuse the multi-source heterogeneous data, resulting in a fused feature vector. The model also includes: The OMI-CNN feature extraction network was initially pre-trained using the ILSVRC2012 dataset from the ImageNet image database. Using the actual image data in the training set as input and the measured index of oyster plumpness level as output, the pre-trained OMI-CNN feature extraction network is trained a second time; the second training includes fine-tuning and optimization. The MTFI-VGG feature extraction network was initially pre-trained using the AudioSet dataset. Using the Markov transition field spectrum image in the training set as input and the measured index of oyster plumpness level as output, the pre-trained MTFI-VGG feature extraction network is trained a second time; the second training includes fine-tuning and correction.

8. A non-destructive testing method for the fullness of closed-shell oysters based on multi-sensor heterogeneous data fusion according to claim 4, characterized in that, The fatness classification model includes: a seventh to ninth fully connected layer and a third output layer using batch normalization and ReLU activation function; the third output layer is a softmax layer with 2 nodes; the seventh to ninth fully connected layers all embed a dropout algorithm; the number of neurons in the seventh to ninth fully connected layers are 2048, 2048 and 1024 respectively; The third output layer is used to output the non-destructive testing results of the target fatness.

9. A non-destructive testing system for the plumpness of closed-shell oysters based on multi-sensor heterogeneous data fusion, characterized in that, The system for implementing the non-destructive detection method for the plumpness of closed-shell oysters based on multi-sensor heterogeneous data fusion as described in claim 1 includes: a data acquisition module, a data preprocessing module, a data fusion model construction module, and a plumpness detection and grading module. The data acquisition module is used to acquire multiple sets of multi-angle images of closed oyster shells, as well as acoustic signals after the oyster shells are struck. The data preprocessing module is used to preprocess the multiple sets of multi-angle images of closed-shell oyster shells and the acoustic signals; The data fusion model construction module is used to establish and save the hybrid data fusion model, and to use the hybrid data fusion model to extract and merge features from the preprocessed multi-group multi-angle closed-shell oyster shell images and the acoustic signals to obtain a fused feature vector. The plumpness detection and grading module is used to establish a plumpness classification model based on the fused feature vector, and to use the plumpness classification model to classify the plumpness of the closed-shell oysters to be detected, and output plumpness level index.

10. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of a non-destructive testing method for the plumpness of closed-shell oysters based on multi-sensor heterogeneous data fusion as described in any one of claims 1 to 8.

Images