Tuna quality grading intelligent detection method based on multivariate information feature fusion

Abstract

The invention discloses a tuna quality grading intelligent detection method based on multivariate information feature fusion, and relates to the technical field of fish quality grading detection. The method comprises the following steps: acquiring fish body images and meat quality images of a plurality of different tunas, collecting related text information of the tunas, including producing areas, varieties, weights and body lengths, and constructing a data set; preprocessing the image and the related text information; generating a multi-modal fusion feature vector through feature extraction and fusion; inputting into a tuna quality grade detection module; and training the deep learning model, and subsequently performing quality grading detection on the to-be-detected tuna. According to the method, the fish body image, the meat quality image and the related text information are fully considered, the multi-modal fusion features including the fish body, the meat quality and the related text information are generated by adopting the multi-modal feature fusion technology, and training is performed through the quality grading detector, so that the robustness and the accuracy of detection are improved, and the detection result is more comprehensive and reliable.

Description

Technical Field

[0001] The present invention relates to the technical field of fish quality grading detection, and in particular to an intelligent detection method for tuna quality grading based on multivariate information feature fusion. Background Art

[0002] Tuna is an oceanic migratory fish, and its common species include yellowfin tuna, bigeye tuna, bluefin tuna, albacore tuna and masu tuna. As a high-value marine economic product, the quality of tuna directly determines the market price and consumer choice. In processing plants, restaurants and other scenarios, accurate quality judgment is crucial to classify tuna according to quality grades, thereby determining its use and formulating corresponding processing procedures.

[0003] However, current quality assessment often relies on manual inspection, which requires inspectors to have rich experience to judge the color, texture and fat content of fish meat, etc., which is inefficient and subjective, and it is difficult to ensure consistency and accuracy. In addition, the existing automatic detection method for tuna quality mainly uses fish tail profile images and fishing vessel information, combined with machine learning for auxiliary judgment, but this method has obvious limitations. First, it only analyzes the tail profile, ignoring the meat texture, color and fat distribution of other parts. In fact, the meat quality information of different parts is crucial to evaluate the overall quality of tuna, and the integrity characteristics of the fish body should also be included in the evaluation scope to ensure the accuracy and comprehensiveness of the test results. Secondly, there are significant differences in the meat quality characteristics of different varieties of tuna. For example, the meat of albacore tuna is lighter in color and has a lower oil content, which is suitable for canning processing, while the meat of bluefin tuna is dark red and has a very high oil content. It is often used to make sashimi, sushi and grilled fish. Therefore, these variety differences also have an important impact on quality evaluation and market pricing. However, existing detection methods mainly rely on information from fishing vessels and fail to combine multi-dimensional data such as origin, species, weight, and body length. This makes it difficult for the model to capture the comprehensive characteristics of tuna quality and cannot meet the precise detection needs of different application scenarios.

[0004] In general, the shortcomings caused by single information source and local image analysis have become the bottleneck of current tuna quality detection. Therefore, a multimodal information fusion solution is urgently needed, which can flexibly analyze the overall image of the fish body and the meat quality image of any part, and combine multi-dimensional structured text data to provide technical support for more accurate evaluation of tuna quality. Summary of the invention

[0005] In order to overcome the shortcomings of the background technology, the present invention provides an intelligent detection method for tuna quality grading based on multivariate information feature fusion, which fully considers fish body images, meat quality images and related text information, adopts multimodal feature fusion technology to generate multimodal fusion features including fish body, meat quality and related text information, and trains through a quality grading detector to improve the robustness and accuracy of detection, making the detection results more comprehensive and reliable.

[0006] To achieve the above object, the present invention adopts the following technical scheme: an intelligent detection method for tuna quality grading based on multivariate information feature fusion, comprising four parts: a multivariate information collection module, a data preprocessing module, a feature extraction and fusion module, and a tuna quality grade detection module, wherein the feature extraction and fusion module comprises a feature extraction model and a feature fusion model, and together with the network structure of the tuna quality grade detection module, constitutes a deep learning model, and the method comprises the following steps:

[0007] Step 1: In the multivariate information collection module, obtain the complete body images of several different tunas and the meat images of any parts, and collect the relevant text information of the tuna, including the origin, species, weight and length, and then build a data set. According to user needs, the quality of tuna is divided into five levels from high to low, and the real grade label is added to each tuna in the data set through manual annotation;

[0008] Step 2: In the data preprocessing module, the fish body images and meat quality images are standardized and preprocessed, all the original images are uniformly adjusted to the target size, and the pixel values ​​are normalized;

[0009] Step 3: In the data preprocessing module, the relevant text information is divided into two categories for preprocessing. Among them, weight and body length are normalized to obtain normalized numerical feature vectors. Origin and variety are converted into dense vector representation using the embedding layer in the deep learning framework PyTorch to obtain embedded discrete feature vectors. The normalized numerical feature vectors and embedded discrete feature vectors are concatenated to obtain the relevant text information feature F. text ;

[0010] Step 4: In the feature extraction and fusion module, the fish image and meat image preprocessed in step 2 are input into the parallel convolutional neural network using the feature extraction model. Several convolutional layers are used to extract the local features of the fish body and meat quality, respectively. The feature dimension is reduced through the pooling layer while retaining the significance of the local features. The multi-dimensional feature map output by the convolutional layer is globally averaged and pooled to generate two fixed-length feature vectors, including the feature vector of the fish body. and the feature vector of meat quality They represent the overall characteristics of the fish body and the overall characteristics of the meat respectively;

[0011] Step 5: In the feature extraction and fusion module, the feature extraction model is used to extract the relevant text information features F in step 3. text Input into the fully connected network to learn its potential features, and output the extracted relevant text information features

[0012] Step 6: In the feature extraction and fusion module, the feature fusion model is used to adopt a multimodal feature fusion strategy to extract the feature vector of the fish body obtained in step 4. and the feature vector of meat quality Related text information features obtained in step 5 The multi-head attention mechanism is used to integrate and fuse the multimodal fusion feature vector F containing fish body images, meat quality images and related text information. global ;

[0013] Step 7: In the tuna quality level detection module, input the multimodal fusion feature vector F obtained in step 6 global ,The network structure of the tuna quality grade detection module is composed of a ,multilayer perceptron and a Softmax layer, and the output is a probability ,distribution representing the probability of the tuna belonging to each of the five grades;

[0014] Step 8: Train the deep learning model and calculate the loss between the probability distribution output in step 7 and the true grade label. The true grade label is given in the form of one-hot encoding, where the position corresponding to the grade of the actual quality of tuna is 1 and the other positions are 0. Then, the weighted cross entropy loss function is used to evaluate the accuracy of the deep learning model. According to the calculated loss value, the parameters of the deep learning model are adjusted using back propagation until the loss value no longer decreases, thereby obtaining the trained deep learning model.

[0015] Step nine: By inputting the sample information of the tuna to be tested into the trained deep learning model obtained in step eight, the quality grading test of the tuna to be tested can be achieved.

[0016] Furthermore, the step 2 specifically includes:

[0017] S2.1. Set the target size. If the original image is not proportional to the target size, fill the original image with white if it is smaller than the target size. If the original image is larger than the target size, scale it proportionally to make its long side consistent with the target size, and then perform center cropping to extract the pixel area in the middle that is consistent with the target size.

[0018] S2.2. Use the normalization formula to adjust the mean of all image pixels after they are resized to the target size to 0 and the standard deviation to 1, and normalize all image pixels to the range of [0,1]. The calculation formula is as follows:

[0019]

[0020] In the formula, normalized pixel Represents normalized pixels, original pixel represents the original pixel, μ is the mean, and σ is the standard deviation.

[0021] Furthermore, the step 4 specifically includes:

[0022] S4.1. For the preprocessed fish image I body , extract the local features of the fish body, including surface texture and damage, through several convolutional layers. Suppose L b The operation of each convolutional layer is:

[0023]

[0024] In the formula, is the feature map generated after the l-th layer convolution of the fish image, is the lth layer convolution operation of the fish image, including the convolution kernel weight and bias. In the first layer, ReLU() represents the activation function;

[0025] After each convolution operation, a pooling layer is used to reduce the spatial dimension of the feature map while retaining the saliency of local features, expressed as:

[0026]

[0027] In the formula, It is the feature map after the fish image is pooled, and Pooling() represents pooling;

[0028] S4.2. Preprocessed meat image I meat , extract the local features of meat quality, including color and fat distribution, through several convolutional layers. Suppose L m The operation of each convolutional layer is:

[0029]

[0030]

[0031] In the formula, is the feature map generated after the l-th layer convolution of the meat image, This is the convolution operation of the meat image layer l, including the convolution kernel weight and bias. In the first layer,

[0032] After each convolution operation, a pooling layer is used to reduce the spatial dimension of the feature map while retaining the saliency of local features, expressed as:

[0033]

[0034] In the formula, It is the feature map after meat image pooling;

[0035] S4.3. Generate two fixed-length feature vectors through global average pooling, where:

[0036] Feature map for all fish images Perform global average pooling to generate the feature vector of the fish body The formula is as follows:

[0037]

[0038] Where H' b and W' b They are the height and width of the feature map after the last layer of pooling of the fish image;

[0039] Feature map for all meat images Perform global average pooling to generate the feature vector of meat quality The formula is as follows:

[0040]

[0041] Where H' m and W' m They are respectively the height and width of the feature map after the last layer of pooling of the meat image.

[0042] Furthermore, the step six specifically includes:

[0043] S6.1. The extracted relevant text information features are transformed into Project to the same dimension as the image feature vector to obtain the relevant text information feature vector Achieve feature dimension alignment;

[0044] S6.2. The feature vector of the fish body Characteristic vector of meat quality and related text information feature vector These three eigenvectors are concatenated into an input matrix F concat , each row represents a feature vector of a modality, and a multi-head attention mechanism is used to fuse the three feature vectors into a multi-modal fusion feature vector. The formula is as follows:

[0045]

[0046] Q=F concat W Q

[0047] K=F concat W K

[0048] V=F concat W V

[0049]

[0050] Head i =Attention(Q i ,K i ,V i )

[0051] MultiHead(Q,K,V)=Concat(Head1,Head2,Head3)W O

[0052] F fused (i,:)=MultiHead(Q,K,V)

[0053]

[0054] In the formula, Q, K, are query, key and value matrices respectively, W Q , W K , are the corresponding learnable linear transformation matrices, Attention() represents the attention mechanism, softmax() normalizes each attention weight to represent its probability, d is the dimension of each feature vector, Head i Represents the output of the i-th attention head in the multi-head attention mechanism, MultiHead() represents the final output of the multi-head attention, and Concat represents concatenating the outputs of multiple heads by column. is the linear transformation matrix of the output, h is the number of heads of multi-head attention, F fused (i,:) represents the fusion feature of the i-th modality, is the multimodal fusion feature vector.

[0055] Compared with the prior art, the present invention has the following beneficial effects:

[0056] 1. The method of the present invention is no longer limited to the input of images of specific parts. Whether it is the image of the whole body of tuna or the image of the meat quality of any part, it can effectively extract deep-level image features. Through the convolutional neural network, local information such as surface texture and damage can be extracted from the overall image of the fish body, and microscopic information such as color and fat distribution can be extracted from the meat quality image. Therefore, even if only the meat quality image of a certain part of the tuna is taken, the model can still give a relatively accurate quality assessment;

[0057] 2. The method of the present invention adopts multimodal feature fusion technology to deeply fuse fish body images, meat quality images with their origin, species, weight, and body length. Through the multi-head attention mechanism, it can effectively capture the relationship between each modal data, make full use of the complementarity of multimodal information, generate a more comprehensive feature representation, significantly improve robustness and accuracy, and make the detection results more comprehensive and reliable. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 It is a flowchart of the method of the present invention;

[0059] Figure 2 is a structural diagram of the deep learning model in the present invention;

[0060] Figure 3 is an example diagram of tuna quality classification in the embodiment. DETAILED DESCRIPTION

[0061] The technical solution of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0062] Traditional tuna quality detection methods mainly rely on manual identification, which is inefficient and prone to errors. At present, automatic identification methods use fish tail profile images and fishing boat information, combined with traditional machine learning models to analyze features. Although they can provide certain quality judgments, they have significant limitations. Specifically, the fish tail profile image can only provide local meat quality information of the fish body, and cannot reflect the state of other parts of the fish body, such as the belly and back, which in turn affects the accuracy of the overall judgment. In addition, the machine learning methods used rely on manually designed features. These feature extraction methods are relatively simple and cannot capture the deep information in the image, especially the complex image patterns and semantic information. Moreover, the method of manually designing features is highly dependent on personnel experience and is difficult to adapt to the diversity of tuna quality in different species and different fishing environments. Especially when relying solely on the fish tail profile image without providing species information, the problem is more significant. The meat of different types of tuna presents different color characteristics. Without species information, it is difficult to give accurate and consistent judgment results. In addition, it is highly dependent on image data, especially on images of specific parts. In actual operation, if the image of the specific part is missing due to shooting conditions or other reasons, the detection performance of this method will drop significantly. This dependence on specific data makes the model's generalization ability weaker, and it is difficult to deal with the problem of missing data in practical applications.

[0063] In view of the shortcomings of the above existing methods, Figure 1-2 As shown in the figure, an intelligent detection method for tuna quality grading based on multivariate information feature fusion is proposed. Figure 1As shown, it includes four parts: multivariate information collection module, data preprocessing module, feature extraction and fusion module and tuna quality level detection module. Among them, the feature extraction and fusion module includes a feature extraction model and a feature fusion model, and together with the network structure of the tuna quality level detection module, it constitutes a deep learning model. In the multivariate information collection module, the system receives the complete body image of tuna and the meat quality image of any part, and simultaneously collects the relevant text information of tuna, including data such as origin, species, weight and body length. These image data combinations provide multimodal data. The body image and meat quality image can capture the image features of tuna, while the relevant text information provides background information and quantitative features. In the data preprocessing module, the system performs data preprocessing on the collected multimodal data. In the feature extraction and fusion module, the system uses a convolutional neural network to process the body image and meat quality image, extracts the key image features of the body and meat quality, and uses a fully connected network to process the relevant text information, extracts numerical features, and then uses a multimodal feature fusion strategy to integrate, and forms a multimodal fusion feature vector containing multi-level information through a multi-head attention mechanism. In the tuna quality level detection module, the multimodal fusion feature vector is input into the quality grading detector for training, and is subsequently used for the detection of tuna quality grading, which specifically includes the following steps:

[0064] Step 1: In the multivariate information collection module, a high-resolution camera is used to obtain complete body images of several different tunas and images of meat quality at any part of the tuna. At the same time, relevant text information of the tuna, including origin, species, weight, and length, is collected to achieve multivariate information collection, and then a data set for tuna quality grading detection is constructed. According to user needs, the quality of tuna is divided into five grades from high to low, namely, premium grade (S grade), excellent grade (A grade), good grade (B grade), qualified grade (C grade), and unqualified grade (D grade). The real grade label is added to each tuna in the data set through manual annotation;

[0065] Step 2: In the data preprocessing module, the fish body images and meat quality images are standardized and preprocessed, all the original images are uniformly adjusted to the target size, and the pixel values ​​are normalized to ensure the consistent distribution of subsequent input data. Specifically:

[0066] S2.1. Set the target size, for example, 224×224 pixels. If the original image is not proportional to the target size, fill the original image with white around the original image that is smaller than the target size. For the original image that is larger than the target size, scale it proportionally to make its long side consistent with the target size, and then perform center cropping to extract the pixel area in the middle that is consistent with the target size, so as to ensure that the adjusted fish body image and meat quality map can maintain their important features, and unify the input size for subsequent processing;

[0067] S2.2. Use the normalization formula to adjust the mean of all image pixels after they are resized to the target size to 0 and the standard deviation to 1, and normalize all image pixels to the range of [0,1]. The calculation formula is as follows:

[0068]

[0069] In the formula, normalized pixel Represents normalized pixels, original pixel represents the original pixel, μ is the mean, and σ is the standard deviation;

[0070] Step 3: In the data preprocessing module, the relevant text information is divided into two categories for preprocessing, where:

[0071] Weight and body length are numerical information, which are normalized to obtain normalized numerical feature vectors to ensure that different features are modeled at the same scale. The calculation formula is as follows:

[0072]

[0073] In the formula, X' is the normalized value, X is the original value, and X min is the minimum value in the data set, X max is the maximum value in the data set.

[0074] Origin and variety are discrete information. The embedding layer in the deep learning framework PyTorch is used to convert each feature into a dense vector representation to obtain a discrete feature vector after embedding.

[0075] Then, the normalized numerical feature vector and the embedded discrete feature vector are concatenated to obtain the relevant text information feature F text ;

[0076] Step 4: In the feature extraction and fusion module, combine Figure 2 As shown in FIG. 1 , the fish image and meat image preprocessed in step 2 are input into a parallel convolutional neural network using a feature extraction model. Several convolutional layers are used to extract local features such as the surface texture and damage of the fish and local features such as the color and fat distribution of the meat. The feature dimension is reduced by a pooling layer while retaining the significance of local features. The multidimensional feature map output by the convolutional layer is globally averaged and pooled to generate two fixed-length feature vectors, including the feature vector of the fish. and the feature vector of meat quality Respectively represent the overall characteristics of the fish body and the overall characteristics of the meat, specifically:

[0077] S4.1. For the preprocessed fish image Ibody , extract the local features of the fish body through several convolutional layers, including surface texture and damage, etc., assuming that L b The operation of each convolutional layer is:

[0078]

[0079]

[0080] In the formula, is the feature map generated after the l-th layer convolution of the fish image, is the lth layer convolution operation of the fish image, including the convolution kernel weight and bias. In the first layer, That is, the fish image itself, and ReLU() represents the activation function.

[0081] After each convolution operation, a pooling layer is used to reduce the spatial dimension of the feature map while retaining the saliency of local features, expressed as:

[0082]

[0083] In the formula, It is the feature map after the fish image is pooled, and Pooling() represents pooling;

[0084] S4.2. Preprocessed meat image I meat , extract the local features of meat quality, including color and fat distribution, through several convolutional layers. Suppose L m The operation of each convolutional layer is:

[0085]

[0086]

[0087] In the formula, is the feature map generated after the l-th layer convolution of the meat image, This is the convolution operation of the meat image layer l, including the convolution kernel weight and bias. In the first layer, That is, the fleshy image itself.

[0088] Similarly, after each convolution operation, a pooling layer is used to reduce the spatial dimension of the feature map while retaining the significance of local features, expressed as:

[0089]

[0090] In the formula, It is the feature map after meat image pooling;

[0091] S4.3. Generate two fixed-length feature vectors through global average pooling, where:

[0092] Feature map for all fish images Perform global average pooling to generate the feature vector of the fish body The formula is as follows:

[0093]

[0094] Where H' b and W' b are the height and width of the feature map after the last layer of pooling of the fish image. Perform global average pooling to generate the feature vector of meat quality The formula is as follows:

[0095]

[0096] Where H' m and W' m They are the height and width of the feature map after the last layer of pooling of the meat image;

[0097] Step 5: In the feature extraction and fusion module, combine Figure 2 As shown, the feature extraction model is used to extract the relevant text information features F in step 3. text Input into the fully connected network to learn its potential features, and output the extracted relevant text information features

[0098] Step 6: In the feature extraction and fusion module, combine Figure 2 As shown in the figure, the feature fusion model is used to adopt the multimodal feature fusion strategy to transform the feature vector of the fish body obtained in step 4. and the feature vector of meat quality Related text information features obtained in step 5 The multi-head attention mechanism is used to integrate and fuse the multimodal fusion feature vector F containing fish body images, meat quality images and related text information. global , specifically:

[0099] S6.1. The extracted relevant text information features are transformed into Project to the same dimension as the image feature vector to obtain the relevant text information feature vector Achieve feature dimension alignment;

[0100] S6.2. The feature vector of the fish body Characteristic vector of meat quality and related text information feature vector These three eigenvectors are concatenated into an input matrix F concat, each row represents the feature vector of a modality. In order to capture the deep interaction between different modalities, a multi-head attention mechanism is used to fuse the three feature vectors into a multi-modal fusion feature vector. The multi-head attention mechanism can adaptively assign attention weights to each modality, thereby capturing the mutual influence between modalities. The formula is as follows:

[0101]

[0102] Q=F concat W Q

[0103] K=F concat W K

[0104] V=F concat W V

[0105]

[0106] Head i =Attention(Q i ,K i ,V i )

[0107] MultiHead(Q,K,V)=Concat(Head1,Head2,Head3)W O

[0108] F fused (i,:)=MultiHead(Q,K,V)

[0109]

[0110] In the formula, Q, K, are query, key and value matrices respectively, W Q , W K , are the corresponding learnable linear transformation matrices, Attention() represents the attention mechanism, softmax() normalizes each attention weight to represent its probability, d is the dimension of each feature vector, Head i Represents the output of the i-th attention head in the multi-head attention mechanism, MultiHead() represents the final output of the multi-head attention, and Concat represents concatenating the outputs of multiple heads by column. is the linear transformation matrix of the output, h is the number of heads of multi-head attention, F fused (i,:) represents the fusion feature of the i-th modality (fish body, meat quality or related text information), is the multimodal fusion feature vector;

[0111] Step 7: In the tuna quality level detection module, input the multimodal fusion feature vector F obtained in step 6 global ,The network structure of the tuna quality grade detection module is composed of a ,multilayer perceptron and a Softmax layer. Firstly, the complex relationship between the ,inputs is captured by the multilayer perceptron, and the Softmax layer is used to ,convert the output into a probability distribution, which represents the ,probability of tuna belonging to each of the five grades (S, A, B, C, D);

[0112] Step 8: Train the deep learning model. To evaluate the performance of the deep learning model, we need to calculate the loss between the probability distribution output in step 7 and the true grade label. The true grade label is given in the form of one-hot encoding, where the position corresponding to the actual quality grade of tuna is 1 and the other positions are 0. Then, we use the weighted cross entropy loss function to evaluate the accuracy of the deep learning model.

[0113] The loss value is calculated and back propagation is used to adjust the parameters of the deep learning model until the loss value no longer decreases, thus obtaining the trained deep learning model.

[0114] Step nine: By inputting the sample information of the tuna to be tested into the trained deep learning model obtained in step eight, the quality grading test of the tuna to be tested can be achieved.

[0115] The present invention learns the quality grading features of tuna based on a deep learning model of multimodal feature fusion, wherein: an image feature extraction network adopts different convolutional layer structures to extract the respective local features of the fish body image and the meat quality image of tuna in parallel according to the different features, wherein the fish body image focuses on the appearance features such as surface texture and damage, while the meat quality image extracts the internal features such as color and fat distribution; a related text information feature extraction network is used to learn the potential features of the related text information and generate a vector representation matching the image features; a feature fusion network uses a multi-head attention mechanism to capture the complex interactive relationship between the fish body image, the meat quality image and the related text information, so as to adaptively focus on the most important features for quality judgment and fully integrate the information of each modality; a detector network is used to output the quality grading result of tuna.

[0116] Different from the existing methods, the method of the present invention comprehensively utilizes fish body images, meat quality images and related text information to achieve a more accurate assessment of tuna quality. Compared with the existing methods that rely solely on fish tail profile images and fishing boat information, the method of the present invention can capture more dimensional information of fish body and meat quality, thereby providing a more comprehensive basis for quality judgment. In terms of feature extraction, the present invention uses different convolutional network structures for fish body images and meat quality images to adapt to the uniqueness of their respective features. The extraction of fish body images focuses on appearance features such as surface texture and damage, while meat quality images focus on internal features such as color and fat distribution. This parallel processing not only improves the accuracy of feature extraction, but also enhances robustness. In addition, a multi-head attention mechanism is introduced to capture the complex relationship between fish body, meat quality and related text information, identify the importance of different features to tuna quality judgment, and further improve the accuracy of the assessment. This innovative method effectively overcomes the limitations of the existing method based on fish tail profile images, making quality grading more comprehensive and accurate, improving the intelligence level of the system, and better meeting the market demand for tuna quality detection.

[0117] Example

[0118] Combination Figure 3 As shown, this embodiment uses a high-resolution camera to obtain the body image and meat quality image of tuna, and collects relevant text information. The obtained image is adjusted to a uniform size of 224×224 pixels, and the pixels are normalized. The collected data information is normalized, and the origin and variety information is converted into a vector representation through the embedding layer. The origin, variety, weight, and body length data are spliced ​​to obtain relevant text information features. A convolutional neural network is used to extract local features of the fish body image and meat quality image, and a multi-head attention mechanism is used to capture the deep interactive information between the image and related text information. The fused multimodal fusion feature vector is sent to the quality grading detector for detection, and the quality grading result of the tuna sample is obtained as premium grade (S grade).

[0119] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other forms of assembly without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered in all respects as exemplary and non-restrictive, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations within the meaning and range of equivalents of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

[0120] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. An intelligent detection method for tuna quality grading based on multivariate information feature fusion, characterized by: The method comprises four parts: a multivariate information collection module, a data preprocessing module, a feature extraction and fusion module, and a tuna quality grade detection module. The feature extraction and fusion module comprises a feature extraction model and a feature fusion model, and together with the network structure of the tuna quality grade detection module, constitutes a deep learning model. The method comprises the following steps: Step 1: In the multivariate information collection module, obtain the complete body images of several different tunas and the meat images of any parts, and collect the relevant text information of the tuna, including the origin, species, weight and length, and then build a data set. According to user needs, the quality of tuna is divided into five levels from high to low, and the real grade label is added to each tuna in the data set through manual annotation; Step 2: In the data preprocessing module, the fish body images and meat quality images are standardized and preprocessed, all the original images are uniformly adjusted to the target size, and the pixel values ​​are normalized; Step 3: In the data preprocessing module, the relevant text information is divided into two categories for preprocessing. Among them, weight and body length are normalized to obtain normalized numerical feature vectors. Origin and variety are converted into dense vector representation using the embedding layer in the deep learning framework PyTorch to obtain embedded discrete feature vectors. The normalized numerical feature vectors and embedded discrete feature vectors are concatenated to obtain the relevant text information feature F. text ; Step 4: In the feature extraction and fusion module, the fish image and meat image preprocessed in step 2 are input into the parallel convolutional neural network using the feature extraction model. Several convolutional layers are used to extract the local features of the fish and meat, respectively. The feature dimension is reduced through the pooling layer while retaining the significance of the local features. The multi-dimensional feature map output by the convolutional layer is globally averaged and pooled to generate two fixed-length feature vectors, including the feature vector of the fish. and the feature vector of meat quality They represent the overall characteristics of the fish body and the overall characteristics of the meat respectively; Step 5: In the feature extraction and fusion module, the feature extraction model is used to extract the relevant text information features F in step 3. text Input into the fully connected network to learn its potential features, and output the extracted relevant text information features Step 6: In the feature extraction and fusion module, the feature fusion model is used to adopt a multimodal feature fusion strategy to extract the feature vector of the fish body obtained in step 4. and the feature vector of meat quality Related text information features obtained in step 5 The multi-head attention mechanism is used to integrate and fuse the multimodal fusion feature vector F containing fish body images, meat quality images and related text information. global ; Step 7: In the tuna quality level detection module, input the multimodal fusion feature vector F obtained in step 6 global ,The network structure of the tuna quality grade detection module is composed of a ,multilayer perceptron and a Softmax layer, and the output is a probability ,distribution representing the probability of the tuna belonging to each of the five grades; Step 8: Train the deep learning model and calculate the loss between the probability distribution output in step 7 and the true grade label. The true grade label is given in the form of one-hot encoding, where the position corresponding to the grade of the actual quality of tuna is 1 and the other positions are 0. Then, the weighted cross entropy loss function is used to evaluate the accuracy of the deep learning model. According to the calculated loss value, the parameters of the deep learning model are adjusted using back propagation until the loss value no longer decreases, thereby obtaining the trained deep learning model. Step nine: By inputting the sample information of the tuna to be tested into the trained deep learning model obtained in step eight, the quality grading test of the tuna to be tested can be achieved.

2. The intelligent detection method for tuna quality grading based on multivariate information feature fusion according to claim 1 is characterized in that: The step 2 specifically includes: S2.

1. Set the target size. If the original image is not proportional to the target size, fill the original image with white if it is smaller than the target size. If the original image is larger than the target size, scale it proportionally to make its long side consistent with the target size, and then perform center cropping to extract the pixel area in the middle that is consistent with the target size. S2.

2. Use the normalization formula to adjust the mean of all image pixels after they are resized to the target size to 0 and the standard deviation to 1, and normalize all image pixels to the range of [0,1]. The calculation formula is as follows: In the formula, normalized pixel Represents normalized pixels, original pixel represents the original pixel, μ is the mean, and σ is the standard deviation.

3. The intelligent detection method for tuna quality grading based on multivariate information feature fusion according to claim 1 is characterized in that: The step 4 specifically includes: S4.

1. For the preprocessed fish image I body , extract the local features of the fish body, including surface texture and damage, through several convolutional layers. Suppose L b The operation of each convolutional layer is: In the formula, is the feature map generated after the l-th layer convolution of the fish image, is the lth layer convolution operation of the fish image, including the convolution kernel weight and bias. In the first layer, ReLU() represents the activation function; After each convolution operation, a pooling layer is used to reduce the spatial dimension of the feature map while retaining the saliency of local features, expressed as: In the formula, It is the feature map after the fish image is pooled, and Pooling() represents pooling; S4.

2. Preprocessed meat image I meat , extract the local features of meat quality, including color and fat distribution, through several convolutional layers. Suppose L m The operation of each convolutional layer is: In the formula, is the feature map generated after the l-th layer convolution of the meat image, This is the convolution operation of the meat image layer l, including the convolution kernel weight and bias. In the first layer, After each convolution operation, a pooling layer is used to reduce the spatial dimension of the feature map while retaining the saliency of local features, expressed as: In the formula, It is the feature map after meat image pooling; S4.

3. Generate two fixed-length feature vectors through global average pooling, where: Feature map for all fish images Perform global average pooling to generate the feature vector of the fish body The formula is as follows: Where H' b and W' b They are the height and width of the feature map after the last layer of pooling of the fish image; Feature map for all meat images Perform global average pooling to generate the feature vector of meat quality The formula is as follows: Where H' m and W' m They are respectively the height and width of the feature map after the last layer of pooling of the meat image.

4. The intelligent detection method for tuna quality grading based on multivariate information feature fusion according to claim 1, characterized in that: The step six specifically includes: S6.

1. The extracted relevant text information features are transformed into Project to the same dimension as the image feature vector to obtain the relevant text information feature vector Achieve feature dimension alignment; S6.

2. The feature vector of the fish body Characteristic vector of meat quality and related text information feature vector These three eigenvectors are concatenated into an input matrix F concat , each row represents a feature vector of a modality, and a multi-head attention mechanism is used to fuse the three feature vectors into a multi-modal fusion feature vector. The formula is as follows: Q=F concat W Q K=F concat W K V=F concat W V Head i =Attention(Q i ,K i ,V i ) MultiHead(Q,K,V)=Concat(Head1,Head2,Head3)W O F fused (i,:)=MultiHead(Q,K,V) In the formula, are query, key, and value matrices respectively, are the corresponding learnable linear transformation matrices, Attention() represents the attention mechanism, softmax() normalizes each attention weight to represent its probability, d is the dimension of each feature vector, Head i Represents the output of the i-th attention head in the multi-head attention mechanism, MultiHead() represents the final output of the multi-head attention, and Concat represents concatenating the outputs of multiple heads by column. is the linear transformation matrix of the output, h is the number of heads of multi-head attention, F fused (i,:) represents the fusion feature of the i-th modality, is the multimodal fusion feature vector.

Images