Agricultural product classification method integrating dual-stream attention integration and cross-modal fusion

Abstract

The present invention is applicable to the field of artificial intelligence technology and specifically provides an agricultural product classification method that integrates dual-stream attention integration and cross-modal fusion. The method comprises the following steps: integrating multiple text feature vectors using adaptive layer weights, integrating multiple image feature vectors using a self-attention mechanism, and weighted summing the integrated text features and image features; performing high-power transformations on the integrated text features and image features, respectively, concatenating the transformed features with the original features to obtain enhanced text features and enhanced image features; constructing a relationship matrix between text and image features by calculating the Pearson correlation coefficient, and weightedly fusing the text and image features according to the relationship matrix to obtain the final fused features; and inputting the fused feature vector into an MLP classifier. The present invention enhances the model's ability to model and learn complex patterns; and uses cross-modal fusion to further improve the accuracy and generalization of agricultural product classification.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to an agricultural product classification method integrating dual-stream attention integration and cross-modal fusion. Background Art

[0002] Current methods for agricultural product classification typically use computer vision technology to extract agricultural product features and classify them based on image information. However, agricultural product images have significant characteristics, such as lighting and shooting environment. These characteristics make it difficult for image data to accurately reflect agricultural product information, further increasing the difficulty of classification.

[0003] Therefore, analysis using only the image module lacks robustness.

[0004] The current image-based agricultural product classification methods have the following shortcomings: First, agricultural products are classified only through a single visual modality, and factors such as lighting, shadows, and shooting angles have a significant impact on visual features. External factors can cause errors in the classification results, making it difficult to meet the needs of modern agricultural product classification; Second, the features extracted based on a single model are usually single and lack diversity, and cannot fully reflect all relevant information of agricultural products, which easily leads to information loss; Third, the features extracted by existing technologies are mostly primitive features, which make it difficult to capture higher-level semantic information of agricultural products; and the interactive relationship between different features is not utilized, making it impossible to effectively model the correlation of multimodal information. Summary of the Invention

[0005] The purpose of the embodiments of the present invention is to provide an agricultural product classification method that integrates dual-stream attention integration and cross-modal fusion, aiming to solve the technical problems raised in the above background technology.

[0006] To achieve the above objectives, the present invention provides the following technical solutions.

[0007] The agricultural product classification method integrating dual-stream attention integration and cross-modal fusion includes the following steps:

[0008] Using the text feature model to extract features from the preprocessed text data to obtain multiple text feature vectors;

[0009] The image feature model is used to extract features from the preprocessed image data to obtain multiple image feature vectors;

[0010] Multiple text feature vectors are integrated through adaptive layer weights, and multiple image feature vectors are integrated through the self-attention mechanism. The weighted summation is used to obtain the integrated text features and image features.

[0011] Perform high-power transformation on the integrated text features and image features respectively, and concatenate the transformed features with the original features to obtain enhanced text features and enhanced image features;

[0012] By calculating the Pearson correlation coefficient, the relationship matrix between text and image features is constructed, and the text and image features are weighted and fused according to the relationship matrix to obtain the final fusion feature;

[0013] The fused feature vector is input into the MLP classifier, and the MLP classifier maps the fused features to the corresponding agricultural product classification category to obtain the agricultural product classification result.

[0014] Furthermore, the step of extracting features from the pre-processed text data using the text feature model includes:

[0015] The TF-IDF model is used to extract the text feature vector T at the statistical information level TF-IDF ;

[0016] The Word2Vec model is used to convert words into high-dimensional vectors. The continuous bag-of-words model (CBOW) is used to predict the central word through the context of the agricultural product dataset. After training the CBOW model, each word will be mapped to a vector of fixed dimension to capture the text feature vector T. Word2Vec ;

[0017] Use the BERT model to extract the deep semantic context information feature vector T BERT .

[0018] Furthermore, the step of extracting features from the pre-processed image data using the image feature model includes:

[0019] The SIFT model is used to construct images of different scales through the Gaussian pyramid. SIFT calculates the local extreme values ​​of the image through the differential Gaussian pyramid DOG. SIFT assigns the direction of the key points, calculates the gradient direction around each key point, and uses the gradient magnitude and gradient direction to calculate the gradient direction histogram of the 16 sub-regions of the key point. Each sub-region has 8 direction components, and finally forms a 128-dimensional feature vector V SIFT ;

[0020] The HOG model is used to capture edge and shape features. HOG calculates the gradient of the image in the X and Y directions, calculates the gradient magnitude and gradient direction, divides the image into 8×8 regions, and calculates the gradient histogram in 9 directions in each region to form a 3780-dimensional HOG feature vector V. HOG ;

[0021] The LBP model is used to extract local texture features and form a low-dimensional feature vector. LBP calculates the LBP value by comparing the relationship between the pixel and its neighboring pixels, and calculates the LBP histogram to form a P+2-dimensional feature vector V LBP .

[0022] Furthermore, the step of integrating multiple text feature vectors through adaptive layer weights includes:

[0023] The text feature vector T TF-IDF 、T Word2Vec 、T BERT Mapped to the same dimension d, it is expressed as:

[0024] ;

[0025] Where W i represents a learnable parameter used to map different features to the same dimension; T i Represents the text feature vector; T' i Represents the text feature vector after dimension mapping;

[0026] Calculate the weight, weight w is expressed as:

[0027] ;

[0028] Among them, w is a three-dimensional vector, each feature has a normalized weight; W attn represents a learnable attention weight matrix;

[0029] After obtaining the normalized weight of each feature, the fused feature vector T is calculated by weighted fusion method. text , expressed as:

[0030] ;

[0031] Where w1, w2 and w3 are normalized weights; T' TF-IDF , T' Word2Vec , T' BERT They all represent the feature vectors after dimension mapping.

[0032] Furthermore, the steps of integrating multiple image feature vectors through the self-attention mechanism include:

[0033] The Query, Key, and Value in the self-attention mechanism come from V SIFT 、V HOG 、V LBP , the query vectors obtained are expressed as: Q SIFT , Q HOG , Q LBP, the obtained Key vector is expressed as: K SIFT , K HOG , K LBP ; The obtained Value vector is expressed as: V' SIFT 、V' HOG 、V' LBP ;

[0034] By calculating the dot product of Query and Key, we can get the similarity score between each pair of features, which is expressed as:

[0035] ;

[0036] Among them, i and j represent the index of the feature, d k is the size of the feature dimension; Q i represents the query vector after the i-th feature is extracted; K j represents the key vector of the jth feature, and T represents the transposition operation;

[0037] Then perform softmax operation on each score to obtain the normalized weight a ij , expressed as: a ij =softmax(Attention Score ij ), where a ij The weight represents the degree of attention of the i-th feature to the j-th feature;

[0038] The features are weighted and summed by calculating the attention weights, which can be expressed as:

[0039] ;

[0040] Among them, α SIFT , α HOG , α LBP Respectively represent the contribution of SIFT, HOG and LBP to the final fusion feature; the Value vectors of the three features are weighted and summed to obtain a weighted fusion image feature vector V image .

[0041] Furthermore, the integrated text features and image features are subjected to high power transformation respectively, and the transformed features are combined with the original features, including: respectively performing high power transformation on the text feature vector T text and image feature vector V image Perform element-wise power operation, expressed as:

[0042] and ;

[0043] Among them, ⊙k represents the element-by-element power operation, for T text and V image Each element of takes k power to obtain the features after high power transformation;

[0044] Text feature vector: T text =[t1,t2,t3], t1 is the word weight extracted by TF-IDF; t2 is the word vector component calculated by Word2Vec; t3 is the contextual semantic feature extracted by BERT; k=2, we can get: ;

[0045] This performs element-by-element power operations to obtain high-power feature expansion:

[0046] {T}^{'}_{power}=[T,{T}^{(2)},{T}^{(3)},...,{T}^{(k)}] ;

[0047] {V}^{'}_{power}=[V,{V}^{(2)},{V}^{(3)},...,{V}^{(k)}] ;

[0048] Among them, T (k) represents the kth text feature vector; V (k) represents the kth image feature vector;

[0049] The high-order features obtained after transformation are concatenated with the original features. The concatenation process is as follows:

[0050] ;

[0051] ;

[0052] Among them, T text and V image is the original feature, T' power and V' power is the power transformation characteristic;

[0053] After splicing, the enhanced feature T' is obtained text and V' image .

[0054] Furthermore, the steps of calculating the Pearson correlation coefficient, constructing a relationship matrix between text and image features, and weighting and fusing text and image features according to the relationship matrix to obtain the final fusion features include:

[0055] Construct the cross-modal relationship matrix R by calculating the Pearson correlation coefficient ρ (e i , e j), which is expressed as follows:

[0056] ;

[0057] Among them, cov(e i , e j ) is e i and e j The covariance of ei and σ ej It is e i and e j The standard deviation of ei and μ ej They are e i and e j The mean of

[0058] T text and V image Substitute e respectively i and e j In, use Construct a relationship fusion matrix;

[0059] Use the Pearson correlation coefficient to obtain the relationship fusion matrix between text features and image features , where m L The value is the number of text and image features; the dimension of the matrix R is m L ×m L , R TI Reflects the relationship between text features and image features; V image and T text Substitute e i and e j Repeat the above steps to get the relationship matrix R between image features and text features IT ;

[0060] After getting the relationship matrix R TI With R IT After that, the image features and text features are weighted respectively; among them, the weighted relationship matrix R from text to image is used TI To weight the image features, it is expressed as: , I fused Represents the image features generated by weighting the text-to-image relationship matrix; the weighted relationship matrix R from image to text IT To weight the text features, it is expressed as: , T fused Represents the text features generated by weighting the image-to-text weighted relationship matrix;

[0061] T' text and V' imageThey are the enhanced text features and image features respectively;

[0062] Finally, the text features and image features are fused together by weighted averaging, which is expressed as: F fused =αT fused +βI fused Among them, F fused For the final fusion feature vector, α and β are hyperparameters that control the contribution of text features and image features in the final fusion features, and their values ​​can be dynamically adjusted according to the importance of each modality.

[0063] Furthermore, in the step of inputting the fused feature vector into the MLP classifier, the MLP classifier maps the fused features to the corresponding agricultural product classification category to obtain the agricultural product classification result. The MLP classifier is designed based on a neural network and includes three layers: input layer, hidden layer, and output layer. The different layers of the MLP neural network are fully connected, where:

[0064] The input layer receives the fused feature vector F fused , and pass it to the first fully connected layer;

[0065] The first fully connected layer takes the input feature vector F fused Mapped to the hidden layer space, expressed as: h1=W1F fused +b1, where W1 is the weight matrix of the first layer, is the bias term, h1 represents the eigenvector F fused Features after linear transformation;

[0066] Then use the activation function ReLU to introduce nonlinearity: , set negative ReLU values ​​to 0 and keep positive values;

[0067] The output of the first layer is passed to the second fully connected layer for processing. The number of neurons in the hidden layer is h2, and the calculation formula is expressed as:

[0068] ;

[0069] Among them, W2 is the weight matrix of the second layer, is the bias term; the output of the second layer is nonlinearly transformed through the ReLU activation function: ;

[0070] The final output layer outputs the results of multiple transformations and obtains the original score of the output layer: , where W3 represents the weight matrix of the output layer; b3 represents the bias term of the output layer; z represents the original score of the output layer;

[0071] The score z of the output layer is converted into the probability of the category through the Softmax activation function, which is expressed as

[0072] ;

[0073] in, represents the score after exponential transformation, e is the base of natural logarithm, Z i The output layer original score is calculated by the output layer of the neural network; Represents the normalization factor used to calculate the sum of the scores of all categories after exponential transformation so that the sum of the probabilities of all categories is equal to 1, where , C represents the number of classification levels of agricultural products, Represents the predicted probability of the i-th category, and the sum of the probabilities of all categories is 1;

[0074] After obtaining the probability of each classification, it is necessary to calculate the cross entropy loss function between the category probability predicted by the model and the one-hot encoding form of the true category to measure the difference between the model output probability and the true label; the cross entropy loss function is:

[0075] ;

[0076] Among them, y i is the one-hot encoding of the true label, is the model’s predicted probability for the i-th class;

[0077] In training the MLP model, the cross-entropy loss function and the back-propagation algorithm are used to obtain the weight gradient corresponding to each loss, and then the weights are dynamically updated; in each training cycle, the prediction results are calculated through forward propagation, and the model error is evaluated based on the loss function; then, the gradient is calculated using the back-propagation algorithm, and the model parameters are updated through the optimization algorithm, and the weights are continuously optimized until the loss function converges.

[0078] Compared with the existing technology, the agricultural product classification method of the present invention that integrates dual-stream attention integration and cross-modal fusion has the following beneficial effects:

[0079] First, the present invention adopts dual-stream attention integration to perform multi-level feature extraction on text modality and image modality respectively; in the processing of text data, TF-IDF, Word2Vec and BERT models are used to extract three different levels of features; in the processing of image data, SIFT, HOG and LBP models are used to extract three different levels of features; then, the text and image features are integrated through the self-attention mechanism and adaptive layer weights to obtain a fusion feature vector of the two modalities; the classification method of the present invention adopts multi-level features to combine shallow and deep features, greatly improving the classification decision-making ability; the classification method of the present invention can also maintain high performance even when some information is missing or the data quality is low, obtain context-related features, and improve the stability of the system;

[0080] Second, the present invention uses a cross-modal relationship matrix to fuse text feature vectors and image feature vectors; by mapping the feature vectors into a high-dimensional space, enhanced text and image features are obtained, thereby improving the model's ability to understand multidimensional information and capturing the correlation between different modalities; the present invention combines multimodal data features to construct a relationship fusion matrix, so that it can adapt to complex environments with unbalanced categories and high data noise, thereby improving the generalization ability of the model; in the processing of large-scale complex data sets, the present invention can effectively calculate the importance of features, weaken the interference of redundant information, thereby improving the discriminative ability of feature representation, and optimizing the accuracy and stability of classification decisions. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention.

[0082] Figure 1 Schematic diagram of the flow of the agricultural product classification method integrating dual-stream attention integration and cross-modal fusion of the present invention;

[0083] Figure 2 This is a logical framework diagram of the agricultural product classification method that integrates dual-stream attention integration and cross-modal fusion in the present invention;

[0084] Figure 3 This is a sub-flowchart of the agricultural product classification method that integrates dual-stream attention integration and cross-modal fusion in the present invention;

[0085] Figure 4 This is another sub-flowchart of the agricultural product classification method that integrates dual-stream attention integration and cross-modal fusion of the present invention. DETAILED DESCRIPTION

[0086] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0087] The specific implementation of the present invention is described in detail below with reference to specific embodiments.

[0088] In one embodiment of the present invention, a method for agricultural product classification that integrates dual-stream attention integration and cross-modal fusion is provided. The method first requires preprocessing the text data and image data of the agricultural products;

[0089] In the process of data preprocessing, the agricultural product text data is subjected to stop words removal, missing values ​​processing, and capitalization unification, and then the text is segmented; the image data is unified in size and then normalized;

[0090] In one implementation of text data processing, the agricultural product text data D t Preprocessing usually involves removing stop words, handling missing values, and format conversion. To remove stop words from text, you can use the sklearn tool. The sklearn tool provides CountVectorizer and TfidfVectorizer, which can directly remove stop words during preprocessing and reduce noise in the text. Missing data may affect model training and prediction, so it needs to be properly filled or deleted. Numerical data should be filled with the mean, and categorical data should be filled with the mode. To ensure consistent data format, numeric fields can be converted to floating point numbers or integers, date fields can be converted to time format, and spaces can be removed and capitalization can be unified.

[0091] In one implementation of image data processing, the agricultural product image data D v Factors such as resolution and size can affect feature extraction during analysis, so image resizing, normalization, and data augmentation are typically required. The image size is standardized to 224×224 or 512×512 to ensure consistent input. The image pixel value range is [0,255]. This large data range affects gradient calculations, so pixel values ​​need to be scaled to [0,1] or [-1,1] to reduce numerical variance. Images are randomly flipped, rotated, and transformed to enhance model robustness.

[0092] Please refer to Figure 1 and Figure 2 The agricultural product classification method provided by the embodiment of the present invention that integrates dual-stream attention integration and cross-modal fusion includes the following steps:

[0093] S1. Using a text feature model to extract features from the preprocessed text data to obtain multiple text feature vectors;

[0094] S2. Using an image feature model to perform feature extraction on the preprocessed image data to obtain multiple image feature vectors;

[0095] Among them, in step S1, the text data D t The text feature models used for feature extraction include TF-IDF model, Word2Vec model and BERT model. Based on the above models, multiple feature vectors T can be obtained. TF-IDF 、T Word2Vec 、T BERT ;

[0096] For details, please refer to Figure 3 The step of extracting features from the pre-processed text data using the text feature model comprises the following steps:

[0097] S11. Use the TF-IDF model to extract the text feature vector T at the statistical information level TF-IDF ;

[0098] The TF-IDF model provided in the embodiments of the present disclosure is used to extract text features at the statistical information level. The specific calculation formula is as follows: ; Among them, the calculation formula of tf is expressed as: tf(t,d)=1 / n; the calculation formula of idf is expressed as: ; In the formula, tf represents term frequency; idf represents inverse document frequency; i represents the number of times word t appears in document d; n represents the total number of words in document d; N represents the total number of documents; df(t) represents the number of documents containing word t; the final feature vector is expressed as:

[0099] .

[0100] S12. Use the Word2Vec model to convert words into high-dimensional vectors. The continuous bag-of-words model (CBOW) is used to predict the central word through the context of the agricultural product dataset. After training the CBOW model, each word will be mapped to a vector of fixed dimension to capture the text feature vector T. Word2Vec ;

[0101] Specifically, the Word2Vec model is a distributed representation method based on neural networks, which can convert words into high-dimensional vectors, so that semantically similar words are closer in the vector space. This paper adopts the CBOW (Continuous Bag of Words) model to predict the central word through the context of the agricultural product dataset. The goal of the CBOW model is to calculate , the formula is: ;

[0102] in, is the target word W t The output word vector of is the average of the context word vectors, and v is the size of the vocabulary;

[0103] After training the CBOW model, each word is mapped to a vector of fixed dimension, thus capturing the feature vector T Word2Vec ;

[0104] S13. Use the BERT model to extract the deep semantic context information feature vector T BERT ;

[0105] Specifically, the BERT model is used to extract deep semantic context information features; the present invention uses the BERT model to obtain a high-dimensional vector representation of each word in the data set, reflecting the text features of deep context semantics;

[0106] For a text dataset, BERT first uses WordPiece Tokenizer to split the input text into word fragments (tokens) and adds special tags [CLS] and [SEP] at the beginning and end, that is, ;

[0107] Then, the word fragment is input into the Embedding layer (word embedding layer), and the final input vector is obtained through embedding;

[0108] Next, a multi-layer Transformer encoder is used to extract contextual features, primarily using self-attention, multi-head attention, and a feedforward neural network (FFN) to obtain a context-aware representation of the text.

[0109] Finally, after the Transformer layer, BERT generates a vector representation T for each word BERT ;

[0110] In step S2, the image data D v The feature vector V is extracted using models such as Scale Invariant Feature Transform (SIFT), Histogram of Oriented Gradients (HOG), and Local Binary Pattern (LBP). SIFT 、V HOG 、V LBP ;

[0111] For details, please refer to Figure 4 The step of extracting features from the pre-processed image data using the image feature model comprises the following steps:

[0112] S21. Use the SIFT model to construct images of different scales through the Gaussian pyramid; SIFT calculates the local extreme values ​​of the image through the differential Gaussian pyramid DOG, SIFT assigns the direction of the key points, calculates the gradient direction around each key point, and uses the gradient magnitude and gradient direction to calculate the gradient direction histogram of the 16 sub-regions of the key point. Each sub-region has 8 direction components, and finally forms a 128-dimensional feature vector V SIFT ;

[0113] Exemplarily, in step S21 of the present invention, SIFT constructs images of different scales through a Gaussian pyramid:

[0114] ;

[0115] Where I(x,y) is the image data D v The input image in G(x, y, σ) is a Gaussian function; L(x, y) represents the smoothed image of the input image at different scales, L(x, y, σ) represents the smoothed version of the image at scale σ, and L(x, y, σ) is the result of convolution of the input image I(x, y) with the Gaussian function G(x, y, σ);

[0116] Subsequently, SIFT calculates the local extreme values ​​of the image through the Difference of Gaussian Pyramid (DOG): D(x,y,σ)=L(x,y,kσ)-L(x,y,σ), where k is the scale change factor;

[0117] Next, SIFT assigns the key point directions and calculates the gradient direction around each key point, as shown in the following formula:

[0118] ;

[0119] ;

[0120] Where m(x,y) is the gradient magnitude, θ(x,y) is the gradient direction;

[0121] Finally, the gradient magnitude m(x,y) and gradient direction θ(x,y) are used to calculate the gradient direction histogram of the 16 sub-regions of the key point:

[0122] ;

[0123] Among them, i represents the direction index of the histogram; h i Represents the cumulative gradient amplitude of the i-th direction in the direction histogram; θ i represents the i-th angle in the direction histogram; δ represents an indicator function, which is a Kronecker Delta function, and the indicator function is used to determine the gradient direction of a pixel point;

[0124] Each sub-region has 8 directional components, which ultimately form a 128-dimensional feature vector: V SIFT= [h1,h2,...,h 128 ];

[0125] S22. Use the HOG model to capture edge and shape features; HOG calculates the gradient of the image in the X and Y directions, calculates the gradient magnitude and gradient direction, divides the image into regions, and calculates the gradient histogram in 9 directions for each region to form a 3780-dimensional HOG feature vector V HOG ;

[0126] In one implementation, HOG is mainly used to capture edge and shape features for agricultural product object detection and classification;

[0127] HOG calculates the gradient of the image in the X and Y directions, as shown in the following formula:

[0128] ;

[0129] ;

[0130] Among them, G x The gradient is calculated in the X direction, that is, the rate of change in the horizontal direction, reflecting the edge information of the image in the left and right directions; G y The gradient is calculated in the Y direction, that is, the rate of change in the vertical direction, reflecting the edge information of the image in the up and down directions;

[0131] Similarly, calculate the gradient magnitude m(x, y) and gradient direction θ(x, y), divide the image into 8×8 regions, and count the gradient histograms of 9 directions in each region, which are expressed as follows:

[0132] ;

[0133] Finally, a 3780-dimensional HOG feature vector is formed: V HOG =[h1,h2,...,h 3780 ];

[0134] S23, using the LBP model to extract local texture features and form a low-dimensional feature vector; wherein, LBP calculates the LBP value by comparing the relationship between the pixel point and its neighboring pixels, and calculates the LBP histogram to form a P+2-dimensional feature vector V LBP .

[0135] Specifically, in one implementation of the present invention, the LBP model is used to extract local texture features to form a low-dimensional feature vector;

[0136] LBP calculates the LBP value by comparing the relationship between the pixel and its neighboring pixels:

[0137] ;

[0138] in, , P is the number of points; LBP (x c ,y c ) means (x c ,y c ) is the LBP value calculated for the center pixel, I p Represents the grayscale value of the neighborhood pixel, I c Represents the grayscale value of the center pixel;

[0139] The LBP histogram is calculated using the following formula:

[0140] ;

[0141] Finally, a P+2-dimensional feature vector is formed (usually the low dimension is 10): V LBP= [h1,h2,...,h 10 ].

[0142] Please continue to refer to Figure 1 and Figure 2 The agricultural product classification method integrating dual-stream attention integration and cross-modal fusion provided by the present invention further includes the following steps:

[0143] S3. Integrate multiple text feature vectors through adaptive layer weights, integrate multiple image feature vectors through the self-attention mechanism, and perform weighted summation to obtain the integrated text features and image features;

[0144] S4, performing high-power transformation on the integrated text features and image features respectively, and concatenating the transformed features with the original features to obtain enhanced text features and enhanced image features;

[0145] S5. Calculate the Pearson correlation coefficient to construct a relationship matrix between text and image features. Then, weight the text and image features according to the relationship matrix to obtain the final fused features.

[0146] S6. Input the fused feature vector into the MLP classifier. The MLP classifier maps the fused feature vector to the corresponding agricultural product classification category to obtain the agricultural product classification result.

[0147] Among them, in step S3 of the present invention, the specific process is divided into two steps: text feature integration and image feature integration;

[0148] First, for complex multi-layer feature fusion, the present invention uses adaptive layer weights (Layer Attention) for feature integration, dynamically assigns weights according to the importance of different layers, and finally performs weighted fusion to obtain more representative features T text ;

[0149] Next, the weight value of each feature is obtained by concatenating the multiple image feature vectors extracted in step S2 and capturing the correlation between different features through the self-attention mechanism;

[0150] Finally, a fused feature vector V is obtained using the weighted summation method. image ;

[0151] Specifically, the steps of integrating multiple text feature vectors through adaptive layer weights include:

[0152] The text feature vector T TF-IDF 、T Word2Vec 、T BERT Mapped to the same dimension d, it is expressed as:

[0153] ;

[0154] Where W i represents a learnable parameter used to map different features to the same dimension; T i Represents the text feature vector; T' i Represents the text feature vector after dimension mapping;

[0155] Calculate the weight, weight w is expressed as:

[0156] ;

[0157] Among them, w is a three-dimensional vector, each feature has a normalized weight; W attn represents a learnable attention weight matrix;

[0158] After obtaining the normalized weight of each feature, the fused feature vector T is calculated by weighted fusion method. text , expressed as:

[0159] ;

[0160] Where w1, w2 and w3 are normalized weights; T' TF-IDF , T' Word2Vec , T' BERT Both represent the feature vectors after dimension mapping;

[0161] The fusion feature T of the embodiment of the present inventiontext It retains the advantages of the three models TF-IDF, Word2Vec, and BERT:

[0162] Term frequency-inverse document frequency (TF-IDF) reflects the sparse information characteristics of the text and can capture keyword information;

[0163] The word vector-based feature extraction model (Word2Vec) provides dense distributed word vector representation;

[0164] The BERT model has the ability to understand context;

[0165] Therefore, T text It combines the stability of traditional methods with the ability of deep semantic modeling, and can achieve higher classification accuracy and generalization ability in agricultural product classification tasks.

[0166] Furthermore, the steps of integrating multiple image feature vectors through the self-attention mechanism include:

[0167] The Query, Key, and Value in the self-attention mechanism come from V SIFT 、V HOG 、V LBP , the query vectors obtained are expressed as: Q SIFT , Q HOG , Q LBP , the obtained Key vector is expressed as: K SIFT , K HOG , K LBP ; The obtained Value vector is expressed as: V' SIFT 、V' HOG 、V' LBP ;

[0168] By calculating the dot product of Query and Key, we can get the similarity score between each pair of features, which is expressed as:

[0169] ;

[0170] Among them, i and j represent the index of the feature, d k is the size of the feature dimension; Q i represents the query vector after the i-th feature is extracted; K j represents the key vector of the jth feature, and T represents the transposition operation;

[0171] Then perform softmax operation on each score to obtain the normalized weight a ij , expressed as: a ij =softmax(Attention Score ij), where a ij The weight represents the degree of attention of the i-th feature to the j-th feature, reflecting the importance of different features;

[0172] The features are weighted and summed by calculating the attention weights, which can be expressed as:

[0173] ;

[0174] Among them, α SIFT , α HOG , α LBP Respectively represent the contribution of SIFT, HOG and LBP to the final fusion feature; the Value vectors of the three features are weighted and summed to obtain a weighted fusion image feature vector V image .

[0175] In step S4 of the present invention, in order to enhance the feature expression capability, it is necessary to text and V image Perform high power transformation and then concatenate it with the original features to obtain the enhanced feature vector 、 .

[0176] Specifically, in step S4 of the present invention, the steps of performing high-power transformation on the integrated text features and image features and splicing the transformed features with the original features include:

[0177] For the text feature vector T text and image feature vector V image Perform element-wise power operation, expressed as:

[0178] and ;

[0179] Among them, ⊙k represents the element-by-element power operation, for T text and V image Each element of takes k power to obtain the features after high power transformation;

[0180] Text feature vector: T text =[t1,t2,t3], t1 is the word weight extracted by TF-IDF; t2 is the word vector component calculated by Word2Vec; t3 is the contextual semantic feature extracted by BERT; k=2, we can get: ;

[0181] This performs element-by-element power operations to obtain high-power feature expansion:

[0182] {T}^{'}_{power}=[T,{T}^{(2)},{T}^{(3)},...,{T}^{(k)}] ;

[0183] {V}^{'}_{power}=[V,{V}^{(2)},{V}^{(3)},...,{V}^{(k)}] ;

[0184] Among them, T (k) represents the kth text feature vector; V (k) represents the kth image feature vector;

[0185] The high-order features obtained after transformation are concatenated with the original features. The concatenation process is as follows:

[0186] ;

[0187] ;

[0188] Among them, T text and V image is the original feature, T' power and V' power is the power transformation characteristic;

[0189] After splicing, the enhanced feature T' is obtained text and V' image , used for subsequent multimodal tasks.

[0190] In step S5 of the present invention, a cross-modal relationship matrix of text features and image features is constructed to deeply analyze the correlation between the two modalities and quantify the weight distribution of each modal feature. Based on this weight information, the enhanced text features and image features are weightedly fused to ultimately obtain a fused feature vector, thereby improving the expressiveness and decision-making performance of multimodal information.

[0191] Specifically, in step S5 of the present invention, the steps of calculating the Pearson correlation coefficient, constructing a relationship matrix between text and image features, and weighting and fusing the text and image features according to the relationship matrix to obtain the final fused features include:

[0192] Construct the cross-modal relationship matrix R by calculating the Pearson correlation coefficient ρ (e i , e j ), which is expressed as follows:

[0193] ;

[0194] Among them, cov(e i , e j ) is ei and e j The covariance of ei and σ ej It is e i and e j The standard deviation of ei and μ ej They are e i and e j The mean of and Substitute e respectively i and e j In, use Construct a relationship fusion matrix;

[0195] Furthermore, in the embodiment of the present invention, the Pearson correlation coefficient is used to obtain the relationship fusion matrix between text features and image features: , where m L The value is the number of text and image features; the dimension of the matrix is ​​m L ×m L , R TI Reflects the relationship between text features and image features;

[0196] Furthermore, in the embodiment of the present invention, V image and T text Substitute e i and e j Repeat the above steps to get the relationship matrix R between image features and text features IT ;

[0197] After getting the relationship matrix R TI With R IT After that, the image features and text features are weighted respectively; among them, the weighted relationship matrix R from text to image is used TI To weight the image features, it is expressed as: , I fused Represents the image features generated by weighting the text-to-image relationship matrix; the weighted relationship matrix R from image to text IT To weight the text features, it is expressed as: , T fused Represents the text features generated by weighting the image-to-text weighted relationship matrix;

[0198] T' text and V' image They are the enhanced text features and image features respectively;

[0199] Finally, the text features and image features are fused together by weighted averaging, which is expressed as: F fused =αT fused+βI fused Among them, F fused For the final fusion feature vector, α and β are hyperparameters that control the contribution of text features and image features in the final fusion features, and their values ​​can be dynamically adjusted according to the importance of each modality.

[0200] In step S6 of the present invention, a multi-layer perceptron (MLP) is used to classify the agricultural product dataset;

[0201] In this invention, the agricultural product classification system is divided into four categories: livestock and poultry, aquatic products, fruits, vegetables, grains and oils, specialty economic crops and others;

[0202] Among them, livestock and poultry aquatic products mainly include livestock and poultry meat, poultry eggs, dairy products, and aquatic products;

[0203] Fruits, vegetables, grains and oils have three subcategories: fruits, vegetables, and grains and oils;

[0204] Specialty economic crops are divided into tea, Chinese herbal medicine, flowers and plants, and nuts;

[0205] Finally, there is the "Other" category, which includes some processed agricultural products and special varieties. This classification method can ensure the comprehensiveness of the classification while improving management and identification efficiency.

[0206] Specifically, in step S6 of the present invention, in the step of inputting the fused feature vector into the MLP classifier, and the MLP classifier mapping the fused feature to the corresponding agricultural product classification category to obtain the agricultural product classification result, the MLP classifier is designed based on a neural network and includes three layers: an input layer, a hidden layer, and an output layer; the different layers of the MLP neural network are fully connected, wherein:

[0207] The input layer receives the fused feature vector F fused , and pass it to the first fully connected layer;

[0208] The first fully connected layer takes the input feature vector F fused Mapped to the hidden layer space, expressed as: h1=W1F fused +b1, where W1 is the weight matrix of the first layer, is the bias term, h1 represents the eigenvector F fused Features after linear transformation;

[0209] Then use the activation function ReLU to introduce nonlinearity: , set negative ReLU values ​​to 0 and keep positive values;

[0210] The output of the first layer is passed to the second fully connected layer for processing. The number of neurons in the hidden layer is h2, and the calculation formula is expressed as:

[0211] ;

[0212] Among them, W2 is the weight matrix of the second layer, is the bias term. Similarly, the output of the second layer is nonlinearly transformed by the ReLU activation function: ;

[0213] The final output layer outputs the results of multiple transformations and obtains the original score of the output layer: ;

[0214] Assume that agricultural products have C classification levels, where W3 represents the weight matrix of the output layer; b3 represents the bias term of the output layer; z represents the original score of the output layer; and the Softmax activation function is used to convert the score z of the output layer into the probability of the category, which is expressed as:

[0215] ;

[0216] in, represents the score after exponential transformation, e is the base of natural logarithm, Z i The output layer original score is calculated by the output layer of the neural network; Represents the normalization factor used to calculate the sum of the scores of all categories after exponential transformation so that the sum of the probabilities of all categories is equal to 1, where , C represents the number of classification levels of agricultural products, Represents the predicted probability of the i-th category, and the sum of the probabilities of all categories is 1;

[0217] After obtaining the probability of each classification, it is necessary to calculate the cross entropy loss function between the category probability predicted by the model and the one-hot encoding form of the true category to measure the difference between the model output probability and the true label;

[0218] Among them, the cross entropy loss function is:

[0219] ;

[0220] Among them, y i is the one-hot encoding of the true label, is the model’s predicted probability for the i-th class;

[0221] Next, the MLP model is further trained, using the cross-entropy loss function and the backpropagation algorithm to obtain the weight gradients corresponding to each loss, and then dynamically updating the weights. During the model training process, the dataset is iterated multiple times. In each training cycle, the prediction results are calculated through forward propagation, and the model error is evaluated based on the loss function. Subsequently, the gradients are calculated using the backpropagation algorithm, and the model parameters are updated through the optimization algorithm. The weights are continuously optimized until the loss function converges, thereby improving the classification performance of the model.

[0222] After the training is completed, the test set is used to further evaluate the generalization ability of the model; specifically, fruits, vegetables, grains and oils are selected from the agricultural product classification system of the present invention for accuracy verification. First, for fruits, vegetables, grains and oils, relatively comprehensive text data and image data on the market are widely collected to ensure the representativeness and diversity of the data. Secondly, 20% of the agricultural product data is randomly divided from the fused feature vector as an independent test set to ensure that the sources of the training set and the test set are consistent. Because the agricultural product data set is large, the present invention uses a direct testing method to input the test set data into the MLP model to calculate parameter values ​​such as accuracy (AC), precision (PE), recall rate (RE), and F1-score. The larger the values ​​of these evaluation indicators, the better the classification performance. The specific definitions are as follows:

[0223] ;

[0224] Among them, TP (True Positive): the number of samples correctly predicted to be of this category; FP (False Positive): the number of samples that do not actually belong to this category but are predicted to be of this category; FN (False Negative): the number of samples that actually belong to this category but are incorrectly predicted to be of other categories; TN (True Negative): the number of samples that do not belong to this category and are not predicted to be of this category.

[0225] Multiple rounds of testing were conducted on fruits, vegetables, grains, and oils, and the resulting test parameters were compared and analyzed with the original method. The experimental results demonstrated that the proposed method significantly improved key evaluation metrics such as accuracy, precision, recall, and F1-score. The same test was then conducted on three other agricultural product categories, and the results were summarized. These results demonstrate that the dual-stream attention mechanism combined with multimodal feature fusion significantly improves the accuracy and robustness of agricultural product classification, demonstrating superior performance and improved generalization when dealing with complex, large-scale datasets.

[0226] In summary, the present invention adopts dual-stream attention integration to perform multi-level feature extraction on text modality and image modality respectively;

[0227] In the processing of text data, TF-IDF, Word2Vec and BERT models are used to extract three different levels of features;

[0228] In image data processing, SIFT, HOG and LBP models are used to extract features at three different levels;

[0229] Then, the text and image features are integrated through the self-attention mechanism and adaptive layer weights to obtain the fused feature vector of the two modalities;

[0230] The classification method of the present invention uses multi-level features to combine shallow and deep features, greatly improving the classification decision-making ability;

[0231] The classification method of the present invention can also maintain high performance even when some information is missing or the data quality is low, obtain context-related features, and improve the stability of the system;

[0232] In addition, the present invention uses a cross-modal relationship matrix to fuse text feature vectors and image feature vectors. By mapping the feature vectors into a high-dimensional space, enhanced text and image features are obtained, improving the model's ability to understand multidimensional information and capturing the correlation between different modalities. The present invention combines multimodal data features to construct a relationship fusion matrix, enabling it to adapt to complex environments with unbalanced categories and high data noise, thereby improving the model's generalization ability.

[0233] In the process of processing large-scale complex data sets, the present invention can effectively calculate the importance of features, reduce the interference of redundant information, thereby improving the discriminative ability of feature representation and optimizing the accuracy and stability of classification decisions.

[0234] The above solutions are only examples of preferred embodiments, but are not limited thereto. When implementing the present invention, appropriate replacements and / or modifications can be made according to user needs.

[0235] The number of devices and processing scales described herein are intended to simplify the description of the present invention. Applications, modifications, and variations of the present invention will be readily apparent to those skilled in the art.

[0236] Although the embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the specification and exemplary embodiments. They can be applied to a variety of fields suitable for the present invention. Further modifications will be readily apparent to those skilled in the art. Therefore, the present invention is not limited to the specific details and illustrations shown and described herein without departing from the general concept defined by the claims and their equivalents.

Claims

1. The agricultural product classification method integrating dual-stream attention integration and cross-modal fusion is characterized by: The following steps are involved: Using the text feature model to extract features from the preprocessed text data to obtain multiple text feature vectors; The image feature model is used to extract features from the preprocessed image data to obtain multiple image feature vectors; Multiple text feature vectors are integrated through adaptive layer weights, and multiple image feature vectors are integrated through the self-attention mechanism. The weighted summation is used to obtain the integrated text features and image features. Among them, the text feature vector is mapped to the same dimension, the weight is calculated, and after obtaining the normalized weight of each feature, the fused feature vector is calculated by the weighted fusion method; the Query, Key and Value in the self-attention mechanism come from V SIFT 、V HOG 、V LBP , the query vectors obtained are expressed as: Q SIFT , Q HOG , Q LBP , the obtained Key vector is expressed as: K SIFT , K HOG , K LBP ; The obtained Value vector is expressed as: V' SIFT 、V' HOG 、V' LBP By calculating the dot product of the query and the key, we get the similarity score between each pair of features, and then perform a softmax operation on each score to get the normalized weight. We use the calculated attention weights to perform weighted summation on the features, and then weighted sum the value vectors of the three features to get a weighted fusion image feature vector. Perform high-power transformation on the integrated text features and image features respectively, and concatenate the transformed features with the original features to obtain enhanced text features and enhanced image features; By calculating the Pearson correlation coefficient, the relationship matrix between text and image features is constructed, and the text and image features are weighted and fused according to the relationship matrix to obtain the final fusion feature; The fused feature vector is input into the MLP classifier, and the MLP classifier maps the fused features to the corresponding agricultural product classification category to obtain the agricultural product classification result.

2. The agricultural product classification method integrating dual-stream attention integration and cross-modal fusion according to claim 1 is characterized in that: The steps of extracting features from preprocessed text data using a text feature model include: Use the TF-IDF model to extract text feature vectors at the statistical information level ; The Word2Vec model is used to convert words into high-dimensional vectors. The continuous bag-of-words model (CBOW) is used to predict the central word through the context of the agricultural product dataset. After training the CBOW model, each word will be mapped to a vector of fixed dimension to capture the text feature vector. ; Using the BERT model to extract deep semantic context information feature vectors .

3. The agricultural product classification method integrating dual-stream attention integration and cross-modal fusion according to claim 2 is characterized in that: The steps of extracting features from the preprocessed image data using the image feature model include: The SIFT model is used to construct images of different scales through the Gaussian pyramid. SIFT calculates the local extreme values ​​of the image through the differential Gaussian pyramid DOG. SIFT assigns the direction of the key points, calculates the gradient direction around each key point, and uses the gradient magnitude and gradient direction to calculate the gradient direction histogram of the 16 sub-regions of the key point. Each sub-region has 8 direction components, and finally forms a 128-dimensional feature vector V SIFT ; The HOG model is used to capture edge and shape features. HOG calculates the gradient of the image in the X and Y directions, calculates the gradient magnitude and gradient direction, divides the image into 8×8 regions, and calculates the gradient histogram in 9 directions in each region to form a 3780-dimensional HOG feature vector V. HOG ; The LBP model is used to extract local texture features and form a low-dimensional feature vector. LBP calculates the LBP value by comparing the relationship between the pixel and its neighboring pixels, and calculates the LBP histogram to form a P+2-dimensional feature vector V LBP .

4. The agricultural product classification method integrating dual-stream attention integration and cross-modal fusion according to claim 3 is characterized in that: In the step of mapping the text feature vector to the same dimension, the text feature vector 、 、 Mapped to the same dimension d, it is expressed as: ； Where, Represents learnable parameters used to map different features to the same dimension; Represents text feature vector; Represents the text feature vector after dimension mapping; In the step of calculating weights, the weight w is expressed as: ； Where w is a three-dimensional vector and each feature has a normalized weight; represents a learnable attention weight matrix; In the step of calculating the fused feature vector by the weighted fusion method, the fused feature vector T text , expressed as: ； Where w1, w2 and w3 are normalized weights; T' TF-IDF , T' Word2Vec , T' BERT They all represent the feature vectors after dimension mapping.

5. The agricultural product classification method integrating dual-stream attention integration and cross-modal fusion according to claim 4 is characterized in that: In the step of obtaining the similarity score between each pair of features by calculating the dot product of the query and the key, the similarity score is expressed as: ； Among them, i and j represent the index of the feature, d k is the size of the feature dimension; Q i represents the query vector after the i-th feature is extracted; K j represents the key vector of the jth feature, and T represents the transposition operation; In the step of obtaining the normalized weight, the normalized weight a ij Expressed as: a ij =softmax(AttentionScore ij ), where a ij The weight represents the degree of attention of the i-th feature to the j-th feature; The weighted summation of the features is performed by calculating the attention weights, which is expressed as: ； in, 、 、 Respectively represent the contribution of the three image features SIFT, HOG and LBP in the final fusion feature, and the Value vectors of the three features are weighted and summed to obtain a weighted fusion image feature vector .

6. The agricultural product classification method integrating dual-stream attention integration and cross-modal fusion according to claim 5 is characterized in that: The steps of performing high-power transformation on the integrated text features and image features and concatenating the transformed features with the original features include: For the text feature vector T text and image feature vector V image Perform element-wise power operation, expressed as: and ; Among them, ⊙k represents the element-by-element power operation, for T text and V image Each element of takes k power to obtain the features after high power transformation; Text feature vector: , is the word weight extracted by TF-IDF; It is the word vector component calculated by Word2Vec; is the contextual semantic feature extracted by BERT, k=2, we can get: ; This performs element-by-element power operations to obtain high-power feature expansion: ； ； Among them, T (k) represents the kth text feature vector; V (k) represents the kth image feature vector; The high-order features obtained after transformation are concatenated with the original features. The concatenation process is as follows: ； Among them, T text and V image is the original feature, T' power and V' power is the power transformation characteristic; After splicing, the enhanced feature T' is obtained text and V' image .

7. The agricultural product classification method integrating dual-stream attention integration and cross-modal fusion according to claim 6 is characterized in that: The steps of calculating the Pearson correlation coefficient, constructing the relationship matrix between text and image features, and weighting and fusing text and image features according to the relationship matrix to obtain the final fusion features include: Construct the cross-modal relationship matrix R by calculating the Pearson correlation coefficient , which is expressed as follows: ； in, yes and The covariance of and yes and The standard deviation of and They are and The mean of T text and V image Substitute separately and In, use Construct a relationship fusion matrix; Use the Pearson correlation coefficient to obtain the relationship fusion matrix between text features and image features , where m L The value is the number of text and image features; the dimension of the matrix is ​​m L ×m L , R TI Reflects the relationship between text features and image features; V image and T text Substitute e i and e j Repeat the above steps to get the relationship matrix R between image features and text features IT ; After getting the relationship matrix R TI With R IT After that, the image features and text features are weighted respectively; among them, the weighted relationship matrix R from text to image is used TI To weight the image features, it is expressed as: , Represents the image features generated by weighting the text-to-image relationship matrix; the weighted relationship matrix from image to text To weight the text features, it is expressed as: , Represents the text features generated by weighting the image-to-text weighted relationship matrix; and They are the enhanced text features and image features respectively; Finally, the text features and image features are fused together by weighted averaging, which is expressed as: ;in, For the final fusion feature vector, α and β are hyperparameters that control the contribution of text features and image features in the final fusion feature.

8. The agricultural product classification method integrating dual-stream attention integration and cross-modal fusion according to claim 7 is characterized in that: In the step of inputting the fused feature vector into the MLP classifier, the MLP classifier maps the fused features to the corresponding agricultural product classification category to obtain the agricultural product classification result. The MLP classifier is designed based on a neural network and includes three layers: input layer, hidden layer, and output layer. The different layers of the MLP neural network are fully connected, where: The input layer receives the fused feature vector F fused , and pass it to the first fully connected layer; The first fully connected layer takes the input feature vector F fused Mapped to the hidden layer space, expressed as: h1=W1F fused +b1, where W1 is the weight matrix of the first layer, is the bias term, h1 represents the eigenvector F fused Features after linear transformation; Then use the activation function ReLU to introduce nonlinearity: , set negative ReLU values ​​to 0 and keep positive values; The output of the first layer is passed to the second fully connected layer for processing. The number of neurons in the hidden layer is h2, and the calculation formula is expressed as: ; Among them, W2 is the weight matrix of the second layer, is the bias term; the output of the second layer is nonlinearly transformed through the ReLU activation function: ; The final output layer outputs the results of multiple transformations and obtains the original score of the output layer: , where W3 represents the weight matrix of the output layer; b3 represents the bias term of the output layer; z represents the original score of the output layer; The score z of the output layer is converted into the probability of the category through the Softmax activation function, which is expressed as: ； in, represents the score after exponential transformation, e is the base of natural logarithm, The output layer original score is calculated by the output layer of the neural network; Represents the normalization factor used to calculate the sum of the scores of all categories after exponential transformation so that the sum of the probabilities of all categories is equal to 1, where , C represents the number of classification levels of agricultural products, Represents the predicted probability of the i-th category, and the sum of the probabilities of all categories is 1; After obtaining the probability of each classification, it is necessary to calculate the cross entropy loss function between the category probability predicted by the model and the one-hot encoding form of the true category to measure the difference between the model output probability and the true label; the cross entropy loss function is: ; in, is the one-hot encoding of the true label, is the model’s predicted probability for the i-th class; In training the MLP model, the cross-entropy loss function and the back-propagation algorithm are used to obtain the weight gradient corresponding to each loss, and then the weights are dynamically updated; in each training cycle, the prediction results are calculated through forward propagation, and the model error is evaluated based on the loss function; then, the gradient is calculated using the back-propagation algorithm, and the model parameters are updated through the optimization algorithm, and the weights are continuously optimized until the loss function converges.

Images