Agricultural product classification method fusing double-flow attention integration and cross-modal fusion

Abstract

The invention is suitable for the technical field of artificial intelligence, and particularly provides an agricultural product classification method fusing double-flow attention integration and cross-modal fusion, which comprises the following steps: integrating a plurality of text feature vectors through adaptive layer weights, integrating a plurality of image feature vectors through a self-attention mechanism, and obtaining a plurality of image feature vectors; carrying out weighted summation to obtain integrated text features and image features; respectively carrying out high-power transformation on the integrated text features and image features, and splicing the transformed features with original features to obtain enhanced text features and enhanced image features; constructing a relation matrix between the text and the image features by calculating a Pearson's correlation coefficient, and performing weighted fusion on the text and the image features according to the relation matrix to obtain a final fusion feature; and inputting the fused feature vector into an MLP classifier. According to the invention, the modeling and learning ability of the model to a complex mode is improved; and the accuracy and generalization ability of agricultural product classification are further improved by using cross-modal fusion.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to an agricultural product classification method integrating dual-stream attention integration and cross-modal fusion. Background Art

[0002] At present, in the research of agricultural product classification, existing methods usually use computer vision technology to extract agricultural product features and realize classification through image information. However, agricultural product related images have some significant characteristics, such as the lighting and shooting environment have a great influence on image features; these characteristics make it difficult for image data to accurately reflect agricultural product information, further increasing the difficulty of classification.

[0003] Therefore, analysis using only the image module lacks robustness.

[0004] At present, the relevant technologies of image-based agricultural product classification methods have the following shortcomings: First, agricultural products are classified only through a single visual modality, and factors such as lighting, shadow, and shooting angle have a great influence on visual features. External factors will cause errors in the classification results, which makes it difficult to meet the needs of modern agricultural product classification; Second, the features extracted based on a single model usually have single features and lack diversity. They cannot fully reflect all relevant information of agricultural products and are prone to information loss; Third, the features extracted by existing technologies are mostly primitive features, which make it difficult to capture higher-level semantic information of agricultural products; and the interactive relationship between different features is not utilized, and the correlation of multimodal information cannot be effectively modeled. Summary of the invention

[0005] The purpose of the embodiments of the present invention is to provide an agricultural product classification method that integrates dual-stream attention integration and cross-modal fusion, aiming to solve the technical problems raised in the above-mentioned background technology.

[0006] To achieve the above object, the present invention provides the following technical solutions.

[0007] The agricultural product classification method integrating dual-stream attention integration and cross-modal fusion includes the following steps: Using the text feature model to extract features from the preprocessed text data, a plurality of text feature vectors are obtained; Using the image feature model to extract features from the preprocessed image data, a plurality of image feature vectors are obtained; Multiple text feature vectors are integrated through adaptive layer weights, and multiple image feature vectors are integrated through the self-attention mechanism. The integrated text features and image features are obtained by weighted summation. Perform high-power transformation on the integrated text features and image features respectively, and concatenate the transformed features with the original features to obtain enhanced text features and enhanced image features; By calculating the Pearson correlation coefficient, the relationship matrix between text and image features is constructed, and the text and image features are weighted and fused according to the relationship matrix to obtain the final fusion feature; The fused feature vector is input into the MLP classifier, and the MLP classifier maps the fused features to the corresponding agricultural product classification category to obtain the agricultural product classification result.

[0008] Furthermore, the step of extracting features from the preprocessed text data using the text feature model includes: The TF-IDF model is used to extract the text feature vector T at the statistical information level. TF-IDF ; The Word2Vec model is used to convert words into high-dimensional vectors. The continuous bag of words model (CBOW) is used to predict the central word through the context of the agricultural product dataset. After the CBOW model is trained, each word will be mapped to a vector of fixed dimension to capture the text feature vector T. Word2Vec ; Use the BERT model to extract the deep semantic context information feature vector T BERT .

[0009] Furthermore, the step of extracting features from the preprocessed image data using the image feature model includes: The SIFT model is used to construct images of different scales through the Gaussian pyramid. SIFT calculates the local extreme value of the image through the differential Gaussian pyramid DOG. SIFT assigns the direction of the key points, calculates the gradient direction around each key point, and uses the gradient amplitude and gradient direction to calculate the gradient direction histogram of the 16 sub-regions of the key point. Each sub-region has 8 direction components, and finally forms a 128-dimensional feature vector V. SIFT ; The HOG model is used to capture edge and shape features. HOG calculates the gradient of the image in the X and Y directions, calculates the gradient amplitude and gradient direction, divides the image into 8×8 regions, and calculates the gradient histogram in 9 directions in each region to form a 3780-dimensional HOG feature vector V HOG ; The LBP model is used to extract local texture features to form a low-dimensional feature vector. LBP compares the relationship between a pixel and its neighboring pixels, calculates the LBP value, and calculates the LBP histogram to form a P+2-dimensional feature vector V LBP .

[0010] Furthermore, the step of integrating multiple text feature vectors through adaptive layer weights includes: The text feature vector T TF-IDF 、T Word2Vec 、TBERT Mapped to the same dimension d, expressed as: ; Where W i represents a learnable parameter used to map different features to the same dimension; T i Represents the text feature vector; T' i Represents the text feature vector after dimension mapping; Calculate the weight, the weight w is expressed as: ; Among them, w is a three-dimensional vector, each feature has a normalized weight; W attn represents a learnable attention weight matrix; After obtaining the normalized weight of each feature, the fused feature vector T is calculated by the weighted fusion method. text , expressed as: ; Where w1, w2 and w3 all represent normalized weights; T' TF-IDF , T' Word2Vec , T' BERT They all represent the feature vectors after dimension mapping.

[0011] Furthermore, the step of integrating multiple image feature vectors through the self-attention mechanism includes: The Query, Key, and Value in the self-attention mechanism come from V SIFT 、V HOG 、V LBP , the obtained Query vectors are expressed as: Q SIFT , Q HOG , Q LBP The obtained Key vector is expressed as: K SIFT , K HOG , K LBP ; The obtained Value vector is expressed as: V' SIFT 、V' HOG 、V' LBP ; By calculating the dot product of Query and Key, we get the similarity score between each pair of features, expressed as: ; Among them, i and j represent the index of the feature, d k is the size of the feature dimension; Q i represents the Query vector after the i-th feature is extracted; K j represents the Key vector of the jth feature, and T represents the transposition operation; Then perform a softmax operation on each score to obtain the normalized weight a ij , expressed as: a ij =softmax(Attention Score ij ), where a ij The weight represents the degree of attention of the i-th feature to the j-th feature; The features are weighted and summed by calculating the attention weights, expressed as: ; Among them, α SIFT , α HOG , α LBP Respectively represent the contribution of SIFT, HOG and LBP to the final fusion feature; the Value vectors of the three features are weighted and summed to obtain a weighted fusion image feature vector V image .

[0012] Furthermore, the integrated text features and image features are respectively subjected to high-power transformation, and the transformed features are concatenated with the original features, including: respectively performing high-power transformation on the text feature vector T text and image feature vector V image Perform element-by-element power operations, expressed as: and ; Among them, ⊙k represents the element-by-element power operation. text and V image Take each element of k to the power and get the features after high-power transformation; Text feature vector: T text =[t1,t2,t3], t1 is the word weight extracted by TF-IDF; t2 is the word vector component calculated by Word2Vec; t3 is the contextual semantic feature extracted by BERT; k=2, we can get: ; This performs element-by-element power operations to obtain high-power feature expansion: {T}^{'}_{power}=[T,{T}^{(2)},{T}^{(3)},...,{T}^{(k)}] ; {V}^{'}_{power}=[V,{V}^{(2)},{V}^{(3)},...,{V}^{(k)}] ; Among them, T (k) represents the kth text feature vector; V (k) represents the kth image feature vector; The high-order features obtained after transformation are concatenated with the original features. The concatenation process is as follows: ; ; Among them, T text and V image is the original feature, T' power and V' power is the power transformation characteristic; After splicing, the enhanced feature T' is obtained text and V' image .

[0013] Furthermore, by calculating the Pearson correlation coefficient, a relationship matrix between text and image features is constructed, and the text and image features are weighted and fused according to the relationship matrix to obtain the final fusion feature, including: Construct the cross-modal relationship matrix R by calculating the Pearson correlation coefficient ρ (e i , e j ), which is expressed as follows: ; Among them, cov(e i , e j ) is e i and e j The covariance of ei and σ ej Yes i and e j The standard deviation of ei and μ ej They are e i and e j The mean of T text and V image Substitute into e respectively i and e j In, use Construct a relationship fusion matrix; Use the Pearson correlation coefficient to obtain the relationship fusion matrix between text features and image features , where m L The value is the number of text and image features; the dimension of matrix R is m L ×m L , R TI Reflects the relationship between text features and image features; V image and T text Substitute e i and e j Repeat the above steps to get the relationship matrix R between image features and text features.IT ; After getting the relationship matrix R TI With R IT After that, the image features and text features are weighted respectively; among them, the weighted relationship matrix R from text to image is used TI To weight the image features, expressed as: , I fused Represents the image features generated by weighting the text-to-image relationship matrix; the weighted relationship matrix R from image to text IT To weight the text features, it is expressed as: , T fused Represents the text features generated by weighting the weighted relationship matrix from image to text; T' text and V' image They are the enhanced text features and image features respectively; Finally, the text features and image features are fused together by weighted averaging, expressed as: F fused =αT fused +βI fused Among them, F fused For the final fused feature vector, α and β are hyperparameters that control the contribution of text features and image features in the final fused features, and their values ​​can be dynamically adjusted according to the importance of each modality.

[0014] Furthermore, in the step of inputting the fused feature vector into the MLP classifier, the MLP classifier maps the fused feature to the corresponding agricultural product classification category to obtain the agricultural product classification result, the MLP classifier is designed based on a neural network and includes three layers: an input layer, a hidden layer, and an output layer; different layers of the MLP neural network are fully connected, where: The input layer receives the fused feature vector F fused , and pass it to the first fully connected layer; The first fully connected layer takes the input feature vector F fused Mapped to the hidden layer space, expressed as: h1=W1F fused +b1, where W1 is the weight matrix of the first layer, is the bias term, h1 represents the eigenvector F fused Features after linear transformation; Then use the activation function ReLU to introduce nonlinearity: , set negative ReLU values ​​to 0 and keep positive values; The output of the first layer is passed to the second fully connected layer for processing. The number of neurons in the hidden layer is h2, and the calculation formula is expressed as: ; Among them, W2 is the weight matrix of the second layer, is the bias term; the output of the second layer is nonlinearly transformed through the ReLU activation function: ; The final output layer outputs the results of multiple transformations, and the original score of the output layer can be obtained: , where W3 represents the weight matrix of the output layer; b3 represents the bias term of the output layer; z represents the original score of the output layer; The score z of the output layer is converted into the probability of the category through the Softmax activation function, which is expressed as ; in, represents the score after exponential transformation, e is the base of natural logarithm, Z i The output layer original score is calculated by the output layer of the neural network; Represents the normalization factor, which is used to calculate the sum of the scores of all categories after exponential transformation so that the sum of the probabilities of all categories is equal to 1, where , C represents the number of classification levels of agricultural products, Represents the predicted probability of the i-th category, and the sum of the probabilities of all categories is 1; After obtaining the probability of each classification, it is necessary to calculate the cross entropy loss function between the category probability predicted by the model and the one-hot encoding form of the true category to measure the difference between the model output probability and the true label; the cross entropy loss function is: ; Among them, y i is the one-hot encoding of the true label, is the model’s predicted probability for the i-th class; In the training MLP model, the cross entropy loss function and the back propagation algorithm are used to obtain the weight gradient corresponding to each loss, and then the weight is dynamically updated; in each training cycle, the prediction result is calculated by forward propagation, and the model error is evaluated based on the loss function; then, the gradient is calculated using the back propagation algorithm, and the model parameters are updated through the optimization algorithm, and the weight is continuously optimized until the loss function converges.

[0015] Compared with the prior art, the agricultural product classification method integrating dual-stream attention integration and cross-modal fusion in the present invention has the following beneficial effects: First, the present invention adopts dual-stream attention integration to perform multi-level feature extraction on text modality and image modality respectively; in the processing of text data, TF-IDF, Word2Vec and BERT models are used to extract three different levels of features; in the processing of image data, SIFT, HOG and LBP models are used to extract three different levels of features; then, the text and image features are integrated through the self-attention mechanism and adaptive layer weights to obtain the fusion feature vector of the two modalities; the classification method of the present invention adopts multi-level features to combine shallow and deep features, greatly improving the classification decision-making ability; the classification method of the present invention can also maintain high performance in the case of partial information missing or low data quality, obtain the relevant features of the context, and improve the stability of the system; Second, the present invention uses a cross-modal relationship matrix to fuse text feature vectors and image feature vectors; by mapping feature vectors into a high-dimensional space, enhanced text and image features are obtained, thereby improving the model's ability to understand multidimensional information and capturing the correlation between different modalities; the present invention combines multimodal data features to construct a relationship fusion matrix, so that it can adapt to complex environments with unbalanced categories and high data noise, and improve the generalization ability of the model; in the processing of large-scale complex data sets, the present invention can effectively calculate the importance of features and weaken the interference of redundant information, thereby improving the discriminative ability of feature representation and optimizing the accuracy and stability of classification decisions. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention.

[0017] Figure 1 A flow chart of the agricultural product classification method integrating dual-stream attention integration and cross-modal fusion according to the present invention; Figure 2 The logical framework diagram of the agricultural product classification method integrating dual-stream attention integration and cross-modal fusion of the present invention; Figure 3 A sub-flow chart of the agricultural product classification method integrating dual-stream attention integration and cross-modal fusion of the present invention; Figure 4 This is another sub-flowchart of the agricultural product classification method that integrates dual-stream attention integration and cross-modal fusion of the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0019] The specific implementation of the present invention is described in detail below in conjunction with specific embodiments.

[0020] In one embodiment of the present invention, a method for classifying agricultural products by integrating dual-stream attention integration and cross-modal fusion is provided. The method firstly requires preprocessing text data and image data of agricultural products; In the process of data preprocessing, the agricultural product text data is subjected to stop words removal, missing values ​​processing, and capitalization unification, and then the text is subjected to word segmentation processing; the image data is subjected to image size unification and then normalization processing; In one implementation of text data processing, the agricultural product text data D t Preprocessing usually involves removing stop words, processing missing values, and format conversion. To remove stop words from text, you can use the sklearn tool. The sklearn tool provides CountVectorizer and TfidfVectorizer, which can directly remove stop words during preprocessing to reduce noise in the text. Missing data may affect the training and prediction of the model, so it needs to be filled or deleted reasonably. Numerical data is filled with the mean, and categorical data is filled with the mode. To ensure consistent data formats, numeric fields can be converted to floating point numbers or integers, date fields can be converted to time format, and spaces can be removed and uppercase and lowercase can be unified. In one implementation of image data processing, the agricultural product image data D v Factors such as resolution and size will affect feature extraction during analysis, so it is usually necessary to unify the image size, normalize, and perform data enhancement. The image size is unified to 224×224 or 512×512 to ensure consistent input; the image pixel value range is [0,255], and the data range is large, which affects the gradient calculation. The pixel value needs to be scaled to [0,1] or [-1,1] to reduce the numerical difference; the image is randomly flipped, rotated, and transformed to enhance the robustness of the model.

[0021] Please refer to Figure 1 and Figure 2 The agricultural product classification method integrating dual-stream attention integration and cross-modal fusion provided by the embodiment of the present invention includes the following steps: S1. Using a text feature model to extract features from the preprocessed text data to obtain multiple text feature vectors; S2, using the image feature model to extract features from the preprocessed image data to obtain multiple image feature vectors; Among them, in step S1, the text data D tThe text feature models used for feature extraction include TF-IDF model, Word2Vec model and BERT model. Based on the above models, multiple feature vectors T can be obtained. TF-IDF 、T Word2Vec 、T BERT ; For details, please refer to Figure 3 The step of extracting features from the preprocessed text data using the text feature model comprises the following steps: S11. Use the TF-IDF model to extract the text feature vector T at the statistical information level TF-IDF ; The TF-IDF model provided in the embodiment of the present disclosure is used to extract text features at the statistical information level, and the specific calculation formula is as follows: ; Among them, the calculation formula of tf is expressed as: tf(t,d)=1 / n; the calculation formula of idf is expressed as: ; In the formula, tf represents term frequency; idf represents inverse document frequency; i represents the number of times word t appears in document d; n represents the total number of words in document d; N represents the total number of documents; df(t) represents the number of documents for word t; the feature vector can be expressed as: .

[0022] S12. Use the Word2Vec model to convert words into high-dimensional vectors. The continuous bag of words model (CBOW) is used to predict the central word through the context of the agricultural product dataset. After training the CBOW model, each word will be mapped to a vector of fixed dimension to capture the text feature vector T. Word2Vec ; Specifically, the Word2Vec model is a distributed representation method based on neural networks, which can convert words into high-dimensional vectors, so that semantically similar words are closer in the vector space; the present invention adopts the CBOW (continuous bag of words) model to predict the central word through the context of the agricultural product data set; the goal of the CBOW model is to calculate , the formula is: ; in, is the target word W t The output word vector of is the average of the context word vectors, and v is the size of the vocabulary; After training the CBOW model, each word is mapped to a vector of fixed dimension, thus capturing the feature vector T Word2Vec ; S13. Use the BERT model to extract the deep semantic context information feature vector T BERT ; Specifically, the BERT model is used to extract contextual information features of deep semantics; the present invention uses the BERT model to obtain a high-dimensional vector representation of each word in the data set, reflecting the text features of deep contextual semantics; For a text dataset, first, BERT uses WordPiece Tokenizer to split the input text into word fragments (tokens) and adds special tags [CLS] and [SEP] at the beginning and end, that is, ; Then, the word fragment is input into the Embedding layer, and the final input vector is obtained through embedding. Next, a multi-layer Transformer encoder is used to extract contextual features, mainly using self-attention, multi-head attention, and feed-forward neural network (FFN) to obtain a context-aware representation of the text; Finally, after the Transformer layer, BERT generates a vector representation T for each word. BERT ; In step S2, the image data D v The feature vector V is extracted using models such as Scale Invariant Feature Transform (SIFT), Histogram of Oriented Gradients (HOG), and Local Binary Pattern (LBP). SIFT 、V HOG 、V LBP ; For details, please refer to Figure 4 The step of extracting features from the preprocessed image data using the image feature model comprises the following steps: S21. Use the SIFT model to construct images of different scales through the Gaussian pyramid; SIFT calculates the local extreme value of the image through the differential Gaussian pyramid DOG, SIFT assigns the direction of the key points, calculates the gradient direction around each key point, and uses the gradient amplitude and gradient direction to calculate the gradient direction histogram of the 16 sub-regions of the key point. Each sub-region has 8 direction components, and finally forms a 128-dimensional feature vector V SIFT ; Exemplarily, in step S21 of the present invention, SIFT constructs images of different scales through a Gaussian pyramid: ; Among them, I(x,y) is the image data D vThe input image in G(x, y, σ) is a Gaussian function; L(x, y) represents the smoothed image of the input image at different scales, L(x, y, σ) represents the smoothed version of the image at scale σ, and L(x, y, σ) is the result of convolution of the input image I(x, y) with the Gaussian function G(x, y, σ); Subsequently, SIFT calculates the local extrema of the image through the difference Gaussian pyramid (DOG): D(x,y,σ)=L(x,y,kσ)-L(x,y,σ), where k is the scale change factor; Next, SIFT assigns the key point directions and calculates the gradient direction around each key point, as shown in the following formula: ; ; Among them, m(x,y) is the gradient magnitude, θ(x,y) is the gradient direction; Finally, the gradient magnitude m(x, y) and gradient direction θ(x, y) are used to calculate the gradient direction histogram of the 16 sub-regions of the key point: ; Among them, i represents the direction index of the histogram; h i Represents the cumulative gradient amplitude of the i-th direction in the direction histogram; θ i represents the i-th angle in the direction histogram; δ represents an indicator function, which is a Kronecker Delta function, and the indicator function is used to determine the gradient direction of a pixel point; Each sub-region has 8 directional components, which ultimately form a 128-dimensional feature vector: V SIFT= [h1,h2,...,h 128 ]; S22, using the HOG model to capture edge and shape features; wherein HOG calculates the gradient of the image in the X and Y directions, calculates the gradient amplitude and gradient direction, divides the image into regions, and calculates the gradient histogram in 9 directions for each region to form a 3780-dimensional HOG feature vector V HOG ; In one implementation, HOG is mainly used to capture edge and shape features for agricultural product object detection and classification; HOG calculates the gradient of the image in the X and Y directions, as shown in the following formula: ; ; Among them, G x The gradient is calculated in the X direction, that is, the rate of change in the horizontal direction, reflecting the edge information of the image in the left and right directions; G yThe gradient is calculated in the Y direction, that is, the rate of change in the vertical direction, reflecting the edge information of the image in the up and down directions; Similarly, the gradient magnitude m(x, y) and gradient direction θ(x, y) are calculated, and the image is divided into 8×8 regions. The gradient histograms of 9 directions are counted in each region, which are expressed as follows: ; Finally, a 3780-dimensional HOG feature vector is formed: V HOG =[h1,h2,...,h 3780 ]; S23, using the LBP model to extract local texture features to form a low-dimensional feature vector; wherein, LBP compares the relationship between the pixel point and its neighboring pixels, calculates the LBP value, and calculates the LBP histogram to form a P+2-dimensional feature vector V LBP .

[0023] Specifically, in one implementation of the present invention, the LBP model is used to extract local texture features to form a low-dimensional feature vector; LBP calculates the LBP value by comparing the relationship between the pixel and its neighboring pixels: ; in, , P is the number of points; LBP (x c ,y c ) means (x c ,y c ) is the LBP value calculated for the center pixel, I p Represents the gray value of the neighborhood pixel, I c Represents the gray value of the center pixel; The LBP histogram is calculated using the following formula: ; Finally, a P+2-dimensional feature vector is formed (usually the low dimension is 10): V LBP= [h1,h2,...,h 10 ].

[0024] Please continue to refer to Figure 1 and Figure 2 The agricultural product classification method integrating dual-stream attention integration and cross-modal fusion provided by the present invention also includes the following steps: S3, integrating multiple text feature vectors through adaptive layer weights, integrating multiple image feature vectors through a self-attention mechanism, and performing weighted summation to obtain integrated text features and image features; S4, performing high-power transformation on the integrated text features and image features respectively, and concatenating the transformed features with the original features to obtain enhanced text features and enhanced image features; S5. By calculating the Pearson correlation coefficient, a relationship matrix between text and image features is constructed, and the text and image features are weighted and fused according to the relationship matrix to obtain the final fusion feature; S6. Input the fused feature vector into the MLP classifier. The MLP classifier maps the fused feature vector to the corresponding agricultural product classification category to obtain the agricultural product classification result.

[0025] Among them, in step S3 of the present invention, the specific process is divided into two steps: text feature integration and image feature integration; First, for complex multi-layer feature fusion, the present invention uses adaptive layer weights (Layer Attention) for feature integration, dynamically assigns weights according to the importance of different layers, and finally performs weighted fusion to obtain more representative features T text ; Next, multiple image feature vectors extracted in step S2 are concatenated, and the correlation between different features is captured through the self-attention mechanism to obtain the weight value of each feature; Finally, a fused feature vector V is obtained using the weighted summation method. image ; Specifically, the steps of integrating multiple text feature vectors through adaptive layer weights include: The text feature vector T TF-IDF 、T Word2Vec 、T BERT Mapped to the same dimension d, expressed as: ; Where W i represents a learnable parameter used to map different features to the same dimension; T i Represents the text feature vector; T' i Represents the text feature vector after dimension mapping; Calculate the weight, the weight w is expressed as: ; Among them, w is a three-dimensional vector, each feature has a normalized weight; W attn represents a learnable attention weight matrix; After obtaining the normalized weight of each feature, the fused feature vector T is calculated by the weighted fusion method. text , expressed as: ; Where w1, w2 and w3 all represent normalized weights; T' TF-IDF , T' Word2Vec , T' BERT Both represent the feature vectors after dimension mapping; The fusion feature T of the embodiment of the present invention text It retains the advantages of the three models TF-IDF, Word2Vec and BERT: Term frequency-inverse document frequency (TF-IDF) reflects the sparse information characteristics of the text and can capture keyword information; The word vector-based feature extraction model (Word2Vec) provides dense distributed word vector representation; The BERT model has the ability to understand context; Therefore, T text It combines the stability of traditional methods with deep semantic modeling capabilities, and can achieve higher classification accuracy and generalization capabilities in agricultural product classification tasks.

[0026] Furthermore, the step of integrating multiple image feature vectors through the self-attention mechanism includes: The Query, Key, and Value in the self-attention mechanism come from V SIFT 、V HOG 、V LBP , the obtained Query vectors are expressed as: Q SIFT , Q HOG , Q LBP The obtained Key vector is expressed as: K SIFT , K HOG , K LBP ; The obtained Value vector is expressed as: V' SIFT 、V' HOG 、V' LBP ; By calculating the dot product of Query and Key, we get the similarity score between each pair of features, expressed as: ; Among them, i and j represent the index of the feature, d k is the size of the feature dimension; Q i represents the Query vector after the i-th feature is extracted; K j represents the Key vector of the jth feature, and T represents the transposition operation; Then perform a softmax operation on each score to obtain the normalized weight a ij , expressed as: a ij =softmax(Attention Score ij ), where aij The weight represents the degree of attention of the i-th feature to the j-th feature, reflecting the importance of different features; The features are weighted and summed by calculating the attention weights, expressed as: ; Among them, α SIFT , α HOG , α LBP Respectively represent the contribution of SIFT, HOG and LBP to the final fusion feature; the Value vectors of the three features are weighted and summed to obtain a weighted fusion image feature vector V image .

[0027] In step S4 of the present invention, in order to enhance the feature expression capability, it is necessary to fused the feature T text and V image Perform a high-power transformation and then concatenate it with the original features to obtain the enhanced feature vector , .

[0028] Specifically, in step S4 of the present invention, the steps of performing high-power transformation on the integrated text features and image features respectively and splicing the transformed features with the original features include: For the text feature vector T text and image feature vector V image Perform element-by-element power operations, expressed as: and ; Among them, ⊙k represents the element-by-element power operation. text and V image Take each element of k to the power and get the features after high-power transformation; Text feature vector: T text =[t1,t2,t3], t1 is the word weight extracted by TF-IDF; t2 is the word vector component calculated by Word2Vec; t3 is the contextual semantic feature extracted by BERT; k=2, we can get: ; This performs element-by-element power operations to obtain high-power feature expansion: {T}^{'}_{power}=[T,{T}^{(2)},{T}^{(3)},...,{T}^{(k)}] ; {V}^{'}_{power}=[V,{V}^{(2)},{V}^{(3)},...,{V}^{(k)}] ; Among them, T (k) represents the kth text feature vector; V (k) represents the kth image feature vector; The high-order features obtained after transformation are concatenated with the original features. The concatenation process is as follows: ; ; Among them, T text and V image is the original feature, T' power and V' power is the power transformation characteristic; After splicing, the enhanced feature T' is obtained text and V' image , used for subsequent multimodal tasks.

[0029] In step S5 of the present invention, by constructing a cross-modal relationship matrix of text features and image features, the correlation between the two modalities is deeply analyzed, and the weight distribution of each modal feature is quantified. Based on this weight information, the enhanced text features and image features are weighted fused to finally obtain a fused feature vector to improve the expression ability and decision-making performance of multimodal information; Specifically, in step S5 of the present invention, by calculating the Pearson correlation coefficient, a relationship matrix between text and image features is constructed, and the text and image features are weighted and fused according to the relationship matrix to obtain the final fusion feature step, including: Construct the cross-modal relationship matrix R by calculating the Pearson correlation coefficient ρ (e i , e j ), which is expressed as follows: ; Among them, cov(e i , e j ) is e i and e j The covariance of ei and σ ej Yes i and e j The standard deviation of ei and μ ej They are e i and e j The mean of and Substitute into e respectively i and e j In, use Construct a relationship fusion matrix; Furthermore, in the embodiment of the present invention, the Pearson correlation coefficient is used to obtain the text feature and image feature relationship fusion matrix: , where m L The value is the number of text and image features; the dimension of the matrix is ​​m L ×m L , R TI It reflects the relationship between text features and image features; Furthermore, in the embodiment of the present invention, V image and T text Substitute e i and e j Repeat the above steps to get the relationship matrix R between image features and text features. IT ; After getting the relationship matrix R TI With R IT After that, the image features and text features are weighted respectively; among them, the weighted relationship matrix R from text to image is used TI To weight the image features, expressed as: , I fused Represents the image features generated by weighting the text-to-image relationship matrix; the weighted relationship matrix R from image to text IT To weight the text features, it is expressed as: , T fused Represents the text features generated by weighting the weighted relationship matrix from image to text; T' text and V' image They are the enhanced text features and image features respectively; Finally, the text features and image features are fused together by weighted averaging, expressed as: F fused =αT fused +βI fused Among them, F fused For the final fused feature vector, α and β are hyperparameters that control the contribution of text features and image features in the final fused features, and their values ​​can be dynamically adjusted according to the importance of each modality.

[0030] In step S6 of the present invention, a multi-layer perceptron (MLP) is used to classify the agricultural product data set; In the present invention, the agricultural product classification system is divided into four categories: livestock and poultry aquatic products, fruits, vegetables, grains and oils, specialty cash crops and others; Among them, livestock and poultry aquatic products mainly include livestock and poultry meat, poultry eggs, dairy products, and aquatic products; The fruit, vegetable, grain and oil category has three subcategories: fruit, vegetable, and grain and oil; Special economic crops are divided into tea, Chinese herbal medicine, flowers and plants, and nuts; Finally, there are other categories, including some processed agricultural products and special varieties. This classification method can not only ensure the comprehensiveness of classification, but also improve management and identification efficiency.

[0031] Specifically, in step S6 of the present invention, in the step of inputting the fused feature vector into the MLP classifier, the MLP classifier maps the fused feature to the corresponding agricultural product classification category, and obtains the agricultural product classification result, the MLP classifier is designed based on a neural network and includes three layers: an input layer, a hidden layer, and an output layer; different layers of the MLP neural network are fully connected, wherein: The input layer receives the fused feature vector F fused , and pass it to the first fully connected layer; The first fully connected layer takes the input feature vector F fused Mapped to the hidden layer space, expressed as: h1=W1F fused +b1, where W1 is the weight matrix of the first layer, is the bias term, h1 represents the eigenvector F fused Features after linear transformation; Then use the activation function ReLU to introduce nonlinearity: , set negative ReLU values ​​to 0 and keep positive values; The output of the first layer is passed to the second fully connected layer for processing. The number of neurons in the hidden layer is h2, and the calculation formula is expressed as: ; Among them, W2 is the weight matrix of the second layer, is the bias term. Similarly, the output of the second layer is nonlinearly transformed through the ReLU activation function: ; The final output layer outputs the results of multiple transformations, and the original score of the output layer can be obtained: ; Assume that agricultural products have C classification levels, where W3 represents the weight matrix of the output layer; b3 represents the bias term of the output layer; z represents the original score of the output layer; and the Softmax activation function is used to convert the score z of the output layer into the probability of the category, which is expressed as: ; in, represents the score after exponential transformation, e is the base of natural logarithm, Z i The output layer original score is calculated by the output layer of the neural network; Represents the normalization factor, which is used to calculate the sum of the scores of all categories after exponential transformation so that the sum of the probabilities of all categories is equal to 1, where , C represents the number of classification levels of agricultural products, Represents the predicted probability of the i-th category, and the sum of the probabilities of all categories is 1; After obtaining the probability of each classification, it is necessary to calculate the cross entropy loss function between the category probability predicted by the model and the one-hot encoding form of the true category to measure the difference between the model output probability and the true label; Among them, the cross entropy loss function is: ; Among them, y i is the one-hot encoding of the true label, is the model’s predicted probability for the i-th class; Next, the MLP model is further trained, using the cross entropy loss function and the back propagation algorithm to obtain the weight gradient corresponding to each loss, and then dynamically update the weight; during the model training process, the data set is iterated for multiple rounds. In each training cycle, the prediction results are calculated through forward propagation, and the model error is evaluated based on the loss function; then, the gradient is calculated using the back propagation algorithm, and the model parameters are updated through the optimization algorithm, and the weights are continuously optimized until the loss function converges, thereby improving the classification performance of the model; After the training is completed, the test set is used to further evaluate the generalization ability of the model; specifically, fruits, vegetables, grains and oils are selected from the agricultural product classification system of the present invention for accuracy verification. First, for fruits, vegetables, grains and oils, relatively comprehensive text data and image data on the market are widely collected to ensure the representativeness and diversity of the data. Secondly, 20% of the agricultural product data is randomly divided from the fused feature vector as an independent test set to ensure that the sources of the training set and the test set are consistent. Because the agricultural product data set is large, the present invention uses a direct testing method to input the test set data into the MLP model to calculate the accuracy (AC), precision (PE), recall rate (RE), F1-score and other parameter values. The larger the value of these evaluation indicators, the better the classification performance. The specific definitions are as follows: ; Among them, TP (True Positive): the number of samples correctly predicted as this category; FP (False Positive): the number of samples that do not actually belong to this category but are predicted as this category; FN (False Negative): the number of samples that actually belong to this category but are incorrectly predicted as other categories; TN (True Negative): the number of samples that do not belong to this category and are not predicted as this category.

[0032] Multiple rounds of tests were conducted on fruit, vegetable, grain and oil agricultural products, and the obtained test parameters were compared and analyzed with the original method. The experimental results show that this method has shown significant improvements in key evaluation indicators such as accuracy, precision, recall and F1-score. Then, the same tests were conducted on the other three types of agricultural products, and the test results were summarized. From the summary results, it can be seen that the use of a dual-stream attention mechanism and the fusion of multimodal features can significantly improve the accuracy and robustness of agricultural product classification, especially when dealing with complex large-scale data sets, showing better performance and stronger generalization ability.

[0033] In summary, the present invention adopts dual-stream attention integration to perform multi-level feature extraction on text modality and image modality respectively; In the processing of text data, TF-IDF, Word2Vec and BERT models are used to extract three different levels of features; In image data processing, SIFT, HOG and LBP models are used to extract features at three different levels; Then, the text and image features are integrated through the self-attention mechanism and adaptive layer weights to obtain the fused feature vector of the two modalities; The classification method of the present invention uses multi-level features to combine shallow and deep features, greatly improving the classification decision-making ability; The classification method of the present invention can also maintain high performance when some information is missing or the data quality is low, obtain the relevant features of the context, and improve the stability of the system; In addition, the present invention uses a cross-modal relationship matrix to fuse text feature vectors and image feature vectors; by mapping feature vectors to a high-dimensional space, enhanced text and image features are obtained, thereby improving the model's ability to understand multidimensional information and capturing the correlation between different modalities; the present invention combines multimodal data features to construct a relationship fusion matrix, so that it can adapt to complex environments with unbalanced categories and high data noise, thereby improving the generalization ability of the model; In the process of processing large-scale complex data sets, the present invention can effectively calculate the importance of features, weaken the interference of redundant information, thereby improving the discriminative ability of feature representation and optimizing the accuracy and stability of classification decisions.

[0034] The above solutions are only an illustration of a preferred example, but are not limited thereto. When implementing the present invention, appropriate replacement and / or modification can be performed according to user needs.

[0035] The number of devices and processing scales described here are used to simplify the description of the present invention. Applications, modifications and variations of the present invention will be obvious to those skilled in the art.

[0036] Although the embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the specification and the embodiments. It can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily realized. Therefore, without departing from the general concept defined by the claims and equivalent scope, the present invention is not limited to the specific details and the illustrations shown and described here.

Claims

1. The agricultural product classification method integrating dual-stream attention integration and cross-modal fusion is characterized by: The following steps are involved: Using the text feature model to extract features from the preprocessed text data, a plurality of text feature vectors are obtained; Using the image feature model to extract features from the preprocessed image data, a plurality of image feature vectors are obtained; Multiple text feature vectors are integrated through adaptive layer weights, and multiple image feature vectors are integrated through the self-attention mechanism. The integrated text features and image features are obtained by weighted summation. Perform high-power transformation on the integrated text features and image features respectively, and concatenate the transformed features with the original features to obtain enhanced text features and enhanced image features; By calculating the Pearson correlation coefficient, the relationship matrix between text and image features is constructed, and the text and image features are weighted and fused according to the relationship matrix to obtain the final fusion feature; The fused feature vector is input into the MLP classifier, and the MLP classifier maps the fused features to the corresponding agricultural product classification category to obtain the agricultural product classification result.

2. The agricultural product classification method integrating dual-stream attention integration and cross-modal fusion according to claim 1 is characterized in that: The steps of extracting features from the preprocessed text data using the text feature model include: The TF-IDF model is used to extract the text feature vector T at the statistical information level. TF-IDF ; The Word2Vec model is used to convert words into high-dimensional vectors. The continuous bag of words model (CBOW) is used to predict the central word through the context of the agricultural product dataset. After the CBOW model is trained, each word will be mapped to a vector of fixed dimension to capture the text feature vector T. Word2Vec ; Use the BERT model to extract the deep semantic context information feature vector T BERT .

3. The agricultural product classification method integrating dual-stream attention integration and cross-modal fusion according to claim 2 is characterized in that: The steps of extracting features from the preprocessed image data using the image feature model include: The SIFT model is used to construct images of different scales through the Gaussian pyramid. SIFT calculates the local extreme value of the image through the differential Gaussian pyramid DOG. SIFT assigns the direction of the key points, calculates the gradient direction around each key point, and uses the gradient amplitude and gradient direction to calculate the gradient direction histogram of the 16 sub-regions of the key point. Each sub-region has 8 direction components, and finally forms a 128-dimensional feature vector V. SIFT ; The HOG model is used to capture edge and shape features. HOG calculates the gradient of the image in the X and Y directions, calculates the gradient amplitude and gradient direction, divides the image into 8×8 regions, and calculates the gradient histogram in 9 directions in each region to form a 3780-dimensional HOG feature vector V HOG ; The LBP model is used to extract local texture features to form a low-dimensional feature vector. LBP compares the relationship between a pixel and its neighboring pixels, calculates the LBP value, and calculates the LBP histogram to form a P+2-dimensional feature vector V LBP .

4. The agricultural product classification method integrating dual-stream attention integration and cross-modal fusion according to claim 3 is characterized in that: The steps of integrating multiple text feature vectors through adaptive layer weights include: The text feature vector T TF-IDF , T Word2Vec , T BERT Mapped to the same dimension d, expressed as: ； Where W i represents a learnable parameter used to map different features to the same dimension; T i Represents the text feature vector; T' i Represents the text feature vector after dimension mapping; Calculate the weight, the weight w is expressed as: ； Among them, w is a three-dimensional vector, each feature has a normalized weight; W attn represents a learnable attention weight matrix; After obtaining the normalized weight of each feature, the fused feature vector T is calculated by the weighted fusion method. text , expressed as: ； Where w1, w2 and w3 all represent normalized weights; T' TF-IDF , T' Word2Vec , T' BERT They all represent the feature vectors after dimension mapping.

5. The agricultural product classification method integrating dual-stream attention integration and cross-modal fusion according to claim 4 is characterized in that: The steps of integrating multiple image feature vectors through the self-attention mechanism include: The Query, Key, and Value in the self-attention mechanism come from V SIFT 、V HOG 、V LBP , the obtained Query vectors are expressed as: Q SIFT , Q HOG , Q LBP The obtained Key vector is expressed as: K SIFT , K HOG , K LBP ; The obtained Value vector is expressed as: V' SIFT 、V' HOG 、V' LBP ; By calculating the dot product of Query and Key, we get the similarity score between each pair of features, expressed as: ； Among them, i and j represent the index of the feature, d k is the size of the feature dimension; Q i represents the Query vector after the i-th feature is extracted; K j represents the Key vector of the jth feature, and T represents the transposition operation; Then perform a softmax operation on each score to obtain the normalized weight a ij , expressed as: a ij =softmax(Attention Score ij ), where a ij The weight represents the degree of attention of the i-th feature to the j-th feature; The features are weighted and summed by calculating the attention weights, expressed as: ； Among them, α SIFT , α HOG , α LBP Respectively represent the contribution of SIFT, HOG and LBP to the final fusion feature; the Value vectors of the three features are weighted and summed to obtain a weighted fusion image feature vector V image .

6. The agricultural product classification method integrating dual-stream attention integration and cross-modal fusion according to claim 5 is characterized in that: The steps of performing high-power transformation on the integrated text features and image features respectively and concatenating the transformed features with the original features include: For the text feature vector T text and image feature vector V image Perform element-by-element power operations, expressed as: and ; Among them, ⊙k represents the element-by-element power operation, for T text and V image Take each element of k to the power and get the features after high-power transformation; Text feature vector: T text =[t1,t2,t3], t1 is the word weight extracted by TF-IDF; t2 is the word vector component calculated by Word2Vec; t3 is the contextual semantic feature extracted by BERT; k=2, we can get: ; This performs element-by-element power operations to obtain high-power feature expansion: ； ； Among them, T (k) represents the kth text feature vector; V (k) represents the kth image feature vector; The high-order features obtained after transformation are concatenated with the original features. The concatenation process is as follows: ； Among them, T text and V image is the original feature, T' power and V' power is the power transformation characteristic; After splicing, the enhanced feature T' is obtained text and V' image .

7. The agricultural product classification method integrating dual-stream attention integration and cross-modal fusion according to claim 6 is characterized in that: The steps of calculating the Pearson correlation coefficient, constructing the relationship matrix between text and image features, and weighting and fusing text and image features according to the relationship matrix to obtain the final fusion features include: Construct the cross-modal relationship matrix R by calculating the Pearson correlation coefficient ρ (e i , e j ), which is expressed as follows: ； Among them, cov(e i , e j ) is e i and e j The covariance of ei and σ ej Yes i and e j The standard deviation of ei and μ ej They are e i and e j The mean of T text and V image Substitute into e respectively i and e j In, use Construct a relationship fusion matrix; Use the Pearson correlation coefficient to obtain the relationship fusion matrix between text features and image features , where m L The value is the number of text and image features; the dimension of the matrix is ​​m L ×m L , R TI Reflects the relationship between text features and image features; V image and T text Substitute e i and e j Repeat the above steps to get the relationship matrix R between image features and text features. IT ; After getting the relationship matrix R TI With R IT After that, the image features and text features are weighted respectively; among them, the weighted relationship matrix R from text to image is used TI To weight the image features, expressed as: , I fused Represents the image features generated by weighting the text-to-image relationship matrix; the weighted relationship matrix R from image to text IT To weight the text features, it is expressed as: , T fused Represents the text features generated by weighting the weighted relationship matrix from image to text; T' text and V' image They are the enhanced text features and image features respectively; Finally, the text features and image features are fused together by weighted averaging, expressed as: F fused =αT fused +βI fused Among them, F fused For the final fused feature vector, α and β are hyperparameters that control the contribution of text features and image features in the final fused features.

8. The agricultural product classification method integrating dual-stream attention integration and cross-modal fusion according to claim 7 is characterized in that: In the step of inputting the fused feature vector into the MLP classifier, the MLP classifier maps the fused features to the corresponding agricultural product classification category to obtain the agricultural product classification result, the MLP classifier is designed based on a neural network and includes three layers: input layer, hidden layer and output layer; the different layers of the MLP neural network are fully connected, where: The input layer receives the fused feature vector F fused , and pass it to the first fully connected layer; The first fully connected layer takes the input feature vector F fused Mapped to the hidden layer space, expressed as: h1=W1F fused +b1, where W1 is the weight matrix of the first layer, is the bias term, h1 represents the eigenvector F fused Features after linear transformation; Then use the activation function ReLU to introduce nonlinearity: , set negative ReLU values ​​to 0 and keep positive values; The output of the first layer is passed to the second fully connected layer for processing. The number of neurons in the hidden layer is h2, and the calculation formula is expressed as: ; Among them, W2 is the weight matrix of the second layer, is the bias term; the output of the second layer is nonlinearly transformed through the ReLU activation function: ; The final output layer outputs the results of multiple transformations, and the original score of the output layer can be obtained: , where W3 represents the weight matrix of the output layer; b3 represents the bias term of the output layer; z represents the original score of the output layer; The score z of the output layer is converted into the probability of the category through the Softmax activation function, which is expressed as ； in, represents the score after exponential transformation, e is the base of natural logarithm, Z i The output layer original score is calculated by the output layer of the neural network; Represents the normalization factor, which is used to calculate the sum of the scores of all categories after exponential transformation so that the sum of the probabilities of all categories is equal to 1, where , C represents the number of classification levels of agricultural products, Represents the predicted probability of the i-th category, and the sum of the probabilities of all categories is 1; After obtaining the probability of each classification, it is necessary to calculate the cross entropy loss function between the category probability predicted by the model and the one-hot encoding form of the true category to measure the difference between the model output probability and the true label; the cross entropy loss function is: ； Among them, y i is the one-hot encoding of the true label, is the model’s predicted probability for the i-th class; In the training MLP model, the cross entropy loss function and the back propagation algorithm are used to obtain the weight gradient corresponding to each loss, and then the weight is dynamically updated; in each training cycle, the prediction result is calculated by forward propagation, and the model error is evaluated based on the loss function; then, the gradient is calculated using the back propagation algorithm, and the model parameters are updated through the optimization algorithm, and the weight is continuously optimized until the loss function converges.

Images