A method and system for representing agricultural product preferences based on multimodal fusion

By using a Transformer encoder and a multimodal fusion method, the problems of semantic misalignment and long-tail distribution between modalities in the modeling of agricultural product preference representation were solved, and stable identification and personalized recommendation of user-product preference levels were achieved.

CN123118218APending Publication Date: 2026-11-17JIANGXI AGRICULTURAL UNIVERSITY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611603256.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-10-16
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing methods for modeling agricultural product preferences suffer from problems such as semantic misalignment between modalities, imbalance of contributions, noise interference, and high computational cost when processing multimodal data. Furthermore, they are difficult to effectively handle long-tail distribution features, leading to unstable preference predictions.

Method used

A multimodal fusion method based on Transformer encoder is adopted. Through query-guided cross-modal gating fusion module, dynamic product graph reasoning module and dynamic preference quantification module, a multimodal dataset is constructed, cross-modal attention calculation and graph structure enhancement are performed to generate user-product joint preference representation, and Focal Loss strategy is introduced for supervised training.

Benefits of technology

It improves the ability to identify weak preference levels under category imbalance conditions, and enhances the accuracy of personalized recommendations and supply-demand matching for agricultural e-commerce.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123118218A_ABST
    Figure CN123118218A_ABST
Patent Text Reader

Abstract

This invention discloses a method and system for representing agricultural product preferences based on multimodal fusion. The method includes: collecting comment text, images, and transaction behavior data from agricultural e-commerce platforms to construct a multimodal dataset; encoding the multimodal data into sequence representations; generating prefix sequences using a query-guided cross-modal gating fusion module; feeding the prefix sequences into the backbone of a large language model, performing intra-layer semantic reasoning within a specific Transformer layer in the order of self-attention, dynamic product graph reasoning, and hybrid expert feedforward; dynamically constructing a neighbor graph based on the cosine similarity of product node features using a dynamic preference quantification decision module; and mapping the global semantic representation and enhanced product node features into user preference vectors and product attribute preference vectors, respectively, and outputting the probability distribution of preference intensity through bidirectional interactive modeling as the preference representation result. This invention can be used for agricultural product user preference prediction, personalized recommendation, and supply-demand matching.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the field of personalized recommendation in agricultural e-commerce, specifically involving a method and system for representing agricultural product preferences based on multimodal fusion. Background Technology

[0002] Modeling agricultural product preferences relies heavily on multimodal data, including review texts, product images, and user behavior. In recent years, large language models, with their powerful contextual modeling and logical reasoning capabilities, have provided a new technological foundation for extracting deep-seated preference cues from unstructured multi-source data, becoming a significant opportunity to enhance the understanding of complex consumer semantics.

[0003] However, existing methods for modeling agricultural product preferences still face many limitations in practical applications: traditional methods often rely on single text features or simple multimodal feature concatenation, making it difficult to effectively handle semantic misalignment, contribution differences, and noise interference between different modalities. When multi-source information is inconsistent, it is easily affected by redundant information, leading to unstable preference prediction. Agricultural product preference judgment is not solely determined by text. If multimodal features are directly input into a large language model, problems such as imbalance of modal contributions, insufficient utilization of product association structure information, and high computational cost will still be encountered. Real e-commerce data exhibits significant long-tail distribution characteristics, with a high proportion of strong preference samples and a relative scarcity of weak and moderate preference samples. Ordinary classification models tend to favor the majority category, severely affecting the recognition performance of minority preference levels.

[0004] Based on the above problems, there is an urgent need to construct a multimodal modeling method that integrates comment text, image information, and behavioral statistical features. Summary of the Invention

[0005] To address the problems of inconsistent evaluation information caused by difficulties in aligning multimodal information modes, explicit modeling of product structure relationships, and imbalanced preference level distribution, this invention achieves the following technical solution: A method for representing agricultural product preferences based on multimodal fusion includes the following steps: Step 1: Construct a large language model that includes a Transformer encoder, a query-guided cross-modal gating fusion module, a dynamic product graph reasoning module, and a dynamic preference quantification module; collect and process user review texts, product main images and customer review images, and historical transaction behavior from e-commerce platforms to construct a multimodal dataset; Step 2: Encode the user review text, product main image, and customer review image in the multimodal dataset with historical transaction behavior to obtain text sequence, image sequence, and behavior sequence respectively; Step 3: Input the text sequence, image sequence, and behavior sequence into the query-guided cross-modal gating fusion module, perform cross-modal attention calculation, and obtain the initial features of the product node after cross-modal fusion; Step 4: Input the initial features of the product nodes after cross-modal fusion into the Transformer encoder. The Transformer encoder is equipped with a dynamic product graph inference module. The enhanced features of the product nodes are obtained through the dynamic product graph inference module. After processing by the Transformer encoder, the hidden features containing multimodal context information are obtained. Perform masked weighted average pooling operation on the effective text feature vectors in the hidden features containing multimodal context information to obtain the global semantic representation. Step 5: Input the enhanced features of the product nodes and the global semantic representation into the dynamic preference quantization module. Map the enhanced features of the product nodes and the global semantic representation to generate product attribute preference feature vectors and user preference feature vectors, respectively. Through bidirectional interactive feature modeling, obtain the user-product joint preference representation. Input the user-product joint preference representation into the rating network and output the three-level user preference intensity classification probability distribution of weak, medium and strong as the preference representation result.

[0006] Further, in step 2, the user review text in the multimodal dataset is processed through the embedding layer of the large language model to obtain a text sequence; the main product image and the customer review image are input into the CLIP-ViT encoder to obtain an image sequence; and the historical transaction behavior is processed through normalization and multilayer perceptron to obtain a behavior sequence.

[0007] Furthermore, in step 3, the text sequence, image sequence, and behavior sequence are input into the query-guided cross-modal gating fusion module to perform cross-modal attention calculation, thereby obtaining the initial features of the product node after cross-modal fusion: ; in, This represents the initial features of the i-th product node after cross-modal fusion; Presentation layer normalization operation; This represents the learnable fusion query vector built into the query-guided cross-modal gating fusion module; , and These represent text sequences, image sequences, and action sequences, respectively. and They respectively represent queries using the fused query vector and queries using the first... Cross-modal attention extraction results for the rth modality features as keys and values; The weight vector representing the gating score calculation is obtained by performing an inner product operation with the hidden layer features after tanh transformation; , , , Indicates the parameters for calculating modal weights; This represents a nonlinear transformation.

[0008] Furthermore, in step 4, the initial features of the product nodes after cross-modal fusion are mapped to multimodal prefix features with the same dimension as the hidden layer of the large language model. The multimodal prefix features are then concatenated as prefix markers at the beginning of the text sequence to form the input sequence of multimodal information. Subsequently, the input sequence of multimodal information is input into the Transformer encoder to obtain hidden features containing multimodal contextual information.

[0009] Furthermore, in step 4, the initial features of the product nodes after cross-modal fusion are input into the dynamic product graph inference module: ; in, Enhance the feature matrix of the (l+1)th layer product node in the c-th relation channel; For the first In the first relationship channel Product node characteristics of the layer; Indicates the first An adjacency matrix of relational channels; Indicates the first Layer Random feature mask in each relation channel; This represents the element-wise multiplication operation; Indicates the first The first relationship channel in the _th relationship channel Layer graph convolution operation; This represents a structure-aware neighborhood aggregation operation. Represents the residual fusion coefficients of the adaptive aggregation result; Presentation layer normalization operation.

[0010] Furthermore, in step 5, bidirectional interactive feature modeling is performed, specifically including concatenating the user preference vector and the product preference vector to obtain the first... User-item joint preference representation for each item: ; in, Let f be the enhanced feature matrix for the (l+1)th layer product node in the c-th relation channel, where c∈{1,2,...,C} represents different types of product relation views; This indicates a user-side weighted pooling operation; For validity mask; Weights for text nodes; The parameter is User preference quantization mapping operation; Indicates the first Calculation of selecting individual product nodes; The parameter is Product preference mapping operation; This represents the vector concatenation operation; This represents the user-item joint preference representation for the i-th item.

[0011] Furthermore, in step 5, the user-item joint preference representation is input into the rating network, which outputs a probability distribution of weak, medium, and strong user preference intensity classifications: ; in, This represents the user-item joint preference representation for the i-th item. For user preference vectors; Let be the product preference vector for the i-th product; This represents the element-wise multiplication operation; This represents the element-wise absolute difference between the user preference vector and the product preference vector. This represents the vector concatenation operation; Representation layer normalization operation; Represents a scoring network; This represents the probability normalization operation; Let represent the probability distribution of the user preference intensity classification of the i-th product into three levels: weak, medium, and strong.

[0012] Furthermore, it also includes supervised training of network modules using the Focal Loss function, which incorporates inverse frequency category weights.

[0013] This invention also provides a multimodal fusion-based agricultural product preference characterization system, comprising: The multimodal data acquisition and encoding module is used to collect user review text, product main images and customer review images, and historical transaction behavior data from agricultural e-commerce platforms, construct a multimodal dataset, and encode the user review text, product main images and customer review images, and historical transaction behavior data in the multimodal dataset into text sequences, image sequences, and behavior sequences, respectively. The query-guided cross-modal gating fusion module is used to receive the text sequence, image sequence and behavior sequence, perform cross-modal attention calculation and adaptive gating weighting through learnable fusion query vectors, and output the initial features of the product node after cross-modal fusion. The dynamic product graph reasoning module is set in a specific layer of the Transformer encoder. It is used to dynamically construct a neighbor graph based on the cosine similarity of product node features, enhance the graph structure of the initial features of the product nodes after cross-modal fusion, and output the enhanced features of the product nodes. The dynamic preference quantization module is used to map the enhanced features of the product node and the global semantic representation into product attribute preference feature vectors and user preference feature vectors, respectively, and generate a user-product joint preference representation through bidirectional interactive feature modeling. The preference level classification module is used to input the user-item joint preference representation into the scoring network and output the probability distribution of three levels of user preference intensity (weak, medium, and strong) as the preference representation result.

[0014] The present invention also provides an electronic device, comprising: a processor and a memory, wherein the memory stores instructions executable by the processor, and the processor is configured to execute the instructions such that the electronic device can implement the aforementioned method for representing agricultural product preferences based on multimodal fusion.

[0015] Compared with the prior art, the present invention has the following beneficial effects: This invention employs a research method combining query-guided cross-modal gating fusion, dynamic product graph reasoning, and dynamic preference quantification. By uniformly aligning the latent space of trimodal agricultural product consumption data, adaptive product graph structure information is injected into the Transformer layer. Furthermore, by explicitly modeling the interaction between user preferences and product attributes, the model not only outputs three preference levels but also provides an analyzable intermediate preference vector. Addressing the issue of a low proportion of weak preference samples in agricultural e-commerce data, a Focal Loss strategy is introduced, effectively improving the ability to stably identify weak and medium preference levels under class imbalance conditions. This provides a quantitative technical solution for personalized product recommendations and intelligent matching of supply and demand in agricultural e-commerce. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the method flow of the present invention.

[0017] Figure 2 This is a schematic diagram of the model structure of the present invention.

[0018] Figure 3 This is a schematic diagram of the improved Transformer structure. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments.

[0020] like Figure 1 As shown, a method for representing agricultural product preferences based on multimodal fusion includes the following steps: Step 1: Construct a large language model that includes a Transformer encoder, a query-guided cross-modal gating fusion module, a dynamic product graph reasoning module, and a dynamic preference quantification module; collect and process user review texts, product main images and customer review images, and historical transaction behavior from e-commerce platforms to construct a multimodal dataset; Step 2: Encode the user review text, product main image, and customer review image in the multimodal dataset with historical transaction behavior to obtain text sequence, image sequence, and behavior sequence respectively; Step 3: Input the text sequence, image sequence, and behavior sequence into the query-guided cross-modal gating fusion module, perform cross-modal attention calculation, and obtain the initial features of the product node after cross-modal fusion; Step 4: Input the initial features of the product nodes after cross-modal fusion into the Transformer encoder. The Transformer encoder is equipped with a dynamic product graph inference module. The enhanced features of the product nodes are obtained through the dynamic product graph inference module. After processing by the Transformer encoder, the hidden features containing multimodal context information are obtained. Perform masked weighted average pooling operation on the effective text feature vectors in the hidden features containing multimodal context information to obtain the global semantic representation. Step 5: Input the enhanced features of the product nodes and the global semantic representation into the dynamic preference quantification module. Map the enhanced features of the product nodes to the visual representation vector of agricultural products to generate product attribute preference feature vector and user preference feature vector respectively. Through bidirectional interactive feature modeling, obtain the user-product joint preference representation. Input the user-product joint preference representation into the scoring network and output the three-level user preference intensity classification probability distribution of weak, medium and strong as the preference representation result.

[0021] like Figure 2As shown, the main product image and customer review images are encoded into an image sequence using ViT (Visual Transformer), user comment text is encoded into a text sequence using the ChatGLM embedding layer, and historical transaction behavior is normalized and mapped to an MLP (Multilayer Perceptron) to form a behavior sequence. The three-modal sequences and a learnable fusion query vector are input into a query-guided cross-modal gating fusion module. After cross-attention and sample adaptive gating weighting, the initial features of the product node after cross-modal fusion are output. These initial features of the product node are mapped to the ChatGLM3-6B hidden dimensions as multimodal prefix features. The multimodal prefix features are then concatenated as prefix markers at the beginning of the text sequence to form a sequence containing multimodal information. Input sequence; The input sequence containing multimodal information is fed into Transformer layers 1 to 28 of the ChatGLM3-6B backbone for layer-by-layer semantic reasoning. In Transformer layers 12, 16, 20, 24, and 28, intra-layer semantic reasoning is performed in the order of self-attention, dynamic item graph reasoning, and hybrid expert feedforward. After processing by all Transformer layers, user preference vectors and item preference vectors are extracted respectively. After bidirectional interactive modeling by the user-item joint preference representation module, they are input into the rating network, which outputs the predicted probability distributions of three levels: low preference, medium preference, and high preference.

[0022] Specifically, in step 1, a large language model is constructed, which includes a Transformer encoder, a query-guided cross-modal gating fusion module, a dynamic product graph reasoning module, and a dynamic preference quantification module. We collected user review texts, product main images and customer review images, and historical transaction behavior data to construct a multimodal dataset and performed data cleaning. For user review texts, we performed deduplication, whitespace removal, and filtering of invalid control characters to ensure semantic purity. For product main images and customer review images, we removed empty images to avoid losing dimensional information. For historical transaction behavior data, we performed score standardization based on the mean and standard deviation of the entire dataset to eliminate differences between different indicators.

[0023] After cleaning and standardization, all samples were divided into training, validation, and test sets in a 7:2:1 ratio. The specific results are shown in Table 1. Table 1. Dataset Sample Distribution

[0024] Specifically, in step 2, feature encoding is performed on user comment text, product main image and customer review image, and historical transaction behavior to obtain text sequence, image sequence and behavior sequence, and the text sequence, image sequence and behavior sequence are input into cross-modal fusion module; The multidimensional feature encoding module is used to extract effective information from product images, product text, and user behavior data; For goods The product images and customer review images are input into a pre-trained Vit visual transformer to obtain an image sequence. Specifically, the main product image and the customer review image are scaled to 224×224 pixels and normalized. They are then encoded into an image feature sequence using ViT. The output of all image block labels is taken as the image block-level feature sequence, and then mapped to the hidden dimension of the large language model through a linear projection layer to obtain an image sequence consistent with the hidden dimension of the language model. For goods Text data is input into the word embedding layer of a large language model to obtain text sequences. Specifically, in this embodiment, ChatGLM3-6B is used as a large language model. The user comment text is segmented by a word segmenter that matches ChatGLM3-6B and then input into the embedding layer of the language model to obtain a text sequence. User's opinion on products Historical transaction behavior is input into a multilayer perceptron and normalized to obtain a behavior sequence. Specifically, the user's collection, add-to-cart, and purchase behaviors for product i are extracted into behavioral statistical vectors. The behavioral statistical vectors include the number of each type of behavior and the time interval between the most recent behavior and the present. These vectors are then mapped to the hidden dimension of the large language model through a multilayer perceptron consisting of two fully connected networks and normalized to obtain the behavior sequence.

[0025] Specifically, in step 3, a query-guided cross-modal gating fusion module is used to perform feature alignment and fusion for image sequences, text sequences, and behavior sequences; Specifically, this embodiment employs QCGF (Query-guided Cross-modal Gated Fusion), within which learnable fusion query vectors are configured. ,use Cross-modal attention is calculated for image sequences, text sequences, and behavior sequences respectively to obtain the corresponding text context features, image context features, and behavior context features.

[0026] Based on the matching degree between the learnable fusion query vector and the context features of each modality, adaptive fusion weights are determined for the image sequence, text sequence, and behavior sequence. Then, the context features of each modality are weighted and summed according to the adaptive fusion weights. Finally, the weighted summation result is residually concatenated with the fusion query vector, and layer normalization is performed to obtain the initial features of the cross-modal fused product node. ; in, This represents the initial features of the i-th product node after cross-modal fusion; Presentation layer normalization operation; This represents the learnable fusion query vector built into the query-guided cross-modal gating fusion module; , and These represent text sequences, image sequences, and action sequences, respectively. and They respectively represent queries using the fused query vector and queries using the first... Cross-modal attention extraction results for the rth modality features as keys and values; This represents the weight vector used to map hidden layer features to scalar attention scores; , , , Indicates the parameters for calculating modal weights; This represents a nonlinear transformation.

[0027] Specifically, in step 4, the initial features of the product nodes obtained after cross-modal fusion are mapped to multimodal prefix features with the same dimension as the hidden layer of ChatGLM3-6B. The multimodal prefix features are concatenated as prefix markers at the beginning of the text sequence to form an input sequence containing multimodal information. Then, the input sequence containing multimodal information is input into the Transformer encoder of ChatGLM3-6B, so that the text sequence, image sequence and behavior sequence are jointly modeled in a unified hidden feature space to obtain hidden features containing multimodal contextual information.

[0028] like Figure 3 As shown, the input sequence containing multimodal information is fed into a specific Transformer layer. First, it undergoes layer normalization, and then a self-attention mechanism is used for global semantic interaction. The self-attention output is added to the input before layer normalization via a residual connection. The added features are normalized again and input into the dynamic product graph inference module. In this module, a Top-K nearest neighbor graph is dynamically constructed based on the product node features, and graph structure enhancement is performed. The enhancement result is written back to the deep feature latent vectors corresponding to the three modal representation units of text, image, and behavior in residual form. The output of the dynamic product graph inference module is normalized and then input into the MoE layer (hybrid expert model layer). The gated routing network selects an activation expert for each token and weights and fuses the outputs of each expert. The output of the MoE layer is added to the output of the DPGR layer via a residual connection, and finally output to the next Transformer layer.

[0029] DPGR (Dynamic ProductGraph Reasoning) is injected into layers 12, 16, 20, 24, and 28 of the Transformer encoder. DPGR first performs labeling and mean pooling on the hidden states of the current layer along the sequence dimension to obtain the global representation of each sample. Then, these representations are aggregated according to the product identifier to construct a graph structure with products as nodes. The association strength between nodes is calculated based on cosine similarity, and the top 8 most similar neighbors of each node are selected to dynamically construct a neighbor graph. A multi-layer graph convolutional network and a structure-aware neighborhood aggregation module are used to enhance the structure of the constructed graph, and the enhancement results are written back to the original hidden state in residual form. For node representations obtained from different graph convolutional layers within the same channel, an adaptive fusion is performed using a structure-aware neighborhood aggregation module to ultimately obtain a multi-channel enhanced node representation. The above process can be expressed by the following formula: ; in, Enhance the feature matrix of the (l+1)th layer product node in the c-th relation channel; For the first In the first relationship channel Product node characteristics of the layer; Indicates the first An adjacency matrix of relational channels; Indicates the first Layer The random feature mask in each relation channel is generated according to the Bernoulli distribution. During training, the node feature dimension is randomly set to zero with a preset probability, and it is turned off during inference. This represents the element-wise multiplication operation; Indicates the first The first relationship channel in the _th relationship channel Layer graph convolution operation; This represents a structure-aware neighborhood aggregation operation; Represents the residual fusion coefficients of the adaptive aggregation result; Presentation layer normalization operation; Let represent the graph convolution operation at layer l in the c-th relation channel. In this embodiment, the normalized form of the adjacency matrix is ​​used as the propagation operator. The node features of the previous layer are multiplied by the learnable weight matrix, and the node features of the current layer are obtained after a nonlinear transformation by the activation function. The normalized form of the adjacency matrix is ​​obtained by symmetric normalization of the adjacency matrix. The dimension of the learnable weight matrix is ​​consistent with the dimension of the node features. The activation function is ReLU. Four relation channels (c∈{1,2,3,4}) are used, and each channel contains two graph convolutions (l∈{1,2}). The transformation formula is as follows: ,in For a symmetric normalized adjacency matrix, The weight matrix is ​​a learnable weight matrix; The structure-aware neighborhood aggregation operation is described in this embodiment. Importance scores are calculated for neighborhood representations with different hop counts. The importance scores of each hop count are normalized by Softmax and used as the fusion weights for the corresponding neighborhood representations. The neighborhood representations with different hop counts are weighted and summed according to the fusion weights to obtain the aggregation result. Indicates the first The adjacency matrix of each relation channel is composed of matrix elements. constitute: ; in, This represents the set of the K neighboring nodes with the highest feature similarity to node i. In this embodiment, K=8. For each product node i in the current batch, the feature cosine similarity between node i and other product nodes in the current batch is calculated, and the top 8 neighboring nodes with the highest cosine similarity are selected to form the neighboring node set. At node j, When the similarity is true, take the cosine similarity between node i and node j; otherwise, take 0. Based on Construct the adjacency matrix of the c-th relation channel ; This represents the initial features of the i-th product node after cross-modal fusion; This represents the initial feature transpose of the i-th product node after cross-modal fusion; This represents the initial features of the j-th product node after cross-modal fusion; This represents the L2 norm of a vector.

[0030] Specifically, in step 5, DPQ (Dynamic Preference Quantification) is used to perform masked weighted average pooling on the effective text tag features output from the last layer of ChatGLM3-6B to obtain the user global semantic representation, and then the user global semantic representation is mapped to the user preference vector through the DPQ decision head.

[0031] The enhanced features of the product nodes are mapped to product preference vectors; the user preference vector and the product preference vector are concatenated to obtain the first... User-item joint preference representation for each item: ; in, For the first Layer, First Enhanced features of product nodes in each channel; This indicates a user-side weighted pooling operation, used based on the validity mask. and text node weights The effective text node features in the node feature matrix are aggregated to obtain user semantic features; This represents the user preference quantization mapping operation, with parameters as follows: The multilayer perceptron is implemented with user semantic features as input and user preference vector as output. The multilayer perceptron contains at least one hidden layer with a dimension smaller than the input dimension, and ReLU is used as the activation function. Indicates the first The product node selection operation is used to extract the first product node from the node feature matrix. Graph reasoning enhancement features corresponding to each product node; By parameters The commodity preference mapping operation consists of a linear layer and a Sigmoid activation function. The input is the visual representation vector of agricultural products, and the output is the commodity attribute preference vector. This represents the vector concatenation operation; This represents the user-item joint preference representation for the i-th item; Specifically, Used for validity mask and text node weights The valid text node features in the node feature matrix are aggregated. The validity mask is used to identify which positions in the node feature matrix belong to real text nodes, excluding image pseudo-markers, behavior pseudo-markers, and graph structure injection positions. The text node weights are obtained by normalizing the node features after linear mapping and are used to measure the importance of each text node in the aggregation. During aggregation, the hidden vector of each valid text node is multiplied by the corresponding validity mask value and the text node weight, and the sum is divided by the sum of the products of the mask value and the weight to obtain the user semantic features.

[0032] Furthermore, element-wise interaction features and element-wise difference features between user preference vectors and product preference vectors are constructed. These user-product joint preference representations, element-wise interaction features, and element-wise difference features are then input into the scoring network to output the first... Probability distribution of each product belonging to different preference levels: ; in, This represents the user-item joint preference representation for the i-th item. For user preference vectors; Let be the product preference vector for the i-th product; The element-wise interaction features of the user preference vector and the product preference vector are used to characterize the matching relationship between user preferences and product features across various representation dimensions. It represents the element-wise absolute difference between the user preference vector and the product preference vector, that is, the element-wise difference feature between the user preference vector and the product preference vector, which is used to characterize the degree of deviation between the two in various representation dimensions; This represents the vector concatenation operation; The representation layer normalization operation is used to unify the scale of the concatenated joint features. Represents a scoring network, with parameters as follows The linear mapping layer is implemented, with the input being the user-item joint preference representation, element-wise interaction features and element-wise difference features, and the output being a three-dimensional unnormalized score for weak, medium and strong preference levels respectively. This represents the probability normalization operation; This represents the predicted probability distribution of the current user's preference for the i-th item as weak, medium, or strong, and serves as the result of preference representation.

[0033] After the probability distribution is processed by argmax, the level with the highest probability is taken as the final preference level.

[0034] In this embodiment, the Focal Loss function with inverse frequency category weights is introduced to perform supervised training of the network modules.

[0035] In this embodiment, ChatGLM3-6B was used as the baseline model. Quantification metrics included accuracy, macro-average precision, macro-average recall, and macro-average F1 score. To analyze the impact of each core module on preference quantification performance, this invention conducted module ablation experiments on an agricultural e-commerce dataset. The experimental setup included the baseline model, variant models that individually incorporated QCGF, DPGR, and DPQ, and a complete model integrating all modules. By comparing the performance of different modules, their independent contributions to cross-modal fusion, preference interaction modeling, and product structure enhancement can be evaluated, as shown in Table 2. Table 2 Ablation Experiment Results

[0036] As shown in Table 2, the results demonstrate that the proposed model exhibits high classification accuracy in preference quantification identification. The baseline model achieved an accuracy of 72.62%, a macro-average F1 score of 37.88%, a macro-average precision of 54.43%, and a macro-average recall of 38.38%. After introducing QCGF, the accuracy improved to 97.26%, the macro-average F1 score to 93.82%, the macro-average precision to 93.27%, and the macro-average recall to 93.91%, indicating that query-guided cross-modal gating fusion effectively aligns trimodal information through sample adaptive gating weighting. When DPQ was introduced alone, the accuracy was 94.93%, the macro-average F1 score was 87.86%, and the macro-average precision was 88.11. The macro average recall rate was 87.66%, while the accuracy rate was 94.47%, the macro average F1 score was 87.12%, the macro average precision was 86.25%, and the macro average recall rate was 88.04% when DPGR was introduced alone. This indicates that the dynamic preference quantification module and the dynamic product graph reasoning module each make independent contributions to preference classification. The complete solution of this invention achieves an accuracy rate of 97.32%, a macro average F1 score of 94.34%, a macro average precision of 93.33%, and a macro average recall rate of 95.49%, which is better than the baseline model and the baseline model that introduces QCGF alone. This demonstrates the effectiveness of the synergistic combination of the query-guided cross-modal gating fusion module, the dynamic product graph reasoning module, and the dynamic preference quantification module.

[0037] To further explore the performance advantages and disadvantages of this invention compared with current preference quantization models, comparative experiments were conducted on the same dataset. BERT-BiLSTM (Bidirectional Long Short-Term Memory Network based on Transformer bidirectional encoder representation), Bi-SRU (Bidirectional Gated Recurrent Unit), ESAFN (Entity Sensitive Attention and Fusion Network), MVAN (Multi-View Attention Network), ITIN (Graph-Text Interaction Network), VAuLT (Enhanced Visual Language Transformer), BridgeTower (Building Bridges Between Encoders in Vision-Language Representation Learning), and DRF (Distribution-Based Feature Recovery and Fusion Model) were trained under the same experimental environment.

[0038] Accuracy, macro-average precision, macro-average recall, and macro-average F1 score were used as quantitative indicators, and the comparison results are shown in Table 3: Table 3 Comparison Results

[0039] As shown in Table 3, the macro-average precision of this invention reaches 93.33%, which is significantly better than BERT-BiLSTM, Bi-SRU, ESAFN, MVAN, ITIN, VAuLT, BridgeTower, and DRF. The macro-average F1 score of this invention reaches 94.34%, and the accuracy of the model is 97.32%, which is better than all the comparison models. In contrast, the second-best model, Bi-SRU, has a macro-average F1 score of only 74.18% and an accuracy of only 86.24%. The results prove that this invention can accurately predict user preference levels under the same multimodal dataset.

[0040] This invention also provides a multimodal fusion-based agricultural product preference characterization system, comprising: The multimodal data acquisition and encoding module is used to collect user review text, product main images and customer review images, and historical transaction behavior data from agricultural e-commerce platforms, construct a multimodal dataset, and encode the user review text, product main images and customer review images, and historical transaction behavior data in the multimodal dataset into text sequences, image sequences, and behavior sequences, respectively. The query-guided cross-modal gating fusion module is used to receive the text sequence, image sequence and behavior sequence, perform cross-modal attention calculation and adaptive gating weighting through learnable fusion query vectors, and output the initial features of the product node after cross-modal fusion. The dynamic product graph reasoning module is set in a specific layer of the Transformer encoder. It is used to dynamically construct a neighbor graph based on the cosine similarity of product node features, enhance the graph structure of the initial features of the product nodes after cross-modal fusion, and output the enhanced features of the product nodes. The dynamic preference quantization module is used to map the enhanced features of the product node and the global semantic representation into product attribute preference feature vectors and user preference feature vectors, respectively, and generate a user-product joint preference representation through bidirectional interactive feature modeling. The preference level classification module is used to input the user-item joint preference representation into the scoring network and output the probability distribution of three levels of user preference intensity (weak, medium, and strong) as the preference representation result.

[0041] The present invention also provides an electronic device, comprising: a processor and a memory, wherein the memory stores instructions executable by the processor, and the processor is configured to execute the instructions such that the electronic device can implement the aforementioned method for representing agricultural product preferences based on multimodal fusion.

[0042] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for representing agricultural product preferences based on multimodal fusion, characterized in that, Includes the following steps: Step 1: Construct a large language model that includes a Transformer encoder, a query-guided cross-modal gating fusion module, a dynamic product graph reasoning module, and a dynamic preference quantification module; collect and process user review texts, product main images and customer review images, and historical transaction behavior from e-commerce platforms to construct a multimodal dataset; Step 2: Encode the user review text, product main image, and customer review image in the multimodal dataset with historical transaction behavior to obtain text sequence, image sequence, and behavior sequence respectively; Step 3: Input the text sequence, image sequence, and behavior sequence into the query-guided cross-modal gating fusion module, perform cross-modal attention calculation, and obtain the initial features of the product node after cross-modal fusion; Step 4: Input the initial features of the product nodes after cross-modal fusion into the Transformer encoder. The Transformer encoder is equipped with a dynamic product graph inference module. The enhanced features of the product nodes are obtained through the dynamic product graph inference module. After processing by the Transformer encoder, the hidden features containing multimodal context information are obtained. Perform masked weighted average pooling on the effective text feature vectors in the hidden features containing multimodal contextual information to obtain the global semantic representation; Step 5: Input the enhanced features of the product nodes and the global semantic representation into the dynamic preference quantization module. Map the enhanced features of the product nodes and the global semantic representation to generate product attribute preference feature vectors and user preference feature vectors, respectively. Through bidirectional interactive feature modeling, obtain the user-product joint preference representation. Input the user-product joint preference representation into the rating network and output the three-level user preference intensity classification probability distribution of weak, medium and strong as the preference representation result.

2. The method for representing agricultural product preferences based on multimodal fusion according to claim 1, characterized in that, In step 2, the user review text in the multimodal dataset is processed by the embedding layer of the large language model to obtain a text sequence; the main product image and the customer review image are input into the CLIP-ViT encoder to obtain an image sequence; and the historical transaction behavior is processed by normalization and multilayer perceptron to obtain a behavior sequence.

3. The method for representing agricultural product preferences based on multimodal fusion according to claim 1, characterized in that, In step 3, the text sequence, image sequence, and behavior sequence are input into the query-guided cross-modal gating fusion module to perform cross-modal attention calculation, obtaining the initial features of the product node after cross-modal fusion: ; in, This represents the initial features of the i-th product node after cross-modal fusion; Presentation layer normalization operation; This represents the learnable fusion query vector built into the query-guided cross-modal gating fusion module; , and These represent text sequences, image sequences, and action sequences, respectively. and They respectively represent queries using the fused query vector and queries using the first... Cross-modal attention extraction results for the rth modality features as keys and values; The weight vector representing the gating score calculation is obtained by performing an inner product operation with the hidden layer features after tanh transformation; , , , Indicates the parameters for calculating modal weights; This represents a nonlinear transformation.

4. The method for representing agricultural product preferences based on multimodal fusion according to claim 1, characterized in that, In step 4, the initial features of the product nodes after cross-modal fusion are mapped to multimodal prefix features with the same dimension as the hidden layer of the large language model. The multimodal prefix features are then concatenated as prefix markers at the beginning of the text sequence to form the input sequence of multimodal information. Subsequently, the input sequence of multimodal information is input into the Transformer encoder to obtain hidden features containing multimodal contextual information.

5. The method for representing agricultural product preferences based on multimodal fusion according to claim 1, characterized in that, In step 4, the initial features of the product nodes after cross-modal fusion are input into the dynamic product graph inference module: ; in, Enhance the feature matrix of the (l+1)th layer product node in the c-th relation channel; For the first In the first relationship channel Product node characteristics of the layer; Indicates the first An adjacency matrix of relational channels; Indicates the first Layer Random feature mask in each relation channel; This represents the element-wise multiplication operation; Indicates the first The first relationship channel in the _th relationship channel Layer graph convolution operation; This represents a structure-aware neighborhood aggregation operation. Represents the residual fusion coefficients of the adaptive aggregation result; Presentation layer normalization operation.

6. The method for representing agricultural product preferences based on multimodal fusion according to claim 1, characterized in that, In step 5, bidirectional interactive feature modeling is performed, specifically including concatenating the user preference vector and the product preference vector to obtain the first... User-item joint preference representation for each item: ; in, Let f be the enhanced feature matrix for the (l+1)th layer product node in the c-th relation channel, where c∈{1,2,..., C} represents different types of product relation views; This indicates a user-side weighted pooling operation; For validity mask; Weights for text nodes; The parameter is User preference quantization mapping operation; Indicates the first Calculation of selecting individual product nodes; The parameter is Product preference mapping operation; This represents the vector concatenation operation; Indicates the first User-product joint preference representation for each product.

7. The method for representing agricultural product preferences based on multimodal fusion according to claim 1, characterized in that, In step 5, the user-item joint preference representation is input into the rating network, which outputs a probability distribution of weak, medium, and strong user preference intensity classifications: ; in, For the obtained first User-product joint preference representation for each product; For user preference vectors; For the first A product preference vector for each product; This represents the element-wise multiplication operation; This represents the element-wise absolute difference between the user preference vector and the product preference vector. This represents the vector concatenation operation; Representation layer normalization operation; Represents a scoring network; This represents the probability normalization operation; Indicates the first Probability distribution of user preference intensity for a product at three levels: weak, medium, and strong.

8. The method for representing agricultural product preferences based on multimodal fusion according to claim 1, characterized in that, It also includes supervised training of network modules using the Focal Loss function, which introduces inverse frequency category weights.

9. A multimodal fusion-based agricultural product preference characterization system, characterized in that, include: The multimodal data acquisition and encoding module is used to collect user review text, product main images and customer review images, and historical transaction behavior data from agricultural e-commerce platforms, construct a multimodal dataset, and encode the user review text, product main images and customer review images, and historical transaction behavior data in the multimodal dataset into text sequences, image sequences, and behavior sequences, respectively. The query-guided cross-modal gating fusion module is used to receive the text sequence, image sequence and behavior sequence, perform cross-modal attention calculation and adaptive gating weighting through learnable fusion query vectors, and output the initial features of the product node after cross-modal fusion. The dynamic product graph reasoning module is set in a specific layer of the Transformer encoder. It is used to dynamically construct a neighbor graph based on the cosine similarity of product node features, enhance the graph structure of the initial features of the product nodes after cross-modal fusion, and output the enhanced features of the product nodes. The dynamic preference quantization module is used to map the enhanced features of the product node and the global semantic representation into product attribute preference feature vectors and user preference feature vectors, respectively, and generate a user-product joint preference representation through bidirectional interactive feature modeling. The preference level classification module is used to input the user-item joint preference representation into the scoring network and output the probability distribution of three levels of user preference intensity (weak, medium, and strong) as the preference representation result.

10. An electronic device, characterized in that, include: The processor and memory, wherein the memory stores instructions executable by the processor, and the processor is configured to, when executing the instructions, enable the electronic device to implement a method for representing agricultural product preferences based on multimodal fusion as described in any one of claims 1 to 8.