Online review helpfulness prediction method based on hierarchical multi-entity multi-modal fusion

Abstract

The application discloses a hierarchical multi-entity multi-modal fusion-based online review usefulness prediction method, comprising the following steps: obtaining original data to be predicted, pre-processing and feature extraction are performed on multi-modal information to obtain text features, image features and data features; the text features, the image features and the data features are spliced into a first input sequence, which is input into a first-level Transformer encoder to generate review entities, user entities and merchant entities; the review entities, the user entities and the merchant entities are stacked to construct a second input sequence, which is input into a second-level Transformer encoder to generate global context features; and the global context features are input into a deep neural network regression predictor to output review usefulness prediction values. The two-stage Transformer design effectively improves the accuracy and robustness of review usefulness prediction.

Description

Technical Field

[0001] This invention belongs to the field of text information mining technology, specifically involving a method for predicting the usefulness of online comments based on hierarchical multi-entity multimodal fusion. Background Technology

[0002] With the booming development of mobile internet and e-commerce, online reviews have become an important reference for consumers' purchasing decisions and a key feedback channel for businesses to improve their services and products. However, the massive amount of user-generated content (UGC) has led to a serious problem of information overload, making it difficult for consumers to sift through thousands of reviews to find truly valuable information. Therefore, automated review helpfulness prediction technology has emerged.

[0003] Existing technologies have the following main drawbacks when predicting the usefulness of comments: (1) Insufficient utilization of modalities: Early studies were mostly limited to text modalities, ignoring the intuitive visual information brought by the comment images, or simply using a splicing method to integrate multimodal data, which could not effectively align the semantic space of text and images.

[0004] (2) Lack of entity interaction modeling: Existing models usually analyze the review content itself in isolation, ignoring the deep internal connection between the "user's" credibility, the attributes of the "Restaurant / Product", and the "Review" content.

[0005] (3) Imbalance in heterogeneous feature dimensions: When fusing text (usually high-dimensional sparse vectors), images (high-dimensional dense vectors) and numerical values ​​(very low-dimensional statistical features, such as ratings and time), low-dimensional numerical features are easily overwhelmed by high-dimensional features, resulting in the loss of key statistical information and affecting prediction accuracy. Summary of the Invention

[0006] To address the aforementioned shortcomings in existing technologies, this invention provides an online review usefulness prediction method based on hierarchical multi-entity multimodal fusion. By constructing a multimodal input sequence containing review, user, and merchant entities and designing a hierarchical Transformer architecture, it prioritizes mining internal modal interactions within entities and then performs cross-entity global modeling, thereby significantly improving the accuracy of usefulness prediction. This solves the problems of insufficient modal utilization, missing entity interaction modeling, and imbalance of heterogeneous feature dimensions in existing technologies for review usefulness prediction.

[0007] To achieve the aforementioned objectives, the technical solution adopted by this invention is: an online review usefulness prediction method based on hierarchical multi-entity multimodal fusion, comprising the following steps: S1. Obtain the raw data to be predicted, including multimodal information of comment entities, user entities, and merchant entities; S2. Preprocess and extract features from multimodal information to obtain text features, image features, and data features; S3. Concatenate the text features, image features, and data features into the first input sequence, and input it into the first-level Transformer encoder to generate aggregated features of a single entity, including comment entity, user entity, and merchant entity; S4. Stack the comment entity, user entity, and merchant entity to construct a second input sequence, and input it into the second-level Transformer encoder to generate global contextual features; S5. Input the global context features into the deep neural network regression predictor and output the predicted value of the comment usefulness.

[0008] Furthermore, S2 includes the following sub-steps: S21. Obtain multimodal information, which includes text data, image data, and numerical data. Preprocess the text data, image data, and numerical data respectively to obtain a token sequence set, standard image data, and standard numerical data respectively. S22. Based on the token sequence set, standard image data, and standard numerical data, feature extraction is performed to obtain text features, image features, and numerical features, respectively.

[0009] Furthermore, in S21, the methods for preprocessing text data include syntactic-aware cleaning of unstructured text and serialization based on overlapping sliding windows; Methods for preprocessing image data include standardization and tensor quantization of unstructured images; Methods for preprocessing numerical data include long-tailed distribution smoothing and distribution standardization.

[0010] Furthermore: In S22, the method for feature extraction based on the token sequence set is as follows: The set of token sequences is input into the CLIP model, the CLIP model's tokenizer is used for word segmentation, and the text features are extracted through the CLIP Text Encoder. The specific method for feature extraction based on standard image data is as follows: Standard image data is input into the CLIP model, and visual features of the image are extracted using the CLIP Image Encoder. The specific method for feature extraction based on standard numerical data is as follows: Standard numerical data is input into a residual projection network for numerical feature enhancement. The residual projection network consists of a linear projection layer, a layer normalization layer, and an activation function layer connected in sequence. Residual connections are introduced to add the input features to the transformed features to prevent information loss when low-dimensional features are mapped to high-dimensional space.

[0011] The beneficial effects of the above-mentioned further solutions are as follows: This invention introduces the CLIP model as a feature extractor, utilizing its zero-shot transfer capability to achieve deep alignment of text and images in the semantic space, which is superior to simple vector concatenation. By designing an adaptive residual dimensionality-upgrading network with layered normalization layers and residual connections, the problem of low-dimensional numerical features being easily submerged in deep networks in traditional methods is effectively solved, thus improving the robustness of features.

[0012] Furthermore: In S3, the first-level Transformer encoder consists of several identical encoder layers stacked together. Each layer includes interconnected multi-head self-attention mechanism sublayers and feedforward neural network sublayers, with addition and normalization operations set after each sublayer.

[0013] The beneficial effects of the above-mentioned further solutions are as follows: This invention adopts a hierarchical Transformer architecture, which conforms to the human cognitive process, that is, first understanding individual information (single entity fusion), and then comprehensively judging the relationships between multiple parties (cross-entity fusion). Experiments show that this architecture can effectively reduce prediction error (MSE) and improve the generalization ability of the model.

[0014] Furthermore: In S3, the workflow of the first-level Transformer encoder is as follows: A1. Input the first input sequence into the embedding layer, and convert it into a continuous high-dimensional vector representation through the embedding layer. Add position encoding to the high-dimensional vector representation to obtain the input representation with position encoding. A2. Input the input representation sequentially to all stacked encoder layers, stack the outputs of each encoder layer, and generate aggregated features for a single entity.

[0015] Furthermore: In A2, the specific workflow of each encoder layer is as follows: A21. Input the input features of the encoder layer into the multi-head self-attention mechanism sub-layer. The attention is calculated in parallel by multiple attention heads. Each attention head obtains the query, key and value respectively through different linear transformation layers. The outputs of all attention heads are concatenated and then fused through a linear layer to obtain the attention output vector. The attention output vector is added to the input features and then normalized to serve as the output of the multi-head self-attention mechanism sub-layer. A22. Input the output of the multi-head self-attention mechanism sublayer into the feedforward neural network, which includes two fully connected layers. Use the activation function to perform non-linear mapping to obtain the output of the feedforward neural network. Add the output of the feedforward neural network to the input and then perform normalization to get the output of the encoder layer.

[0016] Furthermore, in S5, the deep neural network regression predictor includes a linear bottleneck layer, a multilayer perceptron, and an output layer. The multilayer perceptron includes the LeakyReLU activation function and the Dropout regularization operation.

[0017] The beneficial effects of this invention are as follows: (1) This invention provides an online comment usefulness prediction method based on hierarchical multi-entity multimodal fusion, which takes into account both peripheral and core factors affecting comment usefulness. It systematically integrates multimodal data of three key entities: information subject (user), information carrier (comment content), and information object (merchant service or product). This breaks through the limitations of traditional research that only considers limited entities (such as the comment content itself) or limited modal combinations (such as using only comment text, comment images, or a simple combination of the two), and solves the problem of insufficient modal utilization in existing technologies when predicting comment usefulness. This invention systematically examines the mechanism of action of diversified information on comment usefulness prediction from multiple dimensions, which not only provides a more comprehensive theoretical framework for comment usefulness prediction, but also deepens the understanding of the synergistic effect of diversified information in complex prediction tasks, and provides valuable reference for subsequent research.

[0018] (2) To address the challenges of feature fusion for multi-entity and multi-modal data, this invention adopts a two-stage Transformer design. The first stage (Intra-Entity Transformer) focuses on fine-grained interactions of text, images, and numerical values ​​within an entity; the second stage (Inter-Entity Transformer) focuses on global contextual reasoning among the three entities: reviewers, users, and merchants. This design prioritizes capturing high-order interaction relationships between different modal data within the same entity, then performs cross-entity modal feature fusion, and finally achieves effective integration of global information across multiple entities, thereby significantly improving the model's representation ability and prediction performance. To a certain extent, it overcomes the difficulties in aligning modal features and insufficient capture of inter-modal dependencies caused by overly simplistic or complex modal fusion methods. It also solves the problems of missing entity interaction modeling and imbalance of heterogeneous feature dimensions in existing technologies when predicting the usefulness of reviews. Attached Figure Description

[0019] Figure 1 This is a flowchart of the online comment usefulness prediction method based on hierarchical multi-entity multimodal fusion of the present invention.

[0020] Figure 2 This is a model architecture diagram of the present invention. Detailed Implementation

[0021] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0022] like Figure 1 As shown, in one embodiment of the present invention, the online review usefulness prediction method based on hierarchical multi-entity multimodal fusion includes the following steps: S1. Obtain the raw data to be predicted, including multimodal information of the Review entity, User entity, and Restaurant entity; S2. Preprocess and extract features from multimodal information to obtain text features, image features, and data features; S3. Concatenate the text features, image features, and data features into the first input sequence, and input it into the first-level Transformer encoder to generate aggregated features of a single entity, including comment entity, user entity, and merchant entity; S4. Stack the comment entity, user entity, and merchant entity to construct a second input sequence, and input it into the second-level Transformer encoder to generate global contextual features; S5. Input the global context features into the deep neural network regression predictor and output the predicted value of the comment usefulness.

[0023] like Figure 2 As shown, this invention acquires multimodal data of text, images, and numerical values ​​from three types of entities—reviews, users, and merchants—on an online consumption platform. The multimodal data is preprocessed, and a CLIP model (contrastive language-image pre-trained model) is used to extract high-dimensional semantic features of the text and images. Low-dimensional numerical features are enhanced using a residual projection network. After mapping the heterogeneous features to a unified feature space, a first-level Transformer encoder is used for modal fusion within a single entity, followed by a second-level Transformer encoder for global feature fusion across entities. Finally, a deep neural network is used to output a predicted value for the usefulness of the review. Through a multi-layered feature interaction mechanism, the problem of information loss in heterogeneous fusion of multimodal data is effectively solved, significantly improving the accuracy and robustness of the prediction of the usefulness of the review.

[0024] In S1, this embodiment selects a public dataset from a typical online consumer platform (such as Yelp) as the raw data to be predicted as input to the system. The data covers unstructured text, image data, and structured numerical data.

[0025] S2 includes the following steps: S21. Obtain multimodal information, which includes text data, image data, and numerical data. Preprocess the text data, image data, and numerical data respectively to obtain a token sequence set, standard image data, and standard numerical data respectively. S22. Based on the token sequence set, standard image data, and standard numerical data, feature extraction is performed to obtain text features, image features, and numerical features, respectively.

[0026] In S21, the methods for preprocessing text data include syntactic-aware cleaning of unstructured text and serialization based on overlapping sliding windows. In this embodiment, for multi-source text data including review text, user nickname, and merchant tags, the present invention uses the following method to extract a set of token sequences: (1) Syntactic-Aware Cleaning of Unstructured Text. Since the CLIP model's text encoder is based on the Transformer architecture, its pre-training process relies on the contextual dependencies of natural language and is highly sensitive to syntactic structure. This is fundamentally different from the traditional Bag-of-Words (BoW) model's mechanism of relying on keyword co-occurrence. Therefore, this invention abandons the traditional high-intensity stop word filtering strategy and instead adopts a syntactic-aware cleaning strategy: 1) Noise Filtering: Uses regular expressions to remove HTML tags, URL hyperlinks, and non-ASCII characters, removing only formatting noise that disrupts text encoding.

[0027] 2) Syntax Preservation: Considering that stop words such as prepositions and conjunctions play an important syntactic guiding role in the Transformer's self-attention mechanism, this invention fully preserves stop words and basic punctuation marks. This strategy aims to maintain the natural language manifold structure of the text, aligning it to the distribution features of the CLIP pre-training data as closely as possible, and avoiding semantic structure fragmentation due to excessive cleaning.

[0028] (2) Text sequence encoding based on overlapping sliding window. To address the problem that the length of online review text often exceeds the context window limit of the CLIP model (Context Length = 77 tokens), this study designed an "Overlapping Sliding Window" mechanism to replace the traditional information-lossy truncation strategy and ensure the integrity of the semantics of long texts.

[0029] 1) Chunking Strategy: Set the window size W = 77 and the sliding step S = 50. Since S < W, approximately 35% of the context overlap is retained between adjacent windows. Move this window along the text sequence to divide the ultra-long text into a set of N subsequences with overlapping contexts. Among them, for long text reviews, N > 1; for short texts such as user nicknames, it adapts to N = 1. This mechanism can effectively avoid the problem of boundary semantic loss caused by hard truncation.

[0030] 2) Chunk-and-Aggregate: Input the generated set of subsequences into the CLIP text encoder in parallel. For each subsequence, extract the hidden layer vector corresponding to the (End-of-Text) token at its end as the local semantic representation of this window, and then use the mean pooling strategy to fuse all local representations. While retaining the global information of the long text, this method enhances the robustness of local key features through overlapping sampling, and also accommodates the feature extraction of short texts, ensuring the isomorphism of the multi-source text feature space.

[0031] 3) Multi-source text independent stacking strategy: For merchant entities, since their feature attributes contain two groups of text information, "merchant name" and "category label", which have highly semantic complementarity but are distributed independently, simple vector addition or concatenation flattening will blur the semantic boundaries between the two. Therefore, this study adopts a text feature sequence combination strategy of dual-channel parallel encoding. Specifically, use the above general encoding mechanism to extract the merchant name representation and category representation respectively, and then perform a stacking operation in the sequence dimension to construct a text feature sequence containing two independent semantic tokens to retain the complete semantic reference information.

[0032] The methods for preprocessing image data include the standardization and tensorization of unstructured images. In this embodiment, the methods for preprocessing image data are specifically as follows: (1) Standardization and tensorization of unstructured images. To address the problems of different resolutions and aspect ratios of the images uploaded by users, in order to adapt to the fixed input dimensions of the CLIP visual encoder, this invention constructs a standardized visual preprocessing pipeline, and the specific content is as follows: 1) Geometric transformation: First, the short side of the image is scaled to 224 pixels using the bicubic interpolation algorithm. Compared with traditional bilinear interpolation, this strategy uses neighborhood pixels for cubic convolution, which can more effectively suppress aliasing artifacts and retain the high-frequency texture features of the image, ensuring consistency with the data distribution in the CLIP pre-training stage. Subsequently, center cropping is performed to obtain a square region of interest (ROI). This strategy not only unifies the input resolution but also maximally preserves the structural integrity of the main body of the image.

[0033] 2) Pixel-level normalization: Based on the statistical priors of the large-scale ImageNet dataset, Z-Score normalization is performed on the three RGB channels. Specifically, after projecting the pixel values into the interval [0,1], subtract the channel means μ = [0.481, 0.457, 0.408], and divide by the standard deviations σ = [0.268, 0.261, 0.275]. This step aims to eliminate the dimensional differences in illumination and contrast, accelerate the convergence of the visual encoder, and improve the performance of transfer learning.

[0034] (2) Multi-image batch visual coding and feature aggregation. Since there are natural differences in the number of images contained in comments, merchants, and user entities, such as comments often containing multiple real-shot images while users only have a single profile picture, in order to make full use of multi-perspective visual information and address the dimensional alignment problem of variable-length sequence inputs of images from different entities, this study adopts a visual feature sequence combination strategy of multi-perspective parallel coding.

[0035] 1) Different visual input window thresholds L are set for different types of entities. For comment and merchant entities, L = 3 is set to capture rich multi-perspective details; for user entities, L = 1 is set to only retain the core user portrait features.

[0036] 2) For the differences in the number of images N between samples, a standardized preprocessing process is implemented: when the actual number of pictures N > L, the first N high-confidence images are intercepted; when the actual number of pictures N < L, all-zero tensors are introduced to fill in the missing positions to ensure the dimensional consistency of the input tensors.

[0037] 3) At the feature extraction level, to avoid the confusion and loss of key local visual semantics (such as close-ups of dishes, environmental atmosphere, etc.) that may be caused by traditional average pooling or feature flattening, this study strictly maintains visual tokens as independent sequence units during the feature extraction stage. Specifically, firstly, the CLIP visual encoder is used to perform feature mapping on the original image set in parallel to obtain a set of 512-dimensional global representation vectors. Then, the feature vector set is stacked in the sequence dimension to construct a multi-view visual feature tensor of the entity. This strategy ensures that the model can aggregate visual cues from different perspectives, without compressing multiple images into a single vector, but rather constructing it into a visual feature sequence of length . This enables the subsequent Transformer model to dynamically capture the intrinsic relationships between images from different perspectives through a multi-head self-attention mechanism (e.g., the model can focus more on "dish images" than "menu images"), thus building a semantically complete and fine-grained visual representation foundation for multimodal fusion.

[0038] Methods for preprocessing numerical data include long-tailed distribution smoothing and distribution standardization. In this embodiment, the specific method for preprocessing numerical data is as follows: (1) Long-tail distribution smoothing: For features such as the number of usefulness votes for comments and the number of user followers exhibiting significant long-tail distribution characteristics, a smoothing method is adopted. Log1p By changing the dynamic range of the compressed variable, the impact of extreme right skewness of the data on gradient updates can be mitigated.

[0039] (2) Distribution standardization: Differentiated processing is carried out for features with different physical dimensions: StandardScaler is used to standardize features that are approximately normally distributed (such as scores) using Z-Score (mean is 0, variance is 1); MinMaxScaler is used to normalize bounded features (such as years, counts) to the [0, 1] interval to eliminate dimensional differences.

[0040] In S22, the method for feature extraction based on the token sequence set is as follows: The set of token sequences is input into the CLIP model, and the CLIP model's tokenizer is used for word segmentation. The "overlapping sliding window" mechanism is used to replace the traditional information loss truncation strategy to ensure the semantic integrity of long texts. Text features are extracted through CLIPText Encoder. In this embodiment, the generated token sequence set is processed using CLIP's Text Encoder interface to extract a high-dimensional text feature vector (512 dimensions). For the multi-segment sub-sequences segmented by the sliding window, pooling operations are further used to aggregate them into global text features.

[0041] The specific method for feature extraction based on standard image data is as follows: Standard image data is input into the CLIP model, and visual features of the image are extracted using the CLIP Image Encoder. In this embodiment, the preprocessed image tensor is processed using CLIP's Image Encoder interface to extract high-dimensional visual feature vectors (512 dimensions). With the help of CLIP's large-scale contrastive learning pre-trained weights, this invention achieves implicit alignment between text and image in the shared semantic space during the feature extraction stage. Compared with the traditional independent extraction schemes of BERT and ResNet, it has stronger cross-modal semantic association capabilities.

[0042] The specific method for feature extraction based on standard numerical data is as follows: Standard numerical data is input into a residual projection network for numerical feature enhancement. The residual projection network consists of a linear projection layer, a layer normalization layer (LayerNorm), and an activation function layer connected in sequence. Residual connections are introduced to add the input features to the transformed features to prevent information loss when low-dimensional features are mapped to high-dimensional space.

[0043] In this embodiment, a dedicated Adaptive Residual MLP is designed to encode the numerical statistical features of comment entities (such as those containing only duration and rating) and user entities with extremely low dimensionality (such as 2-dimensional or 7-dimensional). The specific structure is as follows: Linear projection layer: For the input data, a linear layer is used to map it to an intermediate dimension (such as 64 dimensions) to avoid information sparsity and feature representation degradation caused by direct mapping to a high-dimensional space.

[0044] Layer normalization and nonlinear activation: Layer normalization (LayerNorm) and the GELU activation function are applied sequentially. LayerNorm is chosen instead of BatchNorm to ensure statistical stability in small-batch training scenarios and prevent model oscillations caused by fluctuations in batch statistics.

[0045] Residual connection mechanism: The identity shortcut is introduced to add the linearly transformed features to the original features (after dimensional adaptation). This design constructs a high-speed information pathway, ensuring that the original weak statistical features are not lost in multi-layer nonlinear transformations, and significantly improving the model's ability to capture low-dimensional numerical information.

[0046] In S3, text features (512-dimensional), image features (512-dimensional), and data features are projected onto a unified target dimension (e.g., 128 or 256) space through a fully connected layer, thereby achieving feature fusion of multi-entity and multi-modal data.

[0047] In S3, the first-level Transformer encoder (Intra-Entity Transformer) consists of several identical encoder layers stacked together. Each layer includes interconnected multi-head self-attention sublayers and feed-forward neural network sublayers, with addition and normalization operations set after each sublayer.

[0048] The first-level Transformer encoder contains multiple stacked multi-head self-attention mechanism layers, with the feedforward network expansion coefficient set to 4, and Dropout (e.g., 0.1) introduced to prevent overfitting.

[0049] In S3, the workflow of the first-level Transformer encoder is as follows: A1. Input the first input sequence into the embedding layer, and convert it into a continuous high-dimensional vector representation. Add position encoding to the high-dimensional vector representation to obtain the input representation with position encoding. A2. Input the input representation sequentially to all stacked encoder layers, stack the outputs of each encoder layer, and generate aggregated features for a single entity.

[0050] In this embodiment, the first-level Transformer encoder utilizes a multi-head attention mechanism to calculate the correlation strength between text, images, and numerical values ​​within the same entity. For example, the model can automatically learn the strong correlation between "negative review text" and "blurry photo".

[0051] In A2, the workflow of each encoder layer is as follows: A21. Input the input features of the encoder layer into the multi-head self-attention mechanism sub-layer. The attention is calculated in parallel by multiple attention heads. Each attention head obtains the query, key, and value respectively through different linear transformation layers. The outputs of all attention heads are concatenated and then fused through a linear layer to obtain the attention output vector. The attention output vector is added to the input features and then normalized to serve as the output of the multi-head self-attention mechanism sub-layer. A22. Input the output of the multi-head self-attention mechanism sublayer into the feedforward neural network, which includes two fully connected layers and uses an activation function (such as ReLU) for nonlinear mapping to obtain the output of the feedforward neural network. Add the output of the feedforward neural network to the input and then perform a normalization operation to obtain the output of the encoder layer.

[0052] In S4, the second-level Transformer encoder (Inter-Entity Transformer) is used for feature fusion across different entities. The second-level Transformer encoder consists of six stacked modules, each with an eight-head multi-head self-attention mechanism. The workflow of the multi-head self-attention mechanism is similar to that of the first-level Transformer encoder. The input to the second-level Transformer encoder is a second input sequence. Based on the multi-head self-attention mechanism, the second-level Transformer encoder captures high-order dependencies between entities, achieving feature fusion across different entities. For example, it captures the specific usefulness patterns implied in long-text reviews posted by "high-reputation users" on "low-rated restaurants." This step achieves deep coupling between the information subject, information carrier, and information object.

[0053] In S5, the deep neural network regression predictor (DNN Predictor) includes a linear bottleneck layer, a multilayer perceptron (MLP), and an output layer. The multilayer perceptron includes the LeakyReLU activation function (with a negative slope of 0.01) and the Dropout regularization operation.

[0054] In this embodiment, the present invention uses a deep neural network (DNN) as the core predictor. The designed DNN structure includes several fully connected layers, each employing the LeakyReLU activation function, and incorporates a Dropout operation to prevent overfitting. The global contextual features obtained above are input into the DNN to predict the number of usefulness votes for comments.

[0055] In the description of this invention, the above are merely preferred embodiments and are not intended to limit the scope of protection of this invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for predicting the usefulness of online reviews based on hierarchical multi-entity multimodal fusion, characterized in that, Includes the following steps: S1. Obtain the raw data to be predicted, including multimodal information of comment entities, user entities, and merchant entities; S2. Preprocess and extract features from multimodal information to obtain text features, image features, and data features; S3. Concatenate the text features, image features, and data features into the first input sequence, and input it into the first-level Transformer encoder to generate aggregated features of a single entity, including comment entity, user entity, and merchant entity; S4. Stack the comment entity, user entity, and merchant entity to construct a second input sequence, and input it into the second-level Transformer encoder to generate global contextual features; S5. Input the global context features into the deep neural network regression predictor and output the predicted value of the comment usefulness.

2. The online review usefulness prediction method based on hierarchical multi-entity multimodal fusion according to claim 1, characterized in that, S2 includes the following steps: S21. Obtain multimodal information, which includes text data, image data, and numerical data. Preprocess the text data, image data, and numerical data respectively to obtain a token sequence set, standard image data, and standard numerical data respectively. S22. Based on the token sequence set, standard image data, and standard numerical data, feature extraction is performed to obtain text features, image features, and numerical features, respectively.

3. The online review usefulness prediction method based on hierarchical multi-entity multimodal fusion according to claim 2, characterized in that, In S21, the methods for preprocessing text data include syntactic-aware cleaning of unstructured text and serialization based on overlapping sliding windows. Methods for preprocessing image data include standardization and tensor quantization of unstructured images; Methods for preprocessing numerical data include long-tailed distribution smoothing and distribution standardization.

4. The online review usefulness prediction method based on hierarchical multi-entity multimodal fusion according to claim 2, characterized in that, In S22, the method for feature extraction based on the token sequence set is as follows: The set of token sequences is input into the CLIP model, the CLIP model's Tokenizer is used for word segmentation, and the text features are extracted through the CLIPText Encoder. The specific method for feature extraction based on standard image data is as follows: Standard image data is input into the CLIP model, and visual features of the image are extracted using the CLIP Image Encoder. The specific method for feature extraction based on standard numerical data is as follows: Standard numerical data is input into a residual projection network for numerical feature enhancement. The residual projection network consists of a linear projection layer, a layer normalization layer, and an activation function layer connected in sequence. Residual connections are introduced to add the input features to the transformed features to prevent information loss when low-dimensional features are mapped to high-dimensional space.

5. The online review usefulness prediction method based on hierarchical multi-entity multimodal fusion according to claim 1, characterized in that, In S3, the first-level Transformer encoder consists of several identical encoder layers stacked together. Each layer includes interconnected multi-head self-attention mechanism sub-layers and feedforward neural network sub-layers, with addition and normalization operations set after each sub-layer.

6. The online review usefulness prediction method based on hierarchical multi-entity multimodal fusion according to claim 5, characterized in that, In S3, the workflow of the first-level Transformer encoder is as follows: A1. Input the first input sequence into the embedding layer, and convert it into a continuous high-dimensional vector representation through the embedding layer. Add position encoding to the high-dimensional vector representation to obtain the input representation with position encoding. A2. Input the input representation sequentially to all stacked encoder layers, stack the outputs of each encoder layer, and generate aggregated features for a single entity.

7. The online review usefulness prediction method based on hierarchical multi-entity multimodal fusion according to claim 6, characterized in that, In A2, the workflow of each encoder layer is as follows: A21. Input the input features of the encoder layer into the multi-head self-attention mechanism sub-layer. The attention is calculated in parallel by multiple attention heads. Each attention head obtains the query, key and value respectively through different linear transformation layers. The outputs of all attention heads are concatenated and then fused through a linear layer to obtain the attention output vector. The attention output vector is added to the input features and then normalized to serve as the output of the multi-head self-attention mechanism sub-layer. A22. Input the output of the multi-head self-attention mechanism sublayer into the feedforward neural network, which includes two fully connected layers. Use the activation function to perform non-linear mapping to obtain the output of the feedforward neural network. Add the output of the feedforward neural network to the input and then perform normalization to get the output of the encoder layer.

8. The online review usefulness prediction method based on hierarchical multi-entity multimodal fusion according to claim 1, characterized in that, In S5, the deep neural network regression predictor includes a linear bottleneck layer, a multilayer perceptron, and an output layer. The multilayer perceptron includes the LeakyReLU activation function and the Dropout regularization operation.

Images