Deep learning method for strengthening key features based on winner effect

Abstract

The invention provides a deep learning method for strengthening key features based on the winner effect, and aims to enhance the extraction capability of a model on the key features and improve the accuracy of multi-modal data prediction by simulating the winner effect. According to the method, by constructing a winner-vanquist effect matrix (WL matrix), the attention weight is dynamically adjusted, and in combination with a self-attention mechanism and a multi-layer neural network, multi-modal learning prediction taking comment usefulness as an example is realized. Experimental results show that the method has significant performance improvement in a multi-modal data prediction task.

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning, and particularly relates to a deep learning method for strengthening key features based on the winner effect, which is particularly applicable to feature extraction and prediction tasks of multimodal data. Background Art

[0002] With the development of big data, a large amount of multimodal data has been accumulated in many industries such as healthcare, power, and business platform reviews in social life. By leveraging the advantages of deep learning technology, implicit feature extraction can be performed on this data, and the comprehensive use of data such as numerical values, texts, pictures, and videos can increase the accuracy of decision-making. Deep learning methods have significant advantages in processing big data of text and picture information. Different from machine learning, in which feature extraction, feature selection, model parameter training and learning, and model prediction are carried out, deep learning can automatically complete feature vector processing and prediction result generation.

[0003] As a research branch of machine learning, using deep learning methods to process multimodal online evaluation data has great advantages. When dealing with these unstructured big data, deep learning methods automatically solve these difficult tasks through parameter learning between hidden layers. Existing deep learning algorithms have achieved remarkable results in fields such as text translation, picture classification, and speech recognition. Recurrent neural networks (such as Lstm and GRU models) can process long text time series information. The BERT model based on the Transformer model using self-attention mechanism can capture the dependencies between overall texts through training with a large-scale corpus and has strong generalization ability, and is the main deep learning tool in many text preprocessing and sentiment analysis. Convolutional neural networks (CNNs) extract local features through convolutional layers, and pooling layers reduce dimensions to capture the complex structure of images. Models such as AlexNet, VGG, and ResNet based on the CNN framework have achieved remarkable results in tasks such as image classification and object detection.

[0004] When using multi-modal data for deep learning model learning and prediction, although multi-modal data provides more information, it also increases the difficulty of feature extraction for various models (such as machine learning and deep learning models) for information. It is necessary to learn high-level features from the limited features that the attention of each hidden layer of deep learning focuses on. The deep learning model based on the attention mechanism is based on this principle. It makes the model focus on some more important parts when learning parameters in each hidden layer, rather than every part of the information contributing to the final decision. For example, in online review data, each person's focus on comments considered useful is different. Mining the attention mechanism in useful comments is more in line with the real human decision-making process. In addition, recent research has found that in the human brain's decision-making process, the determining factors that the brain nerves focus on will enhance the activation degree of this information during the subsequent information transmission process, and there is a "winner effect", which has been confirmed in psychological and physiological research. Therefore, in the deep learning model using the attention mechanism, it is necessary to develop methods that are more in line with consumer decision-making. Summary of the Invention

[0005] To solve the above problems, the present invention discloses a deep learning method for strengthening key features based on the winner effect. By using the winner effect, a learning matrix for enhancing key features is constructed between hidden layers to form a deep learning method for strengthening key features, and the superiority of this method in prediction and decision-making is verified through a multi-modal data set.

[0006] A deep learning method for strengthening key features based on the winner effect includes the following steps:

[0007] Step 1: Construct a WL-EAM model architecture and perform data preprocessing. First, collect multi-modal comments and store them in a database, and preprocess the multi-modal comment data, which includes text data and image data;

[0008] Step 2: For text data, convert the review text into an embedded feature vector through the embeddings of the word embedding model BERT; for image data, perform vectorization representation by extracting the pixel features and RGB space channel features of the image.

[0009] Step 3: For comment image data, use the pre-trained model VGG-16 to convert the picture into an embedded feature vector; use the LSTM model to extract the temporal features of the text and the picture, and capture the temporal dependence relationship in the comment.

[0010] Step 4: Extract the importance weights of each element in the text and picture sequences through the self-attention mechanism to generate an attention weight matrix.

[0011] Step 5: Construct a winner-loser effect matrix to simulate the winner-loser effect. The WL matrix is updated iteratively to record the model's memory of successful experiences and failed lessons during training and dynamically adjust the attention weights;

[0012] Step 6: Verify the effect of the deep learning method with enhanced winner effect features; fuse the feature vectors of text and images, and perform multi-modal comment usefulness prediction through a multi-layer neural network.

[0013] Further, in step 4, three weight matrices are initialized respectively from the sequence embedding features Etext and Eimage obtained from the LSTM layer, including the key matrix K of the input elements, the value matrix V of the learning sequence weights, and the query matrix Q that captures the correlation between the current time step and other time steps; the output vector of the weight features is obtained by equation (1);

[0014]

[0015] In equation (1), first calculate the dot product of the query matrix Q and the key matrix K, then scale it with the dimension of the key matrix K, and then use the softmax activation function to obtain the weights of the value matrix V; taking the element xi in the matrix as an example; the softmax activation function is expressed as equation (2):

[0016]

[0017] where e is the Euler number; by multiplying the weights and the value matrix V, the weighted features, namely the text modality Ztext and the visual modality Zimage, are obtained.

[0018] Further, in step 5, by using the WL matrix as an auxiliary mechanism to adjust self-attention, it helps the model to combine the experience accumulated in previous training during feature extraction, improving the accuracy and robustness of the attention mechanism; specifically including

[0019] Step 51: Extract the attention weight distribution Attention_Weight of the input features through the attention mechanism. Each row in the weight matrix represents the overall representation of a piece of text or an image enhanced by other sequence parts; in the sequence processing task, the magnitude of the attention weight matches the correlation with the contribution degree of the current task;

[0020]

[0021] During the process of training the model, the index positions of the maximum / minimum values in each row of the Attention_Weight matrix are searched and memorized to obtain the index matrices MaxS / MinS; their calculation formulas are shown in Formulas (4) and (5); through the MaxS and MinS matrices, the successful experiences and failure lessons predicted by the model during the learning process are extracted.

[0022]

[0023] The winner-loser effect requires long-term iteration and acts on the current attention mechanism.

[0024] Furthermore, in step 5, an update strategy for the WL matrix is constructed and a rule for its action on the self-attention mechanism is formulated; when the training result is correct, the initial WL matrix is represented by the difference between the initial MaxS and MinS; multiplying the attention weights bit by bit with MaxS / MinS can obtain the weight matrix WL_Attention_Weight adjusted by the winner / loser effect; the calculation formulas are as follows (6)(7):

[0025]

[0026] When the prediction result is correct, MaxS is a favorable factor for the model prediction and MinS is an unfavorable factor.

[0027] Furthermore, by setting the parameter η to represent the degree of inclination of the model towards the winner effect and the loser effect, its range is between 0 and 1. When it is greater than 0.5, it means the model is more inclined to focus on successful experiences. When it is less than 0.5, it means the model is more inclined to focus on failure lessons. When it is equal to 0.5, the two are in an equilibrium state; therefore, the measurement of successful experiences and failure lessons is represented by Formulas (8)(9).

[0028] WL i+1 = WL i + R i WL i ⊙ [ηMaxS i - (1 - η)MinS i , R = 1 (8)

[0029] WL i+1 = WL i + R i WL i ⊙ [(1 - η)MaxS i - ηMinS i , R = 0 (9)

[0030] Furthermore, the self-attention mechanism generates an attention weight matrix through the softmax function, dynamically adjusts the attention weights in combination with the WL matrix, and introduces a dropout parameter θ. The iterative process is shown in formulas (10) and (11).

[0031] WL i+1 = softmax(WL i + dropout(R i WL i ⊙ [ηMaxS i - (1 - η)MinS i , θ)) (10)

[0033] WL i+1 = softmax(WL i + dropout(R i WL i ⊙ [(1 - η)MaxS i - ηMinS i , θ))

[0034] (11).

[0035] Furthermore, before the text is activated by the softmax function, the layer normalization idea is adopted, and the parameter optimization is carried out by the maximum-minimum normalization method. As shown in formulas (11), (12), and (13), where a i_min represents the minimum value of the i-th row of the matrix, and a i_max represents the maximum value:

[0036]

[0037] The continuously iterated WL matrix effectively supplements the attention mechanism. Based on the feature weights of the self-attention mechanism and combined with the enhancement effect of the WL matrix, the final attention weights are shown in formula (14):

[0038]

[0039] Furthermore, the WL-EAM model adopts the Adam optimizer to optimize this multi-modal deep learning framework. The optimization process includes two stages:

[0040] In the first stage, the initial word tokens Dtext of the review text and the RGB matrix Dimage of the review image are used to obtain the embedded features through the pre-trained model. Then, the embedded features of each modality are stacked into the sequential matrices Etext and Eimage;

[0041] In the second stage, WL-EAM captures the temporal features Ztext and Zimage of text and images in the reviews through LSTM. Subsequently, the attention mechanism learns the importance weights of the sequence features to obtain the review feature vectors Xtext and Ximage. Then, through the winner-loser effect empirical matrix, the attention weights of the attention mechanism are parameter-adjusted to coordinate the feature output of the attention, calculate the total loss of each iteration, and use the Adam optimizer to optimize the connection weights w and η; finally, according to the predicted results and the true labels, the parameters of the winner-loser effect empirical matrix are dynamically adjusted.

[0042] Advantages of the present invention:

[0043] 1. The present invention proposes a new attention mechanism and constructs a WL-EAM model for multi-modal data prediction tasks based on this mechanism. Inspired by the "winner-loser" effect, this mechanism can effectively extract key features from complex multi-modal data: (1) By simulating the winner effect, it can identify and strengthen the features that are most important for usefulness prediction; (2) By simulating the loser effect, it can reduce the influence of irrelevant or interfering features. The model constructed based on the "winner-loser" effect to enhance the attention mechanism can dynamically adjust the feature weights and optimize the feature extraction ability of the model.

[0044] 2. To verify the effectiveness of this method, this patent uses a large-scale multi-modal online review dataset on Yelp as an experimental platform because text and image reviews can be collected on this data platform, and single-modal and multi-modal feature sets can be obtained to compare and verify the prediction effect of the model proposed in this patent. The results show that compared with traditional attention mechanisms and models, the model in this paper has achieved a very significant improvement in predicting review usefulness, including higher accuracy, recall rate, and F1 score.

[0045] 3. In addition, the present invention also conducts an exploratory analysis of the model to understand the specific role of the "winner-loser" effect in multi-modal feature extraction. By visualizing and interpreting the attention distribution of the model, it reveals which types of information play a decisive role in review usefulness prediction and how to optimize the model performance by adjusting the attention mechanism. Brief Description of the Drawings

[0046] Figure 1 The architecture diagram of the WL-EAM model of the present invention.

[0047] Figure 2 The architecture diagram of the WL of the present invention.

[0048] Figure 3 The prediction performance diagram of the WL-EAM of the present invention.

[0049] Figure 4 The WL enhancement result graph of the present invention. Specific embodiments

[0050] The present invention will be further clarified below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and not to limit the scope of the present invention. It should be noted that the terms "front", "rear", "left", "right", "upper" and "lower" used in the following description refer to the directions in the accompanying drawings, and the terms "inner" and "outer" respectively refer to the directions towards or away from the geometric center of a specific component.

[0051] A deep learning method based on the winner effect to strengthen key features in this embodiment includes the following steps:

[0052] 1.1 WL-EAM model architecture

[0053] In order to reflect the "experience role" played by the winner-loser effect in the decision-making process and explore its potential effect on the deep learning model to mine multi-modal data features, a new multi-modal learning method called the WL-EAM model is proposed in this paper. A self-attention module enhanced based on the winner-loser effect is designed, which can help the model to "memorize" successful experiences and lessons learned in the multi-modal data prediction task, so that the model can more efficiently and accurately exert its feature extraction ability. Figure 1 The framework of the WL-EAM model.

[0054] As Figure 1 shown, first, multi-modal comments are collected and stored in the database. Secondly, the multi-modal comment data is preprocessed, such as excluding abnormal comment samples (for example, comment length less than 5). For the fields of natural language processing and computer vision, converting different types of data into a format that can be understood by machine learning models is a basic but very crucial task. For text data, in this paper, the comment text is converted into an embedded feature vector through the embeddings of the word embedding model BERT. For image data, the pixel features and RGB space channel features of the image are extracted for vectorization representation. Subsequently, the feature vectors after the conversion of the comment images are used as the input of the pre-trained model VGG-16. Since there is an obvious temporal relationship when users browse comment data, multiple texts and multiple pictures constitute different temporal data, so the LSTM model is used to capture the temporal dependence relationship of each comment modal data.

[0055] To enable the model to better discover factors conducive to task success and noise exacerbating task failure during training, this paper simulates the winner-loser effect by constructing a special self-attention weight matrix to improve the model's feature extraction ability. First, the output of the LSTM is used as the input to the original self-attention layer to obtain the weight distributions of text and image sequences; then, based on the special matrix constructed in this paper, the positive and negative situations learned by the model during training are continuously "remembered" to "increase wins and reduce losses" the output weights of the self-attention module; then, different feature fusion strategies are adopted to fuse the feature vectors of text and images for the multi-modal comment usefulness prediction task; finally, this special matrix is continuously iterated during training to form the long-term influence of the winner-loser effect. The symbols and abbreviations involved in the text are shown in Table 1:

[0056] Table 1. Related Symbol Table in the Design of this Patent

[0057]

[0058]

[0059] 1.2 Winner-Loser Effect Enhanced Attention Mechanism

[0060] Therefore, this paper introduces a self-attention layer to capture the importance weights of each element in the multi-modal comment sequence and improve the model's ability to extract key information. The self-attention mechanism is a weight learning scheme designed to obtain the optimal importance weights of sequential inputs. First, three weight matrices are initialized from the sequence embedding features Etext and Eimage obtained from the LSTM layer, including the key matrix K of the input elements, the value matrix V for learning the sequence weights, and the query matrix Q for capturing the correlation between the current time step and other time steps.

[0061] The output vector of the weight feature can be obtained by Equation (1).

[0062]

[0063] In Equation (1), first calculate the dot product of the query matrix Q and the key matrix K, then scale it by the dimension of the key matrix K, and then use the softmax activation function to obtain the weights of the value matrix V. Taking the element xi in the matrix as an example. The softmax activation function is expressed as Equation (2):

[0064]

[0065] where e is the Euler number. By multiplying the weights and the value matrix V, the weighted features, namely the text modality Ztext and the visual modality Zimage, are obtained.

[0066] The attention mechanism can improve the model's ability to extract features from the currently input data, but it cannot preserve the information of past data, so it has limitations in long-term memory. The winner-loser effect can effectively complement this problem. In biology, for the process of animal competition, previous winning experiences make it easier to win later. In addition, research in the field of neuroscience has shown that there is a neural circuit in the brain that mediates the "winner effect", resulting in previous winning experiences making subsequent victories easier.

[0067] The process of training a deep learning model can be analogized to the process of the model continuously trying a task. In each attempt, the model will face homogeneous but not exactly the same samples. The ordinary self-attention mechanism only focuses on extracting the importance of the current sample data and does not utilize the feature distribution when facing homogeneous data in the past. However, related research on the winner-loser effect reveals that it is possible to assist task decision-making based on past "successful experiences" and "lessons learned from failures". Therefore, by simulating the learning mechanism in a biological neural network and introducing the winner-loser effect into the attention model, the attention mechanism can break through the limitations of input information, enabling the model to not only focus on the key parts of the current input information but also adjust the processing strategy for the current information according to past learning experiences, thus showing better adaptability when dealing with similar tasks.

[0068] To simulate the winner-loser effect, this paper constructs a winner-loser effect matrix (WL) to help the model accumulate learning experiences for sample data during the training process and form the winner-loser effect for the current task. Its structure is as Figure 2 shown. The WL matrix reflects the correlation between the feature distribution focused on by the model during the training process and the success or failure results of the task. By using the WL matrix as an auxiliary mechanism to adjust self-attention, it helps the model combine the experiences accumulated from past training during feature extraction, improving the accuracy and robustness of the attention mechanism.

[0069] Through the attention mechanism, the attention weight distribution Attention_Weight of the input features can be extracted. Each row in the weight matrix represents the overall representation of a word or an image after being enhanced by other parts of the sequence. In sequence processing tasks, a larger attention weight means that the model considers a certain input element (such as a word or a picture pixel) to be more important for the current task. The smaller the weight, the less the model considers this input element to contribute to the current task or its relevance to the task goal is not high.

[0070]

[0071] Based on the results of long-term training, the model can continuously search for information points with larger (smaller) attention weights, forming a learning paradigm based on a memory feedback mechanism, namely the winner (loser) effect. Then, when encountering new data in the same scenario, the model dynamically adjusts the attention distribution based on this learning paradigm, enhancing the information that was more focused on in past experiences and weakening meaningless information, thereby enabling the model to perform better. Therefore, during the process of training the model, this paper searches for and memorizes the index positions of the maximum (minimum) values in each row of the Attention_Weight matrix, obtaining the index matrices MaxS (MinS). Their calculation formulas are shown in Equations (4) and (5). Through the MaxS and MinS matrices, the successful experiences and failure lessons predicted by the model during the learning process can be extracted.

[0072]

[0073] The winner-loser effect requires long-term iteration and acts on the current attention mechanism.

[0074] This paper constructs an update strategy for the WL matrix and formulates rules for its application to the self-attention mechanism. When the training result is correct, the initial WL can be represented by the difference between the initial MaxS and MinS. Multiplying the attention weights element-wise with MaxS (MinS) can obtain the weight matrix WL_Attention_Weight adjusted by the winner (loser) effect. The calculation formulas are as follows (6)(7):

[0075]

[0076] When the prediction result is correct, MaxS is a favorable factor for the model's prediction, and MinS is an unfavorable factor. However, in the actual training process, the training situation of the model fluctuates. It is necessary to distinguish the impacts of MaxS and MinS on the model's downstream tasks based on the model's prediction result R, enabling the model to determine whether MaxS and MinS are factors leading to success or meaningless noise. When the prediction result is correct, it indicates that MaxS is a factor conducive to task completion, while MinS is meaningless; if the prediction result is incorrect, it means that MaxS is significant noise, and the model's attention to this element should be reduced while strengthening the part of MinS.

[0077] In addition, to measure and adjust the impact of the winner effect and the loser effect on the model, this paper sets a parameter η to represent the degree of the model's inclination towards the winner effect and the loser effect. Its range is between 0 and 1. When it is greater than 0.5, it means the model is more inclined to focus on successful experiences; when it is less than 0.5, it means the model is more inclined to focus on lessons from failures; and when it is equal to 0.5, the two are in an equilibrium state. Therefore, the measurement of successful experiences and lessons from failures can be expressed by formulas (8) and (9).

[0078] WL i+1 = WL i + R i WL i ⊙ [ηMaxS i - (1 - η)MinS i , R = 1 (8)

[0079] WL i+1 = WL i + R i WL i ⊙ [(1 - η)MaxS i - ηMinS i , R = 0 (9) In addition, since the model's ability is relatively weak in the early stage of training and it does not learn knowledge sufficiently, the impact of early memories on the current should be lower than that of later memories.

[0080] During the update process of WL, this paper introduces the softmax function for optimization. Through softmax, early experiences are generalized, and at the same time, the problem of the WL matrix potentially facing too small or exploding values during the continuous learning process is avoided. In addition, since the information that affects the model's prediction task often accounts for a small part of the overall information, this paper combines the idea of regularization and introduces a dropout parameter θ (set to 0.5) to shield the influence of some elements. The iterative process is shown in formulas (10) and (11).

[0081] WL i+1 = softmax(WL i + dropout(R i WL i ⊙ [ηMaxS i - (1 - η)MinS i , θ)) (10)

[0082] WL i+1 = softmax(WL i + dropout(R i WL i ⊙ [(1 - η)MaxS i - ηMinS i, θ)) (11) Differences in the training data distribution may cause the parameters of WL to polarize, resulting in the degradation of the features after activation by the softmax function. Before the text is activated by the softmax function, the idea of layer normalization is adopted, and the parameter optimization is carried out by the maximum-minimum normalization method to further improve the stability of WL. As shown in formulas (11), (12), and (13), where a i_min represents the minimum value of the i-th row of the matrix, and a i_max represents the maximum value:

[0083]

[0084] The continuously iterated WL can effectively supplement the attention mechanism. Based on the feature weights of the self-attention mechanism and combined with the enhancement effect of WL, the final attention weights are shown in formula (14):

[0085]

[0086] 1.3 Optimization Algorithm

[0087] Table 2. Training Process of WL-EAM

[0088]

[0089] For the WL-EAM model proposed in this paper, the Adam optimizer is adopted to optimize this multi-modal deep learning framework. The specific process of the algorithm is shown in Table 2. The reason for choosing the Adam optimizer is its simple implementation, high computational efficiency, and low memory requirements, which make it an ideal choice for processing large-scale datasets. The optimization process includes two stages:

[0090] (1) In the first stage, the initial word tokens Dtext of the review text and the RGB matrix Dimage of the review image are used to obtain the embedded features through the pre-trained model. Then, the embedded features of each modality are stacked into the sequential matrices Etext and Eimage.

[0091] (2) In the second stage, WL-EAM captures the temporal features Ztext and Zimage of the text and pictures in the review through LSTM. Subsequently, the attention mechanism learns the importance weights of the sequence features to obtain the review feature vectors Xtext and Ximage. Then, through the winner-loser effect empirical matrix, the attention weights of the attention mechanism are adjusted, the feature output of the attention is coordinated, the total loss of each iteration is calculated, and the connection weights w and η are optimized using the Adam optimizer. Finally, according to the predicted results and the true labels, the parameters of the winner-loser effect empirical matrix are dynamically adjusted.

[0092] 1.4. Prediction Effect of Deep Learning Method with Enhanced Winner Effect Characteristics

[0093] 1.4.1 Experimental Data

[0094] To conduct an empirical evaluation of the WL-EAM model proposed in this patent, 30,000 multimodal online reviews themed on restaurant food were collected from the Yelp website. The release dates of these reviews were between January 2024 and March 2024. Among the multimodal reviews, 13,673 reviews received more than one vote of helpfulness and were labeled as 1. For each useful multimodal review, a corresponding review that received zero votes and was posted on the same day was selected as a useless review sample. Table 3 shows the descriptive statistics of the multimodal review dataset.

[0095] Table 3. Yelp Restaurant Review Data Distribution (2024.1 - 3)

[0096]

[0097] 1.4.2 Experimental Steps

[0098] In this paper, the effectiveness of the WL-EAM method was verified through two experiments. In the first experiment, the proposed method was compared with the state-of-the-art multimodal deep learning methods in terms of prediction performance. Four benchmark methods were identified for comparison. First, the paper conducted comparative experiments with the review text and pictures respectively. The unimodal reviews were input into the LSTM and attention layers, and their helpfulness (unimodal) was predicted. Then, a multimodal learning method based on late fusion was adopted. The representation features of all data modalities were concatenated into a vector, and then the prediction result was generated according to the concatenated vector for predicting the helpfulness of multimodal reviews. Second, a method based on mid-fusion (MF) was adopted, which took the multimodal prediction output and the unimodal prediction output as fusion elements to obtain the final output. Next, a method based on early fusion (EF) was adopted. This method concatenated the embedding features extracted by the BERT and VGG-16 pre-trained models, and then used them as the input of the LSTM and attention layers for modeling. Regarding the number of attention heads in the attention mechanism (a group of K, V, and Q), we incorporated the single-head attention and multi-head attention strategies into the WL-EAM method and the four benchmark methods in this paper.

[0099] In the second experiment, this paper compares the feature learning performance of the WL-EAM model and the baseline model. We use the hidden features of the last layer learned by the WL-EAM method and the output of the baseline model as representation features, and use the downstream prediction model to predict the usefulness of the review. We use three powerful classic prediction models (i.e., SVM, RF, and XGBoost) to evaluate which multimodal representation learning method can produce better prediction performance. For these two experiments, since the experimental data is relatively balanced, ACC is used to evaluate the prediction performance, which can balance the model sensitivity and specificity. All experimental results have been subjected to 5 independent 5-fold cross-validation.

[0100] 1.4.3 Prediction Performance of WL-EAM

[0101] Table 4 and Figure 3 The prediction performance (ACC%) of the proposed WL-EAM method under single-head attention and multi-head attention mechanisms (4-head attention, winner-loser effect parameter η = 0.5) is summarized and compared with baseline deep learning methods based on review text, review image and multimodal reviews.

[0102] Table 4. WL-EAM prediction performance table

[0103]

[0104] The experimental results show that for the study of the usefulness of online reviews, the effect of using only a single modality (text or image) is lower than that of using multiple modalities at the same time. This is because multimodal review data provides more information for deep learning models. By combining the semantics of text and images, it can help the model more fully explore features that are helpful for usefulness classification.

[0105] Different feature fusion strategies have an impact on the performance of the model. In the case of single-head attention, for the task of predicting the usefulness of multimodal online reviews, different feature fusion strategies have a certain impact on the prediction performance of the model. The early fusion strategy achieved the best classification effect (82.79%), followed by late fusion (82.56%) and mid-fusion (81.38%). However, the degree of fluctuation in model performance showed the opposite trend. Since early fusion can process information from different modalities earlier, it can supplement the low-level correlation features between different modalities, thus improving the model performance. However, due to feature concatenation, the feature space becomes high-dimensional, increasing the complexity of subsequent feature extraction for the model. Therefore, there are significant fluctuations in the model performance. The late fusion strategy allows each modality to be processed separately and fuses information from different modalities at a deeper level of the model, which can fully exploit the features of each modality and handle the differences between different data formats and modalities. The mid-fusion strategy fuses information from different modalities at the intermediate level of the model, which is a balance between the early features and late features of different modality data.

[0106] When the multi-head attention mechanism (4-head attention) is adopted, the performance of the model under different feature fusion strategies has been improved. Among them, the mid-fusion has the best model performance (83.44%), which is closely related to the characteristic that the multi-head attention mechanism can learn different feature associations from multiple subspaces. The multi-head attention enhances the model's understanding and generalization ability for complex feature tasks by processing multiple attention patterns in parallel. However, the multi-head attention mechanism also leads to an increase in the performance volatility of late fusion (0.71%) and mid-fusion (0.87%). This may be because the multi-head attention mechanism considers the information between different attention heads, increasing the model complexity to a certain extent. This complexity makes the model more sensitive to small changes in data during the training process, resulting in further fluctuations in the model.

[0107] Through Table 5 and Figure 4It can be found that after incorporating the "winner-loser" effect (WL), the experimental results of the model for single-modal and multi-modal comment usefulness prediction have been significantly improved, and the improvement in the model's performance shows consistency whether using single-head attention or multi-head attention. Further comparing the extent to which the "winner-loser" effect plays in single-modal and multi-modal scenarios, it can be found that when facing single-modal data, the improvement in the model's performance is relatively limited (an average increase of 1.3%), but the stability of the model's performance has been optimized. Except for the increase in the performance fluctuation deviation of the model in the case of text data under the multi-head attention scenario, other situations have been optimized. In the multi-modal scenario, the model's performance has been significantly improved (an average increase of 4.45%), and the late-fusion multi-head attention strategy has the most significant improvement, reaching 5.57%. In addition, whether in the single-modal or multi-modal scenario, under the attention mechanism enhanced by the "winner-loser" effect, the improvement in the model's performance driven by single-head attention is almost lower than that driven by multi-head attention.

[0108] Table 5. WL Enhancement Results

[0109] Single-head Multi-head Average Single-modal 1.27％ 1.34％ 1.30％ Multi-modal 3.99％ 4.92％ 4.45％

[0110] By analyzing the results of the prediction performance experiment of the WL-EAM model, it can be found that the "winner-loser" effect can improve the ability of the attention mechanism to capture potential features of data and can play a better role in the multi-modal scenario. This may be because the iteration of the "winner-loser" effect matrix depends on the self-attention mechanism, and multi-modal data requires more attention matrices for calculation, and also requires a corresponding amount of "winner-loser" memory matrices to store historical information, so it can more comprehensively reflect the potential features affecting the task. Under the multi-head attention mechanism, each attention head can focus on various information blocks, so that the "winner-loser effect" matrix can remember more diverse information in each training, thus further optimizing the model's performance.

[0111] 4.4 Feature Learning Ability of WL-EAM

[0112] To deeply explore the impact of the "winner-loser" effect on the model's feature extraction ability, this paper uses the last layer of features of the trained WL-EAM model as the input of the downstream machine learning model. By comparing single-head attention, multi-head attention, and different feature fusion strategies, and selecting support vector machine (SVM), random forest (RF), and XGBoost as the baseline models, the feature learning performance of the WL-EAM model is analyzed. Table 6 summarizes the prediction performance in the downstream classification task when using the representation features extracted by the WL-EAM model as the feature input of the baseline model.

[0113] Table 6. Feature Extraction Ability of WL-EAM

[0114]

[0115] It can be clearly seen from the results in Table 6 that when using the Random Forest (RF) and XGBoost models, the performance of the WL-EAM model is generally better than that of the Support Vector Machine (SVM), which fully demonstrates the powerful capabilities of the ensemble learning method. In addition, the experimental results also reveal that the features extracted from the review text are superior to those of the review images under both the single-head attention and multi-head attention mechanisms. This may be because traditional machine learning models have natural advantages in processing text data. When comparing the fusion strategies of different modality features, we found that the late fusion method based on the latent space is generally better than the early fusion method based on concatenation and the mid-fusion method based on mixture, indicating that the late fusion method may be more effective for feature extraction in integrating multi-modal information. Regarding the "winner-loser" effect, the model combined with WL enhancement shows significant advantages among all the benchmark methods. In particular, the WL-EAM model is particularly outstanding in the ability to mine latent space features, and the method based on late fusion shows general significance, further proving the powerful feature extraction ability of this effect in multi-modal scenarios.

[0116] The technical means disclosed in the solution of the present invention are not limited to the technical means disclosed in the above embodiments, but also include technical solutions composed of any combination of the above technical features.

Claims

1. A deep learning method for strengthening key features based on the winner effect, characterized in that It includes the following steps: Step 1: Construct the WL-EAM model architecture and perform data preprocessing. First, collect multimodal comments and store them in a database, and preprocess the multimodal comment data, which includes text data and image data; Step 2: For the text data, convert the comment text into an embedded feature vector through the embeddings of the word embedding model BERT; for the image data, perform vectorization representation by extracting the pixel features and RGB space channel features of the image; Step 3: For the comment image data, use the pre-trained model VGG-16 to convert the picture into an embedded feature vector; use the LSTM model to extract the temporal features of the text and the picture, and capture the temporal dependencies in the comment; Step 4: Extract the importance weights of each element in the text and picture sequences through the self-attention mechanism to generate an attention weight matrix; Step 5: Construct a winner-loser effect matrix to simulate the winner-loser effect. The WL matrix is updated iteratively, recording the model's memory of successful experiences and failure lessons during training, and dynamically adjusting the attention weights; Step 6: Verify the effect of the deep learning method with enhanced winner effect features; fuse the feature vectors of the text and the picture using different fusion strategies, and perform multimodal comment usefulness prediction through a multi-layer neural network.

2. The deep learning method for enhancing key features based on the winner effect according to claim 1, wherein, It includes the following steps: In Step 4, first initialize 3 weight matrices from the sequence embedding features Etext and Eimage obtained from the LSTM layer, including the key matrix K of the input elements, the value matrix V for learning the sequence weights, and the query matrix Q for capturing the correlation between the current time step and other time steps; The output vector of the weight features is obtained by Equation (1); In Equation (1), first calculate the dot product of the query matrix Q and the key matrix K, then scale it by the dimension of the key matrix K, and then use the softmax activation function to obtain the weights of the value matrix V; taking the element xi in the matrix as an example, the softmax activation function is expressed as Equation (2): where e is the Euler number; by multiplying the weights and the value matrix V, the weighted features, namely the text modality Ztext and the visual modality Zimage, are obtained.

3. The deep learning method for enhancing key features based on the winner effect according to claim 1, wherein In Step 5, by using the WL matrix as an auxiliary mechanism to regulate self-attention, it helps the model to combine the experience accumulated in previous training during feature extraction, improving the accuracy and robustness of the attention mechanism; specifically including Step 51: Extract the attention weight distribution Attention_Weight of the input features through the attention mechanism. Each row in the weight matrix represents the overall representation of a piece of text or a picture after being enhanced by other parts of the sequence; in the sequence processing task, the magnitude of the attention weight is correlated with the contribution degree of the current task; During the process of training the model in this paper, search for and remember the index positions of the maximum / minimum values in each row of the Attention_Weight matrix to obtain the index matrix MaxS / MinS; Its calculation formulas are shown in Formulas (4) and (5); through the MaxS and MinS matrices, the successful experiences and failure lessons predicted by the model during the learning process are extracted. The winner-loser effect requires long-term iteration and acts on the current attention mechanism.

4. A deep learning method for enhancing key features based on the winner effect according to claim 3, characterized in that In Step 5, an update strategy for the WL matrix is constructed, and rules for its action on the self-attention mechanism are formulated; when the training result is correct, the initial WL matrix is represented by the difference between the initial MaxS and MinS; multiplying the attention weights bit by bit with MaxS / MinS, the weight matrix WL_Attention_Weight adjusted by the winner / loser effect can be obtained; the calculation formulas are as shown below (6)(7): When the prediction result is correct, MaxS is a favorable factor for the model prediction, and MinS is an unfavorable factor.

5. The deep learning method for enhancing key features based on the winner effect according to claim 4, wherein By setting the parameter η to represent the tendency degree of the model towards the winner effect and the loser effect, its range is between 0 and 1. When it is greater than 0.5, it means the model is more inclined to focus on successful experiences. When it is less than 0.5, it means the model is more inclined to focus on failure lessons. When it is equal to 0.5, the two are in an equilibrium state; therefore, the measurement of successful experiences and failure lessons is represented by Formulas (8)(9). WL i+1 = WL i + R i WL i ⊙[ηMaxS i -(1 - η)MinS i , R = 1 (8) WL i+1 = WL i + R i WL i ⊙[(1 - η)MaxS i - ηMinS i , R = 0 (9) 6. The deep learning method for enhancing key features based on the winner effect according to claim 1, wherein The self-attention mechanism generates an attention weight matrix through the softmax function, dynamically adjusts the attention weights in combination with the WL matrix; the dropout parameter θ is introduced; the iteration process is as shown in Formulas (10) and (11); WL i+1 = softmax(WL i + dropout(R i WL i ⊙ [ηMaxS i - (1 - η)MinS i , θ)) (10) WL i+1 = softmax(WL i + dropout(R i WL i ⊙ [(1 - η)MaxS i - ηMinS i , θ)) (11).

7. A deep learning method for enhancing key features based on the winner effect according to claim 1, characterized in that Before the text is activated by the softmax function, the layer normalization idea is adopted, and the parameter optimization is carried out by the maximum-minimum normalization method; as shown in formulas (11), (12), and (13), where a i_min represents the minimum value of the i-th row of the matrix, and a i_max represents the maximum value: The continuously iterated WL matrix effectively supplements the attention mechanism. Based on the feature weights of the self-attention mechanism and combined with the enhancement effect of the WL matrix, the final attention weights are as shown in Formula (14):

8. A deep learning method for enhancing key features based on the winner effect according to claim 1, characterized in that Among them, the WL-EAM model uses the Adam optimizer to optimize this multi-modal deep learning framework, and the optimization process includes two stages: In the first stage, the initial word tokens Dtext of the review text and the RGB matrix Dimage of the review image are used to obtain embedding features through the pre-trained model. Then, the embedding features of each modality are stacked into the sequential matrices Etext and Eimage; In the second stage, WL-EAM captures the temporal features Ztext and Zimage of the text and pictures in the review through LSTM. Subsequently, the attention learns the importance weights of the sequence features to obtain the review feature vectors Xtext and Ximage. Then, through the winner-loser effect experience matrix, the attention weights of the attention mechanism are adjusted parametrically to coordinate the feature output of the attention, calculate the total loss of each iteration, and use the Adam optimizer to optimize the connection weights w and η; finally, according to the predicted results and the true labels, the parameters of the winner-loser effect experience matrix are dynamically adjusted.

Images