Multi-view review quality ranking method based on sub-label enhancement
By introducing a sub-label enhancement method in multi-view characterization learning, using the similarity sub-labels and comment quality score tags generated by CLIP, combining text and image features for information interaction, the problem of insufficient supervision information transmission in multi-view characterization learning is solved, and the problem of insufficient supervision information transmission in multi-view characterization learning is improved. The performance and accuracy of the review quality sorting model.
Patent Information
- Application Number
- CN202410662219.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-27
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2044-05-27
AI Technical Summary
Under the multi-perspective representation learning framework, only using the comment quality score tag as a supervision signal makes it difficult to convey effective supervision information for the learning of each perspective, resulting in the inability to fully tap modal interaction information, affecting the effectiveness and comment quality of multi-perspective representations and impacting the effectiveness of multi-perspective representations and comment quality Performance of sorting models.
Using a method based on sub-label enhancement, the pre-trained model CLIP generates similar sub-labels under multiple perspectives as auxiliary supervision signals, combine text and image features to integrate information interaction modules, generate five perspectives, and use hinge loss function and uniform. The square error loss function updates the model parameters to improve the learning effect of multi-view representation.
It reduces the annotation cost, improves the accuracy of sub-labels, enhances the ability to judge the quality of comments, improves the effectiveness of multi-view representation and the performance of the review quality sorting model.
Smart Images

Figure CN118396475B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the fields of multimodal deep learning and natural language processing, and in particular to a multi-view review quality ranking method based on sub-label enhancement. Background Art
[0002] The quality of reviews on e-commerce platforms is an important factor affecting people's quality of life and market transactions. User reviews can not only reflect the quality of goods, the level of service and consumer satisfaction, but also help other consumers make consumption decisions that suit them, while allowing merchants to identify deficiencies and improve them.
[0003] Existing review quality ranking technologies mainly use deep learning models to extract and vectorize features from reviews, process plain text features using convolution or LSTM (Long Short-Term Memory Network) methods, introduce text character-level features, or use e-commerce review scoring mechanisms to assist in predicting reviews. Currently, on e-commerce platforms, products and reviews often contain both text and images. Compared with single-modal datasets, review quality ranking tasks on multimodal datasets are more realistic. Current multimodal review quality ranking methods build multimodal consistency relationships between reviews and products from different perspectives, and learn multi-perspective representations that can be used to predict review quality scores.
[0004] However, in the framework of multi-view representation learning, only using the review quality score label as the supervision signal for model learning makes it difficult to convey effective supervision information for the learning of each view representation, so that the modal interaction information in each view representation cannot be fully mined, which in turn affects the effectiveness of multi-view representation and the performance of the review quality ranking model. Therefore, the present invention proposes a multi-view review quality ranking method based on sub-label enhancement. Summary of the invention
[0005] In order to solve the above problems, the present invention proposes a multi-view review quality ranking method based on sub-label enhancement, comprising:
[0006] Step S1, obtaining a review quality ranking dataset, generating similarity sub-labels under multiple perspectives, i.e., similarity scores of each perspective as auxiliary supervision signals for training of each perspective;
[0007] Step S2, inputting the trained text and image into a text encoder and an image encoder respectively to obtain text features and image features;
[0008] Step S3: The text features and image features are integrated into the intra-modal and inter-modal features through the information interaction module to obtain five perspective representations of the reviewed products. The five perspective pairs are review image-text pair, review-product text pair, review-product image pair, review text-product image pair, and review image-product text pair.
[0009] Step S4, concatenating the perspective representations to obtain a multi-perspective representation, using the comment quality score labels to supervise the update of the training parameters related to the multi-perspective representation, and at the same time using the generated sub-labels as supervision signals for the training of five different perspective representations to update the training parameters related to each perspective representation;
[0010] Step S5: after the training is completed, the text and image to be tested are passed through a text encoder and an image encoder to generate text features and image features of the text and image to be tested;
[0011] Step S6: Generate each perspective representation through the trained information interaction module using the text features and image features to be tested, then use the concatenation method to obtain the multi-perspective representation, and then input the multi-perspective representation into a four-layer linear neural network for prediction, and finally obtain the comment quality score.
[0012] Furthermore, in the step S1, similarity sub-label generation is accomplished with the help of a pre-trained model CLIP.
[0013] Furthermore, in step S2, the text encoder is implemented by Fast text or Glove, and the image encoder is implemented by Faster R-CNN.
[0014] Furthermore, the step S3 specifically includes:
[0015] The text and image features of products and reviews are input into the information interaction module. First, a linear layer is used to map the text features and image features to the same dimensional space, where the dimension after mapping is 64. Then, the attention mechanism is used to align the features of different modalities to obtain five different perspective representations:
[0016]
[0017] in is the consistent feature representation of the product and review text under the perspective It is the consistent feature representation of the review text and image under the perspective It is the consistent feature representation of the review text-product image under the perspective is the consistent feature representation of the review image-product text pair under different viewing angles It is the consistent feature representation of product and review images under different viewing angles.
[0018] The information interaction module expression is:
[0019]
[0020] in It is the final feature representation of the five perspectives generated by the information interaction module is the attention mechanism operation, are the single-mode features, and They correspond to product text features, review text features, product image features, and review image features respectively.
[0021] Furthermore, the step S4 specifically includes: connecting five different perspective representations in series to obtain a multi-perspective representation , and feed it into a four-layer linear neural network to get the predicted quality score , calculated using the hinge loss function Its true quality score The loss value between ; At the same time, the perspectives generated in step S3 are represented Send them to five different two-layer linear neural networks to get their predicted sub-labels , calculated using the mean square error loss function and its sub-tags The loss value between ; The sum of the two loss values is the final loss value during model training , the loss gradient is passed back to update the relevant parameters.
[0022] Connect the five view features in series to get a multi-view representation The expression is:
[0023]
[0024] Loss value The calculation and parameter update expressions are:
[0025]
[0026] Where N is the number of samples; P is the difference between the true labels of the two samples; Indicates products The quality prediction score of one of the reviews, Indicates that the quality score of the review is large; Indicates products The quality prediction score of another review of Indicates that the quality score of the review is small.
[0027] Loss value The calculation and parameter update expressions are:
[0028]
[0029] in represents the mean square error function It is a product No. Comments View similarity prediction value, It is a product No. Comments View similarity.
[0030] Loss value The final expression is:
[0031]
[0032] in is the predicted quality score of the review and the true quality score of the review The loss value generated between yes Predicted sub-labels of viewpoints and Sub-tags of perspective The loss value generated between .
[0033] The present invention has the following beneficial effects:
[0034] 1. Low labeling cost and high sub-label accuracy. The present invention proposes a multi-view review quality ranking method based on sub-label enhancement, where the sub-labels are generated by the pre-trained model CLIP, and no manual labeling of sub-labels is required. At the same time, CLIP is a powerful image-text pre-training model, and the sub-labels generated by it are highly accurate.
[0035] 2. Strong ability to judge comment quality. In the multi-view reasoning method, it is difficult to convey effective supervision information for the learning of each perspective representation by only using comment quality score labels as the supervision signal for model learning during model training. The present invention introduces sub-labels of different perspectives to effectively guide the model to fully explore the modal interaction information in the representation of each perspective, so as to enhance the effectiveness of multi-view representation and ultimately improve the performance of the comment quality ranking model. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] Figure 1 is a flow chart of the method of the present invention;
[0037] Figure 2 This is a method model diagram of the present invention. DETAILED DESCRIPTION
[0038] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it is obvious that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.
[0039] like Figure 1 As shown, in one embodiment of the present invention, a multi-view review quality ranking method based on sub-tag enhancement includes the following steps:
[0040] Step S1, obtaining a review quality ranking dataset, generating similarity sub-labels under multiple perspectives, i.e., similarity scores of each perspective as auxiliary supervision signals for training of each perspective;
[0041] Each sample in the review quality ranking dataset is a product-review pair on an e-commerce platform, such as the down jacket product-review pair included in the Clothing product category sub-dataset in the public dataset Lazada, where the product information includes photos and product descriptions provided by the seller, and the review information includes the evaluation text and multiple photos of the product.
[0042] The sub-labels under multiple perspectives are similarity scores generated by the pre-trained model CLIP (a pre-trained model based on contrastive text-image pairs) under five perspectives: review image-text pairs, review-product text pairs, review-product image pairs, review text-product image pairs, and review image-product text pairs. They are used as supervision signals for training in each perspective to guide the model to fully mine the modal interaction information in the features of each perspective, ensure the effectiveness of multi-perspective representation, and thus improve the model's ability to predict review quality scores.
[0043] Step S2, inputting the trained text and image into a text encoder and an image encoder respectively to obtain text features and image features;
[0044] The text encoder used in this step is Fast text or Glove, and the image encoder is FasterR-CNN.
[0045] Step S3: The text features and image features are integrated into the intra-modal and inter-modal features through the information interaction module to obtain five perspective representations of the reviewed products. The five perspective pairs are review image-text pair, review-product text pair, review-product image pair, review text-product image pair, and review image-product text pair.
[0046] In this step, the information interaction module, namely the attention mechanism, is used to fully interact the review image and text features with the product image and text features, helping the model to better understand the relationship between reviews and products.
[0047] like Figure 2 As shown, step S4, the perspective representations are connected in series to obtain a multi-perspective representation, and the review quality score labels are used to supervise the update of the training parameters related to the multi-perspective representation, and at the same time, the generated sub-labels are used as supervision signals for the training of five different perspective representations to update the training parameters related to each perspective representation;
[0048] In this step, each perspective representation is input into a different two-layer linear neural network, and the similarity sub-label is used as the supervision signal of each perspective representation to guide the model to continuously update the parameters related to each perspective representation during the training process. The purpose is to help the model fully mine the modal interaction information in the features of each perspective and generate more effective multi-perspective representations, thereby improving the performance of the model in predicting the review quality score.
[0049] Step S5: after the training is completed, the text and image to be tested are passed through a text encoder and an image encoder to generate text features and image features of the text and image to be tested;
[0050] Step S6: Generate each perspective representation through the trained information interaction module using the text features and image features to be tested, then use the concatenation method to obtain the multi-perspective representation, and then input the multi-perspective representation into a four-layer linear neural network for prediction, and finally obtain the comment quality score.
[0051] This step uses the trained model for testing. The model enhanced by similarity sub-labels can better reflect the interactive relationship of the data from various perspectives, generate more effective multi-perspective features for result prediction, and thus obtain a more accurate review quality score.
[0052] A further technical solution is that the specific implementation process in step S2 includes:
[0053] Step S2.1: Use the word embedding model Fast text or Glove to convert each token of the text into a word embedding vector Pre-trained model Faster R-CNN extracts ROI features of images .
[0054] Step S2.2: Learn word embedding Pass it to k-gram CNN and then get the k-gram feature representation . where k∈{1,… } represents the convolution kernel size Indicates the length of text features Refers to using k-gram CNN to embed word vectors Perform semantic encoding. Multi-features are stacked to form text representation H ={ … }. Use self-attention mechanism to focus on ROI features Encode to obtain the semantic representation of visual features .in Represents the image feature length It refers to using the self-attention mechanism to focus on ROI features Carry out strengthening treatment.
[0055] A further technical solution is that the specific implementation process of the information interaction module in step S3 includes:
[0056] The text and image features of products and reviews are input into the information interaction module. First, a linear layer is used to map the text features and image features to the same dimensional space, where the dimension after mapping is 64. Then, the attention mechanism is used to align the features of different modalities to obtain five different perspective representations:
[0057]
[0058] in is the consistent feature representation of the product and review text under the perspective It is the consistent feature representation of the review text and image under the perspective It is the consistent feature representation of the review text-product image under the perspective is the consistent feature representation of the review image-product text pair under different viewing angles It is the consistent feature representation of product and review images under different viewing angles.
[0059] The information interaction modules are as follows:
[0060]
[0061] in It is the final feature representation of the five perspectives generated by the information interaction module is the attention mechanism operation, are the single-mode features, and They correspond to product text features, review text features, product image features, and review image features respectively.
[0062] A further technical solution is that the specific implementation process in step S4 includes:
[0063] Connect five different perspective representations in series to obtain a multi-perspective representation , and feed it into a four-layer linear neural network to get the predicted quality score , calculated using the hinge loss function Its true quality score The loss value between ; At the same time, the perspectives generated in step S3 are represented Send them to five different two-layer linear neural networks to get their predicted sub-labels , calculated using the mean square error loss function and its sub-tags The loss value between ; The sum of the two loss values is the final loss value during model training , the loss gradient is fed back to update the relevant parameters; the input dimension of the two-layer linear neural network (Linear-ReLU-Linear) is set to 64, and the input dimension of the four-layer linear neural network (Linear-ReLU-Linear-ReLU-Linear-ReLU-Linear) is set to 64*5, i.e. 320. After linear transformation, the output dimensions of the two-layer linear neural network and the four-layer linear neural network are both 1.
[0064] Connect the five view features in series to get a multi-view representation The expression is:
[0065]
[0066] Loss value The calculation and parameter update expressions are:
[0067]
[0068] Where N is the number of samples; P is the difference between the true labels of the two samples; Indicates products The quality prediction score of one of the reviews, Indicates that the quality score of the review is large; Indicates products The quality prediction score of another review of Indicates that the quality score of the review is small.
[0069] Loss value The calculation and parameter update expressions are:
[0070]
[0071] in represents the mean square error function It is a product No. Comments View similarity prediction value, It is a product No. Comments View similarity.
[0072] Loss value The final expression is:
[0073]
[0074] in is the predicted quality score of the review and the true quality score of the review The loss value generated between yes Predicted sub-labels of viewpoints and Sub-tags of perspective The loss value generated between .
[0075] The present invention uses CLIP to generate similarity scores as the five categories of similarity sub-labels required for training. By using them as supervision signals for each perspective to guide the model to update training parameters, the model can fully mine the modal interaction information in the features of each perspective, thereby obtaining a more effective multi-perspective representation and ultimately improving the model's comment quality ranking ability.
[0076] The present invention trains multi-view representation based on review quality score labels, adopts hinge loss function and mean square error loss function to guide the model to update training parameters, and uses similarity sub-labels to enhance the learning of each view representation, so that the model can fully mine the modal interaction information in each view feature to obtain more effective multi-view representation, thereby improving the review quality ranking effect.
[0077] The purpose of the sub-label enhancement method is to enable the model to better mine the modal interaction information in different perspectives, thereby obtaining a more effective multi-perspective representation and making the model ranking results more reliable.
[0078] The above description is not intended to limit the present invention in any form. Although the present invention has been disclosed through the above embodiments, it is not intended to limit the present invention. Any technician familiar with the profession can make some changes or modifications to equivalent embodiments of equivalent changes using the technical contents disclosed above without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent change and modification made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the scope of the technical solution of the present invention.
Claims
1. A multi-view review quality ranking method based on sub-label enhancement, characterized in that: The method includes: Step S1, obtaining a review quality ranking dataset, generating similarity sub-labels under multiple perspectives, i.e., similarity scores of each perspective as auxiliary supervision signals for training of each perspective; Step S2, inputting the trained text and image into a text encoder and an image encoder respectively to obtain text features and image features; Step S3, using the information interaction module to fuse the intra-modal and inter-modal features of the text features and the image features to obtain five perspective representations of the reviewed products, wherein the five perspective pairs are review image-text pairs, review-product text pairs, review-product image pairs, review text-product image pairs, and review image-product text pairs; Step S4, concatenating the perspective representations to obtain a multi-perspective representation, using the comment quality score labels to supervise the update of the training parameters related to the multi-perspective representation, and at the same time using the generated sub-labels as supervision signals for the training of five different perspective representations to update the training parameters related to each perspective representation; Step S5: after the training is completed, the text and image to be tested are passed through a text encoder and an image encoder to generate text features and image features of the text and image to be tested; Step S6, the text features and image features to be tested are used to generate each perspective representation through the trained information interaction module, and then a multi-perspective representation is obtained by a series connection method, and then the multi-perspective representation is input into a four-layer linear neural network for prediction to obtain a review quality score; In step S1, the sub-label is generated using the pre-trained model CLIP; In step S2, the text encoder is implemented by Fast text or Glove, and the image encoder is implemented by Faster R-CNN; In step S3, the information interaction module is an attention mechanism, and the graphic features of the product and the review are input into the information interaction module, that is, the attention mechanism is used to align them to obtain five different perspective representations: in is the consistent feature representation of the product and review text under the perspective It is the consistent feature representation of the review text and image under the perspective It is the consistent feature representation of the review text-product image under the perspective is the consistent feature representation of the review image-product text pair under different viewing angles It is the consistent feature representation of product and review images under different viewing angles; The information interaction module expression is: in It is the final feature representation of the five perspectives generated by the information interaction module is the attention mechanism operation, are the single-mode features, and They correspond to product text features, review text features, product image features, and review image features respectively; The specific process of step S4, in which the sub-label guides the model to train each perspective representation and update the parameters related thereto, is as follows: five different perspective representations are connected in series to obtain a multi-perspective representation , and feed it into a four-layer linear neural network to get the predicted quality score , calculated using the hinge loss function Its true quality score The loss value between ; At the same time, the perspectives generated in step S3 are represented Send them to five different two-layer linear neural networks to get their predicted sub-labels , calculated using the mean square error loss function and its sub-tags The loss value between ; The two loss values are added together to get the final loss value during model training , the loss gradient is fed back to update the relevant parameters; the input dimension of the two-layer linear neural network is set to 64, and the input dimension of the four-layer linear neural network is set to 64×5. After linear transformation, the output dimensions of the two-layer linear neural network and the four-layer linear neural network are both 1; Connect the five view features in series to get a multi-view representation The expression is: Loss value The calculation and parameter update expressions are: Where N is the number of samples; P is the difference between the true labels of the two samples; Indicates products The quality prediction score of one of the reviews, Indicates that the quality score of the review is large; Indicates products The quality prediction score of another review of Indicates that the quality score of the review is small; Loss value The calculation and parameter update expressions are: in represents the mean square error function It is a product No. Comments View similarity prediction value, It is a product No. Comments Viewpoint similarity; Loss value The final expression is: in is the predicted quality score of the review and the true quality score of the review The loss value generated between yes Predicted sub-labels of viewpoints and Sub-tags of perspective The loss value generated between .
