Review usefulness prediction method and system based on multimodal information fusion

Abstract

The present invention provides a method and system for predicting review usefulness based on multimodal information fusion, and relates to the technical field of review usefulness prediction. The present invention obtains review usefulness measurement results from a mixed perspective of domain knowledge and product usefulness measurement results from different single perspectives based on multimodal data of product usefulness reviews; and then fuses the review usefulness measurement results from a mixed perspective of domain knowledge and the product usefulness measurement results from different single perspectives to obtain the final evaluation results of the product usefulness reviews. The present invention comprehensively considers multimodal product usefulness review data, not only considering the content of product usefulness reviews, but also considering the initial voting behavior of product usefulness, rationally utilizing the complementarity between information, and avoiding the problem of inaccurate prediction results of product online review usefulness in traditional methods.

Description

Technical Field

[0001] The present invention relates to the technical field of review usefulness prediction, and in particular to a review usefulness prediction method and system based on multimodal information fusion. Background Art

[0002] On online shopping platforms, a single product can receive thousands, tens of thousands, or even hundreds of thousands of reviews, from which consumers can gain a wealth of information to aid in their purchasing decisions. However, this large volume of online product reviews not only leads to information overload, but also to the varying quality of reviews, which inadvertently consumes consumers' time and energy in reading them. Therefore, shopping platforms urgently need a mechanism that can automatically filter out high-quality reviews, thereby providing consumers with effective information quickly and accurately to assist in their purchasing decisions.

[0003] Currently, most shopping platforms have adopted a usefulness voting mechanism, which encourages consumers to vote and like the reviews they read, and leave comments. The cumulative number of consumers' likes on the reviews is then used as the basis for review sorting, thereby predicting the usefulness of online product reviews.

[0004] However, most reviews suffer from non-voting bias (readers rarely vote) and time bias (newly published reviews receive fewer votes due to a lack of time). Therefore, truly valuable reviews are difficult to identify based solely on vote counts. In other words, traditional vote counts cannot truly reflect the usefulness of online product reviews. Furthermore, product usefulness reviews come in a variety of formats (images, text, star ratings, etc.), and existing product usefulness review models are not fully adaptable to the diverse forms of multimodal review data. These factors contribute to inaccurate predictions of online product review usefulness based on traditional usefulness voting mechanisms and a single data format. Summary of the Invention

[0005] (1) Technical problems solved

[0006] In view of the shortcomings of the existing technology, the present invention provides a review usefulness prediction method and system based on multimodal information fusion, which solves the problem of inaccurate review usefulness prediction results in the existing technology.

[0007] (2) Technical solution

[0008] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0009] In a first aspect, the present invention first proposes a review usefulness prediction method based on multimodal information fusion, the method comprising:

[0010] Extracting key features that influence the usefulness of product reviews based on multimodal data of product usefulness reviews according to domain expert knowledge; obtaining review usefulness measurement results from a hybrid perspective of domain knowledge based on the key features;

[0011] Obtain product usefulness review measurement results from different perspectives based on multimodal product usefulness review data;

[0012] The usefulness measurement results of the reviews from the mixed perspective of the domain knowledge are fused with the usefulness measurement results of the product reviews from different single perspectives to obtain the final evaluation results of the product usefulness reviews.

[0013] Preferably, the product usefulness review multimodal data includes review text data, review image data, and user voting behavior data.

[0014] Preferably, the obtaining of product usefulness review measurement results from different single perspectives based on the product usefulness review multimodal data includes:

[0015] Constructing a review usefulness measurement model from the review text perspective based on natural language processing technology and deep learning algorithms, and obtaining product usefulness review measurement results from the text perspective using the review usefulness measurement model from the review text perspective and the review text data;

[0016] Constructing a review usefulness measurement model from the perspective of review images based on a deep neural network, and obtaining product usefulness review measurement results from the perspective of images using the review usefulness measurement model from the perspective of review images and the review image data;

[0017] Based on the user voting behavior data, a usefulness threshold of the number of comment votes is preset, and the user voting behavior data is normalized to obtain a product usefulness review measurement result from the voting behavior perspective.

[0018] Preferably, the step of fusing the usefulness measurement results of the reviews from the mixed perspective of the domain knowledge with the usefulness measurement results of the product reviews from different single perspectives to obtain the final evaluation results of the product usefulness reviews includes:

[0019] S31. Based on the evidential reasoning rule, the usefulness measurement results of the reviews from the mixed perspective of the domain knowledge, the usefulness measurement results of the product reviews from the text perspective, the usefulness measurement results of the product reviews from the image perspective, and the usefulness measurement results of the product reviews from the initial vote count perspective are integrated to obtain an evaluation result of the product usefulness review;

[0020] S32: Sort the evaluation results according to the usefulness probability to obtain the final evaluation result of the product usefulness review.

[0021] In a second aspect, the present invention further proposes a review usefulness prediction system based on multimodal information fusion, the system comprising:

[0022] A mixed-perspective review usefulness measurement result acquisition module is used to extract key features that affect the usefulness of product reviews based on multimodal product usefulness review data and domain expert knowledge; and obtain a review usefulness measurement result from a mixed-perspective domain knowledge perspective based on the key features;

[0023] A module for obtaining product usefulness review measurement results from a single perspective is used to obtain product usefulness review measurement results from different single perspectives based on multimodal product usefulness review data.

[0024] A module for obtaining the final evaluation result of product usefulness reviews, configured to fuse the usefulness measurement results of reviews from the mixed perspective of domain knowledge with the usefulness measurement results of product reviews from different single perspectives, to obtain the final evaluation result of product usefulness reviews;

[0025] Preferably, the product usefulness review multimodal data includes review text data, review image data, and user voting behavior data.

[0026] Preferably, the module for obtaining the product usefulness review measurement result from a single perspective obtains the product usefulness review measurement results from different single perspectives based on the product usefulness review multimodal data, including:

[0027] Constructing a review usefulness measurement model from the review text perspective based on natural language processing technology and deep learning algorithms, and obtaining product usefulness review measurement results from the text perspective using the review usefulness measurement model from the review text perspective and the review text data;

[0028] Constructing a review usefulness measurement model from the perspective of review images based on a deep neural network, and obtaining product usefulness review measurement results from the perspective of images using the review usefulness measurement model from the perspective of review images and the review image data;

[0029] Based on the user voting behavior data, a usefulness threshold of the number of comment votes is preset, and the user voting behavior data is normalized to obtain a product usefulness review measurement result from the voting behavior perspective.

[0030] Preferably, the module for obtaining the final evaluation result of the product usefulness review fuses the usefulness measurement result of the review from the mixed perspective of the domain knowledge with the usefulness measurement results of the product usefulness review from different single perspectives to obtain the final evaluation result of the product usefulness review, including:

[0031] S31. Based on the evidential reasoning rule, the usefulness measurement results of the reviews from the mixed perspective of the domain knowledge, the usefulness measurement results of the product reviews from the text perspective, the usefulness measurement results of the product reviews from the image perspective, and the usefulness measurement results of the product reviews from the initial vote count perspective are integrated to obtain an evaluation result of the product usefulness review;

[0032] S32: Sort the evaluation results according to the usefulness probability to obtain the final evaluation result of the product usefulness review.

[0033] In a third aspect, the present invention further provides a computer-readable storage medium storing a computer program for predicting review usefulness based on multimodal information fusion, wherein the computer program causes a computer to execute the following steps:

[0034] Extracting key features that influence the usefulness of product reviews based on multimodal data of product usefulness reviews according to domain expert knowledge; obtaining review usefulness measurement results from a hybrid perspective of domain knowledge based on the key features;

[0035] Obtain product usefulness review measurement results from different perspectives based on multimodal product usefulness review data;

[0036] The usefulness measurement results of the reviews from the mixed perspective of the domain knowledge are fused with the usefulness measurement results of the product reviews from different single perspectives to obtain the final evaluation results of the product usefulness reviews.

[0037] In a fourth aspect, the present invention further provides an electronic device, comprising:

[0038] one or more processors;

[0039] Memory; and

[0040] One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs comprising steps for performing the following steps:

[0041] Extracting key features that influence the usefulness of product reviews based on multimodal data of product usefulness reviews according to domain expert knowledge; obtaining review usefulness measurement results from a hybrid perspective of domain knowledge based on the key features;

[0042] Obtain product usefulness review measurement results from different perspectives based on multimodal product usefulness review data;

[0043] The usefulness measurement results of the reviews from the mixed perspective of the domain knowledge are fused with the usefulness measurement results of the product reviews from different single perspectives to obtain the final evaluation results of the product usefulness reviews.

[0044] (3) Beneficial effects

[0045] The present invention provides a review usefulness prediction method and system based on multimodal information fusion. Compared with the existing technology, it has the following advantages:

[0046] 1. The present invention obtains the usefulness measurement results of reviews from a mixed perspective of domain knowledge and the usefulness measurement results of products from different single perspectives based on the multimodal data of product usefulness reviews and the knowledge of domain experts; then the usefulness measurement results of reviews from a mixed perspective of domain knowledge and the usefulness measurement results of products from different single perspectives are merged to obtain the final evaluation results of the product usefulness reviews. The present invention comprehensively considers multimodal product usefulness review data, not only the content of product usefulness reviews, but also the initial voting behavior of product usefulness, and rationally utilizes the complementarity between information, avoiding the problem of inaccurate prediction results caused by obtaining the usefulness prediction results of product online reviews based only on voting mechanisms and a single data form. The present invention can provide consumers with high-quality review information and reduce the time cost of consumers browsing and searching for valid reviews.

[0047] 2. The present invention obtains the measurement results of product usefulness reviews from single perspectives such as review text, review image and initial vote number, and simultaneously obtains the measurement results of product usefulness reviews from a mixed perspective (cross-modality) of text and image. This can avoid the non-voting bias and time bias in product usefulness reviews, and can make the final prediction results of the usefulness of product online reviews more accurate.

[0048] 3. The present invention takes into account both the uncertainty of data and the uncertainty of the model, and proposes a multi-model information fusion method based on evidence reasoning, which can flexibly process data information from different modalities and self-learn the reliability of each modal information. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0050] Figure 1 This is an overall flow chart of a method for predicting review usefulness based on multimodal information fusion in an embodiment of the present invention;

[0051] Figure 2 This is a diagram illustrating an embodiment of a method for predicting review usefulness based on multimodal information fusion in an embodiment of the present invention. DETAILED DESCRIPTION

[0052] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0053] The embodiments of the present application provide a review usefulness prediction method and system based on multimodal information fusion, thereby solving the problem of inaccurate review usefulness prediction results in the prior art and enabling consumers to quickly read high-quality product review information.

[0054] The technical solution in the embodiments of the present application is to solve the above technical problems, and the overall idea is as follows:

[0055] To address the inaccurate predictions of usefulness from traditional usefulness voting mechanisms and single-data forms of online product reviews, this technical solution uses multimodal product usefulness review data to obtain product usefulness review metrics from a hybrid domain knowledge perspective and from different single perspectives. All these usefulness measurement results are then integrated to obtain the final evaluation results of the product usefulness reviews. This invention comprehensively considers multimodal product usefulness review data, not only considering the content of product usefulness reviews but also the voting behavior of product users. This rationally leverages the complementarity of information, resulting in more accurate predictions of product usefulness reviews. This provides consumers with high-quality review information and reduces the time it takes to browse and search for valid reviews.

[0056] In order to better understand the above technical solution, the above technical solution will be described in detail below with reference to the accompanying drawings and specific implementation methods.

[0057] Example 1:

[0058] In the first aspect, the present invention first proposes a review usefulness prediction method based on multimodal information fusion, see Figure 1 , the method comprising:

[0059] S1. Extract key features that influence the usefulness of product reviews based on multimodal data of product usefulness reviews according to domain expert knowledge; and obtain review usefulness measurement results from a hybrid perspective of domain knowledge based on the key features.

[0060] S2. Obtain product usefulness review measurement results from different single perspectives based on multimodal data of product usefulness reviews;

[0061] S3. Fusing the usefulness measurement results of the reviews from the mixed perspective of the domain knowledge with the usefulness measurement results of the product reviews from different single perspectives to obtain a final evaluation result of the product usefulness review.

[0062] As can be seen, this embodiment obtains the usefulness measurement results of reviews from a mixed perspective of domain knowledge and the usefulness measurement results of products from different single perspectives based on the multimodal data of product usefulness reviews according to the knowledge of domain experts; and then fuses the usefulness measurement results of reviews from a mixed perspective of domain knowledge and the usefulness measurement results of products from different single perspectives to obtain the final evaluation results of the product usefulness reviews. The present invention comprehensively considers multimodal product usefulness review data, not only considering the content of product usefulness reviews, but also considering the initial voting behavior of product usefulness, rationally utilizing the complementarity between information, and avoiding the problem of inaccurate prediction results caused by obtaining the usefulness prediction results of product online reviews based solely on voting mechanisms and a single data form. The present invention can provide consumers with high-quality review information and reduce the time cost of consumers browsing and searching for valid reviews.

[0063] The following is combined with Figure 1-2 The implementation process of an embodiment of the present invention is described in detail with explanations of the specific steps S1-S3.

[0064] Different reviewers will comment on the usefulness of the product in different forms, thus generating different forms of product usefulness review data, namely, product usefulness review multimodal data. Generally speaking, product usefulness review data includes review text data, review image data, and review voting data, but is not limited to these forms of data. Reviewers can also comment in different forms from other perspectives, thereby generating review data other than the above forms, and these other forms of review data are also fully applicable to the technical solution of the present invention. In this embodiment, we take the three modalities (forms) of product usefulness review data, namely, review text data, review image data, and user voting behavior data, as examples to illustrate the specific implementation process of this technical solution, see Figure 1 and 2 .

[0065] S1. Based on the multimodal data of product usefulness reviews, key features that affect the usefulness of product reviews are extracted according to domain expert knowledge; based on the key features, review usefulness measurement results from a mixed perspective of domain knowledge are obtained.

[0066] Using the key features extracted based on domain knowledge features, traditional machine learning methods are used to predict the usefulness of reviews, and the degree of review usefulness from a mixed perspective of domain knowledge expressed in the form of evidence is output as evidence E1.

[0067] A hybrid perspective does not specifically distinguish the specific form of product usefulness review data, but instead treats the product usefulness review data as a whole to obtain measurement results of the product usefulness review data. It is actually a cross-modal data processing method. Based on the knowledge of domain experts, feature extraction is performed on the multimodal data of product usefulness reviews submitted by users to extract the key features that affect the usefulness of product online reviews from the product online review information submitted by users. The key features obtained mainly include: topic complexity in the review, topic consistency of the review, sentiment bias, review star rating and sentiment consistency, review star rating and review text consistency, text readability, image complexity, image and text similarity, and other information. The specific steps are as follows:

[0068] 1) Vectorize and preprocess the review text, images, structured information, and other information in the multimodal data of product usefulness reviews.

[0069] 1.1) For the review text, extract key feature information such as the complexity of the review text, the consistency of the review topic, sentiment bias, consistency between the review star rating and sentiment, consistency between the review star rating and the review text, sentiment score of the review information, text readability, sentence length of the review text, number of nouns, verbs, personal pronouns, pronouns, adjectives, and adverbs. Specifically,

[0070] The topic complexity of the reviews. Using the LDA topic model, we extract N topics from all reviews, obtain the probability distribution of each review under each topic, and calculate the information entropy of each review. The number of extracted topics is determined by calculating the perplexity and plotting the topic-perplexity curve.

[0071] The topic consistency of the reviews. This is the cosine of the topic probability distribution of the reviews and the average topic probability distribution. The average topic probability distribution is the mean of the probabilities under each topic.

[0072] Sentiment deviation. This is the difference between the sentiment score of each comment and the average sentiment score. Calculate the sentiment score of each comment, represented by α, and the average sentiment score is represented by express,

[0073] Review star rating and sentiment consistency. That is, the difference between the sentiment score of each review and the star rating of the review. Normalize the review star rating and use Indicates that s represents the star rating of the review. represents the mean star rating of reviews for this type of product, σ s Indicates the variance of the star rating; normalize the sentiment score and use χ to represent it. Among them, α represents the sentiment score, represents the average sentiment score, and the review star and sentiment consistency = |χ-β|.

[0074] Review star rating and review text consistency. The difference between the number of nouns, verbs, personal pronouns, pronouns, adjectives, and adverbs in each review and the average number of nouns, verbs, personal pronouns, pronouns, adjectives, and adverbs in the review at the same star rating. i , ε i , γ i ,η i , κ i ,λ i Indicates the number of nouns, verbs, personal pronouns, pronouns, adjectives, and adverbs in each comment. represents the average number of nouns, verbs, personal pronouns, pronouns, adjectives, and adverbs under each star level, σ i δ , σ i ε , σ i γ , σ i η , σ i λ , σ i λ Indicates the standard deviation of nouns, verbs, personal pronouns, pronouns, adjectives, and adverbs at each star rating. Star rating i = 1, 2, 3, 4, 5. Review star and review text consistency The consistency of other parts of speech is consistent with this.

[0075] Text readability. Text readability includes three indicators: Flesch Reading Ease Index, Gunning FOG Index, and Automatic Readability Index. Among them, Flesch Reading Ease = 206.83 – (1.015 × ASL) – (84.6 × ASW). Among them, ASL = average sentence length (number of words divided by number of sentences); ASW = average number of syllables in a word (number of syllables divided by number of words); Gunning FOG calculation formula is 0.4 [(word / sentences) + 100 (complex words / words)]. Among them: complex words refer to the number of words with three or more syllables;

[0076] The Automatic Readability Index (ARI) is calculated as: ARI = 4.71 (characters / words) + 0.5 (words / sentences) - 21.43. Characters represent the number of letters, numbers, and punctuation marks; words are calculated based on the number of spaces.

[0077] 1.2) In terms of images, key feature information such as the number of review images, image complexity, and similarity between images and text is extracted.

[0078] Image complexity. That is, the uncertainty of the object categories contained in the image. Using the VGG16 pre-trained model, the number of object categories is set to 1000, and the probability distribution of the classification is [μ1,μ2,...,μ 1000 ], the complexity of an image is the information entropy under this classification. The greater the information entropy, the greater the uncertainty of the classification information it contains, and the more complex the image is.

[0079] Image and text similarity. Use deep learning networks to learn deep feature representations of images and text and convert them into vectors of the same dimension. The vector representation of an image is [ρ1,ρ2,...,ρ n ], the vector of the text is represented as [v1,ν2,...,ν n ], where n = 256. The similarity between the image and the text is expressed by calculating the cosine value between the two vectors. The calculation formula is as follows:

[0080] 2) Input the extracted key features into the review usefulness measurement model from the domain knowledge hybrid perspective and train the model based on the existing data labels. Once the review usefulness measurement model from the domain knowledge perspective is trained, it can be used to obtain the review usefulness measurement results from the domain knowledge perspective as evidence E1.

[0081] The logistic prediction model (decision model 1) is typically used to measure review usefulness from a hybrid domain knowledge perspective. After this model has been trained to achieve the desired effect, the trained logistic prediction model is used to fit the domain knowledge features of newly input reviews. This yields a prediction of usefulness (expressed as the probability of a review being rated useful), which serves as evidence E1. During model training, the training and test sets are split in a 7:3 ratio. The average accuracy of the prediction model, obtained through ten-fold cross-validation, serves as the reliability of the evidence.

[0082] S2. Obtain product usefulness review measurement results from different single perspectives based on multimodal product usefulness review data.

[0083] Based on natural language processing technology and deep learning algorithms, a review text classification model is trained to obtain a deep feature representation of the review text, and the degree of review usefulness from the text perspective expressed in the form of evidence is output as evidence E2; based on a deep neural network, a review image classification model is trained to obtain a deep feature representation of the review image, and the degree of review usefulness from the image perspective is output as evidence E3.

[0084] To increase the complementarity of the data, in contrast to the aforementioned mixed perspective, we also obtain product usefulness review measurement results from other single perspectives of the product usefulness review data. This means dividing the product usefulness review data into different modalities based on their specific form, and then obtaining measurement results for the product usefulness review data based on the different modalities. This embodiment obtains product usefulness review measurement results from three perspectives: the product review text perspective, the product review image perspective, and the product user voting behavior perspective. Specifically:

[0085] 1) Input the review text into the review usefulness measurement model from the review text perspective. Then, use the trained model to learn and represent the deep features of the review text to obtain the product usefulness measurement results from the review text perspective as evidence E2. Specifically, the Bert-BiLSTM prediction model is used to obtain the product usefulness measurement results from the review text perspective.

[0086] A text-based Bert-BiLSTM prediction model (decision model 2) is constructed, and then the review text is input into the pre-trained BiLSTM model for fitting to obtain the classification results as evidence E2.

[0087] The Bert-BiLSTM prediction model consists of a Bert word embedding layer, an attention layer, a BiLSTM layer, and a fully connected layer. Because the number of words in reviews can vary significantly, the maximum word count is set to 400. This ensures that the model captures as complete information as possible while reducing computational complexity. The Bert word embedding has a dimension of 768, and the output of the attention layer is also 768-dimensional. The BiLSTM layer has one layer, and the output dimension is 128.

[0088] The Bert-BiLSTM prediction model is based on the pytorch framework, uses cross entropy as the loss function, and adopts the stochastic gradient descent (SGD) optimizer to optimize and update the parameters.

[0089] 2) Input the review image into the review usefulness measurement model from the review image perspective. The trained model is then used to learn and represent the deep features of the review image, obtaining the measurement results of the product usefulness reviews from the review image perspective as evidence E3. Specifically, the VGG16 prediction model is used to obtain the measurement results of the product usefulness reviews from the review image perspective.

[0090] Construct a VGG16 prediction model for image perspective (decision model 3). First, concatenate the images contained in the comments and input the concatenated images into a pre-trained VGG16 model for fitting to obtain the classification results, which serve as evidence E3. A two-class image classification network is constructed based on the VGG16 model, and the prediction accuracy of this classification model serves as the reliability of evidence E3.

[0091] 3) Convert the product’s user voting behavior data into evidence E4.

[0092] Using a threshold of 10 votes, comments with more than 10 votes are treated as having 10 votes. The number of votes is then limited to between 0 and 10. The votes are then normalized (i.e., converted to a range of 0-1), and the normalized value is used as the probability of the useful class. For example, if a comment has 8 votes and the normalized result is 0.8, the evidence is represented as (0.2, 0.8). 0.2 represents the probability of the useless class, and 0.8 represents the probability of the useful class.

[0093] S3. Fusing the usefulness measurement results of the reviews from the mixed perspective of the domain knowledge with the usefulness measurement results of the product reviews from different single perspectives to obtain a final evaluation result of the product usefulness review.

[0094] In order to simultaneously consider the uncertainty of data and model, a multi-model information fusion method based on evidential reasoning is proposed, which can flexibly process data information from different modalities and self-learn the reliability of each modal information.

[0095] The number of votes received by the product online reviews is used as the initial usefulness measurement, namely evidence E4. Combined with the above-mentioned evidence E1, E2, and E3, considering the reliability and weight of the evidence, the evidence reasoning rules are used to fuse the review usefulness evidence obtained from different perspectives of multiple modal data (including the initial vote count perspective, the domain knowledge mixed perspective, the review text single perspective, and the review image single perspective). The fusion result is the final usefulness evaluation result, as shown below. Figure 2 shown.

[0096] Evidence fusion. The initial weight of each piece of evidence from the four decision models is set to a random number between 0 and 1. The optimal weight for each piece of evidence is obtained through optimization learning. The usefulness metrics from different models are fused using an evidence inference algorithm to obtain a corrected usefulness score.

[0097] The ER combination rule consists of the following steps:

[0098] 1) Represent the results from each decision model as evidence e i ={(θ,p θ,i )|θ=1,2}, where i=1,…,4, is the number of evidences; θ represents the usefulness level of the review, 1 means useful and 2 means useless. θ,i Indicates the degree on each level.

[0099] 2) Determine the reliability r of the i-th piece of evidence i and weight w iAssume that the uncertainty of the i-th decision model is expressed as r i,m , using the model's classification performance such as accuracy or F value to measure. The corresponding evidence e i The uncertainty of itself is expressed as r i,d , measured by the Gini coefficient, The reliability of evidence is defined as r i =(αr i,m +βr i,d ) / 2, where α and β are coefficients between 0 and 1, and can be learned through optimization. The weight of each piece of evidence is set to be equal to its corresponding reliability.

[0100] 3) Using evidence reasoning rules, the evidence with reliability and weight is integrated. The result of combining two independent pieces of evidence is:

[0101]

[0102]

[0103] m θ,j =w j p θ,j ,j=1,…4

[0104] The process of training the parameters in the ER combination rule is as follows:

[0105] Through the PyTorch framework, we randomly generate the weight of evidence, use cross entropy as the loss function, and use backpropagation to obtain the most appropriate weight of evidence. The loss function calculation formula of the model is as follows:

[0106]

[0107] Among them, y i Indicates the label value of sample i, the positive class is 1, the negative class is 0, p i Represents the probability that sample i is predicted to be a positive class. The SGD optimizer is used to optimize and update the parameters to obtain the minimum loss function value, and the weight when the loss function value is the minimum is used as the final weight value of the model.

[0108] Results output and ranking. The probability of online product reviews falling into the usefulness category is output and used as the corrected usefulness. Finally, the reviews are ranked by the probability of usefulness. This allows users to save time and effort by sorting the evidence and accurately obtain the high-quality, useful product reviews they desire, helping them make purchasing decisions.

[0109] At this point, the entire process of the review usefulness prediction method based on multimodal information fusion of the present invention is completed.

[0110] Example 2:

[0111] In a second aspect, the present invention further provides a review usefulness prediction system based on multimodal information fusion, the system comprising:

[0112] A mixed-perspective review usefulness measurement result acquisition module is used to extract key features that affect the usefulness of product reviews based on multimodal product usefulness review data and domain expert knowledge; and obtain a review usefulness measurement result from a mixed-perspective domain knowledge perspective based on the key features;

[0113] A module for obtaining product usefulness review measurement results from a single perspective is used to obtain product usefulness review measurement results from different single perspectives based on multimodal product usefulness review data.

[0114] A module for obtaining the final evaluation result of product usefulness reviews, configured to fuse the usefulness measurement results of reviews from the mixed perspective of domain knowledge with the usefulness measurement results of product reviews from different single perspectives, to obtain the final evaluation result of product usefulness reviews;

[0115] Optionally, the product usefulness review multimodal data includes review text data, review image data, and user voting behavior data.

[0116] Optionally, the single-perspective product usefulness review measurement result acquisition module acquires product usefulness review measurement results from different single perspectives based on the product usefulness review multimodal data, including:

[0117] Constructing a review usefulness measurement model from the review text perspective based on natural language processing technology and deep learning algorithms, and obtaining product usefulness review measurement results from the text perspective using the review usefulness measurement model from the review text perspective and the review text data;

[0118] Constructing a review usefulness measurement model from the perspective of review images based on a deep neural network, and obtaining product usefulness review measurement results from the perspective of images using the review usefulness measurement model from the perspective of review images and the review image data;

[0119] Based on the user voting behavior data, a usefulness threshold of the number of comment votes is preset, and the user voting behavior data is normalized to obtain a product usefulness review measurement result from the voting behavior perspective.

[0120] Optionally, the module for obtaining the final evaluation result of the product usefulness review fuses the usefulness measurement results of the review from the mixed perspective of the domain knowledge with the usefulness measurement results of the product usefulness review from different single perspectives to obtain the final evaluation result of the product usefulness review, including:

[0121] S31. Based on the evidential reasoning rule, the usefulness measurement results of the reviews from the mixed perspective of the domain knowledge, the usefulness measurement results of the product reviews from the text perspective, the usefulness measurement results of the product reviews from the image perspective, and the usefulness measurement results of the product reviews from the initial vote count perspective are integrated to obtain an evaluation result of the product usefulness review;

[0122] S32: Sort the evaluation results according to the usefulness probability to obtain the final evaluation result of the product usefulness review.

[0123] It can be understood that the review usefulness prediction system based on multimodal information fusion provided by the embodiment of the present invention corresponds to the above-mentioned review usefulness prediction method based on multimodal information fusion. The explanation, examples, beneficial effects, etc. of the relevant contents can refer to the corresponding contents in the review usefulness prediction method based on multimodal information fusion, and will not be repeated here.

[0124] Example 3:

[0125] In a third aspect, the present invention further provides a computer-readable storage medium storing a computer program for predicting review usefulness based on multimodal information fusion, wherein the computer program causes a computer to execute the following steps:

[0126] Extracting key features that influence the usefulness of product reviews based on multimodal data of product usefulness reviews according to domain expert knowledge; obtaining review usefulness measurement results from a hybrid perspective of domain knowledge based on the key features;

[0127] Obtain product usefulness review measurement results from different perspectives based on multimodal product usefulness review data;

[0128] The usefulness measurement results of the reviews from the mixed perspective of the domain knowledge are fused with the usefulness measurement results of the product reviews from different single perspectives to obtain the final evaluation results of the product usefulness reviews.

[0129] Example 4:

[0130] In a fourth aspect, the present invention further provides an electronic device, comprising:

[0131] one or more processors;

[0132] Memory; and

[0133] One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs comprising steps for performing the following steps:

[0134] Extracting key features that influence the usefulness of product reviews based on multimodal data of product usefulness reviews according to domain expert knowledge; obtaining review usefulness measurement results from a hybrid perspective of domain knowledge based on the key features;

[0135] Obtain product usefulness review measurement results from different perspectives based on multimodal product usefulness review data;

[0136] The usefulness measurement results of the reviews from the mixed perspective of the domain knowledge are fused with the usefulness measurement results of the product reviews from different single perspectives to obtain the final evaluation results of the product usefulness reviews.

[0137] In summary, compared with the existing technology, the present invention has the following beneficial effects:

[0138] 1. The present invention obtains the usefulness measurement results of reviews from a mixed perspective of domain knowledge and the usefulness measurement results of products from different single perspectives based on the multimodal data of product usefulness reviews and the knowledge of domain experts; then the usefulness measurement results of reviews from a mixed perspective of domain knowledge and the usefulness measurement results of products from different single perspectives are merged to obtain the final evaluation results of the product usefulness reviews. The present invention comprehensively considers multimodal product usefulness review data, not only the content of product usefulness reviews, but also the initial voting behavior of product usefulness, and rationally utilizes the complementarity between information, avoiding the problem of inaccurate prediction results caused by obtaining the usefulness prediction results of product online reviews based only on voting mechanisms and a single data form. The present invention can provide consumers with high-quality review information and reduce the time cost of consumers browsing and searching for valid reviews.

[0139] 2. The present invention obtains the measurement results of product usefulness reviews from single perspectives such as review text, review image and initial vote number, and simultaneously obtains the measurement results of product usefulness reviews from a mixed perspective (cross-modality) of text and image. This can avoid the non-voting bias and time bias in product usefulness reviews, and can make the final prediction results of the usefulness of product online reviews more accurate.

[0140] 3. The present invention takes into account both the uncertainty of data and the uncertainty of the model, and proposes a multi-model information fusion method based on evidence reasoning, which can flexibly process data information from different modalities and self-learn the reliability of each modal information.

[0141] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0142] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A review usefulness prediction method based on multimodal information fusion, characterized by: The method comprises: Extract key features that influence the usefulness of product reviews based on multimodal data of product usefulness reviews according to domain expert knowledge; obtain review usefulness measurement results from a hybrid perspective of domain knowledge based on the key features, wherein the key features include topic complexity in the reviews, topic consistency of the reviews, sentiment bias, consistency between review star ratings and sentiment, consistency between review star ratings and review text, image and text similarity, text readability, and image complexity; Obtain product usefulness review measurement results from different perspectives based on multimodal product usefulness review data; Based on the rules of evidential reasoning, the usefulness measurement results of reviews from the mixed perspective of domain knowledge, the usefulness measurement results of product reviews from the text perspective, the usefulness measurement results of product reviews from the image perspective, and the usefulness measurement results of product reviews from the initial vote count perspective are integrated to obtain the evaluation results of product usefulness reviews, including: 1) The usefulness measurement results of reviews from the mixed perspective of domain knowledge, the usefulness measurement results of product reviews from the text perspective, the usefulness measurement results of product reviews from the image perspective, and the usefulness measurement results of product reviews from the initial vote count perspective are represented as evidence ,where i=1,…,4,is the number of evidences; θ represents the usefulness level of the review, 1 means useful, 2 means useless, Indicates the degree on each level; 2) Determine the reliability of the i-th piece of evidence and weights : Assume that the uncertainty of the review usefulness measurement model corresponding to the i-th evidence is expressed as , the i-th evidence The uncertainty itself is expressed as , then the reliability of the i-th evidence is , where α and β are coefficients between 0 and 1, and the weight of each piece of evidence is equal to its corresponding reliability; 3) Fusion of evidence with reliability and weight: The synthesis result of two independent pieces of evidence is: in, , , i =1,…,4; The evaluation results are sorted according to the usefulness probability to obtain the final evaluation results of the product usefulness reviews.

2. The method according to claim 1, wherein The product usefulness review multimodal data includes review text data, review image data, and user voting behavior data.

3. The method according to claim 2, wherein The method of obtaining product usefulness review measurement results from different single perspectives based on the product usefulness review multimodal data includes: Constructing a review usefulness measurement model from the review text perspective based on natural language processing technology and deep learning algorithms, and obtaining product usefulness review measurement results from the text perspective using the review usefulness measurement model from the review text perspective and the review text data; Constructing a review usefulness measurement model from the perspective of review images based on a deep neural network, and obtaining product usefulness review measurement results from the perspective of images using the review usefulness measurement model from the perspective of review images and the review image data; Based on the user voting behavior data, a usefulness threshold of the number of comment votes is preset, and the user voting behavior data is normalized to obtain a product usefulness review measurement result from the voting behavior perspective.

4. A review usefulness prediction system based on multimodal information fusion, characterized by: The system comprises: A hybrid perspective review usefulness measurement result acquisition module is used to extract key features that affect the usefulness of product reviews based on multimodal product usefulness review data and domain expert knowledge; based on the key features, a review usefulness measurement result from a hybrid perspective of domain knowledge is acquired, wherein the key features include topic complexity in the review, topic consistency of the review, sentiment bias, consistency between review star rating and sentiment, consistency between review star rating and review text, image and text similarity, text readability, and image complexity; A module for obtaining product usefulness review measurement results from a single perspective is used to obtain product usefulness review measurement results from different single perspectives based on multimodal product usefulness review data. Based on the rules of evidential reasoning, the usefulness measurement results of reviews from the mixed perspective of domain knowledge, the usefulness measurement results of product reviews from the text perspective, the usefulness measurement results of product reviews from the image perspective, and the usefulness measurement results of product reviews from the initial vote count perspective are integrated to obtain the evaluation results of product usefulness reviews, including: 1) The usefulness measurement results of reviews from the mixed perspective of domain knowledge, the usefulness measurement results of product reviews from the text perspective, the usefulness measurement results of product reviews from the image perspective, and the usefulness measurement results of product reviews from the initial vote count perspective are represented as evidence ,where i=1,…,4,is the number of evidences; θ represents the usefulness level of the review, 1 means useful, 2 means useless, Indicates the degree on each level; 2) Determine the reliability of the i-th piece of evidence and weights : Assume that the uncertainty of the review usefulness measurement model corresponding to the i-th evidence is expressed as , the i-th evidence The uncertainty itself is expressed as , then the reliability of the i-th evidence is , where α and β are coefficients between 0 and 1, and the weight of each piece of evidence is equal to its corresponding reliability; 3) Fusion of evidence with reliability and weight: The synthesis result of two independent pieces of evidence is: in, , , i =1,…,4; The evaluation results are sorted according to the usefulness probability to obtain the final evaluation results of the product usefulness reviews.

5. The system according to claim 4, wherein: The product usefulness review multimodal data includes review text data, review image data, and user voting behavior data.

6. The system according to claim 5, wherein: The module for obtaining the product usefulness review measurement result from a single perspective obtains the product usefulness review measurement results from different single perspectives based on the product usefulness review multimodal data, including: Constructing a review usefulness measurement model from the review text perspective based on natural language processing technology and deep learning algorithms, and obtaining product usefulness review measurement results from the text perspective using the review usefulness measurement model from the review text perspective and the review text data; Constructing a review usefulness measurement model from the perspective of review images based on a deep neural network, and obtaining product usefulness review measurement results from the perspective of images using the review usefulness measurement model from the perspective of review images and the review image data; Based on the user voting behavior data, a usefulness threshold of the number of comment votes is preset, and the user voting behavior data is normalized to obtain a product usefulness review measurement result from the voting behavior perspective.

7. A computer-readable storage medium, characterized in that The computer program stores a computer program for predicting review usefulness based on multimodal information fusion, wherein the computer program causes the computer to execute the following steps: Extract key features that influence the usefulness of product reviews based on multimodal data of product usefulness reviews according to domain expert knowledge; obtain review usefulness measurement results from a hybrid perspective of domain knowledge based on the key features, wherein the key features include topic complexity in the reviews, topic consistency of the reviews, sentiment bias, consistency between review star ratings and sentiment, consistency between review star ratings and review text, image and text similarity, text readability, and image complexity; Obtain product usefulness review measurement results from different perspectives based on multimodal product usefulness review data; Based on the rules of evidential reasoning, the usefulness measurement results of reviews from the mixed perspective of domain knowledge, the usefulness measurement results of product reviews from the text perspective, the usefulness measurement results of product reviews from the image perspective, and the usefulness measurement results of product reviews from the initial vote count perspective are integrated to obtain the evaluation results of product usefulness reviews, including: 1) The usefulness measurement results of reviews from the mixed perspective of domain knowledge, the usefulness measurement results of product reviews from the text perspective, the usefulness measurement results of product reviews from the image perspective, and the usefulness measurement results of product reviews from the initial vote count perspective are represented as evidence ,where i=1,…,4,is the number of evidences; θ represents the usefulness level of the review, 1 means useful, 2 means useless, Indicates the degree on each level; 2) Determine the reliability of the i-th piece of evidence and weights : Assume that the uncertainty of the review usefulness measurement model corresponding to the i-th evidence is expressed as , the i-th evidence The uncertainty itself is expressed as , then the reliability of the i-th evidence is , where α and β are coefficients between 0 and 1, and the weight of each piece of evidence is equal to its corresponding reliability; 3) Fusion of evidence with reliability and weight: The synthesis result of two independent pieces of evidence is: in, , , i =1,…,4; The evaluation results are sorted according to the usefulness probability to obtain the final evaluation results of the product usefulness reviews.

8. An electronic device, characterized in that: include: one or more processors; Memory; as well as One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs comprising steps for performing the following steps: Extract key features that influence the usefulness of product reviews based on multimodal data of product usefulness reviews according to domain expert knowledge; obtain review usefulness measurement results from a hybrid perspective of domain knowledge based on the key features, wherein the key features include topic complexity in the reviews, topic consistency of the reviews, sentiment bias, consistency between review star ratings and sentiment, consistency between review star ratings and review text, image and text similarity, text readability, and image complexity; Obtain product usefulness review measurement results from different perspectives based on multimodal product usefulness review data; Based on the rules of evidential reasoning, the usefulness measurement results of reviews from the mixed perspective of domain knowledge, the usefulness measurement results of product reviews from the text perspective, the usefulness measurement results of product reviews from the image perspective, and the usefulness measurement results of product reviews from the initial vote count perspective are integrated to obtain the evaluation results of product usefulness reviews, including: 1) The usefulness measurement results of reviews from the mixed perspective of domain knowledge, the usefulness measurement results of product reviews from the text perspective, the usefulness measurement results of product reviews from the image perspective, and the usefulness measurement results of product reviews from the initial vote count perspective are represented as evidence ,where i=1,…,4,is the number of evidences; θ represents the usefulness level of the review, 1 means useful, 2 means useless, Indicates the degree on each level; 2) Determine the reliability of the i-th piece of evidence and weights : Assume that the uncertainty of the review usefulness measurement model corresponding to the i-th evidence is expressed as , the i-th evidence The uncertainty itself is expressed as , then the reliability of the i-th evidence is , where α and β are coefficients between 0 and 1, and the weight of each piece of evidence is equal to its corresponding reliability; 3) Fusion of evidence with reliability and weight: The synthesis result of two independent pieces of evidence is: in, , , i =1,…,4; The evaluation results are sorted according to the usefulness probability to obtain the final evaluation results of the product usefulness reviews.

Images