Knowledge-enhanced user multimodal online review quality assessment method and system

Abstract

The present invention provides a knowledge-enhanced user multimodal online review quality assessment method, system, storage medium and electronic device, and relates to the technical field of review quality assessment. The present invention constructs a knowledge-enhanced multimodal online review quality assessment model, introduces text part-of-speech features and external quality grade labels in multimodal review data into the model as prior knowledge, and enhances the performance of model quality assessment. A two-layer language-aware text attention mechanism is proposed, which adds the intrinsic part-of-speech features of words as prior knowledge to the word-level and sentence-level attention mechanisms, and uses language features to filter and learn local and global text information; a two-level supervised comparative learning strategy is also proposed, which uses the inherent correlation relationship in multimodal review data to simultaneously learn multimodal semantic feature representations within a single sample instance and between different sample instances.

Description

Technical Field

[0001] The present invention relates to the technical field of review quality assessment, and in particular to a knowledge-enhanced user multimodal online review quality assessment method, system, storage medium and electronic device. Background Art

[0002] Developing new energy vehicles is a major strategic initiative for my country to address climate change and promote green development. The rapid development and deep integration of next-generation information technologies such as big data, artificial intelligence, and cloud computing with intelligent manufacturing have placed higher demands on the rapid iteration and update of new energy vehicle products. With the increasing diversification of user needs, user-centric continuous innovation has become a core competitive advantage for automobile companies. User participation in vehicle development is becoming the primary model for new energy vehicle companies to conduct vehicle R&D. For example, users are involved in product definition, development, verification, configuration, pricing, and improvement. Users and engineers can interact anytime and anywhere, forming a user-generated online community. High-quality user reviews in these communities not only provide valuable reference information for potential buyers, guiding their purchasing decisions, but also significantly assist companies in optimizing and improving their automotive products. However, due to the overload of online reviews and the varying quality of reviews on social media platforms, manual screening and classification of review quality consumes considerable time and effort, is ineffective, and hinders companies from obtaining truly valuable business information in a timely manner. Furthermore, online platforms also contain a large number of reviews that include images, which provide a more intuitive presentation of product information. Therefore, considering the images and text information in online reviews, a multimodal online review quality assessment model is constructed to effectively identify high-quality online reviews, which can fully tap the potential value of user-generated content, provide decision support for car buyers, and provide product optimization ideas for new energy vehicle companies.

[0003] A multimodal online review quality assessment model is proposed in the literature [1] (Xiao S, Chen G, Zhang C, et al. Complementary or substitutive? A novel deep learning method to leverage text-image interactions for multimodal review helpfulness prediction [J]. Expert Systems with Applications, 2022, 208: 118138.). Considering that the information in the text (image) in multimodal reviews can be a supplement or substitute for the corresponding image (text) review information, a new multimodal deep learning method is designed; first, the pre-trained BERT model and VGG model are used to extract the embedded feature vectors of the review text and review image, and then the LSTM network and attention mechanism are used respectively to learn the sequence dependency and importance weight relationship within the single modality of text and image, and finally, the optimized fusion parameters are used to coordinate the complementary effect and substitution effect between the review text and review image to achieve multimodal review quality assessment.

[0004] However, although the above solution takes into account the text and image information in the comments at the same time and effectively overcomes the problems of traditional machine learning methods such as being time-consuming and labor-intensive and heavily relying on manual features constructed based on specific domain knowledge, this method does not fully consider the inherent correlation relationships between and within instances of multimodal data and make full use of external knowledge to mine such correlation relationships. Summary of the Invention

[0005] (1) Technical problems solved

[0006] In response to the shortcomings of the existing technology, the present invention provides a knowledge-enhanced user multimodal online review quality assessment method, system, storage medium and electronic device, which solves the technical problem of not fully considering the inherent correlation relationships between and within instances of multimodal data.

[0007] (2) Technical solution

[0008] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0009] A knowledge-enhanced user multimodal online review quality assessment method is based on a knowledge-enhanced multimodal online review quality assessment model, which includes a multimodal input layer, an intra-modal feature extraction layer, an inter-modal feature interaction layer, and a fusion output layer. The quality assessment method includes:

[0010] S1. Obtaining a user's historical multimodal online comments, where any of the historical multimodal online comments includes a comment text and an associated image;

[0011] S2. In the multimodal input layer, the comment text and the associated image are encoded respectively to obtain text feature representation and image feature representation respectively;

[0012] S3, in the intra-modal feature extraction layer, deep text semantic feature representation is obtained based on the text attention mechanism; and deep image semantic feature representation is obtained based on the image attention mechanism;

[0013] S4. At the inter-modal feature interaction layer, based on the deep text semantic feature representation and the deep image semantic feature representation, a semantic fusion feature vector between modalities is obtained based on a collaborative attention mechanism;

[0014] S5. In the fusion output layer, concatenate the deep text semantic feature representation, the deep image semantic feature representation, and the semantic fusion feature vector to predict the quality assessment level of the historical multimodal online reviews;

[0015] S6. constructing a main loss function based on the predicted and true quality assessment grades; constructing an auxiliary loss function based on the two-level supervised comparative learning strategy within and between instances of the multimodal review data according to each of the historical multimodal online reviews;

[0016] Constructing a total loss function based on the main loss function and the auxiliary loss function; and training the model according to the total loss function until convergence;

[0017] S7. Using the multimodal online review to be evaluated as the input of the converged model to obtain a quality evaluation grade of the multimodal online review to be evaluated.

[0018] Preferably, the main loss function in S6 is The acquisition process includes:

[0019] Given a training instance B in the same batch, which contains |B| training samples, each training sample corresponds to a historical multimodal online review, the main task loss is defined as the cross entropy loss of the multi-task problem.

[0020]

[0021] Among them, y ic represents the true label of the quality level c of the i-th sample; represents the probability value of the i-th sample being identified as quality level c; K is the total number of quality levels.

[0022] Preferably, the auxiliary loss function in S6 The acquisition process includes:

[0023] (1) Intra-instance consistency comparison loss function:

[0024] The deep text semantic feature representation T′ learned by the intra-modal feature extraction layer for each training sample in the training instance B is i , deep image semantic feature representation I′ i Composition of picture and text pairs The image-text pairs whose true labels of the training samples are greater than or equal to the given label threshold are marked as positive samples. The image-text pairs whose true labels of the training samples are less than the given label threshold are marked as negative samples. Then in the training instance B under the same batch, the intra-modal consistency contrast loss is calculated as follows:

[0025]

[0026]

[0027] Where log is the logarithmic function; D = {1, 2, ..., |B|} is the index set of samples in instance B; is the scoring function; exp is the exponential function; sim is the similarity evaluation, sim(A, B) = A·B; τ is the temperature hyperparameter;

[0028] (2) Inter-instance consistency comparison loss function:

[0029] Define the quality level true label corresponding to each training sample in training instance B as (y1, y2, ..., y |B| ), the corresponding feature vectors obtained by each training sample in the inter-modal feature interaction layer are represented by F i , the mutual distance in the embedding space is adjusted, where the distance between feature vectors with the same quality level comments is reduced, and the distance between feature vectors with different quality level comments is increased. Then, in the training instance B under the same batch, the inter-modality consistency contrast loss is calculated as follows:

[0030]

[0031] Among them, C i ={c|y c =y i , c≠i} is the same batch B as y i Samples with the same label, |C i | indicates the corresponding number of samples;

[0032] And the total loss function is expressed as:

[0033]

[0034] Here, λ is a hyperparameter used to adjust the effect of auxiliary contrastive learning loss.

[0035] Preferably, the S3 introduces a two-layer language feature-aware text attention mechanism, which includes word-level and sentence-level attention layers, wherein:

[0036] (1) Obtaining word-level text feature representations based on local language feature perception at the word-level attention layer, including:

[0037] S311. Design word-level part-of-speech embedding methods; including:

[0038] For a given review text S, the j-th review sentence sequence W in the review text S j ={[cls],w1,w2,...,w n-2 , all words in [sep]} are divided into four different categories according to their parts of speech: adjectives, adverbs, verbs and other words;

[0039] Where [cls] and [sep] represent the start and end characters respectively; j = 1, 2, ..., m, where m is the total number of comment sentences; n represents the maximum length of the comment sentence sequence;

[0040] Construct and randomly initialize the part-of-speech embedding matrix Among them, p adj 、p adv 、p verb 、p other Represents the part-of-speech embeddings of adjectives, adverbs, verbs, and other words respectively; d p is the dimension of part-of-speech embedding;

[0041] According to W j The part of speech category corresponding to each word in P vocab W j Converted into word-level part-of-speech embedding representation of the same length as the sentence sequence

[0042] S312, define word-level text feature representation as Will Converted into query vector and key vector And perform scaled dot product calculation with the attention score of word-level part-of-speech embedding Add them together and calculate the word-level text feature attention weights of language feature perception through the softmax activation function

[0043]

[0044] in, is a trainable parameter matrix, dim1 is The dimension of the vector embedding;

[0045] S313, will Convert to the corresponding value vector and through Perform weighted fusion on them and then use residual connection to obtain word-level text feature representation that is aware of local language features

[0046]

[0047] (2) At the sentence-level attention layer, a sentence-level text feature representation based on global language feature perception is obtained and used as the deep text semantic feature representation, specifically including:

[0048] S321, W j Corresponding output Take the average to obtain the sentence-level text feature representation corresponding to the comment text S d represents the dimension of the word feature vector;

[0049]

[0050] Among them, Avg represents average pooling;

[0051] S322, W j ={[cls],w1,w2,...,w n-2 , sum up the part-of-speech categories of all words in [sep]} to obtain W j The number of each part of speech category is N j ={n adj , n adv , n verb , n other}, and the part-of-speech embedding matrix Get the sentence sequence W j Sentence-level part-of-speech embedding representation after corresponding weighted fusion Then the sentence-level part-of-speech embedding corresponding to the comment text S is expressed as

[0052] S323, T s Converted into query vector Q s and key vector K s , and perform a scaled dot product operation with the attention score E of the sentence-level part-of-speech embedding sAdd and use the softmax activation function to calculate the sentence-level text feature attention weights of language feature perception

[0053]

[0054] in, is the trainable parameter matrix, dim2 is Q s , K s The dimension of the vector embedding;

[0055] S324, T s Convert to value vector V s , and use the attention weight A s Perform weighted fusion on it and obtain the global language feature perception sentence-level text feature representation corresponding to the comment text S through residual connection

[0056] T′=T s +A s V s .

[0057] Preferably, the step of obtaining the semantic fusion feature vector F between modalities based on the collaborative attention mechanism in S4 specifically refers to:

[0058]

[0059] Among them, Attention represents the collaborative attention mechanism; Q I′ Represents the query vector corresponding to the deep image semantic feature I′, K T′ 、V T′ Represents the key and value vectors corresponding to the deep text semantic features T′ respectively; dim3 is Q I′ , K T′ The dimension of the vector embedding.

[0060] Preferably, the S5 specifically includes:

[0061] Concatenate the features of T′, I′, and F to obtain the vector Z, and convert the concatenated vector Z into a probability distribution for the multi-classification problem through a linear function and a softmax activation function;

[0062] Z = concat([F, I′, T′])

[0063]

[0064]

[0065] Among them, Wz, b zare the weight parameters and bias terms corresponding to the linear function in the output layer; X = {x1, x2, ..., x K} is converted into a probability distribution represents the probability value of a sample being identified as quality level c; K is the total number of quality levels;

[0066] The quality grade corresponding to the maximum probability value is used as the quality assessment grade prediction result of the historical multimodal online reviews.

[0067] Preferably, in S2:

[0068] A pre-trained BERT model is used to encode the comment text; and / or a pre-trained ResNet50 model is used to encode the associated image.

[0069] A user multimodal online review quality assessment system is based on a knowledge-enhanced multimodal online review quality assessment model. The model includes a multimodal input layer, an intra-modal feature extraction layer, an inter-modal feature interaction layer, and a fusion output layer. The quality assessment system includes:

[0070] An acquisition module is used to acquire a user's historical multimodal online comments, where any of the historical multimodal online comments includes a comment text and an associated image;

[0071] An encoding module, configured to encode the comment text and associated images in a multimodal input layer, respectively, and obtain text feature representation and image feature representation respectively;

[0072] The extraction module is used to obtain deep text semantic feature representation based on the text attention mechanism and deep image semantic feature representation based on the image attention mechanism at the intra-modal feature extraction layer;

[0073] An interaction module is configured to obtain a semantic fusion feature vector between modalities based on the deep text semantic feature representation and the deep image semantic feature representation at the inter-modal feature interaction layer based on a collaborative attention mechanism;

[0074] A prediction module is configured to, at the fusion output layer, combine the deep text semantic feature representation, the deep image semantic feature representation, and the semantic fusion feature vector to predict the quality assessment level of the historical multimodal online reviews;

[0075] A training module is configured to construct a primary loss function based on the predicted and true quality assessment grades; and an auxiliary loss function is constructed based on a two-level supervised comparative learning strategy within and between instances of the multimodal review data, based on each of the historical multimodal online reviews;

[0076] Constructing a total loss function based on the main loss function and the auxiliary loss function; and training the model according to the total loss function until convergence;

[0077] The evaluation module is used to take the multimodal online review to be evaluated as the input of the converged model to obtain the quality evaluation level of the multimodal online review to be evaluated.

[0078] A storage medium stores a computer program for evaluating the quality of user multimodal online reviews for knowledge enhancement, wherein the computer program enables a computer to execute the method for evaluating the quality of user multimodal online reviews as described above.

[0079] An electronic device, comprising:

[0080] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, the programs including a method for executing the user multimodal online review quality assessment method as described above.

[0081] (3) Beneficial effects

[0082] The present invention provides a knowledge-enhanced method, system, storage medium, and electronic device for evaluating the quality of multimodal online user reviews. Compared with existing technologies, the present invention has the following advantages:

[0083] The present invention is based on a knowledge-enhanced multimodal online review quality assessment model. When training the model, a two-level supervised comparative learning strategy is proposed for both inter-instance and intra-instance multimodal review data. When considering the interaction between text and image information, it is no longer just a simple splicing or attention interaction of the features of the text and image in the review. Instead, it fully utilizes the inherent correlation relationship between and within instances of multimodal review data. By using the existing review quality level labels as external knowledge, the consistency between the review text and the review image is distinguished based on the review quality level within a single sample instance. At the same time, the consistency of different review instances is distinguished based on whether the quality levels of multiple sample instances are the same, thereby learning a more comprehensive multimodal feature representation and improving the review quality assessment performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0085] Figure 1A block diagram of a knowledge-enhanced user multimodal online review quality assessment method provided by an embodiment of the present invention;

[0086] Figure 2 A model architecture diagram for multimodal online review quality assessment provided by an embodiment of the present invention;

[0087] Figure 3 An architectural diagram of a dual-layer language feature-aware text attention mechanism provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0088] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0089] The embodiments of the present application solve the technical problem of not fully considering the inherent correlation relationships between and within instances of multimodal data by providing a knowledge-enhanced user multimodal online review quality assessment method, system, storage medium and electronic device.

[0090] The technical solution in the embodiments of the present application is to solve the above technical problems, and the overall idea is as follows:

[0091] The purpose of this invention is to provide a knowledge-enhanced multimodal review quality assessment method. It includes a two-layer language-aware attention mechanism with word-level and sentence-level part-of-speech embeddings and a two-level supervised comparative learning strategy for inter-instance and intra-instance multimodal review data.

[0092] Specifically:

[0093] In a first aspect, an embodiment of the present invention provides a text attention mechanism with dual-layer language feature perception including word-level and sentence-level part-of-speech embedding.

[0094] First, to address the problem of insufficient representation of implicit semantic features in review texts due to sparse content and irregular expressions, we introduce external linguistic knowledge to perform part-of-speech tagging on the review texts. By constructing a part-of-speech embedding matrix and tagging keywords (such as verbs, adjectives, or adverbs) in the review texts, we obtain word-level part-of-speech embedding representations. We then fuse the sentence-level part-of-speech category statistics with the word-level part-of-speech embedding matrix to obtain a sentence-level part-of-speech embedding representation. Together, the word-level and sentence-level part-of-speech embedding representations of the review text constitute its linguistic features.

[0095] Next, in order to further extract local and global features related to language features, a two-layer language feature-aware attention mechanism is proposed. Language features are introduced into the neural network model as a priori knowledge, and word-level part-of-speech embeddings are added to the word-level text attention mechanism. The word-level attention weights of language feature awareness are learned to extract local information related to language features; sentence-level part-of-speech embeddings are added to the sentence-level text attention mechanism, and the sentence-level attention weights of language feature awareness are learned to extract global information related to language features; local and global noises are filtered through layered language features to learn more effective text feature representations.

[0096] On the second aspect, the embodiments of the present invention provide a two-level supervised comparative learning strategy between instances and within instances of multimodal review data. When considering the interaction between text and image information, the existing multimodal review quality assessment methods only perform simple splicing or attention interaction on the features of the text and image in the review, and do not fully utilize the inherent correlation relationship between and within instances of multimodal review data. The present invention proposes a two-level supervised consistency comparative learning strategy, which uses the existing review quality level labels as external knowledge, distinguishes the degree of consistency between the review text and the review image based on the review quality level within a single sample instance, and distinguishes the degree of consistency between different review instances based on whether the quality levels of multiple sample instances are the same, thereby learning a more comprehensive multimodal feature representation.

[0097] In order to better understand the above technical solution, the above technical solution will be described in detail below with reference to the accompanying drawings and specific implementation methods.

[0098] Example:

[0099] like Figure 1 As shown, the embodiment of the present invention provides a knowledge-enhanced user multimodal online review quality assessment method, based on the knowledge-enhanced multimodal online review quality assessment model, such as Figure 2 The model shown includes a multimodal input layer, an intra-modal feature extraction layer, an inter-modal feature interaction layer, and a fusion output layer; the quality assessment method includes:

[0100] S1. Obtaining a user's historical multimodal online comments, where any of the historical multimodal online comments includes a comment text and an associated image;

[0101] S2. In the multimodal input layer, the comment text and the associated image are encoded respectively to obtain text feature representation and image feature representation respectively;

[0102] S3, in the intra-modal feature extraction layer, deep text semantic feature representation is obtained based on the text attention mechanism; and deep image semantic feature representation is obtained based on the image attention mechanism;

[0103] S4. At the inter-modal feature interaction layer, based on the deep text semantic feature representation and the deep image semantic feature representation, a semantic fusion feature vector between modalities is obtained based on a collaborative attention mechanism;

[0104] S5. In the fusion output layer, concatenate the deep text semantic feature representation, the deep image semantic feature representation, and the semantic fusion feature vector to predict the quality assessment level of the historical multimodal online reviews;

[0105] S6. constructing a main loss function based on the predicted and true quality assessment grades; constructing an auxiliary loss function based on the two-level supervised comparative learning strategy within and between instances of the multimodal review data according to each of the historical multimodal online reviews;

[0106] Constructing a total loss function based on the main loss function and the auxiliary loss function; and training the model according to the total loss function until convergence;

[0107] S7. Using the multimodal online review to be evaluated as the input of the converged model to obtain a quality evaluation grade of the multimodal online review to be evaluated.

[0108] The embodiment of the present invention is based on a knowledge-enhanced multimodal online review quality assessment model. When training the model, a two-level supervised comparative learning strategy for multimodal review data instances and within instances is proposed. When considering the interaction between text and image information, it is no longer just a simple splicing or attention interaction of the features of the text and image in the review. Instead, it fully utilizes the inherent correlation relationship between multimodal review data and within instances. By using the existing review quality level labels as external knowledge, the consistency between the review text and the review image is distinguished based on the review quality level within a single sample instance. At the same time, the consistency of different review instances is distinguished based on whether the quality levels of multiple sample instances are the same, thereby learning a more comprehensive multimodal feature representation, thereby improving the review quality assessment performance.

[0109] The following sections describe the various steps of the above solution in detail:

[0110] In step S1, a user's historical multimodal online comments are obtained, where any of the historical multimodal online comments includes a comment text and an associated image.

[0111] Given a multimodal review input, including a review text S and an associated image V, the review text S consists of multiple sentences, represented as S = {W1, W2, ..., W m ), m represents the total number of sentences in a review text.

[0112] In step S2, the comment text and the associated image are encoded separately in the multimodal input layer, and text feature representation and image feature representation are obtained accordingly.

[0113] (1) Use the pre-trained BERT model to encode the review text:

[0114] The j-th comment sentence sequence W in the comment text S j ={[cls], W1, w2, ..., w n-2 , [sep]}, j = 1, 2, ..., m is input to the BERT model, and the corresponding comment text feature representation is obtained by encoding The text feature corresponding to the comment text S is expressed as Where n represents the maximum length of the review sentence sequence, and d represents the dimension of the word feature vector.

[0115] (2) Use the pre-trained ResNet50 model to encode the associated images:

[0116] Image after pixel adjustment As the input of the ResNet50 model, the output of the last convolutional layer of the model is taken as the corresponding comment image feature representation Among them, 49 (7*7) is the product of the height and width of the image output by the last convolutional layer of the ResNet50 model, which can be regarded as the number of visual area blocks in the image.

[0117] In step S3, at the intra-modal feature extraction layer, a deep text semantic feature representation is obtained based on the text attention mechanism; and a deep image semantic feature representation is obtained based on the image attention mechanism.

[0118] In order to solve the problem of insufficient representation of textual implicit semantic features due to sparse content and irregular expression of review texts, and to further extract local and global features related to language features, the embodiment of the present invention also introduces a text attention mechanism that includes word-level and sentence-level part-of-speech embedding and dual-layer language feature perception. This mechanism mainly extracts the implicit semantic feature representation within the review text modality through word-level and sentence-level part-of-speech embedding. The overall attention network design is as follows: Figure 3 As shown in Figure 1, it includes word-level and sentence-level attention layers, where the input is word-level part-of-speech embedding and word-level text representation, and the output is language feature-aware text feature representation.

[0119] (1) Obtaining word-level text feature representations based on local language feature perception at the word-level attention layer:

[0120] Word-level text feature representation The word-level features related to the local information of the review text are included in the model. However, the existing word-level text feature extraction methods do not take into account the implicit semantic information in the text, making it difficult to learn the truly important information. The word part-of-speech category of the text, as a linguistic feature, can well reflect the positions of important words in the review text. Therefore, introducing the part-of-speech information as a priori knowledge into the model can effectively learn the important local information. Figure 3 As shown, in expanded terms:

[0121] S311. Design word-level part-of-speech embedding methods; including:

[0122] For a given review text S, the j-th review sentence sequence W in the review text S j ={[cls],w1,w2,...,w n-2 , all words in [sep]} are divided into four different categories according to their parts of speech: adjectives, adverbs, verbs and other words;

[0123] Wherein, [cls] and [sep] represent the start and end characters respectively; j = 1, 2, ..., m;

[0124] Construct and randomly initialize the part-of-speech embedding matrix Among them, p adj 、p adv 、p verb 、p other Represents the part-of-speech embeddings of adjectives, adverbs, verbs, and other words respectively; d p is the dimension of part-of-speech embedding;

[0125] According to W j The part of speech category corresponding to each word in P vocab W j Converted into word-level part-of-speech embedding representation of the same length as the sentence sequence

[0126] S312, define word-level text feature representation as Will Converted into query vector and key vector And perform scaled dot product calculation with the attention score of word-level part-of-speech embedding Add them together and calculate the word-level text feature attention weights of language feature perception through the softmax activation function

[0127]

[0128] in, is a trainable parameter matrix, dim1 is The dimension of the vector embedding;

[0129] S313, will Convert to the corresponding value vector and through Perform weighted fusion on them and then use residual connection to obtain word-level text feature representation that is aware of local language features

[0130] (2) At the sentence-level attention layer, a sentence-level text feature representation based on global language feature perception is obtained and used as the deep text semantic feature representation:

[0131] The part-of-speech category of a text can not only reflect the important word positions in the review text, but also highlight the important sentence regions in the text; therefore, introducing this language feature as prior knowledge can also enable the model to effectively learn sentence-level text feature representations related to global information. Figure 3 As shown, in expanded terms:

[0132] S321, W j Corresponding output Take the average to obtain the sentence-level text feature representation corresponding to the comment text S

[0133]

[0134] Among them, Avg represents average pooling;

[0135] S322, W j ={[cls],w1,w2,...,w n-2 , sum up the part-of-speech categories of all words in [sep]}, and obtain the number of each part-of-speech category corresponding to Wj, recorded as N j ={n adj , n adv , n verb , n other}, and the part-of-speech embedding matrix Get the sentence sequence W j Sentence-level part-of-speech embedding representation after corresponding weighted fusion Then the sentence-level part-of-speech embedding corresponding to the comment text S is expressed as

[0136] S323, T s Converted into query vector Q s and key vector K s , and perform a scaled dot product operation with the attention score E of the sentence-level part-of-speech embedding sAdd and use the softmax activation function to calculate the sentence-level text feature attention weights of language feature perception

[0137]

[0138] in, is the trainable parameter matrix, dim2 is Q s , K s The dimension of the vector embedding;

[0139] S324, T s Convert to value vector V s , and use the attention weight A s Perform weighted fusion on it and obtain the global language feature perception sentence-level text feature representation corresponding to the comment text S through residual connection

[0140] T′=T s +A s V s .

[0141] In addition, for the image feature I, the self-attention mechanism is used to mine the deep semantic information of the image to obtain the learned deep image semantic feature representation I′, which is shown as follows:

[0142]

[0143] I′=Attention(Q I , K I , V I )

[0144] in, is the trainable weight parameter matrix, d h is the dimension of the hidden layer in the attention mechanism, dim is Q I , K I The dimension of the vector embedding.

[0145] In step S4, at the inter-modal feature interaction layer, according to the deep text semantic feature representation and the deep image semantic feature representation, a semantic fusion feature vector between modalities is obtained based on a collaborative attention mechanism.

[0146] In this step, the semantic fusion feature vector F between modalities is obtained based on the collaborative attention mechanism, specifically:

[0147]

[0148] Among them, Attention represents the collaborative attention mechanism; QI′ Represents the query vector corresponding to the deep image semantic feature I′, K T′ 、V T′ Represents the key and value vectors corresponding to the deep text semantic features T′ respectively; dim3 is Q I′ , K T′ The dimension of the vector embedding.

[0149] In step S5, at the fusion output layer, the deep text semantic feature representation, the deep image semantic feature representation, and the semantic fusion feature vector are spliced ​​together to predict the quality assessment level of the historical multimodal online reviews. Specifically, the steps include:

[0150] Concatenate the features of T′, I′, and F to obtain the vector Z, and convert the concatenated vector Z into a probability distribution for the multi-classification problem through a linear function and a softmax activation function;

[0151] Z = concat([F, I′, T′])

[0152]

[0153]

[0154] Among them, W z 、b z are the weight parameters and bias terms corresponding to the linear function in the output layer; X = {x1, x2, ..., x K} is converted into a probability distribution represents the probability value of a sample being identified as quality level c; K is the total number of quality levels;

[0155] The quality grade corresponding to the maximum probability value is used as the quality assessment grade prediction result of the historical multimodal online reviews.

[0156] In step S6, a main loss function is constructed based on the predicted and true grades of the quality assessment; an auxiliary loss function is constructed based on the two-level supervised comparative learning strategy within and between instances of the multimodal review data according to each of the historical multimodal online reviews;

[0157] A total loss function is constructed based on the main loss function and the auxiliary loss function; and the model is trained until convergence based on the total loss function.

[0158] Specifically, the main loss function The acquisition process includes:

[0159] Given a training instance B in the same batch, which contains |B| training samples, each training sample corresponds to a historical multimodal online review, the main task loss is defined as the cross entropy loss of the multi-task problem.

[0160]

[0161] Among them, y ic represents the true label of the quality level c of the i-th sample; represents the probability value of the i-th sample being identified as quality level c; K is the total number of quality levels.

[0162] The two-level supervised contrastive learning strategy is based on the inherent correlation relationship between and within instances of multimodal review data, while considering the consistency of images and texts within a single sample instance and the consistency of different review instances between multiple sample instances to extract the multimodal semantic feature representation of the reviews.

[0163] Based on mutual information theory, separating positive and negative examples by similarity during model training can enhance the model's learned representations and improve the performance of downstream tasks. Therefore, leveraging the inherent correlations between and within instances of multimodal review data to construct positive and negative examples during model training can better assist the model in perceiving the quality of multimodal reviews. To calculate the comparison score for each sample pair, a commonly used exponential function was selected as the scoring function. The specific formula is as follows:

[0164]

[0165] Here, similarity is measured by the commonly used sim(A, B) = A·B; is the scoring function; exp is the exponential function; sim is the similarity evaluation, sim(A, B) = A·B; τ is the temperature hyperparameter.

[0166] Correspondingly, the auxiliary loss function The acquisition process includes:

[0167] (1) Intra-instance consistency comparison loss function:

[0168] The deep text semantic feature representation T′ learned by the intra-modal feature extraction layer for each training sample in the training instance B is i , deep image semantic feature representation I′ i Composition of picture and text pairs The image-text pairs whose true labels of the training samples are greater than or equal to the given label threshold are marked as positive samples. The image-text pairs whose true labels of the training samples are less than the given label threshold are marked as negative samples. Then in the training instance B under the same batch, the intra-modal consistency contrast loss is calculated as follows:

[0169]

[0170] Wherein, D={1, 2, ..., |B|} is the index set of samples in instance B.

[0171] (2) Inter-instance consistency comparison loss function:

[0172] Define the quality level true label corresponding to each training sample in training instance B as (y1, y2, ..., y |B| ), the corresponding feature vectors obtained by each training sample in the inter-modal feature interaction layer are represented by F i , the mutual distance in the embedding space is adjusted, where the distance between feature vectors with the same quality level comments is reduced, and the distance between feature vectors with different quality level comments is increased. Then, in the training instance B under the same batch, the inter-modality consistency contrast loss is calculated as follows:

[0173]

[0174] Among them, C i ={c|y c =y i , c≠i} is the same batch B as y i Samples with the same label, |C i | indicates the corresponding number of samples;

[0175] Then the total loss function is expressed as:

[0176]

[0177] Here, λ is a hyperparameter used to adjust the effect of auxiliary contrastive learning loss.

[0178] In step S7, the multimodal online review to be evaluated is used as the input of the converged model to obtain the quality evaluation level of the multimodal online review to be evaluated.

[0179] In order to better help the embodiments of the present invention provide the advantages of the solution, the following specific examples are provided, including:

[0180] (1) Using Python web crawler technology, we crawled review data related to automobile products on platforms such as Autohome and Dongchedi. The main crawled content includes review pictures, review texts, and review likes, forming a multimodal review dataset.

[0181] (2) Preprocess the data in the multimodal review dataset.

[0182] ① Remove comments with repeated content or less than 4 words from the comment data, and divide the comment text into sentences so that each comment text S can be represented as several sentences.

[0183] ② Use the Natural Language Toolkit (NLTK) to tag all text content, including punctuation, with adjectives, adverbs, verbs, and other parts of speech. For example, given the sentence "the car runs smoothly.", the corresponding tag sequence is (other, other, verb, adv, other).

[0184] (3) Divide the dataset into training and test sets to evaluate and select the model.

[0185] (4) Use the pytorch framework to build a deep neural network model and train it on the training set.

[0186] ① Segment the comment text into words and convert the comment images into pixels, and then feed them into the pre-trained BERT model and pre-trained ResNet50 model respectively to learn text and image features.

[0187] ② Construct a part-of-speech embedding matrix. This is combined with the previously learned text features to form a word-level attention mechanism. This word-level part-of-speech embedding adjusts the text attention weights. The sentence's part-of-speech category summary information is calculated. For example, the part-of-speech summary information for (other, other, verb, adv, other) is (0, 1, 1, 3). Based on the part-of-speech embedding matrix, a sentence-level part-of-speech embedding is constructed. This is then combined with the sentence-level text features learned from the word-level attention mechanism to form a sentence-level attention mechanism. Image features are updated using the self-attention mechanism.

[0188] ③The updated text features and image features are updated through a collaborative attention mechanism, where text features are keys and values, and image features are queries.

[0189] ④ For the text feature and image feature vectors obtained in step 2, an intra-modal consistency comparison loss function is constructed based on the true value of usefulness. For the feature vectors obtained in step 3, an inter-modal consistency comparison loss function is constructed based on the true value of usefulness.

[0190] ⑤ Concatenate the text feature and image feature vectors obtained in step 2 with the feature vector obtained in step 3, and convert them into binary probability values ​​through a linear layer and a sigmoid function, and construct a cross-entropy loss function based on the true value of usefulness.

[0191] ⑥ The cross entropy loss is used as the main loss of the model, and the consistency contrast loss obtained in step 4 is used as the auxiliary loss of the model. The auxiliary loss is adjusted using hyperparameters; the optimizer is used to backpropagate the model and update the gradient.

[0192] (5) The performance of the model is evaluated on the test set using three indicators: accuracy, F1-Score, and AUC. The model parameters are tuned based on these indicators, and the tuned model is trained and continuously adjusted until the performance of the model reaches the optimal level. The calculation formulas for the three indicators are shown below.

[0193]

[0194]

[0195] Among them, TP means that the sample is actually positive and predicted to be positive, that is, a true positive example; TN means that the sample is actually negative and predicted to be negative, that is, a true negative example; FP means that the sample is actually negative and predicted to be positive, that is, a false positive example; FN means that the sample is actually positive and predicted to be negative, that is, a false negative example.

[0196] The embodiment of the present invention provides a knowledge-enhanced user multimodal online review quality assessment system. The knowledge-enhanced multimodal online review quality assessment model includes a multimodal input layer, an intra-modal feature extraction layer, an inter-modal feature interaction layer, and a fusion output layer. The quality assessment system includes:

[0197] An acquisition module is used to acquire a user's historical multimodal online comments, where any of the historical multimodal online comments includes a comment text and an associated image;

[0198] An encoding module, configured to encode the comment text and associated images in a multimodal input layer, respectively, and obtain text feature representation and image feature representation respectively;

[0199] The extraction module is used to obtain deep text semantic feature representation based on the text attention mechanism and deep image semantic feature representation based on the image attention mechanism at the intra-modal feature extraction layer;

[0200] An interaction module is configured to obtain a semantic fusion feature vector between modalities based on the deep text semantic feature representation and the deep image semantic feature representation at the inter-modal feature interaction layer based on a collaborative attention mechanism;

[0201] A prediction module is configured to, at the fusion output layer, combine the deep text semantic feature representation, the deep image semantic feature representation, and the semantic fusion feature vector to predict the quality assessment level of the historical multimodal online reviews;

[0202] A training module is configured to construct a primary loss function based on the predicted and true quality assessment grades; and an auxiliary loss function is constructed based on a two-level supervised comparative learning strategy within and between instances of the multimodal review data, based on each of the historical multimodal online reviews;

[0203] Constructing a total loss function based on the main loss function and the auxiliary loss function; and training the model according to the total loss function until convergence;

[0204] The evaluation module is used to take the multimodal online review to be evaluated as the input of the converged model to obtain the quality evaluation level of the multimodal online review to be evaluated.

[0205] An embodiment of the present invention provides a storage medium storing a computer program for quality assessment of user multimodal online reviews for knowledge enhancement, wherein the computer program enables a computer to execute the above-described method for quality assessment of user multimodal online reviews.

[0206] An embodiment of the present invention provides an electronic device, including:

[0207] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, the programs including a method for executing the user multimodal online review quality assessment method as described above.

[0208] It is understandable that the knowledge-enhanced user multimodal online review quality assessment system, storage medium and electronic device provided in the embodiments of the present invention correspond to the knowledge-enhanced user multimodal online review quality assessment method provided in the embodiments of the present invention. The explanations, examples and beneficial effects of the relevant contents can refer to the corresponding parts in the user multimodal online review quality assessment method, and will not be repeated here.

[0209] In summary, compared with the existing technology, the present invention has the following beneficial effects:

[0210] 1. It takes into account both text and image information in reviews, making better use of data on online platforms, improving review quality assessment performance, and helping customers and businesses find useful information.

[0211] 2. Language features are constructed based on the part-of-speech tags of the text, and are introduced into the quality assessment model as a priori knowledge through a two-layer language feature-aware text attention mechanism. This can effectively extract the implicit semantic features of the review text and timely filter out noise information in the review text.

[0212] 3. The two-level supervised contrastive learning strategy can fully utilize the external label knowledge and the inherent consistency matching relationship in multimodal review data to learn a more comprehensive multimodal feature representation.

[0213] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0214] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A knowledge-enhanced user multimodal online review quality assessment method, characterized by: A knowledge-enhanced multimodal online review quality assessment model, which includes a multimodal input layer, an intra-modal feature extraction layer, an inter-modal feature interaction layer, and a fusion output layer; The quality assessment method includes: S1. Obtaining a user's historical multimodal online comments, where any of the historical multimodal online comments includes a comment text and multiple associated images; S2. In the multimodal input layer, the comment text and the associated image are encoded respectively to obtain text feature representation and image feature representation respectively; S3, in the intra-modal feature extraction layer, deep text semantic feature representation is obtained based on the text attention mechanism; and deep image semantic feature representation is obtained based on the image attention mechanism; S4. At the inter-modal feature interaction layer, based on the deep text semantic feature representation and the deep image semantic feature representation, a semantic fusion feature vector between modalities is obtained based on a collaborative attention mechanism; S5. In the fusion output layer, concatenate the deep text semantic feature representation, the deep image semantic feature representation, and the semantic fusion feature vector to predict the quality assessment level of the historical multimodal online reviews; S6. constructing a main loss function based on the predicted and true quality assessment grades; constructing an auxiliary loss function based on the two-level supervised comparative learning strategy within and between instances of the multimodal review data according to each of the historical multimodal online reviews; Constructing a total loss function based on the main loss function and the auxiliary loss function; and training the model according to the total loss function until convergence; S7. Using the multimodal online review to be evaluated as the input of the converged model to obtain a quality evaluation grade of the multimodal online review to be evaluated.

2. The method for evaluating the quality of user multimodal online reviews according to claim 1, wherein: The main loss function in S6 The acquisition process includes: Given a training instance B in the same batch, which contains |B| training samples, each training sample corresponds to a historical multimodal online review, the main task loss is defined as the cross entropy loss of the multi-task problem. Among them, y ic represents the true label of the quality level c of the i-th sample; represents the probability value of the i-th sample being identified as quality level c; K is the total number of quality levels.

3. The method for evaluating the quality of multimodal online user reviews according to claim 2, wherein: The auxiliary loss function in S6 The acquisition process includes: (1) Intra-instance consistency comparison loss function: The deep text semantic feature representation T learned by the intra-modal feature extraction layer for each training sample in the training instance B is i ′, deep image semantic feature representation I i 'composes a picture-text pair The image-text pairs whose true labels of the training samples are greater than or equal to the given label threshold are marked as positive samples. The image-text pairs whose true labels of the training samples are less than the given label threshold are marked as negative samples. Then in the training instance B under the same batch, the intra-modal consistency contrast loss is calculated as follows: Where log is the logarithmic function; D = {1, 2, ..., |B|} is the index set of samples in instance B; is the scoring function; exp is the exponential function; sim is the similarity evaluation, sim(A,B) = A·B; τ is the temperature hyperparameter; (2) Inter-instance consistency comparison loss function: Define the quality level true label corresponding to each training sample in training instance B as (y1,y2,…,y |b| ), the corresponding feature vectors obtained by each training sample in the inter-modal feature interaction layer are represented by F i , the mutual distance in the embedding space is adjusted, where the distance between feature vectors with the same quality level comments is reduced, and the distance between feature vectors with different quality level comments is increased. Then, in the training instance B under the same batch, the inter-modality consistency contrast loss is calculated as follows: Among them, C i ={c|y c =y i ,c≠i} is the same batch B as y i Samples with the same label, |C i | indicates the corresponding number of samples; And the total loss function is expressed as: Here, λ is a hyperparameter used to adjust the effect of auxiliary contrastive learning loss.

4. The method for evaluating the quality of multimodal online user reviews according to claim 1, wherein: S3 introduces a two-layer language feature-aware text attention mechanism, which includes word-level and sentence-level attention layers, where: (1) Obtaining word-level text feature representations based on local language feature perception at the word-level attention layer, including: S311. Design word-level part-of-speech embedding methods; including: For a given review text S, the j-th review sentence sequence W in the review text S j ={[cls],w1,w2,…,w n-2 All words in [sep]} are divided into four different categories according to their parts of speech: adjectives, adverbs, verbs and other words; Where [cls] and [sep] represent the start and end characters respectively; j = 1, 2, ..., m, where m is the total number of comment sentences; n represents the maximum length of the comment sentence sequence; Construct and randomly initialize the part-of-speech embedding matrix Among them, p adj 、p adv 、p verb 、p other Represents the part-of-speech embeddings of adjectives, adverbs, verbs, and other words respectively; d p is the dimension of part-of-speech embedding; According to W j The part of speech category corresponding to each word in P vocab W j Converted into word-level part-of-speech embedding representation of the same length as the sentence sequence S312, define word-level text feature representation as T j w , T j w Converted into query vector and key vector K j w , and perform scaled dot product calculation with the attention score E of the word-level part-of-speech embedding j w Add them together and calculate the word-level text feature attention weights of language feature perception through the softmax activation function in, is a trainable parameter matrix, dim1 is K j w The dimension of the vector embedding; S313, T j w Converted into the corresponding value vector V j w and through Perform weighted fusion on them and then use residual connection to obtain word-level text feature representation that is aware of local language features (2) At the sentence-level attention layer, a sentence-level text feature representation based on global language feature perception is obtained and used as the deep text semantic feature representation, specifically including: S321, W j Corresponding output T j w′ Take the average to obtain the sentence-level text feature representation corresponding to the comment text S d represents the dimension of the word feature vector; Among them, Acg represents average pooling; S322, W j ={[cls],w1,w2,…,w n-2 ,[sep]}, and the part-of-speech categories of all words in the word list are summarized to obtain W j The number of each part of speech category is N j ={n adj ,n adv ,n verb ,n other }, and the part-of-speech embedding matrix Get the sentence sequence W j Sentence-level part-of-speech embedding representation after corresponding weighted fusion Then the sentence-level part-of-speech embedding corresponding to the comment text S is expressed as S323, T s Converted into query vector Q s and key vector K s , and perform a scaled dot product operation with the attention score E of the sentence-level part-of-speech embedding s Add and use the softmax activation function to calculate the sentence-level text feature attention weights of language feature perception in, is the trainable parameter matrix, dim2 is Q s , K s The dimension of the vector embedding; S324, T s Convert to value vector V s , and use the attention weight A s Perform weighted fusion on it and obtain the global language feature perception sentence-level text feature representation corresponding to the comment text S through residual connection T′=T s +A s V s 。 5. The method for evaluating the quality of user multimodal online reviews according to claim 1, wherein: The semantic fusion feature vector F between modalities is obtained based on the collaborative attention mechanism in S4, specifically: Among them, Attention represents the collaborative attention mechanism; Q I′ Represents the query vector corresponding to the deep image semantic feature I', K T′ 、V T′ Represents the key and value vectors corresponding to the deep text semantic features T'; dim3 is Q I′ , K T′ The dimension of the vector embedding.

6. The method for evaluating the quality of user multimodal online reviews according to claim 5, wherein: The S5 specifically includes: Concatenate T', I', and F to obtain the vector Z, and convert the concatenated vector Z into a probability distribution for multi-classification problems through a linear function and a softmax activation function. Z = concat([F, I′, T′]) X=W z T Z+b z Among them, W z 、b z are the weight parameters and bias terms corresponding to the linear function in the output layer; X = {x1, x2, ..., x K } is converted into a probability distribution represents the probability value of a sample being identified as quality level c; K is the total number of quality levels; The quality grade corresponding to the maximum probability value is used as the quality assessment grade prediction result of the historical multimodal online reviews.

7. The method for evaluating the quality of user multimodal online reviews according to any one of claims 1 to 6, wherein: In S2: A pre-trained BERT model is used to encode the comment text; and / or a pre-trained ResNet50 model is used to encode the associated image.

8. A user multimodal online review quality assessment system, characterized by: A knowledge-enhanced multimodal online review quality assessment model, which includes a multimodal input layer, an intra-modal feature extraction layer, an inter-modal feature interaction layer, and a fusion output layer; The quality assessment system includes: An acquisition module is used to acquire a user's historical multimodal online comments, where any of the historical multimodal online comments includes a comment text and an associated image; An encoding module, configured to encode the comment text and associated images in a multimodal input layer, respectively, and obtain text feature representation and image feature representation respectively; The extraction module is used to obtain deep text semantic feature representation based on the text attention mechanism and deep image semantic feature representation based on the image attention mechanism at the intra-modal feature extraction layer; An interaction module is configured to obtain a semantic fusion feature vector between modalities based on the deep text semantic feature representation and the deep image semantic feature representation at the inter-modal feature interaction layer based on a collaborative attention mechanism; A prediction module is configured to, at the fusion output layer, combine the deep text semantic feature representation, the deep image semantic feature representation, and the semantic fusion feature vector to predict the quality assessment level of the historical multimodal online reviews; A training module is configured to construct a primary loss function based on the predicted and true quality assessment grades; and an auxiliary loss function is constructed based on a two-level supervised comparative learning strategy within and between instances of the multimodal review data, based on each of the historical multimodal online reviews; Constructing a total loss function based on the main loss function and the auxiliary loss function; and training the model according to the total loss function until convergence; The evaluation module is used to take the multimodal online review to be evaluated as the input of the converged model to obtain the quality evaluation level of the multimodal online review to be evaluated.

9. A storage medium, characterized in that: The computer program for evaluating the quality of user multimodal online reviews based on knowledge enhancement is stored therein, wherein the computer program enables a computer to execute the method for evaluating the quality of user multimodal online reviews based on any one of claims 1 to 7.

10. An electronic device, characterized in that: include: one or more processors; Memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the programs include a method for executing the user multimodal online review quality assessment method according to any one of claims 1 to 7.

Images