Multi-modal food false comment detection method based on graph attention network

Abstract

The invention provides a multi-modal food false comment detection method based on a graph attention network. The method comprises the following four main steps: firstly, obtaining text and picture data of food comments from a comment platform through a web crawler technology, and performing data cleaning by utilizing a Python toolkit to obtain a standard data format; secondly, extracting text and image features by using a pre-trained feature extractor, and converting the features into a modal diagram structure; secondly, mapping the text and the visual modal graph into a unified feature space, and realizing full fusion of multi-modal features through a graph attention network; and finally, extracting global features through global average pooling operation, classifying the global features by using a full-connection classifier, and optimizing the model in combination with a cross entropy loss function so as to realize accurate detection of the authenticity of food comments. According to the method, the defects of a multi-modal fusion method are overcome, so that the accuracy and robustness of food false comment detection are effectively improved.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence, deep learning and natural language processing, and in particular to a multimodal food false review detection method based on a graph attention network. The method integrates multimodal data such as text and images, and uses graph neural network technology to identify and detect false reviews. Background Art

[0002] In the digital age, online reviews posted by users on various review platforms have a profound impact on consumer decision-making and trust building, especially food reviews, which not only provide consumers with reference information about the quality of restaurants, food or services, but also have a significant impact on the economic benefits of businesses. Although online reviews play a key role in building consumer trust and promoting purchasing decisions, the prevalence of false reviews is gradually eroding the credibility of user review content. False reviewers often manipulate consumer cognition by posting content with extremely positive or negative comments, thereby raising or lowering the ratings of businesses and distorting the true situation of the market. Especially in the catering industry, a small increase in online ratings can bring significant revenue growth. This phenomenon further stimulates the motivation to generate and spread false food reviews. Therefore, using technical means to effectively detect false reviews and purify the online review environment has become an urgent problem to be solved.

[0003] At present, the research on food fake review detection technology is mainly divided into two types: single-modal-based fake review detection methods and multi-modal-based fake review detection methods. Single-modal-based detection methods focus on review text analysis, such as judging the authenticity of reviews by analyzing features such as vocabulary selection, syntactic structure, and sentiment polarity of reviews. However, single-modal data is difficult to effectively capture the complex features of fake reviews and can be easily deceived by well-trained fake reviewers. In contrast, multi-modal-based detection methods can more comprehensively identify fake reviews by fusing multi-source information such as text and images. However, existing multi-modal methods have certain irrationalities in feature representation and fusion strategies, which limits the further improvement of their detection effect.

[0004] Therefore, for the detection of false online food reviews, it is necessary to extract the feature information of heterogeneous modalities in false reviews in a fine-grained manner and support fine-grained interactions between and within modalities to ensure that text and image features can fully exchange information. By introducing the graph attention mechanism, key features can be adaptively captured and false reviews can be identified more accurately. Summary of the invention

[0005] In order to solve the above problems, the present invention provides a multimodal food fake review detection method based on graph attention network. By performing unimodal encoding on the acquired food review text and image information respectively, the heterogeneous modal information is fused to form a fine-grained and more comprehensive food review feature representation, so as to more accurately identify fake reviews.

[0006] The present invention provides a multimodal food false review detection method based on graph attention network, comprising the following steps:

[0007] Step S1: Obtain the user food review data to be tested from the review platform, including the text and pictures of the food review, and then use the data cleaning toolkit provided by Python to clean the obtained review data to obtain a standardized data format;

[0008] Step S2: By using pre-trained text and image feature extractors, deep semantic and visual features are extracted from the text and images of food reviews, respectively, and the semantic features and visual features are converted into text modal graphs and visual modal graphs, respectively;

[0009] Step S3: Map the extracted text modal graph and visual modal graph into a unified feature space, introduce cross-modal connections, construct a multimodal graph, and then use the graph attention network to adaptively adjust the inter-modal and intra-modal weights to enhance the fusion representation of information;

[0010] Step S4: Perform a global average pooling operation on the fused multimodal features, and input the obtained global feature vector into a fully connected classifier for training. Use the cross entropy loss function to optimize the model classification performance in order to achieve the best detection effect for new comments.

[0011] Compared with the prior art, the present invention has the following advantages:

[0012] 1. This method does not rely solely on single-modal information, but combines text and image information. This allows the model to improve robustness through multimodal interaction and effectively respond to complex false review behaviors even if false comments are disguised in language or vision.

[0013] 2. The present invention innovatively integrates the features of food review text and images into a shared space, allowing two different types of information to be complementary analyzed on the same dimension, significantly improving the accuracy of false review detection and information capture capabilities.

[0014] 3. Compared with traditional methods, the present invention introduces a graph attention network, which can adaptively assign weights to key nodes, effectively capture hidden cross-modal associations in comments, and improve the ability to recognize complex false comment patterns. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 Flow chart of a multimodal false food review detection method based on graph attention network in an embodiment of the present invention;

[0016] Figure 2 It is a technical framework diagram of a multimodal food false review detection method based on a graph attention network in an embodiment of the present invention;

[0017] Figure 3 This is a flowchart of step S1 in a multimodal food false review detection method based on a graph attention network in an embodiment of the present invention: obtaining user food review data to be detected from a review platform, including text and pictures of food reviews, and then using a data cleaning toolkit provided by Python to clean the obtained review data to obtain a standardized data format;

[0018] Figure 4 A detailed flow chart of crawling food review data in a multimodal food false review detection method based on a graph attention network in an embodiment of the present invention;

[0019] Figure 5 A flowchart of step S2 in a multimodal food false review detection method based on a graph attention network in an embodiment of the present invention: extracting deep semantic and visual features from the text and pictures of food reviews respectively by using pre-trained text and image feature extractors, and converting the semantic features and visual features into text modal graphs and visual modal graphs respectively;

[0020] Figure 6 A flowchart of step S3 in a multimodal food fake review detection method based on a graph attention network in an embodiment of the present invention: mapping the extracted text modal graph and visual modal graph into a unified feature space, introducing cross-modal connections, constructing a multimodal graph, and then using the graph attention network to adaptively adjust inter-modal and intra-modal weights to enhance the fusion representation of information;

[0021] Figure 7 This is a flowchart of step S4 in a multimodal food false review detection method based on a graph attention network in an embodiment of the present invention: performing a global average pooling operation on the fused multimodal features, inputting the obtained global feature vector into a fully connected classifier for training, and using a cross entropy loss function to optimize the model classification performance in order to achieve the best detection effect for new reviews. DETAILED DESCRIPTION

[0022] The present invention provides a multimodal food false review detection method based on graph attention network. Firstly, the text and image data of food reviews are obtained from the platform, and the review data are cleaned using the cleaning toolkit provided by Python to obtain a standardized data format; then, the pre-trained models are used to extract deep semantic and visual features from the text and image respectively; then, the extracted text and image features are mapped to a unified public space for fusion, thereby obtaining a multimodal feature map combining feature information of different modalities; finally, the fused features are used for false review detection by a trained classifier to obtain the authenticity result of the review.

[0023] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below through specific implementations and in conjunction with the accompanying drawings.

[0024] Embodiment 1

[0025] like Figure 1 , Figure 2 As shown, in this embodiment, the present invention provides a multimodal food false review detection method based on a graph attention network, comprising the following steps:

[0026] Step S1: Obtain the user food review data to be tested from the review platform, including the text and pictures of the food review, and then use the data cleaning toolkit provided by Python to clean the obtained review data to obtain a standardized data format;

[0027] Step S2: By using pre-trained text and image feature extractors, deep semantic and visual features are extracted from the text and images of food reviews, respectively, and the semantic features and visual features are converted into text modal graphs and visual modal graphs, respectively;

[0028] Step S3: Map the extracted text modal graph and visual modal graph into a unified feature space, introduce cross-modal connections, construct a multimodal graph, and then use the graph attention network to adaptively adjust the inter-modal and intra-modal weights to enhance the fusion representation of information;

[0029] Step S4: Perform a global average pooling operation on the fused multimodal features, and input the obtained global feature vector into a fully connected classifier for training. Use the cross entropy loss function to optimize the model classification performance in order to achieve the best detection effect for new comments.

[0030] like Figure 3 , Figure 4As shown, in this embodiment, the above step S1: obtains the user food review data to be detected from the review platform, including the text and pictures of the food review, and then uses the data cleaning toolkit provided by Python to clean the obtained review data to obtain a standardized data format, specifically including:

[0031] Step S11: crawling food review data on the Yelp.com platform in batches by using a web crawler tool to obtain text content of food reviews and corresponding review image data;

[0032] Step S12: storing the captured comment text and image data in a database and a file system respectively, and establishing a corresponding relationship between the comment text and the image to form a preliminary data set;

[0033] Step S13: Use numpy and pandas toolkits provided by python to clean the text data, remove HTML tags, special symbols, emoticons and other irrelevant information, and normalize the text data;

[0034] Step S14: pre-process the image data, use the OpenCV toolkit to standardize the size of the image to a uniform size, check whether the image is damaged or missing, delete invalid images, and normalize the image data;

[0035] Step S15: After the step-by-step cleaning, the overall cleaning is performed. If a comment lacks pictures or text, the incomplete data record is deleted, so that the text data of each comment is matched with the corresponding picture data one by one, and finally the food review multimodal dataset R is established. i ={T i ,V i}, where R i Represents a comment data, T i is the comment text of the data, V i This is the image that corresponds to this comment.

[0036] like Figure 5 As shown, in this embodiment, the above step S2: by using pre-trained text and image feature extractors, extracting deep semantic and visual features from the text and pictures of food reviews respectively, and converting the semantic features and visual features into text modal graphs and visual modal graphs respectively, specifically includes:

[0037] Step S21: Use the pre-trained Transformer model to extract features from the comment text and process the comment sample into a series of feature sequences T = {t1, t2, …, t k}, the output context feature representation t k It is given by the following formula:

[0038] {t0,t1,t2,…,t k}=Transformer[CLS],w1,w2,…w k )

[0039] Among them, [CLS] is the overall representation of the text sequence, w k represents the kth token of the original text, t0 represents the comprehensive features of the entire text sequence, and t i Represents the feature information of the kth token in the text;

[0040] Step S22: The extracted text features are converted into a graph structure through a graph network, so that the marked features in each sequence are used as a node in the graph, and the nodes are connected through contextual semantic information to form a fully connected text modal graph G T =(N T ,E T ), where N T Represents a text node in the graph, E T The edges representing the nodes in the graph;

[0041] Step S23: Use the pre-trained Vision Transformer model to extract features from the image, divide the image into several image blocks, extract the visual information of each image block, and generate the corresponding visual feature vector. The calculation formula is as follows:

[0042]

[0043] Among them, ViT represents the Vision Transformer model, Represents the feature vector of the i-th image block, I i represents the i-th image block of the input;

[0044] Step S24: The extracted image features are converted into a graph structure through a graph network. Each image block in the image is regarded as a node. The nodes are connected through the local and global relationships of the visual features to form a fully connected visual modal graph G. V =(N V ,E V ), where N V Represents a node in the graph, E V An edge representing a node in a graph.

[0045] like Figure 6 As shown, in this embodiment, the above step S3: mapping the extracted text modal graph and visual modal graph into a unified feature space, introducing cross-modal connections, constructing a multimodal graph, and then using a graph attention network to adaptively adjust inter-modal and intra-modal weights, thereby strengthening the fusion representation of information, specifically includes:

[0046] Step S31: Projecting the text modality graph and the visual modality graph into a unified shared feature space through a fully connected layer, so that the features of the two modalities can be compared and fused in the same dimension;

[0047] Step S32: Use the ELU nonlinear activation function to perform nonlinear transformation on the projected features to enhance the expressive power of the features. The process is represented by the following formula:

[0048]

[0049] Among them, W MM and b MM are the parameters of the projection layer, and Respectively represent the visual features v after nonlinear transformation i and text feature t j ;

[0050] Step S33: By introducing unweighted bidirectional edge connections between text and visual nodes, a comprehensive multimodal graph G is constructed. MM =(N MM ,E MM ), where N MM Represents a node in a multimodal graph, E V Edges representing nodes in a multimodal graph;

[0051] Step S34: Use the graph attention network to perform feature fusion on the multimodal graph, and further enhance the fusion representation of feature information by adaptively adjusting the weights between nodes.

[0052] like Figure 7 As shown, in this embodiment, the above step S4: performing a global average pooling operation on the fused multimodal features, inputting the obtained global feature vector into a fully connected classifier for training, and using a cross entropy loss function to optimize the model classification performance in order to achieve the best detection effect for new comments, specifically includes:

[0053] Step S41: performing a global average pooling operation on the fused features to aggregate the embedding vectors of all nodes into a global feature vector containing multimodal information;

[0054] Step S42: input the global feature vector into a fully connected classifier for training. The classifier consists of two layers of fully connected feedforward networks. A ReLU nonlinear activation function is added after each layer, and the dimension of the output layer is set to 2, corresponding to the categories of real and fake reviews.

[0055] Step S43: Use the cross entropy loss function to perform model training so that the model parameters are optimized. The optimization process is expressed by the following formula:

[0056]

[0057] Among them, N represents the total number of samples, Y i represents the true category of the sample, represents the predicted probability of a true review, represents the predicted probability of a fake review;

[0058] Step S44: After the training is completed, the classifier is applied to new comments, and the output dimension described in step S42 is converted into true or false probability using the Softmax function. Whether the comment is a false comment is determined based on the set threshold (0.5), thereby obtaining the detection result.

Claims

1. A multimodal food fake review detection method based on graph attention network, characterized in that: include: Step S1: Obtain the user food review data to be tested from the review platform, including the text and pictures of the food review, and then use the data cleaning toolkit provided by Python to clean the obtained review data to obtain a standardized data format; Step S2: By using pre-trained text and image feature extractors, deep semantic and visual features are extracted from the text and images of food reviews, respectively, and the semantic features and visual features are converted into text modal graphs and visual modal graphs, respectively; Step S3: Map the extracted text modal graph and visual modal graph into a unified feature space, introduce cross-modal connections, construct a multimodal graph, and then use the graph attention network to adaptively adjust the inter-modal and intra-modal weights to enhance the fusion representation of information; Step S4: Perform a global average pooling operation on the fused multimodal features, and input the obtained global feature vector into a fully connected classifier for training. Use the cross entropy loss function to optimize the model classification performance in order to achieve the best detection effect for new comments.

2. The multimodal food false review detection method based on graph attention network according to claim 1 is characterized in that: Step S1: obtaining the user food review data to be tested from the review platform, including the text and pictures of the food review, and then using the data cleaning toolkit provided by Python to clean the obtained review data to obtain a standardized data format, including: Step S11: crawling food review data on the Yelp.com platform in batches by using a web crawler tool to obtain text content of food reviews and corresponding review image data; Step S12: storing the captured comment text and image data in a database and a file system respectively, and establishing a corresponding relationship between the comment text and the image to form a preliminary data set; Step S13: Use numpy and pandas toolkits provided by python to clean the text data, remove HTML tags, special symbols, emoticons and other irrelevant information, and normalize the text data; Step S14: pre-process the image data, use the OpenCV toolkit to standardize the size of the image to a uniform size, check whether the image is damaged or missing, delete invalid images, and normalize the image data; Step S15: After the step-by-step cleaning, the overall cleaning is performed. If a comment lacks pictures or text, the incomplete data record is deleted, so that the text data of each comment is matched with the corresponding picture data one by one, and finally the food review multimodal dataset R is established. i ={T i ,V i }, where R i Represents a comment data, T i is the comment text of the data, V i This is the image that corresponds to this comment.

3. The multimodal food false review detection method based on graph attention network according to claim 1 is characterized in that: The step S2: extracting deep semantic and visual features from the text and pictures of the food reviews respectively by using pre-trained text and image feature extractors, and converting the semantic features and visual features into text modal graphs and visual modal graphs respectively, comprises: Step S21: Use the pre-trained Transformer model to extract features from the comment text and process the comment sample into a series of feature sequences T = {t1, t2, …, t k }, the output context feature representation t k It is given by the following formula: {t0,t1,t2,…,t k }=Transformer([CLS],w1,w2,…w k ) Among them, [CLS] is the overall representation of the text sequence, w k represents the kth token of the original text, t0 represents the comprehensive features of the entire text sequence, and t i Represents the feature information of the kth token in the text; Step S22: The extracted text features are converted into a graph structure through a graph network, so that the marked features in each sequence are used as a node in the graph, and the nodes are connected through contextual semantic information to form a fully connected text modal graph G T =(N T ,E T ), where N T Represents a text node in the graph, E T The edges representing the nodes in the graph; Step S23: Use the pre-trained Vision Transformer model to extract features from the image, divide the image into several image blocks, extract the visual information of each image block, and generate the corresponding visual feature vector. The calculation formula is as follows: Among them, ViT represents the Vision Transformer model, Represents the feature vector of the i-th image block, I i represents the i-th image block of the input; Step S24: The extracted image features are converted into a graph structure through a graph network. Each image block in the image is regarded as a node. The nodes are connected through the local and global relationships of the visual features to form a fully connected visual modal graph G. V =(N V ,E V ), where N V Represents a node in the graph, E V An edge representing a node in a graph.

4. The multimodal food false review detection method based on graph attention network according to claim 1 is characterized in that: The step S3: mapping the extracted text modal graph and visual modal graph into a unified feature space, introducing cross-modal connections, constructing a multi-modal graph, and then using a graph attention network to adaptively adjust inter-modal and intra-modal weights, thereby strengthening the fusion representation of information, including: Step S31: Projecting the text modality graph and the visual modality graph into a unified shared feature space through a fully connected layer, so that the features of the two modalities can be compared and fused in the same dimension; Step S32: Use the ELU nonlinear activation function to perform nonlinear transformation on the projected features to enhance the expressive power of the features. The process is represented by the following formula: Among them, W MM and b MM are the parameters of the projection layer, and Respectively represent the visual features v after nonlinear transformation i and text feature t j ; Step S33: By introducing unweighted bidirectional edge connections between text and visual nodes, a comprehensive multimodal graph G is constructed. MM =(N MM ,E MM ), where N MM Represents a node in a multimodal graph, E V Edges representing nodes in a multimodal graph; Step S34: Use the graph attention network to perform feature fusion on the multimodal graph, and further enhance the fusion representation of feature information by adaptively adjusting the weights between nodes.

5. The multimodal food false review detection method based on graph attention network according to claim 1 is characterized in that: The step S4: performing a global average pooling operation on the fused multimodal features, inputting the obtained global feature vector into a fully connected classifier for training, and using a cross entropy loss function to optimize the model classification performance in order to achieve the best detection effect for new comments, including: Step S41: performing a global average pooling operation on the fused features to aggregate the embedding vectors of all nodes into a global feature vector containing multimodal information; Step S42: input the global feature vector into a fully connected classifier for training. The classifier consists of two layers of fully connected feedforward networks. A ReLU nonlinear activation function is added after each layer, and the dimension of the output layer is set to 2, corresponding to the categories of real and fake reviews. Step S43: Use the cross entropy loss function to perform model training so that the model parameters are optimized. The optimization process is expressed by the following formula: Among them, N represents the total number of samples, Y i represents the true category of the sample, represents the predicted probability of a true review, represents the predicted probability of a fake review; Step S44: After the training is completed, the classifier is applied to new comments, and the output dimension described in step S42 is converted into true or false probability using the Softmax function. Whether the comment is a false comment is determined based on the set threshold (0.5), thereby obtaining the detection result.

Images