Online scientific research comment information topic analysis and hotspot evaluation method

Abstract

The invention belongs to the field of social calculation research, and discloses a scientific research comment information topic analysis and hotspot evaluation method based on natural language processing and big data, which comprises a data acquisition module, a topic analysis module, a hotspot evaluation module and a user interaction module. The data acquisition module adopts an advanced web crawler technology to realize automatic acquisition of online scientific research comment information, and stores data to a MySQL database. The topic analysis module deploys an efficient scientific research topic analysis model, and identifies and classifies main topics in scientific research comment information. And the hotspot evaluation module quantifies and evaluates the popularity level of the scientific research comment topic by using a calculation model based on the comment analysis result. The user interaction module provides a user-friendly front-end interface with complete functions, a user can conveniently carry out data acquisition, topic analysis and hotspot assessment operation, and all topic analysis and risk assessment results are displayed through a visual visualization chart. By establishing a comprehensive scientific research information topic analysis and hotspot evaluation system, real-time monitoring and accurate analysis of scientific research comment information in an innovative network are realized, related departments and enterprises are helped to quickly respond, and scientific research hotspot topics are effectively tracked and analyzed. The method is beneficial to formulating and implementing coping strategies aiming at new technology evolution, enhancing communication and exchange with science and technology worker groups, enhancing science and technology trend cognition and understanding, predicting and analyzing scientific research hotspots, and finally achieving the purpose of assisting in evaluating the scientific research trend.

Description

Technical Field

[0001] The present invention relates to the field of social computing research, mainly to big data analysis technology, natural language processing technology and knowledge graph technology, and is a method for online scientific research review information topic analysis and hot spot evaluation. Technical Background

[0002] As online platforms become a new area for scientific research co-creation, experts and scholars can instantly share their views, discuss scientific research trends and directions, and express their opinions on various scientific research issues. Although the popularity of this form of participation has enhanced the breadth of scientific research innovation, it has also brought challenges to the integration and analysis of scattered information.

[0003] The demand for analysis and evaluation of research hot topics has also increased. Research institutions and enterprises need to keep abreast of changes in research trends, evaluate the opportunities implied by research hotspots, and take corresponding countermeasures. An effective research hotspot monitoring and evaluation system can help decision makers discover and track potential hot trends in a timely manner, which is of great significance for promoting scientific research and innovation.

[0004] In the current field of social network public opinion analysis, although a variety of technologies have been applied to the topic analysis and risk assessment of online social comment information, they generally have some significant limitations. First, the shortness of social network texts and the sparseness of data pose challenges to traditional probabilistic topic models. These models often find it difficult to capture accurate word co-occurrence patterns when processing short texts, resulting in poor quality of generated topics and insufficient consistency and diversity of topics. In addition, the lack of contextual information in short texts makes it difficult for the model to accurately infer the topic structure of the document, thus affecting the accuracy of topic analysis. In the prior art, although neural topic models have alleviated these problems to a certain extent by leveraging the advantages of deep learning, they still have deficiencies in processing the semantic association relationship between words within the document, which limits the model's ability to deeply understand the text content. In addition, when evaluating the popularity of hot topics in the prior art, the evaluation indicators considered in the evaluation index are relatively single, and cannot fully reflect the complexity and diversity of the required hot direction requirements. Summary of the invention

[0005] The present invention integrates network platform comment text analysis, deep learning technology, pre-trained language models, and multi-granularity feature fusion technologies to form a complete set of scientific research comment topic analysis and hot spot evaluation solutions. Through automated data collection, application of knowledge graphs, enhanced topic extraction, multi-granularity public opinion sentiment analysis, and risk assessment models, this method can monitor, deeply analyze, and accurately evaluate comments on the network platform in real time, providing scientific decision support for enterprises. The present invention is implemented through the following technical solutions: S1. Collect and clean online scientific research review information based on crawler technology; S2. Use knowledge graphs to supplement entity concept knowledge for short texts; S3. Integrate short text semantic features to extract topics from online review texts; S4. Use the BERT model to embed online review text at the character level, and combine it with the GloVe model to embed it at the word level, while considering part-of-speech information; S5. Extract local features and context features of the text, integrate the features to perform sentiment analysis on the comment text, and determine whether the commentary supports or denies the topic; S6. Evaluate the popularity level of scientific research review topics based on the sentiment and topic tags of online social comments; S7. Interact with users through the front-end interactive interface and use charts and other methods to clearly and intuitively display the results of scientific research review topic analysis.

[0006] The specific process of step S1 includes the following steps: S1-1. Concatenate the URL address based on the crawler target information submitted by the user; S1-2. Using web crawler technology, automatically collect research-related comment data from online platforms based on preset data sources, keywords, and topics; S1-3. Clean the collected raw data, including removing irrelevant noise data, correcting format errors and deleting duplicate entries.

[0007] The specific process of step S2 includes the following steps: S2-1. Perform preprocessing operations such as word segmentation and stop word filtering on the short text to obtain the word set of the text; S2-2. Use the TextRank algorithm to extract key entities from texts and obtain the conceptual information of these keywords in the knowledge graph as external knowledge of short texts; S2-3. Establish edges between words and between words and concepts, and construct a text word semantic association graph through two semantic association calculations.

[0008] The specific process of step S3 includes the following steps: S3-1. Use the GCN model to extract association features and capture the comprehensive feature representation of local semantic association characteristics. Finally, the output of GCN is pooled, and the feature vector of the word semantic association graph is obtained by adding the node feature vectors; S3-2. Map text data to latent topic space and generate topic vectors by building a deep learning encoder network. S3-3. Use the decoder network to reconstruct the latent topic vector back to the word distribution representation of the text, and train the parameters of the variational autoencoder by minimizing the reconstruction error and KL divergence.

[0009] The specific process of step S4 includes the following steps: S4-1. Use Chinese word segmentation tools (such as Jieba library) to segment and tag the preprocessed text, and encode the tagged parts of speech; S4-2. For each word in the text, calculate the position encoding according to its position index and dimension index; S4-3. Input the comment text, pass it through the word embedding layer, fuse the part-of-speech encoding obtained in step S4-1 and the position encoding obtained in step S4-2, and obtain the GloVe word embedding sequence and the BERT word embedding sequence.

[0010] The specific process of step S5 includes the following steps: S5-1. Use convolution kernels with different windows to extract local features of text and adopt k-max-average pooling strategy; S5-2. Combine the BiGRU model and the attention mechanism to extract the contextual information of the text; S5-3. Fuse the feature vectors obtained by the TextCNN layer in step S5-1 and the BiGRU-ATT layer in step S5-2, and use the Softmax classifier for sentiment classification.

[0011] The specific process of step S6 includes the following steps: S6-1. Use cosine similarity to calculate the degree of relevance between each scientific research review topic and the original text of the review information; S6-2. Calculate the number of positive support and negative comments under each scientific research review topic, and finally calculate the heat value of the scientific research review topic according to the heat formula; S6-3. Based on the heat value calculated in S6-2, the scientific research review topics are divided into different heat levels, such as low heat, medium heat and high heat.

[0012] The specific process of step S7 includes the following steps: S7-1. Design a data collection request form on the front end, where users can input key information and implement two-way data binding and dynamic page updates; S7-2, implement asynchronous file upload and download based on Ajax, allowing users to upload and download data sets and texts in a non-blocking manner; S7-3, store the data in a MySQL database and process it on the backend, and integrate the JavaScript visualization library to visualize the analysis results on the frontend. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a framework diagram of the online scientific research review information topic analysis and hot spot evaluation system of the present invention; Figure 2 It is a schematic diagram of the process of collecting and storing information of online social scientific research reviews. DETAILED DESCRIPTION

[0014] See attached Figure 1 In this embodiment, a method for analyzing and evaluating online scientific research review information topics is described as follows: (1) Collect and clean online social comment information based on crawler technology (1-1) Based on the crawler target information submitted by the user on the front end, the Uniform Resource Locator (URL) of the target web page is spliced ​​and request parameters are added to imitate the browsing behavior of normal users to ensure that the request can be successfully accepted by the Web server; (1-2) Send an HTTP request to the Web server and use Python's Requests library to send a request to the Weibo server. The Requests library supports HTTPS and can automatically manage cookies and sessions. After the request is sent, the server returns a response containing the data to be collected. In the implementation, multiple sets of data are collected by simulating page turning or sliding page turning. According to the page turning characteristics of the Weibo web client, dynamically construct an HTTP request to obtain all search result data, and parse the response data to extract useful information.

[0015] (1-3) First, use regular expressions to identify and remove all emoticons and URL links. The purpose of this step is to reduce the interference of irrelevant content on text analysis; second, clean up the user tags (@username) in the text to ensure the simplicity and coherence of the text content, thereby avoiding the influence of user tags on semantic understanding; finally, remove all special characters and redundant punctuation marks, and only retain basic punctuation marks to separate sentences.

[0016] (2) Use knowledge graphs to supplement entity concept knowledge for short texts.

[0017] (2-1) The UMAP algorithm is used to reduce the dimensionality of the comment vector. First, the distance between each point in the data set and its nearest neighbor is calculated, and then a local neighborhood graph is constructed. This process involves using the k-nearest neighbor algorithm to identify the neighbors of each point and adjusting the weights through Gaussian distribution to accurately reflect the local distance between the points. Furthermore, the high-dimensional comment vector is mapped into a low-dimensional space representation, which ensures the structural integrity of the data points by minimizing the cross entropy of the neighborhood graphs in high-dimensional and low-dimensional spaces.

[0018] (2-2) The TextRank algorithm is used to extract key entities from texts. The graph model is constructed using the preprocessed text word set. The weight of the graph is then iteratively calculated until the model converges to obtain the importance ranking of text words. Finally, the top three keywords are selected to supplement the conceptual knowledge using the knowledge graph.

[0019] (2-3) Use word embedding models such as GloVe to convert the keywords in the original text and the words in their context into vectors, and then calculate the cosine similarity between these vectors and the candidate concept vectors and these word vectors. If their similarity exceeds the threshold, an edge is established between them. The points and edges composed of the above points and concepts are used to form a text semantic association graph.

[0020] (3) Integrate the semantic features of short texts to extract topics from online scientific research review texts.

[0021] (3-1) Use (2-3) to get the word semantic association graph, use the pre-trained word embedding model to initialize the node features, and then use the multi-layer convolution operation of GCN to achieve layer-by-layer abstraction of features and aggregation of neighbor node features to capture local semantic associations. During the feature propagation process, each layer of GCN updates the node representation to reflect its local structure in the graph. All node features are integrated through feature pooling to form a global feature vector. (3-2) Create an encoder network that can receive text data and map it into a latent topic space, generate topic vectors, embed the text using the SBERT model, generate text vectors that can capture rich contextual information, and use these vectors as the input of the encoder. Combine the semantic association features extracted by the graph convolutional neural network (GCN) in (3-1) with the text embedding vectors generated by SBERT to form the final input of the encoder, thereby enriching the semantic representation of the text.

[0022] (3-3) Construct a decoder network to convert the latent topic vector into the word distribution of the text to reconstruct the original document. The reconstruction accuracy of the model is measured by calculating the reconstruction error, and the model parameters are optimized using variational inference to minimize the sum of the KL divergence and the reconstruction error, ensuring that the model effectively captures the key information of the data and generates high-quality topic representations; in addition, the back propagation and gradient descent algorithms are used to iteratively update the network parameters until the model is stable on a specific data set, thereby achieving optimized training of the model.

[0023] (4) Use the BERT model to embed online review text at the word level, and how to use the GloVe model to embed the text at the word level and combine it with part-of-speech information.

[0024] (4-1) Perform word segmentation and part-of-speech tagging on the preprocessed text, which includes identifying the part-of-speech category of each word in the text, such as noun, verb, adjective, etc., and encoding the marked part-of-speech to ensure that the dimension of each part-of-speech vector is consistent with the dimension of the GloVe word vector.

[0025] (4-2) For each word in the text, calculate the position encoding based on its position index and dimension index, and enhance the model Understanding of long-distance dependencies.

[0026] (4-3) Fusion step (5-1) obtained part of speech embedding and step (5-2) obtained position encoding into GloVe word embedding to obtain enhanced GloVe word embedding sequence; use the pre-trained BERT model to process the text, get the output from the last hidden layer of the BERT model, and generate a BERT word embedding sequence.

[0027] (5) The review text is converted into two different types of word embeddings through the GloVe model and the pre-trained BERT model.

[0028] (5-1) The convolution kernel height is set from 3 to 3, 4, and 5 respectively, forming 3 channels to extract local features of public opinion text. The width of each convolution kernel is set to be consistent with the dimension of the input word vector to ensure that the convolution kernel can fully and effectively cover and process all information in the word vector matrix. The number of convolution kernels of each height is set to 128. The local extraction calculation of the word embedding sequence input to the convolution layer is shown in formula (1), where is the feature map, is a nonlinear activation function, is the convolution kernel, and is the bias value Then use k-max-average pooling to calculate their average value as the output result. k-max-average pooling obtains the top K largest values ​​from the pooling window. Its calculation formula is shown in the formula, where is the output of the convolution operation: After pooling, the outputs of different windows are connected to finally obtain the multi-scale local features of the public opinion text.

[0029] (5-2) Use BiGRU and attention mechanism to extract contextual information of text. Specifically, the BiGRU model is first used to process text data. The model can simultaneously consider the previous and subsequent information of words in the text to capture long-distance dependencies. Then, the attention mechanism is introduced on the basis of BiGRU, which enables the model to identify and focus on key emotional words in the text. Through attention weighting, the output features of BiGRU are further optimized to enhance the model's ability to recognize emotional tendencies. (5-3) For the input comment text, the part-of-speech embedding obtained in step (5-1) and the position encoding obtained in step S5-2 are fused with the GloVe word embedding to enhance the model's understanding of the text semantics, and a fused GloVe word embedding sequence is obtained; the comment text is processed using the pre-trained BERT model, and the output is obtained from the last hidden layer of the BERT model to obtain a BERT word embedding sequence.

[0030] (6) Evaluate the popularity level of scientific research review topics based on the sentiment and topic tags of online social comments.

[0031] (6-1) The topic label of each data can be obtained from step (3.3), and the data is classified according to the topic label. The comment data under the same topic label constitutes a scientific research review topic. Then the keyword of the scientific research review topic and the corresponding comment information text are input into the SBERT model to obtain the topic embedding of the scientific research review topic and the corresponding comment information text embedding respectively. Furthermore, the cosine similarity is used to calculate the correlation between each scientific research review topic and the original comment information.

[0032] (6-2) Based on the policy public opinion sentiment analysis results obtained in (5.3), the sum of the number of positive support and negative negation comments under each comment topic is obtained, and finally the heat value of the scientific research comment topic is calculated according to the risk formula (3). The formula definition is as follows, where is the total number of comments under the topic. (6-3) The topic heat level is evaluated based on the research review topic heat value R obtained in (6-2). The heat level is divided into four levels: Level I (ordinary), with a corresponding risk value R between 0 and 40 (inclusive); Level II (low heat), with a corresponding risk value R between 40 and 60 (inclusive); Level III (medium heat), with a corresponding risk value R between 60 and 80 (inclusive); Level IV (high heat), with a corresponding risk value R between 80 and 100.

[0033] (7) The processing results can be visualized through the front-end interactive interface, and the analysis results of the scientific research review can be clearly and intuitively displayed using charts and other means.

[0034] (7-1) The front-end interface designs a data collection request form, where users can enter key information such as keywords and date ranges. The form uses HTML / CSS for layout and style design, binds the form input to the data model, and implements two-way data binding and dynamic page updates.

[0035] (7-2) Implement an Ajax-based asynchronous file upload and download, allowing users to upload and download online review datasets and related topic texts in a non-blocking manner.

[0036] (7-3) The data uploaded in (7-2) and the online scientific research review data crawled in (1) are stored in a MySQL database and processed by the back-end service through a preset data analysis process. The front-end page integrates Echart, a chart component for displaying the analysis results, and uses Ajax requests to obtain the analysis result data from the background and visualize it on the front-end.

Claims

1. A method for analyzing and evaluating online scientific research review information topics, comprising the following steps: (1) Use web crawler technology to automatically collect online scientific research review information and store the data in a MySQL database; (2) Train and deploy topic analysis models to identify and classify major topics in policy opinion; (3) Based on the results of topic analysis, use computational models to quantify and evaluate the popularity of scientific research review topics; (4) Provide a user-friendly user interaction interface to enable interaction between users and data and visual display of analysis results.

2. The data collection method as described in claim 1, through web crawler technology, according to the collection target specified by the user or the preset target, by simulating the browsing behavior of normal users, sending HTTP requests to obtain the target web page content, and dynamically constructing requests to adapt to the page turning mechanism of different network platforms, automatically collects online scientific research review data from the network platform.

3. The topic analysis method as described in claim 1 introduces a Chinese concept knowledge graph to supplement the entity concept knowledge in the short text, so as to enhance the semantic information of the text and solve and alleviate the sparsity problem of short text data.

4. The online scientific research review hotspot evaluation method described in claim 1 quantifies the popularity level of each scientific research review topic by defining an accurate mathematical formula, and comprehensively considers the degree of semantic correlation between the scientific research review topic and the review information text.

5. The online scientific research review topic heat evaluation method as claimed in claim 1 maps the calculated heat value to a predefined heat level to provide an intuitive heat evaluation result. The heat level is divided into four levels: Level I (normal), Level II (low heat), Level III (medium heat), and Level IV (high heat).

6. The user interaction function as claimed in claim 1 dynamically updates the front-end page through the JavaScript framework, and supports data set uploading and visualization through the Ajax asynchronous upload interface, and can perform the following operations: (1) Submit a data collection request, including entering key information to automatically collect research-related review data from online platforms; (2) Upload scientific research review data for customized topic analysis and popularity assessment; (3) Access and download research review datasets stored in the system; (4) View and obtain the visualization results of scientific research review topic analysis and popularity assessment obtained through system analysis.

Images