A distributed public opinion information monitoring method and system
Through the distributed public opinion information monitoring method, deep learning models and blockchain technology are used to solve the problems of low information collection efficiency and insufficient storage security of public opinion system, efficient information collection and secure storage are achieved, and accountability process is simplified.
Patent Information
- Application Number
- CN202411231694.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-04
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-09-04
AI Technical Summary
The existing public opinion system has problems such as inefficient information collection and insufficient security of information storage, making it difficult to hold accountable and trace the source.
Using distributed public opinion information monitoring method, the URL sent by the task scheduling center is received through the work node, the content of the web page is analyzed, the public opinion information is analyzed using the deep learning model, the negative coefficient is calculated and stored in the blockchain, and combined with the database, the efficient collection of information and Secure storage.
It improves information collection efficiency, ensures the security of information storage, and simplifies the accountability and traceability process.
Smart Images

Figure CN118733896B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer and information security technology, and in particular to a distributed public opinion information monitoring method and system. Background Art
[0002] Public opinion is an important carrier for netizens to express their emotions and spread their ideas. However, with the popularization of Internet technology, the amount of information has exploded exponentially. In order to safeguard the legitimate interests of netizens and maintain a good network environment, the public opinion system has become a hot topic in related fields.
[0003] However, the existing public opinion systems often do not consider the division of labor and cooperation between hosts, but only one host parses the web page content, obtains resources and performs analysis, or although the division of labor and cooperation between hosts is considered, an efficient management and coordination mechanism is not designed, resulting in low efficiency in information collection. In addition, the collected information is often stored in a database, which leads to security risks and makes it difficult to hold people accountable and trace the source. Summary of the invention
[0004] The purpose of the present invention is to overcome the deficiencies in the prior art and to provide a distributed public opinion information monitoring method and system to solve the problems of low efficiency in information collection, lack of security in information storage, and difficulty in accountability and tracing.
[0005] To solve the above technical problems, the present invention is implemented by adopting the following solutions:
[0006] The present invention provides a distributed public opinion information monitoring method, which is executed by a working node and includes:
[0007] Receive the URL sent by the task scheduling center, parse the URL and collect public opinion information from the web page corresponding to the URL;
[0008] Use deep learning models to analyze and calculate public opinion information, obtain the negative coefficient of public opinion information, and identify negative information based on the negative coefficient and the set negative coefficient threshold;
[0009] Count the number of negative information that meets the set conditions within a set time period, calculate the negative information propagation coefficient based on the number of negative information that meets the set conditions within the set time period, and identify negative information that exceeds the predetermined propagation power based on the negative information propagation coefficient and the set propagation coefficient threshold;
[0010] Calculate the real heat and expected heat of negative information respectively, and identify negative information that exceeds the predetermined heat according to the difference between the real heat and the expected heat and the set difference threshold;
[0011] Negative information that exceeds the expected influence is identified based on negative information that exceeds the expected spread and negative information that exceeds the expected popularity;
[0012] Negative information that exceeds the predetermined influence is packaged into blocks, which are broadcast and connected to the blockchain.
[0013] Furthermore, the URL sent by the task scheduling center is received, the URL is parsed, and public opinion information in the web page corresponding to the URL is collected, including:
[0014] The working node obtains the URL from the tail of the pending queue of the task scheduling center and inserts the obtained URL into the head of the pending double-ended queue of the working node;
[0015] The working node maintains the load parameters updated in real time and reports the load parameters to the task scheduling center at regular intervals. The working node obtains the URL according to the URL acquisition quantity allocated by the task scheduling center based on the load parameters of the working node;
[0016] When the pending double-ended queue is not empty, a URL is popped from the tail of the pending double-ended queue and parsed: if the URL obtained is a navigation page URL, all URLs in the navigation page are returned to the task scheduling center and added to the head of the pending queue; if the URL obtained is a detail page URL, the detail page is parsed;
[0017] Process the data according to the type of data obtained through analysis to obtain public opinion information;
[0018] When a worker node's pending double-ended queue is empty, the worker node broadcasts a message that the work has been completed. The busy worker node that receives the message pops the unprocessed URL from the tail of its pending double-ended queue and sends it to the worker node whose pending double-ended queue is empty for processing;
[0019] Among them, the data is processed according to the type of data obtained by analysis, including: if the data type is text, retain the text; if the data type is audio, convert the audio into text according to the sound characteristics of the audio; if the data type is an image, convert the image into binary form according to the pixel points of the image.
[0020] Furthermore, the deep learning model is used to analyze and calculate the public opinion information to obtain the negative coefficient of the public opinion information, and the negative information is identified according to the negative coefficient and the set negative coefficient threshold, including:
[0021] Input the public opinion information into the deep learning model to obtain the negative coefficient N of the public opinion information;
[0022] According to the negative coefficient N and the set negative coefficient threshold N0, the information satisfying N>N0 is identified as negative information;
[0023] Among them, the deep learning model is an improved convolutional neural network model, and the negative coefficient It is equal to the probability value of the negative sentiment tendency output by the improved convolutional neural network model, and the negative coefficient threshold N0 is adjusted according to the actual situation of the current network public opinion.
[0024] Furthermore, it also includes:
[0025] The working node uses the SHA-256 algorithm to process the string of the account information, release time, and information content of the negative information in order to obtain the Hash value of the negative information, and stores the account information, release time, information content, and Hash value of the negative information in the database DB. i in database DB i Before the storage capacity of the database is 0, the working node will i The content is sent to the database DB w , databaseDB w Determine whether the negative information is stored in the database repeatedly according to the Hash value of the received negative information w , if so, discard the received negative information;
[0026] Among them, the database Represents the database maintained by the i-th working node, database DB w Represents a database that is independent of the worker node.
[0027] Furthermore, the number of negative information that meets the set conditions within a set time period is counted, the negative information propagation coefficient is calculated according to the number of negative information that meets the set conditions within the set time period, and the negative information that exceeds the predetermined propagation power is identified according to the negative information propagation coefficient and the set propagation coefficient threshold, including:
[0028] Each working node monitors for the set duration The two counters it maintains are used to count and , The working nodes after statistics are completed will The broadcast is sent out, and other working nodes that receive the broadcast will count it. Updated to + , continuous statistics and counter updates, to obtain a common ;
[0029] Calculate the negative information propagation coefficient ;
[0030] According to the negative information transmission coefficient and the propagation coefficient threshold set , will satisfy > Negative information is considered to be beyond the expected spread;
[0031] in, is the number of identical or similar negative messages received by each working node, is the number of identical or similar items in all negative information.
[0032] Furthermore, the calculation formula for the real heat of negative information is:
[0033] ;
[0034] In the formula, for The true heat of negative information at all times, , and They are , and The heat of negative information at all times, , and pass D(t) ≈ ∑ t t+Δt [ C R (x) +10 * C L (x)+100 * C C (x)+100 * C S (x)] Δt Calculated, express The heat of negative information at all times, , , and Respectively The number of views, likes, comments and reposts of negative information at any time. Indicates the time period, Indicates the time interval, Less than , and Assigned to working nodes by the scheduling center;
[0035] The calculation formula for the expected heat of negative information is:
[0036] ;
[0037] In the formula, For The expected heat of negative information at all times, for hour The value of ;
[0038] The difference between the actual heat of negative information and the expected heat , will satisfy Negative information is identified as negative information exceeding the predetermined heat level and the real heat level of the negative information is recorded as the heat coefficient;
[0039] in, The difference threshold assigned by the scheduling center to the working node.
[0040] Furthermore, negative information that exceeds the predetermined spread power and negative information that exceeds the predetermined popularity is identified as negative information that exceeds the predetermined influence, including:
[0041] Set the label of negative information that exceeds the predetermined spread as =1, set the label of negative information exceeding the predetermined heat level to =1;
[0042] Calling smart contracts calculate and The result of AND operation , will verify through =1 is considered as negative information that exceeds the expected influence;
[0043] Pack negative information that exceeds the predetermined influence into blocks, broadcast the blocks and connect them to the blockchain, including:
[0044] All through smart contracts All verified negative information is uploaded to the blockchain and the blockchain ledger is maintained;
[0045] Among them, maintaining the blockchain ledger includes: a working node receives the token sent by the task scheduling center and obtains the blockchain accounting right, and other working nodes will use smart contracts to The negative information that is verified to exceed the predetermined influence is sent to the working node with the accounting right. The working node with the accounting right performs a de-redundancy operation on all the received information according to the hash value of the negative information that exceeds the predetermined influence.
[0046] Furthermore, a block includes a block header and a block body. Multiple pieces of public opinion information that appear within a set time are stored in the same block body and organized into a Merkle Tree. The root hash value of the Merkle Tree is stored in the block header. A piece of public opinion information includes the account information, negative coefficient, propagation coefficient, heat coefficient, information content, and the hash value of the information content of the negative information that exceeds the predetermined influence.
[0047] When tracing the source of a public opinion event, the blockchain is queried through the public opinion information corresponding to the public opinion event to obtain the source of the public opinion event, including: each working node maintains a Map container, the key of the Map container is the Hash value of the information content, and the value is the Hash value of a certain block. When tracing the source of a public opinion event, the information content of the public opinion information corresponding to the public opinion event is processed using the SHA-256 algorithm to obtain the Hash value of the information content, and the Map container is queried to obtain the Hash values of several blocks. All records in the block with the smallest timestamp TS are traversed to obtain its account information and find the source of the public opinion event.
[0048] The present invention also provides a distributed public opinion information monitoring system, comprising a task scheduling center and a working node, wherein the task scheduling center and a plurality of the working nodes form a tree network structure, and each of the working nodes forms a P2P network structure, wherein the working nodes include:
[0049] The data collection module is used to receive the URL sent by the task scheduling center, parse the URL and collect public opinion information from the web page corresponding to the URL;
[0050] The first data processing module is used to analyze and calculate the public opinion information using a deep learning model to obtain the negative coefficient of the public opinion information, and identify the negative information according to the negative coefficient and a set negative coefficient threshold;
[0051] The second data processing module is used to count the number of negative information that meets the set conditions within a set time period, calculate the negative information propagation coefficient according to the number of negative information that meets the set conditions within the set time period, and identify negative information that exceeds a predetermined propagation power according to the negative information propagation coefficient and a set propagation coefficient threshold;
[0052] The third data processing module is used to calculate the real popularity and expected popularity of negative information respectively, and identify negative information that exceeds a predetermined popularity according to the difference between the real popularity and the expected popularity and a set difference threshold;
[0053] A fourth data processing module is used to identify negative information that exceeds a predetermined influence based on negative information that exceeds a predetermined spread power and negative information that exceeds a predetermined popularity;
[0054] The data storage module is used to package negative information that exceeds the predetermined influence into blocks, broadcast the blocks and connect them to the blockchain.
[0055] Furthermore, the task scheduling center is configured as follows:
[0056] In rounds, at the beginning of each round, the new URL is header removed, domain name case changed, parameter information removed, formatted, and processed using the SHA-256 algorithm. The processing results are input into multiple hash functions set in the Bloom filter to obtain several position indexes in the Bloom filter, and check whether each position is 1. If not, the URL is added to the head of the queue to be processed and the values stored in the corresponding positions of the several position indexes are set to 1;
[0057] Receive the send load parameters of the working node, according to the working node number Maintain a priority queue with the load parameters, and assign the corresponding URL acquisition quantity to the working node according to the priority queue;
[0058] When more than a predetermined number of working nodes have completed information collection, the next round of information collection begins;
[0059] After a round of information collection is completed, a broadcast instruction notifies the working nodes to perform information analysis;
[0060] Setting negative coefficient threshold Broadcast to working nodes;
[0061] Set monitoring duration , propagation coefficient threshold Send to the working node;
[0062] Set time period , time interval Sum difference threshold Sent to the working nodes.
[0063] Compared with the prior art, the beneficial effects achieved by the present invention are as follows: the working nodes of the present invention independently complete the monitoring process of public opinion information under the coordination of the task scheduling center and the communication and cooperation with each other, thereby improving the efficiency of information collection; blockchain and database are used in combination to store information according to the degree of influence of the public opinion information. While blockchain ensures the security of information storage, it is also easy to hold accountable and trace the source. BRIEF DESCRIPTION OF THE DRAWINGS
[0064] Figure 1 It is a flow chart of a distributed public opinion information monitoring method provided by an embodiment of the present invention;
[0065] Figure 2 It is an architecture diagram of the information collection part in a distributed public opinion information monitoring method provided by an embodiment of the present invention;
[0066] Figure 3 It is a working principle diagram of a Bloom filter in a distributed public opinion information monitoring method provided by an embodiment of the present invention;
[0067] Figure 4This is a principle diagram of counting the number of identical or similar negative information in a distributed public opinion information monitoring method provided by an embodiment of the present invention;
[0068] Figure 5 This is an architecture diagram of a blockchain in a distributed public opinion information monitoring method provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0069] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.
[0070] Embodiment 1:
[0071] This embodiment provides a distributed public opinion information monitoring system, which consists of working nodes and a task scheduling center, and is characterized in that each part uses a distributed architecture. There is a tree-like network structure between the task scheduling center and each working node to transmit information and send control commands to each working node; there is a P2P network structure between the working nodes for relevant communication and collaboration. Each working node can independently complete the information monitoring process under the coordination of the task scheduling center and the communication and collaboration with each other.
[0072] The working nodes include:
[0073] The data collection module is used to receive the URL sent by the task scheduling center, parse the URL and collect public opinion information from the web page corresponding to the URL;
[0074] The first data processing module is used to analyze and calculate the public opinion information using a deep learning model to obtain the negative coefficient of the public opinion information, and identify the negative information according to the negative coefficient and a set negative coefficient threshold;
[0075] The second data processing module is used to count the number of negative information that meets the set conditions within a set time period, calculate the negative information propagation coefficient according to the number of negative information that meets the set conditions within the set time period, and identify negative information that exceeds a predetermined propagation power according to the negative information propagation coefficient and a set propagation coefficient threshold;
[0076] The third data processing module is used to calculate the real popularity and expected popularity of negative information respectively, and identify negative information that exceeds a predetermined popularity according to the difference between the real popularity and the expected popularity and a set difference threshold;
[0077] A fourth data processing module is used to identify negative information that exceeds a predetermined influence based on negative information that exceeds a predetermined spread power and negative information that exceeds a predetermined popularity;
[0078] The data storage module is used to package negative information that exceeds the predetermined influence into blocks, broadcast the blocks and connect them to the blockchain.
[0079] The task scheduling center is configured to work in rounds. At the beginning of each round, the new URL is header-removed, domain name case-converted, parameter information removed, formatted, and processed using the SHA-256 algorithm. The processing results are input into multiple hash functions set in the Bloom filter to obtain several position indexes in the Bloom filter. Check whether each position is 1. If not, the URL is added to the head of the queue to be processed and the values stored in the corresponding positions of several position indexes are set to 1; receive the sending load parameters of the working node , according to the working node number and load parameters Maintaining a priority queue , according to the priority queue Assign the corresponding number of URL acquisitions to the working nodes; start the next round of information collection when more than a predetermined number of working nodes have completed information collection; after a round of information collection is completed, broadcast instructions to notify the working nodes to perform information analysis; set the negative coefficient threshold Broadcast to working nodes; set monitoring duration , propagation coefficient threshold Send to working node; set time period , time interval Sum difference threshold Sent to the working node. The URL is the URL of the web page published by major websites.
[0080] Embodiment 2:
[0081] Based on the distributed public opinion information monitoring system described in Example 1, this embodiment provides a distributed public opinion information monitoring method, including: information collection, information analysis and information storage. The main process is as follows: Figure 1 shown.
[0082] The following is an explanation of the characteristics and workflow of each part:
[0083] like Figure 2 As shown in the figure, the information collection part adopts a distributed collection architecture, including a task scheduling center node and several working nodes. Among them, the task scheduling center is responsible for the overall planning of the system's collection process, and each working node is responsible for the specific public opinion information collection work in this part.
[0084] The target of the collection is various web pages (including but not limited to media platforms such as TikTok and Weibo). Web pages are divided into navigation pages and detail pages. Navigation pages refer to pages containing various URLs (Uniform Resource Locators), and detail pages refer to pages that do not contain URLs but consist of specific data.
[0085] The task scheduling center maintains a queue to be processed Each working node maintains a double-ended queue to be processed locally . All unprocessed URLs are stored in the task scheduling center. The task scheduling center works in rounds. At the beginning of each round, the scheduling center needs to Add new URLs to the header of the task scheduler. These new URLs are generally the URLs of newly released web pages on major websites. Then, each work node Get some URLs at the end of the head.
[0086] In addition, if Figure 3 As shown in Figure 2, the task scheduling center also maintains a Bloom filter BF and sets multiple hash functions When the system is first started, the value of each position of BF is set to 0. When adding a new URL, the URL is preprocessed first, and then the processing results are input into each hash function to obtain several position indexes in the Bloom filter { }, check whether each position is 1, if not, it means that the URL has not been collected yet, add it , and { }The value stored in the corresponding position is set to 1.
[0087] The function of a hash function is to output a corresponding value for any information input. According to different requirements, the hash function can be customized to output a value that meets the requirements. For example, the SHA-256 hash function will output a 256-bit garbled code for any information input. The advantage of a hash function is that it can quickly determine whether a piece of information has been "encountered" before.
[0088] Here, L hash functions are prepared , the output of these hash functions is a number, which represents the index of the Bloom filter. A Bloom filter is an array, which can be understood as a container of boxes arranged one by one. Each grid stores a binary number (either 1 or 0), and each grid has a subscript (or index). For example, the index of the first grid is 1, the index of the second grid is 2, and the index of the 100th grid is 100. When the system is first used, the values of all grids in the Bloom filter are set to 0. After that, each time a URL is encountered, it is processed separately using L hash functions. Because these hash functions are different, L different indexes will be obtained. }. When this URL is processed, the values stored in the cells corresponding to these indexes are changed to 1.
[0089] There is a flaw in the hash function: hash collisions may occur. For example, two different inputs A and B, the hash function produces the same output C. Although the probability is small, it objectively exists. This will lead to misjudgment. B was not received, but the hash positions of A and B are the same, so it will be mistakenly believed that B has been received, and B will be discarded. In order to reduce the probability of misjudgment, Bloom filters use many hash functions. Even if a hash conflict occurs in one of the hash functions, as long as one of the corresponding values in all the hashed positions is 0, it means that the URL has not been processed. Then the URL is preprocessed and the hash value of the URL is further calculated.
[0090] The above-mentioned preprocessing of the URL and further calculating the hash value of the URL include the following steps:
[0091] The first step is to remove the http header of the URL, including http: / / and https: / / .
[0092] This is because all URLs contain an http header, so the http header has no effect on determining whether the URLs are identical and should be removed.
[0093] The second step is to change all the uppercase English letters in the domain name part of the URL to lowercase letters.
[0094] This is because URLs are not case-sensitive. For example, https: / / www.bilibili.com and https: / / www.Bilibili.com point to exactly the same web pages. They are the same URL, but after being processed by the Bloom filter, they map out a series of different locations, which is unreasonable.
[0095] The third step is to remove all parameter information in the URL.
[0096] This is because when a user interacts with a page, some parameters may be sent to the server, and the URL of the webpage will change, but the user will still stay on the page. In other words, whether the URL contains parameter information or whether the parameter information is the same cannot be used as a basis for determining whether the URLs are the same.
[0097] The fourth step is to format the URL, including: replacing all backslashes with forward slashes and removing the trailing slash.
[0098] This is because URLs do not distinguish between forward and backward slashes, and all URLs contain an extra slash at the end to mark the end of the URL.
[0099] The fifth step is to pre-process the URL and use the pre-processed URL as the input information of the SHA-256 function. The SHA-256 function is used to process the pre-processed URL to obtain the hash value of the URL. The specific steps are as follows:
[0100] (1) Run the Python interpreter and input the input information to be processed into the interpreter.
[0101] (2) Import Python’s hashlib library and use the SHA-256 function to perform hash operations on the input information.
[0102] (3) Use the hexdigest function to convert the result of the hash operation into a readable form. Specifically, it can be converted into a 64-bit hexadecimal string.
[0103] It should be noted that all steps involving processing input information using the SHA-256 function in the embodiments of the present invention are the same as those herein, except that the input information itself is different.
[0104] The above five steps are further illustrated below with reference to specific examples:
[0105] Assume that the URL to be processed is: https:\\www.BILIBILI.com\video\X\?spm_id_from=333.337.search-card.all.click\;
[0106] After the first step of processing, we can get: www.BILIBILI.com\video\X\?spm_id_from=333.337.search-card.all.click\;
[0107] After the second step, we can get: www.bilibili.com\video\X\?spm_id_from=333.337.search-card.all.click\;
[0108] After the third step, we can get: www.bilibili.com\video\X\;
[0109] After the fourth step, we can get: www.bilibili.com / video / X;
[0110] After the fifth step, we can get: 93e8678ebdc27f4fb8853e66362dd1a5e7e841b9a3113c4789488267aed935cf.
[0111] When the local node If it is not empty, the URL will be popped up from the end and parsed. There are two cases: if the URL of the navigation page is obtained, all the URLs contained in the navigation page will be returned to the task scheduling center and added. header; if the URL of the detail page is obtained, the information will be processed and temporarily stored.
[0112] The specific data on the details page includes text, audio, and images. Zhihu articles and Tieba comments belong to the text category, music, voice, and the dubbing part of videos belong to the audio category, and the visual part of pictures and videos belongs to the image category. Different processing methods are used for different types of data. For text, no additional processing is required; for audio, it is converted into text according to its sound characteristics; for images, it is converted into binary form according to its pixel points. In order to ensure the accuracy of the subsequent analysis process, the data collected at this time should be compressed as little as possible.
[0113] In addition to a one-to-many tree-like network structure between the task scheduling center and each working node, there is also a P2P network structure between each working node. This is because the task scheduling center allocates collection tasks in units of URLs, but different URLs often have different amounts of data (some web pages have a lot of resources, while others have very little). In order to reduce the pressure on nodes with many tasks, the node with an empty local double-ended queue broadcasts a message FA, indicating that it has completed all tasks. At this time, the busy node that receives FA can pop some unprocessed URLs from the tail and send them to the node to improve the overall processing efficiency of the system.
[0114] During the above collection process, each working node maintains its own real-time updated load parameters . Assume that the CPU usage, memory usage, and network usage of each working node are , and , then the load parameter Each working node needs to Report to the task scheduling center regularly. The task scheduling center maintains a priority queue , where each element is a tuple consisting of a worker node number and a load parameter , elements according to The size of the task is maintained in a monotonically increasing order. For the first two tuples, the task scheduling center will assign the work node number. The corresponding working nodes are assigned more URLs.
[0115] The task scheduling center decides when to start the next round of collection tasks based on the completion status of each working node. The task scheduling center needs to wait for most nodes rather than all nodes to complete the task. This is due to the following two factors: first, the working node may be down due to a fault, and then the task scheduling center's waiting will be useless; second, considering the queue Due to the first-in-first-out feature, nodes that have not completed their tasks can always be processed first in the next round.
[0116] After a round of collection tasks is completed, the task scheduling center broadcasts the command Proceed, and each work node begins to analyze the public opinion information. It should be noted that the collection and analysis of information are independent modules of each work node, which means that the work node collects the next round of information while analyzing the previous round of information.
[0117] For the information analysis part, the specific analysis is as follows:
[0118] The task scheduling center sets a negative coefficient threshold And broadcast it to all working nodes. All working nodes maintain the same deep learning model locally, input all collected and processed public opinion information into the model, and obtain the output negative coefficient .
[0119] The aforementioned deep learning model can use the improved convolutional neural network model CNN (hereinafter referred to as the improved CNN model), which includes the following parts:
[0120] (1) Input layer: The collected public opinion information is further processed to obtain the word vector matrix Matrix, which serves as the raw data input of the improved CNN model.
[0121] (2) Convolutional layer: Use several convolution kernels Ker of different sizes to perform convolution operations on the word vector matrix Matrix generated by the input layer to generate several feature maps.
[0122] (3) Pooling layer: Pooling operation is performed on the feature map generated by the convolutional layer. Specifically, the maximum pooling method can be used to select the maximum value in each feature map as the representative value of the feature map.
[0123] (4) Fully connected layer: connects the eigenvalues output by the pooling layer to form a feature vector.
[0124] (5) Output layer: The feature vector of the fully connected layer is used as the input of the softmax function to obtain the probability values of positive and negative sentiment tendencies, where the probability value of negative sentiment tendency is used as the negative coefficient N of public opinion information. N is output as the final output result of the improved CNN model.
[0125] The difference between the improved CNN model and the traditional CNN model is that the output result of the improved CNN model is not a Boolean type data to represent the emotional polarity (True represents positive, False represents negative), but the probability value of negative emotional tendency obtained by the softmax function is directly used as the negative coefficient Output. Any > All information will be identified as negative information, and its label will be set =1 (default is 0). The advantage is that when encountering certain public opinion events and needing to strengthen or reduce supervision of specific public opinions, managers can directly adjust the dispatch center's allocation of It is more convenient to adjust the value of , rather than adjusting the individual worker nodes and the deep learning models they maintain.
[0126] The characteristics of the information storage part are: using database (such as MySQL, Oracle) and blockchain to store information at the same time. The information stored in the blockchain refers to negative information, and positive information does not need to be stored.
[0127] The process of information tracing is the process of accountability. There is a lot of similar information on the Internet, and most of the information is shared and forwarded by users. It is unrealistic to hold everyone accountable. Therefore, in this embodiment, we only focus on the source of negative information, obtain its user information, and then hold it accountable. This requires recording the complete path of information dissemination and ensuring the integrity and security of the information. However, the database can only record negative information, which makes it difficult to ensure the security of information. The stored information is easily tampered with, which affects the accountability process. Blockchain is a distributed ledger supported by a consensus mechanism, which is jointly maintained by multiple computer nodes. The data stored on it is extremely safe and difficult to tamper with, which just meets this requirement. However, blockchain has the characteristics of high redundancy. When storing the same information, the required space is much higher than that of the database, which determines that the data stored on the blockchain must be small, important and concise. Therefore, this embodiment combines the traditional database with the blockchain. For negative information that does not cause a large impact, the database can be used for storage, while for information that needs to be traced and held accountable and has a significant impact, the blockchain is used for storage.
[0128] Each working node in the system maintains a database ,in Indicates The system is also equipped with a database independent of each working node. , It can be understood as a warehouse that stores records of negative information sent by all nodes. = 1, store the account information Info, release time T, information content C and the Hash value of the information in Among them, Info is the information of the account homepage of the sender of the information, T is the information sending time displayed on the website, and C is the information content of the information. C should be greatly compressed to reduce the storage capacity; the hash value of the information is the result of processing the string of Info, T and C in sequence using the SHA-256 function.
[0129] To save resources, The capacity is small and data needs to be cleared from time to time. Before clearing data, you need to Send the content in to Considering that the system adopts a distributed architecture, it is inevitable that each working node collects, processes and stores a small part of exactly the same information during the actual working process. According to the Hash value of the received information, determine whether the information is consistent with SHA-256 is a well-known hash algorithm that can convert a message (no matter how long) into a 256-bit garbled code after processing. The SHA-256 function can ensure that any different message (even if only slightly different, such as Hello and hello) will have different garbled codes after processing.
[0130] because The capacity is small and data needs to be sent to However, it may happen that a node A and B send exactly the same information (for example, a comment is sent, and this comment is the same information posted by the same user at the same time). In order to prevent information redundancy, Only one copy of the same information should be kept. However, the information itself contains a lot of content, and direct comparison is very time-consuming. The garbled code after hash processing is fixed to 256 bits, which is very short, so you only need to compare the hash value to know whether Info, T, and C are the same. If they are the same, it means that the two messages sent are the same, and one of them should be discarded.
[0131] It should be noted that the only basis for judging the above-mentioned "exactly the same information" is the Hash value of the information. The meaning of the word "same" in "exactly the same information" is different from the meaning of "same" in "same or similar information" in all the following descriptions. The former means that the Hash value of the information is the same, that is, all the contents of the information, including Info, T and C are the same; while the "same" in all the following descriptions only refers to the same information content C.
[0132] On the Internet, the appearance of a large amount of repeated negative information indicates the emergence of a hot topic of public opinion. Therefore, each work node needs to continuously monitor and communicate the sending of the same or similar information within a period of time. The task scheduling center sets a monitoring time. Sent to each working node, such as Figure 4 As shown, each working node maintains its own statistics and Two counters, is the number of identical or similar negative information received by each node. is the number of identical or similar items in all negative information. The nodes that have completed the statistics will After broadcasting, other nodes will count the Change to + Finally, each worker node gets a common value Calculate the negative information propagation coefficient The task scheduling center sets a propagation coefficient threshold Sent to each working node. > Negative information will be considered as highly contagious negative information. =1 (default is 0).
[0133] It should be noted that the “same or similar information” counted by the above working nodes is the same type. For example, when a node and other nodes count a piece of negative information A, The same or similar number of nodes in time, then it will and other nodes will be The number of all negative information A collected within the time period is sent out, and finally all nodes get a common value, which is the number of negative information A and has nothing to do with other negative information. In addition, current computers are parallel systems, which can count the number of negative information A while counting the number of negative information B without conflict.
[0134] The attributes of public opinion information itself can be used as a criterion for judging the popularity of the information, such as the number of views, likes, comments, and reposts of the information, but two issues need to be considered here: First, the popularity of information is not a simple addition of the number of these attributes. For example, the impact of a video being reposted is greater than the impact of a video being viewed. Second, the popularity of information is not for a certain moment, but for a certain period of time. For example, a comment made many years ago will naturally have a lot of views due to its long time ago, but its popularity today is almost 0. Taking the above issues into consideration, the formula for the information popularity at a certain moment, which is approximately expressed by a time period, is given:
[0135] D(t) ≈ ∑ t t+Δt [ C R (x) +10 * C L (x)+100 * C C (x)+100 * C S (x)] Δt ;
[0136] in, express The information popularity of a certain public opinion information at a certain moment, , , and Respectively The number of views, likes, comments and reposts of the information at that moment. Represents a time period, which is allocated to a working node by the scheduling center.
[0137] In order to minimize the impact of accidental factors, it is necessary to calculate , and Information popularity at three moments , and , taking the average value as Real heat at all times ,Right now:
[0138] ,
[0139] in, Indicates the time interval, Greater than , which is assigned to the working nodes by the scheduling center.
[0140] Any public opinion event will experience a period of rising popularity from the time it is released until it reaches a peak, and then experience a long period of decline. High-heat public opinion events generally have the following characteristics: the popularity does not decrease for a long time after reaching the peak, or decreases very little, or even increases suddenly after a period of decline. Therefore, the speed of decline can be used as an indicator to measure whether the popularity of public opinion information is too high.
[0141] It is believed here that the heat of normal public opinion information is an exponential decay function with respect to time, and the relationship is as follows:
[0142]
[0143] in, For The remaining heat of public opinion information at any time is the expected heat. For a certain moment before, it should be satisfied , for hour The value of .
[0144] If there is a difference between the actual heat and the expected heat , threshold If the dispatch center assigns it to a work node, it is considered that the public opinion information is too hot, and the =1, and its real heat is used as the heat coefficient Record.
[0145] It should be noted that the above calculation of the propagation coefficient of negative information and heat coefficient This negative information is not taken out from the database and then calculated. Instead, when a piece of information is identified as negative information after the two steps of collection and analysis, it is immediately processed in the working node and its value is calculated. and Storing information in a database only serves to record the information.
[0146] Both popularity and communication power are the basis for judging influence. The specific process of uploading influential information to the blockchain is as follows: Each node maintains a ledger of the blockchain network. For the negative information stored, the smart contract is called Verify whether it meets the condition of high influence. calculate and The result of AND operation , and determine whether it satisfies =1, if so, the verification is passed, and the information is packaged into blocks and connected to the blockchain. It should be noted here that all the same or similar information that has passed the verification should be uploaded to the blockchain, rather than just storing one copy of the data because they are the same or similar. This is because their release time is different, and the release time is an important basis for judging the source of information.
[0147] It should be noted that since the system adopts a distributed architecture, the information obtained, processed and stored by each working node is different, but the blockchain is a public distributed ledger, and it must be ensured that the blockchain ledger stored in each node is exactly the same. Therefore, the following method is used to ensure the consistency of the ledger: the task scheduling center sends a token to a certain working node, and broadcasts the node number to all other working nodes. The node holding the token obtains the right to record the blockchain, and all other nodes must record all the data stored through the smart contract. The verified information is sent to the node. The node needs to perform redundancy operations on all information, using the hash value of the information as the basis for judgment. The process of handling redundant information is similar and will not be described here.
[0148] like Figure 5 As shown in the figure, the block consists of two parts: the block header and the block body. The block header includes: ① the parent block hash value Pre_Hash, ② the current block hash value Hash, ③ the timestamp TS, and ④ the root hash value Merkle_Root_Hash of the Merkle tree. Hash is the result of processing ③④ using SHA-256. A piece of public opinion information is a record, which contains account information Info, negative coefficient N, propagation coefficient S, heat coefficient D, information content C, and information content Hash value. Account information Info includes account ID, IP location, location, mobile phone number, associated account (such as WeChat), etc. Multiple pieces of public opinion information that appear at the same or almost the same time will be stored in the same block body and organized into a Merkle Tree. The root hash value is calculated and stored in the block header, which is conducive to saving blockchain space.
[0149] When tracing the source and holding accountable is required, it is necessary to query the relevant records in the blockchain. All blocks in the blockchain can be sequentially traversed to find the earliest timestamp block in the same or similar information, and the account information in the record can be obtained for relevant professionals to handle. However, there is a problem with this method: as time goes by, the length of the blockchain, that is, the number of blocks, will continue to increase, and the efficiency of this sequential search method will become lower and lower. Therefore, the Map container is used here to perform query operations. In addition to maintaining the ledger of the blockchain, each working node also maintains a Map container. From the perspective of use, Map is a container with a binary tuple (key, value) as the basic unit, which can quickly query the corresponding value according to the key. From the bottom-level perspective, different Map containers use different data structures, and can use binary search trees, red-black trees, B-trees and other data structures. Here, the bottom layer of Map is implemented with a hash table. Specifically, an array is maintained. For any input key, the key is mapped to a position in the array and the value is written to the position.
[0150] In blockchain query, key is the hash value of the information content, and value is the hash value of a block. The importance of the hash value of a block is that as long as the hash value of the block is known, the specific location of the block in the blockchain is known, so the hash is also called a pointer (ptr).
[0151] A public opinion event can be understood as a collection of a large number of identical or similar public opinion information. For example, when a social hotspot occurs, someone makes an inappropriate comment on it, and then others keep forwarding it. A large amount of information forms a public opinion event. We need to trace the source of the public opinion event, that is, the first information sent, and then hold the publisher accountable. Therefore, when it is necessary to trace the source of a public opinion event, it is only necessary to process the information content of the public opinion information corresponding to the public opinion event with SHA-256 to obtain the Hash value of the information content, and then query the Map container to obtain the Hash values of several blocks, and then traverse all records in the block with the smallest timestamp TS, that is, the block with the earliest position relative to the blockchain, to obtain the required records. The efficiency of the entire query is much higher than that of sequential search.
[0152] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. A distributed public opinion information monitoring method, characterized in that: The method is executed by a working node and includes: Receive the URL sent by the task scheduling center, parse the URL and collect public opinion information from the web page corresponding to the URL; Use deep learning models to analyze and calculate public opinion information, obtain the negative coefficient of public opinion information, and identify negative information based on the negative coefficient and the set negative coefficient threshold; Count the number of negative information that meets the set conditions within a set time period, calculate the negative information propagation coefficient based on the number of negative information that meets the set conditions within the set time period, and identify negative information that exceeds the predetermined propagation power based on the negative information propagation coefficient and the set propagation coefficient threshold; Calculate the real heat and expected heat of negative information respectively, and identify negative information that exceeds the predetermined heat according to the difference between the real heat and the expected heat and the set difference threshold; Negative information that exceeds the expected influence is identified based on negative information that exceeds the expected spread and negative information that exceeds the expected popularity; Pack negative information that exceeds the predetermined influence into blocks, broadcast the blocks and connect them to the blockchain; Count the number of negative information that meets the set conditions within a set time period, calculate the negative information propagation coefficient based on the number of negative information that meets the set conditions within the set time period, and identify negative information that exceeds the predetermined propagation power based on the negative information propagation coefficient and the set propagation coefficient threshold, including: Each working node monitors for the set duration The two counters it maintains are used to count and , The working nodes after statistics are completed will The broadcast is sent out, and other working nodes that receive the broadcast will count it. Updated to + , continuous statistics and counter updates, to obtain a common ; Calculate the negative information propagation coefficient ; According to the negative information transmission coefficient and the propagation coefficient threshold set , will satisfy > Negative information is considered to be beyond the expected spread; in, is the number of identical or similar negative messages received by each working node, is the same or similar number of all negative information; The calculation formula for the true heat of negative information is: ; In the formula, for The true heat of negative information at all times, , and They are , and The heat of negative information at all times, , and pass Calculated, express The heat of negative information at all times, , , and Respectively The number of views, likes, comments and reposts of negative information at any time. Indicates the time period, Indicates the time interval, Less than , and Assigned to working nodes by the scheduling center; The calculation formula for the expected heat of negative information is: ; In the formula, For The expected heat of negative information at all times, for hour The value of ; The difference between the actual heat of negative information and the expected heat , will satisfy Negative information is identified as negative information that exceeds the predetermined heat level and the actual heat level of the negative information is recorded as the heat coefficient; in, The difference threshold assigned to the working node by the scheduling center; Negative information that exceeds the expected spread and negative information that exceeds the expected popularity will be identified as negative information that exceeds the expected influence, including: Set the label of negative information that exceeds the predetermined spread as =1, set the label of negative information exceeding the predetermined heat level to =1; Calling smart contracts calculate and The result of AND operation , will verify through =1 is considered as negative information that exceeds the expected influence; Pack negative information that exceeds the predetermined influence into blocks, broadcast the blocks and connect them to the blockchain, including: All through smart contracts All verified negative information is uploaded to the blockchain and the blockchain ledger is maintained; Among them, maintaining the blockchain ledger includes: a working node receives the token sent by the task scheduling center and obtains the blockchain accounting right, and other working nodes will use smart contracts to Verify that the negative information that exceeds the predetermined influence is sent to the work node with the right to record. The work node with the right to record performs redundancy operations on all the received information based on the hash value of the negative information that exceeds the predetermined influence. The task scheduling center is configured as follows: In rounds, at the beginning of each round, the new URL is header removed, domain name case changed, parameter information removed, formatted, and processed using the SHA-256 algorithm. The processing results are input into multiple hash functions set in the Bloom filter to obtain several position indexes in the Bloom filter. Check whether each position is 1. If not, add the URL to the head of the queue to be processed and set the values stored in the corresponding positions of several position indexes to 1; Receive the send load parameters of the working node, according to the working node number Maintain a priority queue with the load parameters, and assign the corresponding URL acquisition quantity to the working node according to the priority queue; When more than a predetermined number of working nodes have completed information collection, the next round of information collection begins; After a round of information collection is completed, a broadcast instruction notifies the working nodes to perform information analysis; Setting negative coefficient threshold Broadcast to working nodes; Set monitoring time , propagation coefficient threshold Send to the working node; Set time period , time interval Sum difference threshold Sent to the working nodes.
2. The distributed public opinion information monitoring method according to claim 1 is characterized in that: Receive the URL sent by the task scheduling center, parse the URL and collect public opinion information from the web page corresponding to the URL, including: The working node obtains the URL from the tail of the pending queue of the task scheduling center and inserts the obtained URL into the head of the pending double-ended queue of the working node; The working node maintains the load parameters updated in real time and reports the load parameters to the task scheduling center at regular intervals. The working node obtains the URL according to the URL acquisition quantity allocated by the task scheduling center based on the load parameters of the working node; When the pending double-ended queue is not empty, a URL is popped from the tail of the pending double-ended queue and parsed: if the URL obtained is a navigation page URL, all URLs in the navigation page are returned to the task scheduling center and added to the head of the pending queue; if the URL obtained is a detail page URL, the detail page is parsed; Process the data according to the type of data obtained through analysis to obtain public opinion information; When a worker node's pending double-ended queue is empty, the worker node broadcasts a message that the work has been completed. The busy worker node that receives the message pops the unprocessed URL from the tail of its pending double-ended queue and sends it to the worker node whose pending double-ended queue is empty for processing; Among them, the data is processed according to the type of data obtained by analysis, including: if the data type is text, retain the text; if the data type is audio, convert the audio into text according to the sound characteristics of the audio; if the data type is an image, convert the image into binary form according to the pixel points of the image.
3. The distributed public opinion information monitoring method according to claim 1 is characterized in that: Use deep learning models to analyze and calculate public opinion information to obtain the negative coefficient of public opinion information. According to the negative coefficient and the set negative coefficient threshold, negative information is identified, including: Input the public opinion information into the deep learning model to obtain the negative coefficient N of the public opinion information; According to the negative coefficient N and the set negative coefficient threshold N0, the information satisfying N>N0 is identified as negative information; Among them, the deep learning model is an improved convolutional neural network model, and the negative coefficient It is equal to the probability value of the negative sentiment tendency output by the improved convolutional neural network model, and the negative coefficient threshold N0 is adjusted according to the actual situation of the current network public opinion.
4. The distributed public opinion information monitoring method according to claim 1 is characterized in that: Also includes: The working node uses the SHA-256 algorithm to process the string of the account information, release time, and information content of the negative information in order to obtain the Hash value of the negative information, and stores the account information, release time, information content, and Hash value of the negative information in the database DB. i in database DB i Before the storage capacity of the database is 0, the working node will i The content is sent to the database DB w , databaseDB w Determine whether the negative information is stored in the database repeatedly according to the Hash value of the received negative information w , if so, discard the received negative information; Among them, the database Represents the database maintained by the i-th working node, database DB w Represents a database independent of the worker node.
5. The distributed public opinion information monitoring method according to claim 1 is characterized in that: The block includes a block header and a block body. Multiple pieces of public opinion information that appear within a set time are stored in the same block body and organized into a Merkle Tree. The root hash value of the Merkle Tree is stored in the block header. A piece of public opinion information includes the account information, negative coefficient, propagation coefficient, heat coefficient, information content, and the hash value of the information content of the negative information that exceeds the predetermined influence. When tracing the source of a public opinion event, the blockchain is queried through the public opinion information corresponding to the public opinion event to obtain the source of the public opinion event, including: each working node maintains a Map container, the key of the Map container is the Hash value of the information content, and the value is the Hash value of a certain block. When tracing the source of a public opinion event, the information content of the public opinion information corresponding to the public opinion event is processed using the SHA-256 algorithm to obtain the Hash value of the information content, and the Map container is queried to obtain the Hash values of several blocks. All records in the block with the smallest timestamp TS are traversed to obtain its account information and find the source of the public opinion event.
6. A distributed public opinion information monitoring system, characterized in that: It includes a task scheduling center and working nodes. The task scheduling center and multiple working nodes form a tree network structure. Each working node forms a P2P network structure. The working nodes include: The data collection module is used to receive the URL sent by the task scheduling center, parse the URL and collect public opinion information from the web page corresponding to the URL; The first data processing module is used to analyze and calculate the public opinion information using a deep learning model to obtain the negative coefficient of the public opinion information, and identify the negative information according to the negative coefficient and a set negative coefficient threshold; The second data processing module is used to count the number of negative information that meets the set conditions within a set time period, calculate the negative information propagation coefficient according to the number of negative information that meets the set conditions within the set time period, and identify negative information that exceeds a predetermined propagation power according to the negative information propagation coefficient and a set propagation coefficient threshold; The third data processing module is used to calculate the real popularity and expected popularity of negative information respectively, and identify negative information that exceeds a predetermined popularity according to the difference between the real popularity and the expected popularity and a set difference threshold; A fourth data processing module is used to identify negative information that exceeds a predetermined influence based on negative information that exceeds a predetermined spread power and negative information that exceeds a predetermined popularity; A data storage module, used to package negative information that exceeds a predetermined influence into blocks, broadcast the blocks and connect them to the blockchain; Count the number of negative information that meets the set conditions within a set time period, calculate the negative information propagation coefficient based on the number of negative information that meets the set conditions within the set time period, and identify negative information that exceeds the predetermined propagation power based on the negative information propagation coefficient and the set propagation coefficient threshold, including: Each working node monitors for the set duration The two counters it maintains are used to count and , The working nodes after statistics are completed will The broadcast is sent out, and other working nodes that receive the broadcast will count it. Updated to + , continuous statistics and counter updates, to obtain a common ; Calculate the negative information propagation coefficient ; According to the negative information transmission coefficient and the propagation coefficient threshold set , will satisfy > Negative information is considered to be beyond the expected spread; in, is the number of identical or similar negative messages received by each working node, is the same or similar number of all negative information; The calculation formula for the true heat of negative information is: ; In the formula, for The true heat of negative information at all times, , and They are , and The heat of negative information at all times, , and pass Calculated, express The heat of negative information at all times, , , and Respectively The number of views, likes, comments and reposts of negative information at any time. Indicates the time period, Indicates the time interval, Less than , and Assigned to working nodes by the scheduling center; The calculation formula for the expected heat of negative information is: ; In the formula, For The expected heat of negative information at all times, for hour The value of ; The difference between the actual heat of negative information and the expected heat , will satisfy Negative information is identified as negative information exceeding the predetermined heat level and the real heat level of the negative information is recorded as the heat coefficient; in, The difference threshold assigned to the working node by the scheduling center; Negative information that exceeds the expected spread and negative information that exceeds the expected popularity will be identified as negative information that exceeds the expected influence, including: Set the label of negative information that exceeds the predetermined spread as =1, set the label of negative information exceeding the predetermined heat level to =1; Calling smart contracts calculate and The result of AND operation , will verify through =1 is considered as negative information that exceeds the expected influence; Pack negative information that exceeds the predetermined influence into blocks, broadcast the blocks and connect them to the blockchain, including: All through smart contracts All verified negative information is uploaded to the blockchain and the blockchain ledger is maintained; Among them, maintaining the blockchain ledger includes: a working node receives the token sent by the task scheduling center and obtains the blockchain accounting right, and other working nodes will use smart contracts to Verify that the negative information that exceeds the predetermined influence is sent to the work node with the right to record. The work node with the right to record performs redundancy operations on all the received information based on the hash value of the negative information that exceeds the predetermined influence. The task scheduling center is configured as follows: In rounds, at the beginning of each round, the new URL is header removed, domain name case changed, parameter information removed, formatted, and processed using the SHA-256 algorithm. The processing results are input into multiple hash functions set in the Bloom filter to obtain several position indexes in the Bloom filter. Check whether each position is 1. If not, add the URL to the head of the queue to be processed and set the values stored in the corresponding positions of several position indexes to 1; Receive the send load parameters of the working node, according to the working node number Maintain a priority queue with the load parameters, and assign the corresponding URL acquisition quantity to the working node according to the priority queue; When more than a predetermined number of working nodes have completed information collection, the next round of information collection begins; After a round of information collection is completed, a broadcast instruction notifies the working nodes to perform information analysis; Setting negative coefficient threshold Broadcast to working nodes; Set monitoring time , propagation coefficient threshold Send to the working node; Set time period , time interval Sum difference threshold Sent to the working nodes.
