A high-real-time social network data collection method for public safety

Abstract

The present invention provides a high-real-time social network data collection method for public security, belonging to the technical field of network data collection, and solving typical problems in existing methods such as node homogeneity, redundant noise, and large time delay. The method comprises: constructing an initial monitoring node set of the social network, selecting and expanding the fourth-order neighbors of the nodes based on the four-dimensional space principle of the small-world theory to obtain a candidate monitoring node set, calculating and removing nodes with high homogeneity to obtain a final monitoring node set; then collecting text information of the social network content of each node, clustering the text information, obtaining topic information and time-series changes of propagation characteristics, obtaining potential topics and feature words, and then constructing a data collection request to collect data on a specified topic. By updating the monitoring nodes, the present invention can efficiently obtain information related to the event before the event propagates on the social network, providing technical and data guarantees for subsequent applications.

Description

Technical Field

[0001] The present invention belongs to the technical field of network data collection and is applied to social networks related to public safety. Specifically, it is a high-real-time social network data collection method for public safety. Background Art

[0002] With the development of network information technology, social networks represented by Weibo, WeChat, Toutiao and Douyin are becoming increasingly popular. Major mainstream traditional media including People's Daily and CCTV have also opened accounts on social networks and used them to transmit information. Social networks have gradually replaced traditional media forms and become the main medium for people to obtain information.

[0003] Hundreds of millions of pieces of information are published and disseminated on social networks every day. This vast amount of information encompasses a wide range of topics, with information on public safety being particularly crucial. Hidden within this information lies immense value and crucial decision-making implications. Efficiently filtering and collecting this information has long been a research topic for those skilled in the art.

[0004] In the existing technology, the data collection methods for social networks still have the following deficiencies:

[0005] 1. Targeting specific interesting content on social networks, a "snowball sampling" approach is used to collect data based on the friend relationships of the collection nodes. While this data collection process is targeted, it also suffers from the disadvantage of high homogeneity. Furthermore, this collection method does not differentiate between the types of data being collected, collecting a large amount of noise data that is irrelevant to subsequent applications and actual needs, such as daily life information between friends, advertising information, and gossip about celebrities.

[0006] 2. For specific events, when collecting event data, users are required to enter "keywords" related to the event in advance in order to collect information about events that have become hot topics. This will cause subsequent applications of event collection, such as public opinion analysis, to have a very large time delay, making it impossible to effectively capture the development dynamics of events in social networks in real time. At present, there is still no efficient and feasible method for the information judgment process that may become a hot topic and for collecting data based on this judgment process.

[0007] Therefore, the above two deficiencies can be summarized into two typical problems:

[0008] First, the existing technology lacks an effective monitoring node selection strategy, the monitoring nodes are highly homogeneous, and the collected data contains a large amount of redundant data and noise data.

[0009] Second, when data is collected from social network events based on static feature word technology, the data collection time delay is large, and the events in the social network can no longer provide real-time technical support for subsequent applications of the collection program.

[0010] Regarding the first question, since there are a huge number of users in social networks and the information generators and transmitters in social networks meet the 80 / 20 principle, selecting a suitable and representative set of user nodes in social networks as the basis for data collection and timely grasping the information being disseminated in social networks have always been key and difficult issues that need to be urgently addressed in the social network data collection process.

[0011] Regarding the second problem, existing methods require the collector to input feature words related to the event when collecting data. As a result, existing data collection programs can only collect data that has been widely circulated in social networks and has a large impact, but are unable to perceive and process events in a timely manner in the early stages of the spread of related events. Summary of the Invention

[0012] The purpose of the present invention is to solve the above-mentioned two problems existing in existing social network data collection methods: namely, to disclose a method for how to effectively select representative key nodes in a social network with many active users, and how to collect key information that may become hot topics in the social network in real time; the present invention uses a dynamic update algorithm for monitoring nodes to further select topic events with large change trends, extract subject keywords, and construct collection requests, thereby more comprehensively collecting relevant data on events that may become hot topics, providing real-time and comprehensive data support for subsequent applications such as public opinion analysis and public opinion guidance.

[0013] The present invention adopts the following technical solutions to achieve the purpose:

[0014] A high-real-time social network data collection method for public safety consists of two sequential processes: a dynamic update process of monitoring nodes based on user activity and an event collection strategy execution process based on window heat.

[0015] The monitoring node dynamic update process includes the following steps:

[0016] S11, constructing an initial set of monitoring nodes of the social network;

[0017] S12. According to the four-dimensional space principle of the small-world theory, select the fourth-order neighbors of all nodes in the initial monitoring node set, expand the set, and obtain the candidate monitoring node set;

[0018] S13. Calculate the degree of homogeneity among the nodes in the candidate monitoring node set, remove nodes with high homogeneity, and obtain the final monitoring node set;

[0019] The event collection strategy execution process includes the following steps:

[0020] S21. Collect text information corresponding to the social network content of each node based on the final set of monitoring nodes;

[0021] S22. Clustering the collected text information using a clustering model to obtain topic information currently being disseminated on the social network;

[0022] S23. Obtaining a potential topic set based on the temporal changes in the propagation characteristics of the topic information;

[0023] S24, using an information data mining algorithm to obtain characteristic words of each topic in the topic information;

[0024] S25. Based on the feature words, a data collection request is constructed, and data on the specified topic is obtained through the search tool of the social network to realize data collection.

[0025] Furthermore, in step S11, the social user list in the social network is monitored through a data collection program; when a user node in the list changes, the user node information in the changed list is collected and the user node information is added to the initial monitoring node set V.

[0026] Furthermore, in step S12, based on the initial monitoring node set, the four-dimensional space principle of the small-world theory is adopted to obtain the fourth-order neighbor nodes of each node in the set; all the fourth-order neighbor nodes are added to the initial monitoring node set to expand the set and obtain the candidate monitoring node set V′.

[0027] Furthermore, in step S22, the BerTopic clustering model is used to cluster the text information in the acquired information set. The specific process is: based on the collected text information, the pre-trained framework BERT is used to represent the text information, and the word embedding vector of the corresponding information is obtained to form an embedding set; the embedding set is input into the BerTopic clustering model to obtain a cluster information set G; among the multiple topics in the set, each topic is formed by combining the word embedding vectors of the same type of text information.

[0028] Furthermore, in step S23, the changes in the propagation attributes of each topic in the cluster information set G at a specific time interval are counted as the corresponding propagation feature change value; an initial threshold is also given. When the propagation feature change value of a topic exceeds the initial threshold, the information of the corresponding topic is collected to form a potential topic set.

[0029] Furthermore, in step S24, based on the potential topic set, the BerTopic clustering model is used again to represent multiple pieces of information in the set as a single document; then the cluster-based word frequency-inverse word frequency segmentation algorithm and word algorithm in the information data mining algorithm are used to extract the feature word set W that represents the topic class.

[0030] In summary, due to the adoption of this technical solution, the beneficial effects of the present invention are as follows:

[0031] The method of the present invention can quickly and efficiently obtain information that is being disseminated in social networks, and based on the identification of potential topics, can achieve timely and comprehensive acquisition of information related to an event before it is widely disseminated in social networks, thereby providing technical and data guarantees for subsequent applications.

[0032] The present invention collects hot events in social networks in real time by monitoring the updates of node sets, and selects subjects with larger change trends based on the changing trends of propagation characteristics within a specified window, and constructs data collection requests by extracting keyword feature words. Therefore, it can more comprehensively collect relevant data of possible hot events, which is conducive to carrying out public opinion discovery, analysis and processing based on the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 Schematic diagram of the overall data collection process of the method of the present invention. DETAILED DESCRIPTION

[0034] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0035] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention as claimed, but rather merely represents selected embodiments of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort shall fall within the scope of protection of the present invention.

[0036] Example

[0037] A high-real-time social network data collection method for public safety. Figure 1 The overall process of this method is shown. This method generally consists of two processes that are carried out in sequence, namely the dynamic update process of monitoring nodes based on user activity and the event collection strategy execution process based on window heat.

[0038] The dynamic update process of the monitoring node focuses on social public security events and user activity. It selects social events that are of great interest to netizens on social networks and have a large social network reach and influence as data collection targets. In this embodiment, Sina Weibo is used as an example of a social network to explain the method of this embodiment in detail.

[0039] The dynamic update process of the monitoring node includes the following steps:

[0040] S11, constructing an initial set of monitoring nodes of the social network;

[0041] S12. According to the four-dimensional space principle of the small-world theory, select the fourth-order neighbors of all nodes in the initial monitoring node set, expand the set, and obtain the candidate monitoring node set;

[0042] S13. Calculate the degree of homogeneity between nodes in the candidate monitoring node set, remove nodes with high homogeneity, and obtain the final monitoring node set.

[0043] The execution process of the event collection strategy is to collect hot events in social networks in real time. Based on natural language processing technology and clustering technology in machine learning technology, data collection requests are constructed through changing trends.

[0044] The event collection strategy execution process includes the following steps:

[0045] S21. Collect text information corresponding to the social network content of each node based on the final set of monitoring nodes;

[0046] S22. Clustering the collected text information using a clustering model to obtain topic information currently being disseminated on the social network;

[0047] S23. Obtaining a potential topic set based on the temporal changes in the propagation characteristics of the topic information;

[0048] S24, using an information data mining algorithm to obtain characteristic words of each topic in the topic information;

[0049] S25. Based on the feature words, a data collection request is constructed, and data on the specified topic is obtained through the search tool of the social network to realize data collection.

[0050] The following is a detailed example description and detailed introduction of each of the above steps in this embodiment. Figure 1 The processes shown are explained in sequence.

[0051] (1) Generate an initial set of monitoring nodes;

[0052] In this embodiment, the data collection program monitors the social user list of "V Influence" on Sina Weibo in real time; once a user node in this list is found to have changed, the user node information of the changed list is immediately collected and the obtained user node information is added to the initial monitoring node set V = {u1, u2, ..., u m}; among them, u i is the user identifier of the i-th user, and m is the total number of users in the list.

[0053] (2) Based on the four-dimensional principle of the small-world theory, a set of candidate monitoring nodes is generated;

[0054] In this embodiment, based on the initial monitoring node set, the fourth-order neighbor nodes of each node in the set are obtained; all the fourth-order neighbor nodes are added to the initial monitoring node set to expand the set, and the candidate monitoring node set V′={u1, u2, ..., u n}; where n is the number of nodes in the initial monitoring node set and the sum of the number of nodes of all fourth-order neighbor nodes.

[0055] (3) Based on the degree of node homogeneity, similar monitoring nodes in the candidate monitoring node set are filtered out to obtain the final monitoring node set;

[0056] In this embodiment, based on the candidate monitoring node set, the degree of homogeneity between nodes is calculated, and nodes with high homogeneity with existing nodes in the set are removed to obtain the final monitoring node set V″={u1, u2, ..., u o}; where o is the total number of nodes in the final monitoring node set, and the degree of homogeneity sim between nodes is calculated as follows:

[0057]

[0058] Where u i ,u j Indicates the user identification number of user node i and user node j; {follower(u i )} means follow user u i The user node collection, {followee(u i )} represents user u i The collection of user nodes followed; the symbol |…| indicates obtaining the number of elements in the collection.

[0059] (4) Based on the final monitoring node, obtain the data being spread on the social network;

[0060] In this embodiment, the information set M published by each user node at a specific time interval is obtained as follows:

[0061] M={info1, info2,...,info i ; c1, c2, ..., c i ;f1,f2,...,f i ;s1,s2,...,s i}

[0062] Among them, the subscript i represents the serial number of the same type of elements in the set; info i represents the i-th message published by the user in a specific event δ, c i represents the number of comments on the i-th message, f i Indicates the number of times the i-th message is forwarded, s i Indicates the number of likes corresponding to the i-th information; the above-obtained information set is stored in the database in the form of text information.

[0063] (5) Based on the BerTopic clustering model, the collected information is clustered to obtain the main discussion topics in the social network;

[0064] In this embodiment, the BerTopic clustering model is used to cluster the text information in the acquired information set. The specific process is as follows: based on the collected text information, the pre-trained framework BERT is used to represent the text information, and the word embedding vector (word embedding) of the corresponding information is obtained to form an embedding set; the embedding set is input into the BerTopic clustering model to obtain the cluster information set G = {g1, g2, ..., g i}; where g i Represents the i-th topic obtained by clustering. Each topic is formed by combining the word embedding vectors of similar text information.

[0065] (6) Based on the changes in the statistical characteristics of the dissemination of topic information, obtain potential topics that may become hot topics in social networks;

[0066] In this embodiment, the change of the propagation attribute of each topic in the cluster information set G at a specific time interval is counted as the corresponding propagation characteristic change value H p ; In addition, an initial threshold is given. If the propagation feature change value is greater than the initial threshold, it means that this topic is likely to become a hot topic. Therefore, when the propagation feature change value of a topic exceeds the initial threshold, the information of the corresponding topic is collected to form a potential topic set. The propagation feature change value H p The calculation formula is as follows:

[0067]

[0068] In the formula, p represents a category in the cluster; c pi(t) represents the number of comments on the i-th original information in the p-th category at time t; f pi (t) represents the number of forwarding of the i-th original information in the p-th category at time t; s pi (t) represents the number of likes for the i-th original information in the p-th category at time t.

[0069] (7) Based on the potential topics, obtain the characteristic words that can represent the subject, construct a data collection request, and obtain the potential topic information in real time and comprehensively;

[0070] In this embodiment, based on the potential topic set, the BerTopic clustering model is used again to represent multiple pieces of information in the set as a single document. Then, the cluster-based word frequency-inverse word frequency segmentation algorithm, i.e., the c-TF-IDF algorithm, in the information data mining algorithm, is used to extract the feature word set W that best represents the topic class. In this embodiment, the extracted W = (w1, w2, w3, w4, w5), a total of 5 words. The calculation formula for the relationship between each feature word in the set and the category representation is as follows:

[0071]

[0072] In the formula, x represents a feature word in the document; tf x，c represents the frequency of feature word x in category c; f x It represents the frequency of the feature word x in all the collected subject categories; A represents the average frequency of the feature word x in each subject category.

[0073] By using and calculating the extracted feature word vectors, a data collection request Req related to the corresponding features is constructed to obtain data related to the potential topic. The data collection request is executed using a search tool in a social network. In this embodiment, the search on Sina Weibo is used as an example. The request construction process is as follows:

[0074] Req=https: / / weibo.com / ajax / statuses / search? q=w1+w2+w3+w4+w5

[0075] Through the constructed request, search results related to the potential topic are presented, and the data of all search results are collected and stored in the database again as the final social network data collection result.

Claims

1. A high-real-time social network data collection method for public safety, characterized by: The method consists of two sequential processes: a dynamic update process of monitoring nodes based on user activity and an event collection strategy execution process based on window heat; The monitoring node dynamic update process includes the following steps: S11, constructing an initial set of monitoring nodes of the social network; S12. According to the four-dimensional space principle of the small-world theory, select the fourth-order neighbors of all nodes in the initial monitoring node set, expand the set, and obtain the candidate monitoring node set; S13. Calculate the degree of homogeneity among the nodes in the candidate monitoring node set, remove nodes with high homogeneity, and obtain the final monitoring node set; The event collection strategy execution process includes the following steps: S21. Collect text information corresponding to the social network content of each node based on the final set of monitoring nodes; S22. Clustering the collected text information using a clustering model to obtain topic information currently being disseminated on the social network; S23. Obtaining a potential topic set based on the temporal changes in the propagation characteristics of the topic information; S24, using an information data mining algorithm to obtain characteristic words of each topic in the topic information; S25. Based on the feature words, a data collection request is constructed, and data on the specified topic is obtained through the search tool of the social network to realize data collection; In step S23, the clustering information set is statistically analyzed. The change of propagation attributes of each topic in a specific time interval is used as the corresponding propagation feature change value ; In addition, an initial threshold is given. When the change value of the propagation characteristic of a topic exceeds the initial threshold, the information of the corresponding topic is collected to form a potential topic set; Propagating feature change values The calculation formula is as follows: Where, Represents a category in the cluster; express Moment in the category Number of comments on the original message; express Moment in the category Number of reposts of the original message; express Moment in the category The number of likes for the original post.

2. The method for collecting high-real-time social network data for public safety according to claim 1, characterized in that: In step S11, the social user list in the social network is monitored through the data collection program; when the user node in the list changes, the user node information in the changed list is collected , and add the user node information to the initial monitoring node set Among them, For the User identifier for each user, The total number of users in the list.

3. The method for collecting high-real-time social network data for public safety according to claim 2, characterized in that: In step S12, based on the initial monitoring node set, the four-dimensional space principle of the small-world theory is used to obtain the fourth-order neighbor nodes of each node in the set; all the fourth-order neighbor nodes are added to the initial monitoring node set to expand the set and obtain the candidate monitoring node set. ;in, is the sum of the number of nodes in the initial monitoring node set and the number of nodes of all fourth-order neighbor nodes.

4. The method for collecting high-real-time social network data for public safety according to claim 3, characterized in that: In step S13, based on the candidate monitoring node set, the degree of homogeneity between nodes is calculated, and nodes with high homogeneity with existing nodes in the set are removed to obtain the final monitoring node set. ;in, is the total number of nodes in the final monitoring node set, and the degree of homogeneity between nodes The calculation formula is as follows: Where, Represents a user node and user nodes User identification number; Show concern for users The collection of user nodes, Represents a user Collection of user nodes followed; symbol Gets the number of elements in a collection.

5. The method for collecting high-real-time social network data for public safety according to claim 1, characterized in that: In step S21, the information set published by each user node at a specific time interval is obtained ,as follows: Among them, the subscript Indicates the ordinal number of the same type of elements in the set; Indicates that the user is in a specific event The first Messages, Indicates the Number of comments on the message, Indicates the Number of times the message was forwarded, Indicates the The number of likes corresponding to the information; the above information set obtained is stored in the database in the form of text information.

6. The method for collecting high-real-time social network data for public safety according to claim 5, characterized in that: In step S22, the text information in the acquired information set is clustered using the BerTopic clustering model. The specific process is as follows: based on the collected text information, the pre-trained framework BERT is used to represent the text information, and the word embedding vector of the corresponding information is obtained to form an embedding set; the embedding set is input into the BerTopic clustering model to obtain a clustered information set. ;in, Indicates the clustering result topics, each of which is formed by combining word embedding vectors of similar text information.

7. The method for collecting high-real-time social network data for public safety according to claim 1, characterized in that: In step S24, based on the potential topic set, the BerTopic clustering model is used again to characterize multiple pieces of information in the set into a single document; then the cluster-based word frequency-inverse word frequency segmentation algorithm in the information data mining algorithm is used to extract the characteristic word set representing the topic class. ; The calculation formula for each feature word and category representation relationship in the set is as follows: Where, Represents a feature word in the document; Characteristic words In category Frequency of occurrence in Characteristic words frequency of occurrence across all thematic categories collected; Characteristic words The average frequency of occurrence in each theme category.

8. The method for collecting high-real-time social network data for public safety according to claim 7, characterized in that: In step S25, the feature words representing the subject class are used to form corresponding feature word vectors, thereby constructing a data collection request related to the potential topic; After executing the data collection request using the search tool in the social network, the search results related to the potential topic are presented, and the data of all the search results are collected and stored in the database again as the final social network data collection results.

Images