A traffic safety public opinion analysis method based on an SQ-LDA topic model

Abstract

The present application relates to the technical field of data mining, and especially relates to a traffic safety public opinion analysis method based on an SQ-LDA topic model. The present application comprises the following steps: S1, obtaining traffic safety social software public opinion data; S2, preprocessing the traffic safety social software public opinion data; S3, extracting feature topics by using the SQ-LDA topic model; and S4, visualizing traffic safety social software public opinion hotspots. The sample quality negative sampling method in the SQ-LDA topic model of the present application calculates sample information quantity according to a gradient, so as to distinguish high and low quality negative samples, filter out low quality negative samples, maximize positive sample probability, minimize high quality negative sample probability, and further effectively mine and analyze traffic safety social software public opinion data. The present application sets a keyword frequency probability threshold value under a topic, and when the frequency probability of a keyword exceeds the set frequency probability threshold value, the keyword public opinion early warning under the topic will be triggered, thereby providing necessary technical support for social public opinion early warning.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data mining, and particularly relates to a traffic safety public opinion analysis method based on an SQ-LDA topic model. BACKGROUND

[0002] With the rapid development of social media, various social network platforms have become important channels for the public to obtain and publish various public opinion related information. Transportation is the "advance guard" of economic and social development. In recent years, the integration of transportation and emerging technologies such as big data and artificial intelligence, and the in-depth promotion of intelligent transportation, traffic safety events have been widely spread on social network platforms, forming traffic safety network public opinion.

[0003] In order to effectively explore traffic safety network public opinion, various probability topic models have been proposed by scholars at home and abroad. There are currently two kinds of probability topic models: the probability latent semantic analysis model (PLSA) and the traditional latent Dirichlet allocation (LDA) topic model. Among them, the PLSA model usually uses the EM algorithm to solve the model, which has given semantic information, but this semantic information cannot explain the polysemy of text, and the amount of calculation is large, and the understanding of the theme information is not intuitive enough. The traditional LDA topic model adds Dirichlet prior distribution on the basis of the PLSA model, effectively solving the problem of polysemy in large-scale text sets.

[0004] Because syntax and phrases are the key to capturing text semantics, based on the traditional LDA model, many improved LDA topic models have been proposed by researchers at home and abroad in the aspect of semantic optimization, such as word weight, graph mining, sampling method, etc. In 2019, Yuaya et al. proposed an LDA topic model based on dynamic weight by dynamically obtaining word weight through semantic information. In 2022, Li Shan et al. proposed an LDA topic model based on graph mining by effectively combining the idea of semantic relevance. In the same year, Li Mingchao et al. fully considered the association relationship between words, replaced the original Gibbs sampling method with word pairs generated by taking the co-occurrence degree between words as the standard, and proposed an improved LDA topic model.

[0005] With the deepening of the research of LDA topic model, many improved LDA topic models have been applied to text clustering, personalized recommendation and so on. But for some professional texts in certain fields, such as traffic safety public opinion texts, water conservancy engineering construction management texts and so on, the improved traditional LDA topic model has problems such as fuzzy theme division and chaotic context logic. In order to solve these problems, Liu Jinxia et al. proposed an LDA topic model based on random negative sampling, which uses the negative sampling method to clearly describe the relationship between context words and effectively distinguish theme words. However, in most of the research process, the quality of negative samples is different, the information amount of high-quality negative samples and the information amount carried by positive samples are small, and the information amount of low-quality negative samples and the information amount carried by positive samples are large, that is, low-quality negative samples have small optimization effect on the model. However, the current sample sampling method cannot distinguish high and low quality negative samples, therefore the sampling method of the improved LDA topic model is improved, and the sample quality negative sampling is helpful to strengthen the connection between the mined word information and improve the text information mining efficiency. SUMMARY

[0006] The purpose of the present application is to solve the problem that the traditional LDA topic model in the prior art does not consider the influence of distinguishing high and low quality negative samples on the correct analysis of semantics of the model, and a traffic safety public opinion analysis method based on SQ-LDA topic model is proposed.

[0007] In order to achieve the above-mentioned purpose of the application, the technical scheme adopted is as follows:

[0008] A traffic safety public opinion analysis method based on SQ-LDA topic model, comprising the following steps: S1, obtaining traffic safety social software public opinion data; S2, preprocessing traffic safety social software public opinion data; S3, using SQ-LDA topic model to extract feature topics; S4, visualizing traffic safety social software public opinion hotspots.

[0009] As a preferred technical scheme of the present application, the step S1 specifically comprises the following steps:

[0010] S1.1, searching for topics about "traffic safety" in the search box of the social software, and obtaining network links url corresponding to the content;

[0011] S1.2, using Python tool, introducing requests module and encapsulating network link url, requesting data Data from the server;

[0012] S1.3, encapsulating Data as json data, and defining as Data_json;

[0013] S1.4, obtain the public opinion content under each "traffic safety" topic in the Data_json data set;

[0014] S1.5, introduce the Beautiful Soup module, parse the scraped public opinion content into objects for processing, and a total of 10000 comment data about "traffic safety" are crawled;

[0015] S1.6, obtain the original public opinion data set Text.

[0016] As a preferred technical solution of the application: the step S2 specifically comprises the following steps:

[0017] S2.1, define the original public opinion data set Text, that is:

[0018] Text={text1, text2, …, text 10000}

[0019] Where text i is the i(i=1, 2, …, 10000) comment content data;

[0020] S2.2, judge whether the subscript i satisfies i≤10000; if i≤10000 is satisfied, enter S2.3; if i≤10000 is not satisfied, enter S2.5;

[0021] S2.3, data cleaning is performed on the original public opinion data set Text; read the text i data, use special symbols (such as #) to locate the effective comment position of the comment content data, and remove meaningless emoticons, names, beginning and ending punctuation marks, color words, websites, etc.;

[0022] S2.4, establish a stop list S, delete the invalid characters in the text i data, and enter S2.2;

[0023] S2.5, obtain the cleaned and reorganized public opinion data set CText.

[0024] As a preferred technical solution of the application: the step S3 specifically comprises the following steps:

[0025] S3.1, define the cleaned and reorganized public opinion data set CText, that is:

[0026] CText={ctext1, ctext2, …, ctext 9823}

[0027] Where, ctext i is the i(i=1, 2, …, 9823) comment content data after cleaning;

[0028] S3.2, Use the jieba word segmentation tool to segment ctext i Perform word segmentation;

[0029] S3.3, Define the keyword w in ctexti ij (1≤j≤m), where m is the number of keywords in each comment;

[0030] S3.4, Import the deactivation table S;

[0031] S3.5 Determine if the index i satisfies i≤10000; if it satisfies i≤10000, proceed to S3.6; if it does not satisfy i≤10000, proceed to S3.8.

[0032] S3.6 Determine if the index j satisfies j≤m; if j≤m, proceed to S3.7; if j≤m, proceed to S3.5.

[0033] S3.7, Determine the keyword w ij Is it a stop word? Return to S3.6; if the keyword w ij If it is a stop word, delete the keyword; otherwise, keep the keyword.

[0034] S3.8 Obtain the keyword table Key, and define Key = {key1, key2, ..., key...} 9789}, where key i Let key be the keyword group of the i-th comment data. i ={w i1 w i2 , ..., w im};

[0035] S3.9 Suppose there are k topics related to "traffic safety". t (t = 1, 2, ..., k);

[0036] S3.10, Loop through the keyword table Key, and assign each keyword w ij Assign a topic (Topic) with the same keywords (w) ij It can be assigned to different topics;

[0037] S3.11, Traverse the keyword phrase key i Define and calculate keyword w ij Not belonging to the topic t The probability P(w) ij |Topic t ) and Topic t In the corpus key iThe probability P in Topic t |key i );

[0038] S3.12. Let α and β be the prior parameters of the Dirichlet function;

[0039] S3.13, Define keyword group (key) i Chinese keyword w ij Topic t The probability P(w) ij |Topic t key i ),Right now:

[0040] P(w ij |Topic t key i ) = [P(Topic t |key i )+α]·[P(w ij |Topic t )+β];

[0041] S3.14. Using cosine similarity, find keywords w ij The 2c most similar keywords w i(j+h) This means that these (2c+1) keywords have similar themes;

[0042] S3.15, Define keyword w ij The 2c keywords most similar to it i(j+h) The similarity probability P(w) i(j(j+h) |w ij -c≤h≤c, h≠0;

[0043] S3.16. Utilize negative sampling based on sample quality to make the probability P(w) i(j+h) |w ij Minimize, that is:

[0044]

[0045] Among them, Ew g ~P n (w) represents the expected value of a word not appearing in the context of a Huffman tree, where G is the number of noisy keywords. g Noise is a key word. For noise keywords w g The vector representation of vw ij For the keyword w ij Vector representation of;

[0046] S3.17, define the average log probability function APF, that is:

[0047]

[0048] wherein N=9789;

[0049] S3.18, maximize the log probability function APF;

[0050] S3.19, distinguish the keyword w ij from G noise keywords w g , and increase the P(w i(j+h) |w ij ) probability value;

[0051] S3.20, judge whether the iteration number s satisfies s≥120, if s≥120 is satisfied, enter S3.21; if s≥120 is not satisfied, enter S3.11, and update the iteration number, that is, s=s+1;

[0052] S3.21, merge the subject keywords of the optimized similar keywords;

[0053] S3.22, find the optimal subject number E, and find the keyword w t corresponding to the frequency probability P(w ij |Topic ij , key t ) under each subject Topic i .

[0054] As a preferred technical solution of the present application: the step S4 specifically comprises the following steps:

[0055] S4.1, build a web system of Django framework;

[0056] S4.2, use ECharts and pyLDAvis to visualize the hot topics of traffic safety public opinion;

[0057] S4.3, analyze the keyword corresponding frequency probability P(w ij |Topic t , key i ) (0<P(w ij |Topic t , key i )<1) under different hot topics;

[0058] S4.4, set the frequency probability threshold of the keyword as 0.5; when the keyword corresponding frequency probability P(w ij |Topic t , key iWhen the value of the frequency probability of the keyword is greater than 0.5, the system carries out public opinion early warning.

[0059] The traffic safety public opinion analysis method based on the SQ-LDA topic model has the following technical effects compared with the prior art:

[0060] (1) The SQ-LDA topic model is proposed based on the traditional LDA topic model, and a new sample quality negative sampling method is proposed, which can filter low-quality negative samples that have little effect on model optimization, and then use the remaining samples to train the model. At the same time, the model also considers the cosine similarity, clusters the high-similarity keywords, and can quickly merge the hotspots of traffic safety public opinion, further reducing the computational cost. Compared with the LDA topic model based on random negative sampling, the sample quality negative sampling method in the SQ-LDA topic model calculates the sample information according to the gradient to distinguish high-quality and low-quality negative samples, filters out low-quality negative samples, maximizes the positive sample probability and minimizes the high-quality negative sample probability, and further effectively mines and analyzes the traffic safety social software public opinion data.

[0061] (2) The frequency probability threshold of the keyword under the theme is set, when the frequency probability of the keyword exceeds the set frequency probability threshold, the keyword public opinion early warning under the theme will be triggered, and necessary technical support is provided for social public opinion early warning. BRIEF DESCRIPTION OF DRAWINGS

[0062] Figure 1 A flowchart of the traffic safety public opinion analysis method based on the SQ-LDA topic model is provided for the present application;

[0063] Figure 2 A flowchart of the traffic safety public opinion analysis method based on the SQ-LDA topic model is provided for the present application;

[0064] Figure 3 A flowchart of the traffic safety public opinion analysis method based on the SQ-LDA topic model is provided for the present application;

[0065] Figure 4 A flowchart of the traffic safety public opinion analysis method based on the SQ-LDA topic model is provided for the present application;

[0066] Figure 5 A flowchart of the traffic safety public opinion analysis method based on the SQ-LDA topic model is provided for the present application. DETAILED DESCRIPTION

[0067] The present invention will be further explained in detail below with reference to the accompanying drawings, so that those skilled in the art can better understand and implement the present invention. However, the following examples are only used to explain the present invention and are not intended to limit the present invention.

[0068] In specific implementation, such as Figure 1 As shown, a traffic safety public opinion analysis method based on the SQ-LDA topic model includes the following steps: S1: acquiring traffic safety Weibo public opinion data; S2: preprocessing traffic safety Weibo public opinion data; S3: extracting feature topics using the SQ-LDA topic model; S4: visualizing traffic safety Weibo public opinion hotspots.

[0069] like Figure 2 As shown, step S1 specifically includes the following steps:

[0070] S1.1 Search for the topic "traffic safety" in the Weibo search box and obtain the corresponding web link URL;

[0071] S1.2 Using Python tools, import the requests module and encapsulate the network link URL to request data from the server;

[0072] S1.3. Encapsulate the Data as JSON data and define it as Data_json;

[0073] S1.4. Retrieve the public opinion content under each "Traffic Safety" topic in the Data_json dataset;

[0074] S1.5. The BeautifulSoup module was introduced to parse the crawled public opinion content into objects for processing. A total of 10,000 comments on "traffic safety" were crawled.

[0075] S1.6 Obtain the original public opinion dataset Text.

[0076] like Figure 3 As shown, step S2 specifically includes the following steps:

[0077] S2.1 Define the original public opinion dataset Text, namely:

[0078] Text={text1, text2,…, text 10000}

[0079] text i This represents the content data of the i-th comment (i = 1, 2, ..., 10000);

[0080] S2.2, judge whether subscript i satisfies i≤10000; if i≤10000 is satisfied, enter S2.3; if i≤10000 is not satisfied, enter S2.5;

[0081] S2.3, data cleaning is performed on the original public opinion data set Text; read text i data, and the effective comment position of the comment content data is located by using a special symbol (such as #), meaningless emoticons, names, head and tail punctuation marks, color words, website addresses and the like are removed;

[0082] S2.4, a stop list S is established, and invalid characters in the text i data are deleted, and S2.2 is entered;

[0083] S2.5, a reorganized public opinion data set CText after cleaning is obtained.

[0084] As Figure 4 shown, step S3 specifically includes the following steps:

[0085] S3.1, the reorganized public opinion data set CText after cleaning is defined, that is:

[0086] CText={ctext1,ctext2,…,ctext 9823}

[0087] Wherein, ctext i is the i-th (i=1, 2, …, 9823) comment content data after cleaning;

[0088] S3.2, ctext i is segmented by using a jieba segmentation tool;

[0089] S3.3, the keywords w ij (1≤j≤m) in ctexti are defined, and m is the number of keywords of each comment data;

[0090] S3.4, the stop list S is imported;

[0091] S3.5, judge whether subscript i satisfies i≤10000; if i≤10000 is satisfied, enter S3.6; if i≤10000 is not satisfied, enter S3.8;

[0092] S3.6, judge whether subscript j satisfies j≤m; if j≤m is satisfied, enter S3.7; if j≤m is not satisfied, enter S3.5;

[0093] S3.7, judge whether the keyword w ij is a stop word, and return to S3.6; if the keyword w ijIf it is a stop word, delete the keyword; otherwise, keep the keyword.

[0094] S3.8 Obtain the keyword table Key, and define Key = {key1, key2, ..., key...} 9789}, where key i Let key be the keyword group of the i-th comment data. i ={w i1 w i2 , ..., w im};

[0095] S3.9 Suppose there are k topics related to "traffic safety". t (t = 1, 2, ..., k);

[0096] S3.10, Loop through the keyword table Key, and assign each keyword w ij Assign a topic (Topic) with the same keywords (w) ij It can be assigned to different topics;

[0097] S3.11, Traverse the keyword phrase key i Define and calculate keyword w ij The probability P(w) that does not belong to the Topic ij |Topic t ) and Topic t In the corpus key i The probability P in Topic t |key i );

[0098] S3.12. Let α and β be the prior parameters of the Dirichlet function;

[0099] S3.13, Define keyword group (key) i Chinese keyword w ij Topic t The probability P(w) ij |Topic t key i ),Right now:

[0100] P(w ij |Topic t key i ) = [P(Topic t |key i )+α]·[P(w ij |Topic t )+β];

[0101] S3.14, find the 2c most similar keywords w ij to the keyword w i(j+h) , i.e. the (2c+1) keywords are similar in topic;

[0102] S3.15, define the keyword w ij and its 2c most similar keywords w i(j+h) , and the similarity probability P(w i(j+h) |w ij ), -c≤h≤c, h≠0;

[0103] S3.16, minimize the probability P(w i(j+h) |w ij ) by negative sampling of the sample quality, i.e.

[0104]

[0105] where Ew g ~P n (w) is the expected value of the context not appearing in the Huffman tree, G is the number of noise keywords, w g is the noise keyword, is the vector representation of the noise keyword w g , and w ij is the vector representation of the keyword w ;

[0106] S3.17, define the average log probability function APF, i.e.

[0107]

[0108] where N=9789;

[0109] S3.18, maximize the log probability function APF;

[0110] S3.19, distinguish the keyword w ij from the G noise keywords w g , and increase the P(w i(j+h) |w ij ) probability value;

[0111] S3.20, determine whether the iteration number s satisfies s≥120, if s≥120, go to S3.21; if s≥120 is not satisfied, go to S3.11, and update the iteration number, i.e. s=s+1;

[0112] S3.21, merge the optimized similar keywords in topic;

[0113] S3.22, find the optimal number of topics E, and the keywords in each topic Topict Below, keyword w ij The corresponding frequency probability P(w) ij |Topic t key i ).

[0114] like Figure 5 As shown, step S4 specifically includes the following steps:

[0115] S4.1 Build a web system using the Django framework;

[0116] S4.2 Utilize ECharts and pyLDAvis to visualize hot topics in public opinion regarding traffic safety;

[0117] S4.3 Analyze the frequency probability P(w) of keywords under different hot topics. ij |Topic t key i )(0<P(w ij |Topic t key i <1);

[0118] S4.4, Set the frequency probability threshold for keywords to 0.5; when the frequency probability P(w) corresponding to a certain keyword is 0.5, ij |Topic t key i When the value is greater than 0.5, the system will issue a public opinion warning.

[0119] In the specific implementation, the symbols are explained in Table 1:

[0120] Table 1. Symbol Explanation

[0121]

[0122]

[0123] The application provides a traffic safety public opinion analysis method based on an SQ-LDA topic model. On the basis of the traditional LDA topic model, a new sample quality negative sampling method is proposed, which can filter low-quality negative samples that have little effect on model optimization, and then use the remaining samples to train the model.

[0124] The application sets a keyword frequency probability threshold under a topic, and when the keyword frequency probability exceeds the set keyword frequency probability threshold, the keyword public opinion early warning under the topic is triggered, thereby providing necessary technical support for social public opinion early warning.

[0125] The above specific embodiments further illustrate the purpose, technical solutions and advantages of the application, and it should be understood that the above description is only a specific embodiment of the application, and is not intended to limit the scope of the application. Any equivalent changes and modifications made by those skilled in the art without departing from the concept and principles of the application shall fall within the scope of the application.

Claims

1.A traffic safety public opinion analysis method based on an SQ-LDA topic model, characterized in that, The method comprises the following steps: S1, obtaining traffic safety social software public opinion data; S2, preprocessing traffic safety social software public opinion data; S3, extracting feature topics by using an SQ-LDA topic model; The step S3 specifically comprises the following steps: S3.1, define the cleaned recombined public opinion data set That is: ； wherein, the first day after cleaning comment content data; S3.2, using the jieba word segmentation tool to perform word segmentation on ; S3.3, Definition the keywords in the , is the number of keywords for each review data; S3.4, importing a stop list S; S3.5, determining the subscript whether the condition is satisfied if the condition is satisfied then go to S3.6; if the condition is not satisfied then go to S3.8; S3.6, determining the index whether the condition is satisfied if the condition is satisfied proceed to S3.7; if the condition is not satisfied proceed to S3.5; S3.7, Determine keywords Is it a stop word? Return to S3.6; if keyword If it is a stop word, delete the keyword; otherwise, keep the keyword. S3.8, obtaining a keyword list , defining wherein is a keyword group of the th comment data, and ; S3.9, assuming there are topics about "Traffic safety" ; S3.10, loop through keyword list , assign a topic to each keyword , same keyword can be assigned to different topics , same keyword can be assigned to different topics; S3.11, Traversing Keyword Phrase Define and calculate keywords Not related to the topic probability and theme In the corpus The probability of ; S3.12, record prior parameters that are Dirichlet functions; S3.13, define key phrase in the middle key word belongs to the subject probability i.e.: ； S3.14, find the 2c keywords most similar to the keyword using cosine similarity S3.15, find the 2c keywords most similar to the keyword using cosine similarity i.e. the topics of these (2c+1) keywords are similar; S3.15, define keywords 2c keywords most similar to it probability of similarity ; S3.16, with negative sampling of sample qualities, the probability is minimized, i.e.: ； wherein, represents the expected value of a certain word not appearing in the context of the Huffman tree, is the number of noise keywords, is a noise keyword, is a noise keyword is a vector representation of, is a vector representation of the keyword is a vector representation of the keyword S3.17, defining the average log probability function i.e.: ； Wherein, N=9789; S3.18, maximizing the log-probability function ; S3.19, distinguishing keywords with one noise keyword and cause the probability value to increase; S3.20, judge the iteration number whether the condition is satisfied if the condition is satisfied go to S3.21; if the condition is not satisfied go to S3.11 and update the iteration number, i.e. ; S3.21, merging and optimizing the topic words of similar keywords; S3.22, find the optimal number of topics , and under each topic , the keywords corresponding frequency probability ; S4, visualizing traffic safety social software public opinion hotspots. 2.The traffic safety public opinion analysis method based on the SQ-LDA topic model according to claim 1, characterized in that, The step S1 specifically comprises the following steps: S1.1, searching for topics about "traffic safety" in a social software search box, obtaining network links url of corresponding contents; S1.2, using Python tools, importing the requests module and encapsulating the network link url, requesting data Data from the server; S1.3, encapsulating Data as json data and defining it as Data_json; S1.4, obtaining public opinion contents under each "traffic safety" topic in the Data_json data set; S1.5, importing the BeautifulSoup module, parsing the scraped public opinion contents into objects for processing, a total of 10000 pieces of comment data about "traffic safety" are crawled; S1.6, obtaining the original public opinion data set Text. 3.The traffic safety public opinion analysis method based on SQ-LDA topic model according to claim 1, characterized in that, The step S2 specifically comprises the following steps: S2.1, define the original public opinion data set That is: ； wherein is a first comment content data; S2.2, judging the subscript whether the condition is satisfied ; if the condition is satisfied , go to S2.3; if the condition is not satisfied , go to S2.5; S2.3, on the original public opinion data set Data cleaning; read data, use special symbols to locate the effective comment position of the comment content data, remove meaningless emoticons, names, beginning and end punctuation marks, color words, and website addresses; S2.4, build the inactivation table S, delete invalid words in the data and go to S2.2; S2.5, obtaining the cleaned recombined public opinion dataset . 4.The traffic safety public opinion analysis method based on the SQ-LDA topic model of claim 1, wherein, The step S4 specifically comprises the following steps: S4.1, building a web system of Django framework; S4.2, using ECharts and pyLDAvis to visualize the hot topic of traffic safety public opinion; S4.3, analyze the frequency probability of keywords corresponding to different hot topics ( ); S4.4, set the keyword frequency probability threshold to 0.5; when the relative frequency probability of a keyword is greater than 0.5, the system carries out public opinion early warning. when the relative frequency probability of a keyword is greater than 0.5, the system carries out public opinion early warning.

Images