A Spatial Gene Identification and Extraction Method Based on Social Media Text Data

Abstract

This invention discloses a method for spatial gene identification and extraction based on social media text data, comprising the following steps: collecting online text data about a city, then preprocessing the data to obtain dataset D1; constructing a dictionary and vector space in analysis software, introducing an LDA topic model, and classifying the obtained dataset D1 into topics; merging synonyms in each topic, and performing synonym replacement in dataset D1 to obtain dataset D2; counting the co-occurrence frequency of keywords in dataset D2 and constructing a co-occurrence matrix M; and using a hierarchical clustering model to cluster the semantic network analysis results to obtain spatial combination patterns, i.e., spatial genes. This invention collects online text data about a specific city from multiple social media platforms, providing a practical technical means for urban researchers to identify urban spatial genes by obtaining rich, non-invasive data.

Description

Technical Field

[0001] This invention belongs to the field of urban planning technology, specifically relating to a spatial gene identification and extraction method based on social media text data. Background Technology

[0002] Since the beginning of the 21st century, rapid urban expansion has led to increasingly homogenized physical spatial forms, making the exploration of urban characteristics and the inheritance of cultural heritage a challenging task. The concept of "spatial genes" was proposed to address this issue. It shifts the research perspective from the "built form" of physical space to its underlying "deep structure." It argues that the key to achieving cultural heritage inheritance in urban development is not the protection or replication of historical forms and symbols themselves, but rather the continuation of the regional combination patterns of spatial elements and their inherent generative mechanisms. Only by forming methods of morphological organization and place creation can a win-win situation be achieved for the inheritance of historical heritage and urban development.

[0003] Currently, spatial gene identification and extraction based on public perception typically employs traditional sociological methods such as questionnaires and interviews. However, limitations in data collection make this step overly reliant on the abilities and experience of planners and designers, making it susceptible to subjective influences and prone to misunderstandings of public perception in practice. With the continuous advancement of technological globalization, the emergence of social media has broken down geographical and temporal barriers, lowering the threshold for public expression and significantly expanding the channels for collecting public perception information. Compared to traditional sociological methods, methods based on social media text to obtain public perception data effectively improve the efficiency and authenticity of information collection: on the one hand, traditional surveys such as distributing questionnaires or conducting interviews in the field have low information acquisition efficiency and require significant upfront investment of human and material resources, while social media text data is easier to obtain and offers a richer sample size; on the other hand, survey questionnaires with pre-set question options may miss detailed and in-depth information, resulting in data lacking real flexibility, while through social media text data, researchers can obtain a large amount of authentic, non-intrusive sample data, thereby better identifying spatial genes. Summary of the Invention

[0004] In view of the shortcomings of the prior art, the purpose of this invention is to provide a spatial gene identification and extraction method based on social media text data, so as to solve the problems mentioned in the background art.

[0005] The objective of this invention can be achieved through the following technical solutions:

[0006] A spatial gene identification and extraction method based on social media text data includes the following steps:

[0007] Step 1: Collect online text data about the city, and then preprocess the data to obtain dataset D1;

[0008] Step 2: Construct a dictionary and vector space in the analysis software, introduce the LDA topic model, and perform topic classification on the obtained dataset D1 to obtain k topic classes, denoted as Z1, Z2...Z k From these, select themes related to spatial elements, and use the words contained in these themes as keywords, denoted as keyword sets C1, C2...C1. n , where k≥n;

[0009] Step 3: In the keyword set C1, C2...C n In dataset D1, synonym replacement is performed simultaneously. The resulting set of keywords is denoted as keyword set C', and the keywords contained therein are denoted as a1, a2, ..., a p The resulting dataset is denoted as D2;

[0010] Step 4: Count the keywords a1, a2...a in dataset D2. p Count the number of co-occurrences of each pair of words and construct a co-occurrence matrix M;

[0011] Step 5: Using a hierarchical clustering model, cluster the semantic network analysis results to obtain spatial combination patterns, i.e., spatial genes.

[0012] Preferably, the preprocessing operations performed on the network text data in step 1 include filtering invalid characters using regular expressions, sentence segmentation, removal of stop words, and marking of reserved words.

[0013] Preferably, in step 2, the LDA model uses a bag-of-words model to treat each text as a word frequency vector, transforming textual information into easily modelable numerical information. The word size is defined as L, and an L-dimensional vector (1,0,0,…,0,0) represents a word. A comment consisting of N words is denoted as d = (W1, W2,…W…). N The text dataset D consists of q comments, denoted as D = (d1, d2, ..., dq). q ), where q comments are distributed across k topics, and let Z denote the text topics. i (i = 1, 2, ..., k), let α and β be the prior parameters of the Dirichlet function, θ be the parameters of the multinomial distribution of topics in the document, which follows a Dirichlet prior distribution with hyperparameter α, φ be the parameters of the multinomial distribution of words in the topic, which follows a Dirichlet prior distribution with hyperparameter β, and z be the hidden variable.

[0014] Preferably, the mixing ratio of text in the LDA model follows a multinomial distribution, as shown in the following formula:

[0015] z|θ=Multinomial(θ) (1)

[0016] The word mixing ratio in the thematic vocabulary also follows a multinomial distribution, as shown in the following formula:

[0017] w|z,φ=Multinomial(φ) (2)

[0018] In text d j Word w generated under conditions i The probability is expressed as follows:

[0019]

[0020] In the formula, P(w) i |z=s) represents the word w i The probability of belonging to the s-th topic is P(z=s|d). j ) indicates that the s-th topic is in text d j The probability of it.

[0021] Preferably, the topic extraction process of the LDA model in step 2 includes:

[0022] Input the number of topics k and the number of comments q, then initialize the parameters α and β;

[0023] Start the EM algorithm iterative loop, initialize all variational parameters θ, φ, z, and perform the expectation step (E step) iterative loop of LDA until the variational parameters θ, φ, z converge to obtain the optimal variational parameters;

[0024] Perform the maximum step (M step) of LDA, and iterate the cyclic model under the condition of optimal variational parameters until α and β converge.

[0025] The document topic distribution θ is obtained by analyzing the topics of each word in each text within the statistical data set. q The distribution φ of topics and words in LDA was obtained. k It is then output as a visual format.

[0026] Preferably, among all the said themes, themes Z1, Z2...Z1 related to spatial elements are printed. k Let the keywords contained within them be denoted as keyword sets C1, C2, ..., C6. n , where k≥n.

[0027] Preferably, step 4, the co-word matrix construction process, includes:

[0028] For each text in dataset D2, construct a two-dimensional array formatted_data;

[0029] Let the keyword set C' be denoted as word_data;

[0030] Construct a two-dimensional matrix matrix with a size of (number of keywords + 1) × (number of keywords + 1), i.e., a co-occurrence matrix, and use all keywords as the first row and first column of the co-occurrence matrix;

[0031] Set the diagonal of the matrix to 0;

[0032] Iterate through formatted_data, combining the extracted row keywords and column keywords. If a set of keywords co-occurs, increment the corresponding matrix by 1, and finally obtain the co-occurrence matrix M.

[0033] Preferably, the process of obtaining the space gene in step 5 includes:

[0034] The keywords are vectorized, and the distance L(x,y) between each word vector is as follows:

[0035] L(x,y)=(x[1]-u[1]) 2 +(x[1]-y[2]) 2 +…+(x[n]-y[n]) 2 (4)

[0036] Create n classes, each containing only one keyword;

[0037] Calculate the distance between any two clusters, and merge the two closest or most similar clusters to create a new cluster;

[0038] Repeat the above steps until you get clustering results of the appropriate categories. Each clustering result is a stable spatial combination pattern that is locally representative and forms a broad consensus, which is called spatial gene.

[0039] The beneficial effects of this invention are:

[0040] The method of this invention collects online text data about a research city from multiple social media platforms. By obtaining a large sample size of non-intrusive data, it provides urban researchers with a practical technical means to identify the spatial genes of cities and truly and fully reflects the public's perception of urban space. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a flowchart of the method in an embodiment of the present invention;

[0043] Figure 2 It is the visualization diagram of topic distribution in the embodiments of the present invention;

[0044] Figure 3 It is the visualization result diagram of Topic 1 obtained in the embodiments of the present invention. Detailed implementation manners

[0045] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0046] Please refer to Figure 1 As shown, this embodiment provides a method for identifying and extracting spatial genes based on social media text data, including the following steps:

[0047] Step 1: Collect network text data about a certain research city from multi-source social media platforms, preprocess the data, and obtain a data set D1;

[0048] The preprocessing operations performed on the network text data include filtering invalid characters through regular expressions, sentence tokenization, and removing stop words, and marking the retained words. The invalid characters include emoticons, punctuation marks, etc. The stop words, for example: um, yeah, we, where, etc. The retained words include proper place names, scenic spot names, hotel names, etc.

[0049] Step 2: Build a dictionary and a vector space in an analysis software, introduce the LDA topic model, classify the obtained data set D1 into topics, and obtain k categories of topics, denoted as Z1, Z2... Z k , select the topics related to spatial elements from them, and use the words included in the above topics as keywords, denoted as keyword sets C1, C2... C n , where k≥n; [[ID=3?]]

[0050] Build a dictionary and a vector space in an analysis software, (using the Scikit-learn library of Python) build an LDA topic model, set the number of topics, and output the results in a visualization format;

[0051] Specifically as follows:

[0052] Step 2.1: Build a dictionary and a vector space in an analysis software, (using the Scikit-learn library of Python) build an LDA topic model;

[0053] Detailed as follows:

[0054] The LDA model uses a bag-of-words (BOW) model to treat each text as a word frequency vector, thus transforming textual information into numerical information that is easy to model. In the model, the word size is defined as L, and an L-dimensional vector (1,0,0,…,0,0) represents a word. A comment consisting of N words is denoted as d = (W1, W2,…W…). N Suppose a text dataset D for a certain city consists of q comments (usually one text per line), denoted as D = (d1, d2, ..., dq). q The q comments are distributed across k topics, denoted as Z. i (i = 1, 2, ..., k). Let α and β be the prior parameters of the Dirichlet function, θ be the parameters of the multinomial distribution of topics in the document, which follows a Dirichlet prior distribution with hyperparameter α, φ be the parameters of the multinomial distribution of words in the topic, which follows a Dirichlet prior distribution with hyperparameter β, and z be the hidden variable.

[0055] The LDA model assumes that each text is a random mixture of various topics in a certain proportion, and the mixing proportion follows a multinomial distribution, denoted by equation (1):

[0056] z|θ=Multinomial(θ) (1)

[0057] Each theme is composed of words from the vocabulary in a certain proportion, and the proportion of the mixture also follows a multinomial distribution, denoted as Equation (2):

[0058] w|z,φ=Multinomial(φ) (2)

[0059] In text d j Word w generated under conditions i The probability is expressed as equation (3):

[0060]

[0061] In the formula, P(w) i |z=s) represents the word w i The probability of belonging to the s-th topic is P(z=s|d). j ) indicates that the s-th topic is in text d j The probability of it.

[0062] Step 2.2: Extract topics from the LDA model using the EM algorithm. In each round, the text in the acquired text dataset D is traversed only once. After multiple rounds of iteration on all texts, the document-topic distribution θ is obtained. q and topic-term distribution φ k ;

[0063] Details are as follows:

[0064] Input the number of topics k and the number of comments q, then initialize the parameters α and β;

[0065] Start the EM algorithm iterative loop, initialize all variational parameters θ, φ, z, and perform the expectation step (E step) iterative loop of LDA until the variational parameters θ, φ, z converge to obtain the optimal variational parameters;

[0066] Perform the maximum step (M step) of LDA, and iterate the cyclic model under the condition of optimal variational parameters until α and β converge.

[0067] The document topic distribution θ is obtained by analyzing the topics of each word in each text within the statistical data set. q The distribution φ of topics and words in LDA was obtained. k It is then output as a visual format.

[0068] Step 2.3: Print the topics Z1, Z2...Z1 related to spatial genes (spatial elements) from all topics. k Let the keywords contained within them be denoted as keyword sets C1, C2, ..., C6. n , where k≥n.

[0069] Step 3: In the keyword set C1, C2...C n In dataset D1, synonym replacement is performed simultaneously. The resulting set of keywords is denoted as keyword set C', and the keywords contained therein are denoted as a1, a2, ..., a p The resulting dataset is denoted as D2;

[0070] Step 4: Count the keywords a1, a2...a in dataset D2. p Count the number of co-occurrences of each pair of words and construct a co-occurrence matrix M;

[0071] Details are as follows:

[0072] For each text in dataset D2, construct a two-dimensional array formatted_data;

[0073] Let the keyword set C' be denoted as word_data;

[0074] Construct a two-dimensional matrix `matrix` with a size of (number of keywords + 1) × (number of keywords + 1), i.e., a co-occurrence matrix. Then, use all keywords as the first row and first column of the co-occurrence matrix.

[0075] Set the diagonal of the matrix to 0;

[0076] Iterate through formatted_data, combining the extracted row keywords and column keywords. If a set of keywords co-occurs, increment the corresponding matrix by 1, and finally obtain the co-occurrence matrix M.

[0077] Step 5: Use the hierarchical clustering model to cluster the semantic network analysis results to obtain the spatial combination pattern.

[0078] Details are as follows:

[0079] Vectorize the keywords, and the distance L(x,y) between each word vector is as shown in Equation (4):

[0080] L(x,y) = (x[1] - y[1]) 2 + (x[1] - y[2]) 2 + … + (x[n] - y[n]) 2 (4)

[0081] Construct n classes, and each class contains only one keyword;

[0082] Calculate the distance between every two clusters, and merge the two clusters with the closest or most similar distance to construct a new class;

[0083] Repeat the above steps until the clustering results of the appropriate class are obtained. Each set of clustering results is a set of spatial combination patterns that stably exist, have local representativeness, and form a broad consensus, which is the spatial gene.

[0084] For a clearer understanding of the technical features, objectives, and effects of the present invention, the microblog text data is taken as an example.

[0085] 1. Collect the network text data about a certain research city from the microblog, and preprocess the data to obtain the data set D1.

[0086] The preprocessing operations performed on the network text data include filtering invalid characters through regular expressions, sentence tokenization, removing stop words, and marking the reserved words.

[0087] The invalid characters include emoji, punctuation marks, etc. The stop words, for example: um, yeah, we, where, etc. The reserved words include proper place names, scenic spot names, hotel names, etc. For the embodiment, the reserved words added include words such as "Penglai Fairy Island", "Yellow Sea and Bohai Sea Demarcation Coordinates", and "Hometown of Qi Jiguang". The processed microblog text data D1 (partial example) is shown in Table 1.

[0088] Table 1

[0089]

[0090] 2. Introduce the LDA topic model to classify the data set D1 into k topics, denoted as Z1, Z2... Z kSelect themes related to spatial elements from these themes, and denote the words contained within these themes as keywords, denoted as keyword sets C1, C2...C1. n , where k≥n.

[0091] The LDA topic model was constructed using the Scikit-learn library, with 3 input topics. Three topic classes were obtained, denoted as Z1, Z2, and Z3. The top 50 words from the two spatial element-related topics Z1 and Z3 were selected as keywords, denoted as C1 and C3, respectively. The resulting topic-classified dataset is shown in Table 2, and the distribution of the three topics is as follows: Figure 2 As shown, the visualization results for topic Z1 are as follows: Figure 3 As shown.

[0092] Table 2

[0093]

[0094]

[0095] 3. Merge synonyms in the keyword sets of C1 and C3, merge and update the keyword set, denoted as C', and perform synonym replacement in dataset D1 as shown in Table 3 to obtain dataset D2.

[0096] Table 3

[0097]

[0098] 4. In dataset D2, count the number of co-occurrences of each keyword in keyword set C', and construct a co-occurrence matrix M;

[0099] For each text in dataset D2, construct a two-dimensional array formatted_data;

[0100] Let the keyword set C' be denoted as word_data;

[0101] Construct a two-dimensional matrix matrix with a size of (number of keywords + 1) × (number of keywords + 1), i.e., a co-occurrence matrix, and use all keywords as the first row and first column of the co-occurrence matrix;

[0102] Set the diagonal of the matrix to 0;

[0103] Iterate through formatted_data, combining the extracted row keywords and column keywords. If a set of keywords co-occurs, increment the corresponding matrix by 1. Finally, obtain the co-occurrence matrix M, as shown in Table 4.

[0104] Table 4

[0105]

[0106]

[0107] 5. Using a hierarchical clustering model, cluster analysis was performed on the semantic network analysis results to obtain the spatial genes of Penglai. The clustering results of this example are shown in Table 5:

[0108] Table 5

[0109]

[0110]

[0111] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0112] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0113] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0114] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0115] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0116] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for spatial gene identification and extraction based on social media text data, characterized in that, Includes the following steps: Step 1: Collect online text data about the city, and then preprocess the data to obtain dataset D1; Step 2: Construct a dictionary and vector space in the analysis software, introduce the LDA topic model, and perform topic classification on the obtained dataset D1 to obtain k topic classes, denoted as Z1, Z2...Z k From these, select themes related to spatial elements, and use the words contained in these themes as keywords, denoted as keyword sets C1, C2...C1. n , where k≥n; Step 3: In the keyword set C1, C2...C n In dataset D1, synonym replacement is performed simultaneously. The set of keywords after replacement is denoted as keyword set C', and the keywords contained in it are denoted as a1, a2, ..., a p The resulting dataset is denoted as D2; Step 4: Count the keywords a1, a2...a in dataset D2. p Count the number of co-occurrences of each pair of words and construct a co-occurrence matrix M; Step 5: Using a hierarchical clustering model, cluster the semantic network analysis results to obtain spatial combination patterns, i.e., spatial genes; Step 4, the co-word matrix construction process, includes: For each text in dataset D2, construct a two-dimensional array formatted_data; Let the keyword set C' be denoted as word_data; Construct a two-dimensional matrix matrix with a size of (number of keywords + 1) × (number of keywords + 1), i.e., a co-occurrence matrix, and use all keywords as the first row and first column of the co-occurrence matrix; Set the diagonal of the matrix to 0; Iterate through formatted_data, combining the extracted row keywords and column keywords. If a set of keywords co-occurs, increment the corresponding matrix by 1, and finally obtain the co-occurrence matrix M. The process of obtaining the space gene in step 5 includes: The keywords are vectorized, and the distance L(x,y) between each word vector is as follows: L(x,y)=(x[1]-y[1]) 2 +(x[1]-y[2]) 2 +…+(x[n]-y[n]) 2 (4) Create n classes, each containing only one keyword; Calculate the distance between any two clusters, and merge the two closest or most similar clusters to create a new cluster; Repeat the above steps until you get clustering results of the appropriate categories. Each clustering result is a stable spatial combination pattern that is locally representative and forms a broad consensus, which is called spatial gene.

2. The spatial gene identification and extraction method based on social media text data according to claim 1, characterized in that, The preprocessing operations performed on the network text data in step 1 include filtering invalid characters using regular expressions, sentence segmentation, removal of stop words, and marking of reserved words.

3. The spatial gene identification and extraction method based on social media text data according to claim 1, characterized in that, In step 2, the LDA topic model uses a bag-of-words model to treat each text as a word frequency vector, transforming textual information into easily modelable numerical information. The word size is defined as L, and an L-dimensional vector (1,0,0,…,0,0) represents a word. A comment consisting of N words is denoted as d = (W1, W2,…W…). N The text dataset D consists of q comments, denoted as D = (d1, d2, ..., dq). q ), where q comments are distributed across k topics, and let Z denote the text topics. i (i = 1, 2, ..., k), let α and β be the prior parameters of the Dirichlet function, θ be the parameters of the multinomial distribution of topics in the document, which follows a Dirichlet prior distribution with hyperparameter α, φ be the parameters of the multinomial distribution of words in the topic, which follows a Dirichlet prior distribution with hyperparameter β, and z be the hidden variable.

4. The spatial gene identification and extraction method based on social media text data according to claim 3, characterized in that, The mixing ratio of text in the LDA topic model follows a multinomial distribution, as shown in the following formula: z|θ=Multinomial(θ) (1) The word mixing ratio in the thematic vocabulary also follows a multinomial distribution, as shown in the following formula: w|z,φ=Multinomial(φ) (2) In text d j Word w generated under conditions i The probability is expressed as follows: In the formula, P(w) i |z=s) represents the word w i The probability of belonging to the s-th topic is P(z=s|d). j ) indicates that the s-th topic is in text d j The probability of it.

5. The spatial gene identification and extraction method based on social media text data according to claim 4, characterized in that, The topic extraction process of the LDA topic model in step 2 includes: Input the number of topics k and the number of comments q, then initialize the parameters α and β; Start the EM algorithm iterative loop, initialize all variational parameters θ, φ, z, and perform the expectation step iterative loop of LDA until the variational parameters θ, φ, z converge to obtain the optimal variational parameters; Perform maximum steps of LDA, and iterate the cyclic model under the condition of optimal variational parameters until α and β converge; The document topic distribution θ is obtained by analyzing the topics of each word in each text within the statistical data set. q The distribution φ of topics and words in LDA was obtained. k It is then output as a visual format.

6. The spatial gene identification and extraction method based on social media text data according to claim 5, characterized in that, Print themes Z1, Z2...Z related to spatial elements among all the described themes. k Let the keywords contained within them be denoted as keyword sets C1, C2, ..., C6. n , where k≥n.

7. A computer-readable storage medium storing program code that, when executed by a processor, implements the method as claimed in any one of claims 1 to 6.

8. A computing device comprising a processor and a storage medium storing program code, said program code, when executed by the processor, implementing the method as claimed in any one of claims 1 to 6.

Images