A Method and System for Detecting Diverse Demands of Subway Passengers Based on BERTopic Model

Abstract

This invention discloses a method and system for detecting diverse passenger demands in subway systems based on the BERTopic model. First, the original passenger text is collected and segmented, then transformed into high-dimensional semantic vectors using an embedding model. Next, nonlinear dimensionality reduction is performed using the UMAP model, followed by density clustering of the low-dimensional vectors using the HDBSCAN algorithm to form topic clusters. Then, the original text undergoes Chinese preprocessing, including cleaning and word segmentation, to obtain standardized text. Based on the clustering results and standardized text, topic keywords are extracted for each cluster using CountVectorizer and c-TF-IDF, and the MMR method is used to optimize the keywords to improve their representativeness and diversity. Finally, a visual report is generated to output the analysis results. This invention organically combines deep semantic modeling with the characteristics of Chinese text processing. Through modular modeling and system integration of various functional areas, it can automatically and efficiently identify semantically consistent and interpretable diverse topics from massive amounts of passenger demands, providing accurate data support for subway operation decisions.

Description

Technical Field

[0001] This invention relates to the field of intelligent transportation information processing and natural language analysis technology, specifically to a method and system for detecting diverse demands of subway passengers based on the BERTopic model, which is used to improve the intelligent analysis and management level of subway operation services. Background Technology

[0002] With the acceleration of urbanization, subways have become a major mode of urban public transportation, and the surge in passenger flow has made issues related to passenger service experience and operational management increasingly prominent. Textual data from passengers through questionnaires, hotlines, and online comments provides crucial information for analyzing service quality. Traditional analysis methods rely on manual classification or word frequency statistical models based on LDA and NMF, which suffer from shallow semantic understanding and low clustering accuracy. In recent years, deep semantic models based on BERT and Sentence-BERT, as well as the HDBSCAN density clustering algorithm, have been introduced into the field of text topic analysis. Research shows that the BERTopic model significantly outperforms traditional LDA methods in terms of short text semantic modeling and clustering robustness.

[0003] While existing passenger complaint analysis methods based on topic models such as LDA and NMF offer some interpretability, they suffer from limitations including shallow semantic expression, poor modeling performance with short texts, and strong parameter dependence. These methods simplify text to a bag-of-words model, neglecting contextual semantics and easily leading to topic confusion; furthermore, they require manual setting of the number of topics, resulting in high subjectivity. In subway passenger complaint analysis, research has largely focused on satisfaction statistics and sentiment analysis, lacking an intelligent modeling system capable of automating large-scale text processing. Therefore, there is an urgent need to introduce an improved BERTopic model that combines a domain dictionary with an adaptive clustering mechanism to enhance the accuracy of passenger complaint identification and the intelligence level of subway service decision-making. Summary of the Invention

[0004] This invention addresses the aforementioned problems and aims to provide a method and system for detecting diverse passenger demands in subway systems based on the BERTopic model. The main objectives are: (1) A modular modeling method that combines deep semantic representation, dimensionality reduction clustering and topic optimization is provided to improve the accuracy and robustness of detecting the diverse demands of subway passengers; (2) Establish a semantic adaptation mechanism (custom user dictionary) and a custom stop word system for subway scenarios to improve the model’s ability to recognize words in the rail transit field; (3) Design an interpretable theme visualization module to facilitate subway operation managers to intuitively identify passenger concerns, facility defects and service shortcomings, thereby providing technical support for subway operation service quality optimization and emergency response.

[0005] This invention provides a method for detecting diverse demands of subway passengers based on the BERTopic model, characterized by the following steps: Step 1, Raw text data acquisition and source segmentation: Acquire raw text data from subway passengers and convert the raw text data into high-dimensional semantic vectors using an embedding model; Step 2, Nonlinear dimensionality reduction: Based on the high-dimensional semantic vectors, use the UMAP model to perform nonlinear dimensionality reduction on the high-dimensional semantic vectors to obtain low-dimensional vector representations; Step 3, Density clustering: Use the HDBSCAN density clustering algorithm to cluster the low-dimensional vectors to obtain clustering results containing at least one topic cluster; Step 4, Chinese preprocessing: Perform Chinese preprocessing on the raw text data to obtain standardized text, and the Chinese preprocessing package... The process includes: cleaning, regular expression normalization, stop word and user dictionary maintenance, Jieba word segmentation, word segmentation result caching and version management; Step 5, topic feature representation: based on clustering results and canonical text, a keyword weight matrix is ​​generated for each cluster. First, word frequency statistics are performed using CountVectorizer, and then topic representation is calculated based on c-TF-IDF to extract keywords representing the topic; Step 6, topic optimization and merging: for the extracted keywords, the MMR method is used to optimize the keywords, automatically selecting keywords with strong representativeness and semantic diversity to obtain an optimized set of topic keywords; Step 7, visualization and reporting: based on the optimized set of topic keywords, a topic analysis visualization report is generated and output.

[0006] The subway passenger multi-request detection method based on the BERTopic model provided in this invention may also have the following features: In step 1, requests from different sources are collected, desensitized, and labeled, and written into their respective queues and parameter files according to data source type. The process is as follows: Constructing a semantic embedding module: Calling Bert-Base-Chinese or equivalent Chinese Transformer to generate text semantic vectors, and completing local caching and vector index management. In the BERTopic model, using Embedding word embedding models such as Bert-Base-Chinese to transform passenger request text into high-dimensional semantic vectors, generating document representations through mean pooling or context encoding, and combining UMAP dimensionality reduction and HDBSCAN clustering to achieve semantically consistent topic recognition. The core principle formula is as follows: In the formula, A high-dimensional semantic vector representation of passenger appeal text; For the first in the text i One word; This indicates that the Bert-Based-Chinese model is used for context encoding of words; This represents the text length.

[0007] The subway passenger multi-demand detection method based on the BERTopic model provided in this invention may also have the following features: In step 2, parameterized interfaces n_components, min_dist, and random_state are provided, and low-dimensional embeddings are output for clustering, including the following steps: Step 2-1, dimensionality reduction: The high-dimensional semantic vector is reduced using the nonlinear dimensionality reduction model UMAP; Step 2-2, the above steps are achieved by setting at least one of the following parameters: n_components: Specifies the target dimension after dimensionality reduction, which is 2 or 5; min_dist: Controls the minimum distance between points after dimensionality reduction, the smaller the value, the denser the local structure; random_state: Ensures the reproducibility of the results. In step 2-1, the essence of the UMAP model is to minimize the cross-entropy difference between the high-dimensional and low-dimensional similarity distributions: This yields a low-dimensional representation that preserves the semantic neighborhood structure, providing a highly discriminative feature base for subsequent HDBSCAN clustering.

[0008] The subway passenger multi-demand detection method based on the BERTopic model provided by this invention may also have the following features: In step 3, cluster labels, noise point identifiers, or core point information are output, and the parameters are read from the default archive according to the data source. The process is as follows: Step 3-1, in the BERTopic model, the HDBSCAN density clustering algorithm is used to cluster the document vectors after UMAP dimensionality reduction, and the high-density regions in the semantic space are automatically identified as topic clusters through hierarchical density connection trees; Step 3-2, the above steps are achieved by setting at least one of the following parameters: min_cluster_size: defines the minimum number of documents in a cluster. The larger the value, the more macroscopic the topic; min_samples: controls the number of neighborhood samples of the core point, affecting the determination of noise points, and the output is the cluster label of the document. In step 3-1, the core idea of ​​HDBSCAN is based on reachability distance and minimum spanning tree hierarchical clustering. The most critical mathematical definition is as follows: In the formula, For the sample Euclidean distance in UMAP reduced space; :point The core distance is defined as its first core distance. The nearest neighbor samples to The distance; The "reachable distance" between two points is used to construct a weighted complete graph and generate a minimum spanning tree.

[0009] The subway passenger multi-demand detection method based on the BERTopic model provided in this invention may also have the following features: Step 4 consists of the following steps: Step 4-1, analyzing data characteristics and classifying data sources according to their origin and proportion; Step 4-2, performing data cleaning for different categories of data, including noise filtering, stop word processing, and retention of proper nouns; Step 4-3, using the Jieba word segmentation tool to implement basic word segmentation, with the cleaned and segmented text cached for direct use by the BERTopic model, thereby improving clustering efficiency and topic recognition accuracy. The mathematical formula is: In the formula, The original text sequence; The optimal word segmentation sequence is to be determined. Indicator The preceding word The conditional probability of its subsequent occurrence; The word segmentation scheme with the highest probability is the optimal word segmentation result.

[0010] The subway passenger multi-demand detection method based on the BERTopic model provided in this invention may also have the following features: In step 4-2, noise filtering involves removing irrelevant characters and fixed template content from the original text. Irrelevant characters include at least special symbols and garbled characters, and fixed template content includes at least serial numbers and data sources. Stop word processing involves comparing the stop word list with the custom stop words to comprehensively filter out meaningless words. The stop word list is the Harbin Institute of Technology stop word library, and the custom stop words are high-frequency words with low semantic value. Proper noun retention involves introducing the UserDict dictionary to forcibly retain proper nouns and fixed collocations in the rail transit industry, thereby avoiding semantic loss caused by excessive word segmentation.

[0011] The subway passenger multi-demand detection method based on the BERTopic model provided by this invention may also have the following features: In step 5, a keyword weight matrix is ​​generated for each cluster, and dynamic stop word addition is supported. The specific process is as follows: Step 5-1, in the clustering results, the word segmented text within each topic is statistically analyzed using CountVectorizer to obtain high-frequency words within the topic. Then, the topic representation is calculated based on c-TF-IDF. The text of the same cluster is regarded as a document. By measuring the distribution difference of words inside and outside the topic, the keywords that best represent the topic semantics are extracted to achieve high-quality topic representation; Step 5-2, the above steps are achieved by setting at least one of the following parameters: stop_words: remove stop words and filter out meaningless words in the text; min_topic_size or min_cluster_size: merge topics with fewer than this value; nr_topics: limit the final number of topics, which is achieved by merging similar topics. In the formula, For words On the topic The number of times it appears in; Theme The sum of the occurrences of all words within the range is used for word frequency normalization; The total number of all topics; For words containing The number of topics; To represent words On the topic Importance score.

[0012] The BERTopic-based method for detecting diverse demands of subway passengers provided in this invention may also have the following feature: In step 6, the topic representation optimization model uses the MMR method to optimize keywords and reduce redundant words. In the formula, A set of candidate keywords; The set of selected keywords; The central vector of the topic; Keywords Semantic similarity to the central theme; Keywords With selected keywords semantic similarity; These are the balancing parameters.

[0013] The method for detecting diverse demands of subway passengers based on the BERTopic model provided by this invention may also have the following features: In step 7, the topic analysis visualization report includes any one or more of the following: output topic proportion, two-dimensional distribution of topic spacing, and hierarchical clustering diagram.

[0014] This invention provides a method for detecting diverse demands of subway passengers based on the BERTopic model, characterized by the following features: Module 1, Topic Clustering Module: used to acquire raw text data from subway passengers, and convert the raw text data into high-dimensional semantic vectors using an embedding model; based on the high-dimensional semantic vectors, a UMAP model is used to perform nonlinear dimensionality reduction on the high-dimensional semantic vectors to obtain a low-dimensional vector representation; the HDBSCAN density clustering algorithm is used to cluster the low-dimensional vectors to obtain clustering results containing at least one topic cluster; Module 2, Word Segmentation Model: used to perform Chinese preprocessing on the raw text data to obtain standardized text; Chinese preprocessing includes cleaning, correcting... The system includes: Module 1, Word Frequency Statistics Model: This module generates a keyword weight matrix for each cluster based on clustering results and standardized text; it uses CountVectorizer for word frequency statistics, calculates topic representations based on c-TF-IDF, and extracts keywords representing the topic; it optimizes the extracted keywords using the MMR method, automatically selecting keywords with strong representativeness and semantic diversity to obtain an optimized set of topic keywords; Module 2, Subject Visualization: This module generates and outputs a topic analysis visualization report based on the optimized set of topic keywords. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the overall technical architecture of the subway passenger multi-demand detection method based on the BERTopic model in this embodiment of the invention; Figure 2 This is a schematic diagram of high-frequency words within the theme in an embodiment of the present invention; Figure 3 This is a visualization diagram of the subject spacing in an embodiment of the present invention; Figure 4 This is a hierarchical clustering diagram of the topic suggestions for the satisfaction survey in an embodiment of the present invention. Detailed Implementation

[0016] To make the technical means, creative features, objectives and effects of this invention easy to understand, the following embodiments, in conjunction with the accompanying drawings, specifically illustrate a method and system for detecting diverse demands of subway passengers based on the BERTopic model.

[0017] Example 1 This embodiment provides a method for detecting diverse demands of subway passengers based on the BERTopic model.

[0018] This embodiment provides an automatic detection technology for diverse passenger demands in subway systems based on BERTopic. Its core process is "semantic embedding → nonlinear dimensionality reduction → density clustering → topic representation / optimization → visualization and reporting." This four-stage technical path and theoretical basis can be supported by the modular description of BERTopic and the "embedding-dimensionality reduction-clustering-representation" framework found in the literature.

[0019] Unlike traditional topic models, this embodiment prioritizes the "semantic preservation principle" for Chinese preprocessing, performing only necessary cleaning, stop word filtering, and text normalization to avoid over-preprocessing that could damage contextual information, thus matching the input features of BERT-like embedding models.

[0020] In the dimensionality reduction stage, UMAP is preferred (PCA / Truncated SVD, etc. can also be used as alternatives). The recommended min_dist value for UMAP is 0.0 to significantly reduce outliers. In the clustering stage, HDBSCAN is preferred, and the cluster granularity and noise identification can be controlled by min_cluster_size and min_samples. In the topic representation stage, CountVectorizer + c-TF-IDF is used. In the topic optimization stage, MMR is introduced to improve keyword diversity and readability and support topic merging (nr_topics).

[0021] In response to the differences in data sources, this embodiment divides the data into "satisfaction survey suggestion type" (accounting for 96.97%) and "third-party review type", and sets different cleaning and parameter files for each type to obtain stable and interpretable subject results.

[0022] Figure 1 This is a schematic diagram of the overall technical architecture of the subway passenger multi-demand detection method based on the BERTopic model in this embodiment of the invention.

[0023] like Figure 1 As shown, the method for detecting diverse demands of subway passengers based on the BERTopic model involved in this embodiment includes the following steps: Step S1, Raw Text Data Acquisition and Source Segmentation: Acquire raw text data from subway passengers and transform it into high-dimensional semantic vectors using an embedding model. This step employs multi-source data access and source segmentation management, along with deep semantic embedding. During the text embedding stage, Chinese BERT / equivalent Chinese Transformer are used to process the raw text, generating dense sentence vectors (typically 768 dimensions), which are then cached for reuse. In Chinese scenarios, the embedding model is replaced with BERT-Base-Chinese, along with appropriate word segmentation / vectorization logic.

[0024] Specifically, in this embodiment, requests from different sources are collected, de-identified, and tagged, and written into their respective queues and parameter files according to data source type. The process is as follows: Construct a semantic embedding module: This module calls Bert-Base-Chinese or an equivalent Chinese Transformer to generate text semantic vectors and manages local caching and vector indexing. In the BERTopic model, BERT-Base-Chinese and other embedding word embedding models are used to transform passenger request text into high-dimensional semantic vectors. Document representations are generated through mean pooling or context encoding, and UMAP dimensionality reduction and HDBSCAN clustering are combined to achieve semantically consistent topic recognition. Semantic vectors are cached locally for fast retrieval, improving modeling and clustering efficiency. The core principle and formula are as follows: In the formula, A high-dimensional semantic vector representation of passenger appeal text; For the first in the text i One word; This indicates that the Bert-Based-Chinese model is used for context encoding of words; This represents the text length.

[0025] Step S2, Nonlinear Dimensionality Reduction: Based on the high-dimensional semantic vector, the UMAP model is used to perform nonlinear dimensionality reduction on the high-dimensional semantic vector to obtain a low-dimensional vector representation. In this embodiment, parameterized interfaces n_components, min_dist, and random_state are provided, and low-dimensional embeddings are output for clustering. Specifically, the steps include: Step S2-1, Dimensionality Reduction: The high-dimensional semantic vector is reduced using the nonlinear dimensionality reduction model UMAP. Compared with traditional dimensionality reduction techniques (such as PCA and t-SNE), this method can better balance the preservation of local and global structure, providing low-dimensional and highly discriminative feature representations for subsequent HDBSCAN clustering, thereby effectively improving the efficiency and effectiveness of topic modeling.

[0026] Step S2-2, parameter adjustment: n_components: Specifies the target dimension after dimensionality reduction, usually 2 or 5, and 5 in this example; min_dist: Controls the minimum distance between points after dimensionality reduction. The smaller the value, the denser the local structure (default 0.1). This is the default parameter for the BERTopic sub-model (the BERTopic framework sets this value to 0.0, which can significantly reduce outliers; in this embodiment, this parameter is consistent with the official BERTopic parameter).

[0027] random_state: Ensures the reproducibility of the results; in this embodiment, it is set to a fixed integer of 42.

[0028] The essence of the UMAP model is to minimize the cross-entropy difference between high-dimensional and low-dimensional similarity distributions: This yields a low-dimensional representation that preserves the semantic neighborhood structure, providing a highly discriminative feature base for subsequent HDBSCAN clustering.

[0029] In summary, this embodiment performs UMAP dimensionality reduction on the embedding vectors to preserve the local / global structure. Preferred key parameters are: min_dist = 0.0, n_components = 5 (visually friendly), and random_state fixed at 42 to ensure reproducibility.

[0030] Step S3, Density Clustering (HDBSCAN): The HDBSCAN density clustering algorithm is used to cluster low-dimensional vectors to obtain clustering results containing at least one topic cluster. In this embodiment, cluster labels, noise point identifiers, or core point information are output. Parameters are read from the default archive according to the data source (differentiated for satisfaction surveys / third-party reviews in this embodiment). The process is as follows: Step S3-1: In the BERTopic model, the HDBSCAN density clustering algorithm is used to cluster the document vectors after dimensionality reduction from UMAP. A hierarchical density connection tree is used to automatically identify high-density regions in the semantic space as topic clusters. This method does not require a preset number of clusters, is robust to noise, and can adaptively discover different density structures, providing a high-quality grouping foundation for subsequent topic extraction.

[0031] The core idea of ​​HDBSCAN is based on reachability distance and minimum spanning tree (MST) hierarchical clustering. The most crucial mathematical definition is as follows: In the formula, For the sample Euclidean distance in UMAP reduced space; :point The core distance is defined as its first core distance. The nearest neighbor samples to The distance; The "reachable distance" between two points is used to construct a weighted complete graph and generate a minimum spanning tree.

[0032] Step S3-2, parameter adjustment: min_cluster_size: Defines the minimum number of documents in a cluster. The larger the value, the more macroscopic the topic. In this embodiment, the default value of 10 is used. min_samples: controls the number of neighborhood samples of the core point, which affects the determination of noise points. (In this embodiment, it is the same as min_cluster_size by default.) The output is the cluster label of the document (including the noise point label - 1).

[0033] In this embodiment, HDBSCAN clustering is performed in the reduced-dimensional space, outputting cluster labels and noise points (-1). Key parameters: min_cluster_size (default 10, the larger the value, the more macroscopic the topic; in this embodiment, it is set to 20), defines the minimum number of neighbors required for the core point; min_samples=1 is set to reduce outliers (noise).

[0034] Step S4, Chinese preprocessing: The original text data is preprocessed to obtain standardized text. Chinese preprocessing includes cleaning, regular expression normalization, stop word and user dictionary maintenance, Jieba word segmentation, word segmentation result caching, and version management. The specific process is as follows: Step S4-1: Analyze the characteristics of the data and classify the data sources according to their sources and proportions. In this embodiment, passenger complaint texts are divided into two categories according to their sources: satisfaction survey suggestions and third-party reviews. A "source type" field and parameter files are created in the database for each category (for subsequent differentiated preprocessing and clustering parameter selection).

[0035] Step S4-2: Data cleaning is performed for different categories of data. In this embodiment, the operations performed on the original text include noise reduction and template removal, stop word filtering, typo / abbreviation correction, and number and symbol standardization.

[0036] Noise filtering: Remove irrelevant characters (special symbols, garbled text) and fixed template content (serial numbers, data sources) from the original text.

[0037] After retaining the passenger request data in plain text format, stop word processing is performed: The stop word list (Harbin Institute of Technology stop word library) and custom stop words (high-frequency but low semantic value words) are compared to comprehensively filter out meaningless words. The stop word list includes general stop words and domain-specific stop words (such as line names, station numbers, etc., high-frequency, low-information words) to reduce topic drift and maintain contextual integrity, satisfying the contextual dependence of BERT-like models.

[0038] Preservation of proper nouns: The UserDict dictionary is introduced to force the retention of proper nouns in the rail transit industry (such as "platform door" and "customer service center") and fixed collocations (such as "phone speaker"), thereby avoiding semantic loss caused by excessive word segmentation.

[0039] Specifically, the operations performed on the original text in this embodiment include noise reduction and template removal, stop word filtering, typo / abbreviation correction, and number and symbol normalization.

[0040] Step S4-3 utilizes the Jieba word segmentation tool to implement basic word segmentation functionality. This tool is based on two main modules: a statistical language model and a rule engine. It achieves accuracy and efficiency in Chinese word segmentation through the maximum matching algorithm and HMM out-of-vocabulary word recognition. Further, a user dictionary is used to expand proper noun segmentation (e.g., "transfer" and "delay compensation"). An exact mode + HMM configuration is employed to balance accuracy and domain-specific adaptability, effectively supporting the semantic representation requirements of subsequent BERTopic topic modeling. The Harbin Institute of Technology stop word list and user dictionary are used to iteratively optimize the cleaning effect, ensuring the semantic integrity of the text. The cleaned and segmented text is cached for direct use by the BERTopic model, thereby improving clustering efficiency and topic recognition accuracy. The mathematical formula is: In the formula, The original text sequence; The optimal word segmentation sequence is to be determined. Indicator The preceding word The conditional probability of subsequent occurrence (estimated by the language model); The word segmentation scheme with the highest probability is the optimal word segmentation result.

[0041] This formula demonstrates that Jieba segmentation achieves optimal segmentation of Chinese text by maximizing sentence probability. Combined with the HMM out-of-vocabulary word recognition mechanism and user dictionary updates, it ensures the semantic integrity of the segmentation results and provides high-quality input for subsequent BERTopic semantic modeling.

[0042] Step S5, Topic Feature Model (CountVectorizer + c-TF-IDF) Representation: Based on the clustering results and canonical text, a keyword weight matrix is ​​generated for each cluster. First, word frequency statistics are performed using CountVectorizer, and then the topic representation is calculated based on c-TF-IDF to extract keywords representing the topic. Words that are "high frequency within a class of documents but low frequency in other classes" are highlighted as topic keywords. In this embodiment, a keyword weight matrix is ​​generated for each cluster, and dynamic stop word addition is supported. The specific process is as follows: Step S5-1: In the clustering results, perform CountVectorizer word frequency statistics on the segmented text within each topic to obtain high-frequency words within the topic. Then, calculate the topic representation based on c-TF-IDF. Treat the text in the same cluster as a document. By measuring the distribution difference of words inside and outside the topic, extract the keywords that best represent the topic semantics to achieve high-quality topic representation. Step S5-2, parameter adjustment: `stop_words`: Removes stop words (e.g., a list of commonly used Chinese stop words). Its purpose is to filter out meaningless words in the text, improving the quality and efficiency of text analysis, especially in domain-specific applications. Setting `stop_words` prevents high-frequency but low-information words, such as line names, from affecting the analysis results. Stop words reappear at this stage to improve topic differentiation and efficiency.

[0043] min_topic_size: Topics with fewer than this value will be merged (10 by default in this example). Essentially, it is the same parameter as min_cluster_size, so either one can be called. In this example, min_topic_size is called.

[0044] nr_topics: Limits the final number of topics, achieved by merging similar topics (in this example, setting it to None means there is no limit). In the formula, For words On the topic The number of times it appears in (obtained by CountVectorizer); Theme The sum of the occurrences of all words within the range is used for word frequency normalization; The total number of all topics; For words containing The number of topics; To represent words On the topic Importance score.

[0045] This formula demonstrates that this embodiment achieves highly discriminative topic keyword extraction through an intra-class TF-IDF mechanism. Compared with the traditional TF-IDF method of document-level statistics, it can more accurately reflect the semantic features of each topic in the demands of subway passengers, and improve the accuracy and interpretability of topic representation.

[0046] Step S6, Topic Optimization and Merging (MMR / nr_topics): For the extracted keywords, the MMR (Maximal Marginal Relevance) method is used to optimize the keywords and reduce redundant words. Keywords with strong representativeness and semantic diversity are automatically selected. While ensuring relevance, the diversity of topic representation is enhanced, and the representation quality and interpretability of each topic keyword are improved. Thresholding similarity is supported to merge small topics into larger topics. The final optimized set of topic keywords is obtained.

[0047] In the formula, A set of candidate keywords; The set of selected keywords; This is the topic center vector (in this embodiment, it is represented by the topic document vector or cluster center); Keywords Semantic similarity to the central theme; Keywords With selected keywords semantic similarity; To balance the parameters, the weights of "correlation" and "diversity" are controlled.

[0048] This formula reflects how the MMR optimization mechanism in this embodiment balances "relevance" and "diversity" in keyword selection. It effectively reduces topic redundancy, improves semantic representativeness and topic interpretability, and provides theoretical support for understanding and displaying the clustering results of subway passenger demands.

[0049] Figure 2 This is a schematic diagram of high-frequency words within the theme in an embodiment of the present invention; Figure 3 This is a visualization diagram of the subject spacing in an embodiment of the present invention; In this embodiment, MMR is used to balance relevance and diversity, reducing keyword redundancy and resulting in 36 sub-topics for satisfaction survey suggestions. The top 5 words (or phrases) for the first 8 satisfaction survey suggestion topics are as follows: Figure 2 As shown; when there are too many sub-topics, hierarchical merging can be performed by setting nr_topics or a semantic similarity threshold to obtain a macro-topic. Setting the parameter nr_topics=9 manually merges 36 sub-topics into 9 large topics for satisfaction survey suggestions (including outliers) to improve analysis efficiency (corresponding to Figure 3 ).

[0050] Step S7, Visualization and Reporting: Based on the optimized set of topic keywords, output the topic percentage, two-dimensional distribution of topic spacing, and hierarchical clustering diagram to present the key passenger demands with good separation. Based on the key passenger demands, conduct data statistics on each subway line, analyze the primary passenger demands of each line, provide rectification suggestions for each line, assist in operational decision-making, so as to solve the main problems and effectively improve service quality.

[0051] Figure 4 This is a hierarchical clustering diagram of the topic suggestions for the satisfaction survey in an embodiment of the present invention.

[0052] In this embodiment, the proportion of output topics (e.g.) Figure 2 ), two-dimensional distribution of subject spacing (such as) Figure 3 ), hierarchical clustering diagrams and other views (such as Figure 4 ),like Figure 4 As shown in the figure, similar topics are closer in the topic hierarchical clustering graph. A vertical line drawn at position 1 on the horizontal axis intersects with nine horizontal lines in the figure. The topics connected to the left of these intersect to form a key passenger demand, exhibiting good separation. Based on these nine key passenger demands, data was statistically analyzed for each subway line to identify the primary passenger demands for each line. Improvement suggestions were provided for each line to assist operational decisions, thereby resolving major issues and effectively improving service quality.

[0053] Example 2 This embodiment provides a subway passenger multi-demand detection system based on the BERTopic model.

[0054] like Figure 1 As shown, the subway passenger multi-demand detection system based on the BERTopic model in this embodiment includes the following parts: Module 1: Topic Clustering Model. This module processes raw text and includes steps S1, S2, and S3 from Example 1. It mainly comprises three functional areas: 1. An Embedding word embedding model to generate semantic vectors; 2. A nonlinear dimensionality reduction model using the UMAP model to reduce the dimensionality of semantic vectors; 3. A density clustering model using the HDBSCAN density clustering algorithm to cluster the document vectors after UMAP dimensionality reduction.

[0055] Module Two: Word Segmentation Model. This module includes step S4 from Example 1, primarily implementing text segmentation. First, the text is preprocessed using a comprehensive stop word list. It innovatively introduces custom stop words and a user dictionary to iteratively optimize and clean the text, ensuring semantic integrity. The Jieba word segmentation tool is used to implement basic word segmentation. The cleaned and segmented text is cached for direct use by the BERTopic model, thereby improving clustering efficiency and topic recognition accuracy.

[0056] Module 3: Word Frequency Statistics Model. This module processes segmented text, including steps S5 and S6 in Example 1, and mainly comprises three functional areas: 1. The word frequency statistics model performs CountVectorizer word frequency statistics on the segmented text within each topic in the clustering results to obtain high-frequency words within the topic; 2. The topic feature representation model calculates topic representations based on c-TF-IDF, extracts keywords that best represent the topic semantics, and achieves high-quality topic representation; 3. The topic representation optimization model uses the MMR (Maximal Marginal Relevance) method to optimize keywords, effectively reducing topic redundancy and improving semantic representativeness and topic interpretability.

[0057] Module 4: Subject Visualization. This module includes step S7 in Example 1. Its function is to output the subject percentage, the two-dimensional distribution of subject spacing, and the hierarchical clustering diagram, presenting the key passenger demands with good separation, and providing clues for rectification suggestions to the operation side.

[0058] The above embodiments are preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention.

[0059] This invention first establishes independent data processing pipelines for different data sources (satisfaction surveys, third-party reviews, online comments, etc.) to achieve source-specific modeling; adopting the principle of "semantic preservation first," it designs a multi-level cleaning and word segmentation process (denoising, stop word filtering, misspelling correction, template removal, Jieba word segmentation + user dictionary); and introduces a domain-defined stop word system to filter high-frequency, low-information words such as subway line names, station numbers, and template fields.

[0060] Secondly, the Bert-Based-Chinese deep semantic model is introduced to generate dense text vectors (semantic vectors), and combined with UMAP dimensionality reduction and HDBSCAN clustering to form an integrated framework of "embedding-dimensionality reduction-clustering-representation". The robustness of short text clustering is improved by adjusting the parameters (adaptive `min_dist=0.0`, `min_cluster_size`, `min_samples`). In the topic representation stage, CountVectorizer + c-TF-IDF is combined to automatically generate a highly distinguishable set of topic keywords.

[0061] Finally, based on c-TF-IDF, the MMR (Maximal Marginal Relevance) algorithm is introduced to balance the relevance and diversity of topic keywords; hierarchical automatic merging of subtopics with high semantic similarity (`nr_topics = K`) is implemented to improve the clarity of topic expression and visualization quality; the output results include a topic distance plot, a hierarchical clustering tree, and an interpretable keyword table. The following technical effects are achieved: 1. This invention constructs a modular system analysis process, which integrates four major models (modules) and allocates six functional areas to achieve three main functions: topic clustering, word segmentation, and topic representation. It is efficient, has a clear structure, and is easy to maintain and expand.

[0062] 2. This invention effectively improves the semantic understanding capability of short texts and significantly enhances the accuracy and discriminative power of topic recognition by constructing a semantic embedding system specifically designed for the subway field (Bert-Base-Chinese + general stop word list + custom stop words + user dictionary).

[0063] 3. This invention introduces a user dictionary function, adding proper nouns and fixed collocations to ensure the integrity of important information in the text. Word frequency statistics, combined with a domain stop word mechanism (filtering subway line names, fixed template words, etc.), reduce a large number of low-information words during model operation, reducing noise interference and improving clustering stability.

Claims

1. A method for detecting diverse demands of subway passengers based on the BERTopic model, characterized in that, Includes the following steps: Step 1, Raw text data collection and source segmentation: Obtain raw text data from subway passengers and use an embedding model to transform the raw text data into a high-dimensional semantic vector; Step 2, Nonlinear Dimensionality Reduction: Based on the high-dimensional semantic vector, the UMAP model is used to perform nonlinear dimensionality reduction on the high-dimensional semantic vector to obtain a low-dimensional vector representation; Step 3, density clustering: The low-dimensional vectors are clustered using the HDBSCAN density clustering algorithm to obtain clustering results containing at least one topic cluster; Step 4, Chinese preprocessing: Perform Chinese preprocessing on the original text data to obtain standardized text. The Chinese preprocessing includes cleaning, regular expression normalization, stop word and user dictionary maintenance, Jieba word segmentation, word segmentation result caching and version management. Step 5, Topic Feature Representation: Based on the clustering results and the canonical text, a keyword weight matrix is ​​generated for each cluster. First, the word frequency is counted using CountVectorizer, and then the topic representation is calculated based on c-TF-IDF to extract the keywords representing the topic. Step 6, Topic Optimization and Merging: For the extracted keywords, the MMR method is used to optimize the keywords, automatically selecting keywords that are highly representative and semantically diverse, to obtain an optimized set of topic keywords; Step 7, Visualization and Reporting: Based on the optimized set of topic keywords, generate and output a topic analysis visualization report.

2. The method for detecting diverse demands of subway passengers based on the BERTopic model according to claim 1, characterized in that: in, In step 1, requests from different sources are collected, anonymized, and tagged, and written into their respective queues and parameter files according to data source type. The process is as follows: Construct a semantic embedding module: This module calls Bert-Base-Chinese or an equivalent Chinese Transformer to generate text semantic vectors and manages local caching and vector indexing. In the BERTopic model, BER-Base-Chinese and other embedding word embedding models are used to transform passenger request text into high-dimensional semantic vectors. Document representations are generated through mean pooling or context encoding, and UMAP dimensionality reduction and HDBSCAN clustering are combined to achieve semantically consistent topic recognition. The core principle and formula are as follows: In the formula, A high-dimensional semantic vector representation of passenger appeal text; For the first in the text i One word; This indicates that the Bert-Based-Chinese model is used for context encoding of words; This represents the text length.

3. The method for detecting diverse demands of subway passengers based on the BERTopic model according to claim 1, characterized in that: in, Step 2 provides parameterized interfaces n_components, min_dist, and random_state, and outputs low-dimensional embeddings for clustering, including the following steps: Step 2-1, Dimensionality Reduction: Use the nonlinear dimensionality reduction model UMAP to reduce the dimensionality of the high-dimensional semantic vector; Step 2-2, the above steps are achieved by setting at least one of the following parameters: n_components: Specifies the target dimension after dimensionality reduction, which is 2 or 5; min_dist: controls the minimum distance between points after dimensionality reduction. The smaller the value, the denser the local structure. random_state: Ensures the reproducibility of results. In step 2-1, the essence of the UMAP model is to minimize the cross-entropy difference between the high-dimensional and low-dimensional similarity distributions: This yields a low-dimensional representation that preserves the semantic neighborhood structure, providing a highly discriminative feature base for subsequent HDBSCAN clustering.

4. The method for detecting diverse demands of subway passengers based on the BERTopic model according to claim 1, characterized in that: in, In step 3, cluster labels, noise point identifiers, or core point information are output. Parameters are read from the default file according to the data source. The process is as follows: Step 3-1: In the BERTopic model, the HDBSCAN density clustering algorithm is used to cluster the document vectors after UMAP dimensionality reduction. The hierarchical density connection tree automatically identifies high-density regions in the semantic space as topic clusters. Step 3-2, the above steps are achieved by setting at least one of the following parameters: min_cluster_size: Defines the minimum number of documents in a cluster; a larger value indicates a more macro-level topic. min_samples: Controls the number of neighborhood samples of the core point, affecting the determination of noise points, and outputs the cluster labels of the document. In step 3-1, the core idea of ​​HDBSCAN is based on reachability distance and minimum spanning tree hierarchical clustering. The most crucial mathematical definition is as follows: In the formula, For the sample Euclidean distance in UMAP reduced space; :point The core distance is defined as its first core distance. The nearest neighbor samples to The distance; The "reachable distance" between two points is used to construct a weighted complete graph and generate a minimum spanning tree.

5. The method for detecting diverse demands of subway passengers based on the BERTopic model according to claim 1, characterized in that: in, Step 4 is as follows: Step 4-1: Analyze the characteristics of the data and classify the data sources according to their sources and proportions. Step 4-2: Perform data cleaning for different categories of data, including noise filtering, stop word processing, and retention of proper nouns. Step 4-3: Use the Jieba word segmentation tool to implement basic word segmentation. The cleaned and segmented text is cached for direct use by the BERTopic model, thereby improving clustering efficiency and topic recognition accuracy. The mathematical formula is: In the formula, The original text sequence; The optimal word segmentation sequence is to be determined. Indicator The preceding word The conditional probability of its subsequent occurrence; The word segmentation scheme with the highest probability is the optimal word segmentation result.

6. The method for detecting diverse demands of subway passengers based on the BERTopic model according to claim 5, characterized in that: in, In step 4-2, the noise filtering involves removing irrelevant characters and fixed template content from the original text. The irrelevant characters include at least special symbols and garbled text, and the fixed template content includes at least serial numbers and data sources. The stop word processing involves comparing a stop word list with a user-defined stop word list to comprehensively filter out meaningless words. The stop word list is the Harbin Institute of Technology stop word database, and the user-defined stop words are high-frequency words with low semantic value. The retention of proper nouns involves introducing a user dictionary (UserDict) to forcibly retain proper nouns and fixed collocations in the rail transit industry, thereby avoiding semantic loss caused by excessive word segmentation.

7. The method for detecting diverse demands of subway passengers based on the BERTopic model according to claim 1, characterized in that: in, In step 5, a keyword weight matrix is ​​generated for each cluster, and dynamic stop word addition is supported. The specific process is as follows: Step 5-1: In the clustering results, perform CountVectorizer word frequency statistics on the segmented text within each topic to obtain high-frequency words within the topic. Then, calculate the topic representation based on c-TF-IDF. Treat the text in the same cluster as a document. By measuring the distribution difference of words inside and outside the topic, extract the keywords that best represent the topic semantics to achieve high-quality topic representation. Step 5-2, the above steps are achieved by setting at least one of the following parameters: stop_words: Removes stop words, filtering out meaningless words in the text; min_topic_size or min_cluster_size: Topics with fewer than this value will be merged. nr_topics: Limits the final number of topics, achieved by merging similar topics. In the formula, For words On the topic The number of times it appears in; Theme The sum of the occurrences of all words within the range is used for word frequency normalization; The total number of all topics; For words containing The number of topics; To represent words On the topic Importance score.

8. The method for detecting diverse demands of subway passengers based on the BERTopic model according to claim 1, characterized in that: in, In step 6, the topic representation optimization model uses the MMR method to optimize keywords and reduce redundant words: In the formula, A set of candidate keywords; The set of selected keywords; The central vector of the topic; Keywords Semantic similarity to the central theme; Keywords With selected keywords semantic similarity; These are the balancing parameters.

9. The method for detecting diverse demands of subway passengers based on the BERTopic model according to claim 1, characterized in that: in, In step 7, the topic analysis visualization report includes any one or more of the following: output topic percentage, two-dimensional distribution of topic spacing, and hierarchical clustering diagram.

10. A subway passenger multi-demand detection system based on the BERTopic model, characterized in that, include: Module 1, Topic Clustering Module: This module acquires raw text data from subway passengers, transforms the raw text data into high-dimensional semantic vectors using an embedding model, performs nonlinear dimensionality reduction on the high-dimensional semantic vectors using the UMAP model to obtain low-dimensional vector representations, and then uses the HDBSCAN density clustering algorithm to cluster the low-dimensional vectors to obtain clustering results containing at least one topic cluster. Module 2, Word Segmentation Model: This module performs Chinese preprocessing on the original text data to obtain standardized text. The Chinese preprocessing includes cleaning, regular expression normalization, stop word and user dictionary maintenance, Jieba word segmentation, word segmentation result caching, and version management. Module 3, Word Frequency Statistics Model: This model generates a keyword weight matrix for each cluster based on the clustering results and the canonical text; it uses CountVectorizer to perform word frequency statistics, calculates topic representations based on c-TF-IDF, and extracts keywords representing the topic; it then uses the MMR method to optimize the extracted keywords, automatically selecting keywords that are highly representative and semantically diverse to obtain an optimized set of topic keywords. Module 4, Subject Visualization: This module generates and outputs a subject analysis visualization report based on the optimized set of subject keywords.

Images