Open domain long text classification method and apparatus based on topic analysis

Abstract

The application relates to an open domain long text classification method and device based on theme analysis. The method comprises the following steps: constructing a field self-adaptive word table; preprocessing original long text to obtain purified text; using an LDA model to construct a latent theme space; extracting a core sentence group from the long text through semantic clustering to generate an abstract; mapping the abstract to the theme space to calculate a correlation degree score and output a theme identification; and converting the theme identification into a business classification label by querying a theme-label mapping table. The application effectively solves the problems of complex long text semantics and dynamic label system changes, and improves classification accuracy and expansibility.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, in particular to a long text classification method and device in an open domain based on topic analysis. BACKGROUND

[0002] With the rapid development of the Internet and big data technology, text data is growing explosively. As the most widely distributed and largest data carrier, how to efficiently organize and manage these massive texts has become a key problem to be solved. Text classification, as a basic task in the field of natural language processing, aims to systematically organize and classify text resources, and is an important means to alleviate information overload.

[0003] Existing text classification techniques mainly focus on short texts (usually single sentences or a few hundred words) in a closed set environment. However, in the task of long text classification in an open domain, due to the characteristics of long texts, such as coexistence of information redundancy and noise, multiple themes and complex structure, and strong semantic dependency, existing methods face severe challenges. First, there is a significant data labeling bottleneck. High-quality labeled data is a core element to promote the development of artificial intelligence. However, in the context of long texts with heterogeneous content, manual labeling is inefficient and the quality is difficult to guarantee. Second, model training faces an expansion dilemma. As information iteration accelerates, the label system needs to be updated constantly. Traditional supervised learning methods can only identify existing labels, and once new categories are added, the data must be re-labeled and the model must be trained, which is inefficient and cannot meet the dynamic expansion needs in practical applications.

[0004] Therefore, it is necessary to provide a text classification method that can efficiently process long texts, adapt to dynamic label systems, and reduce dependence on labeled data. SUMMARY

[0005] Therefore, it is necessary to provide a text classification method that can efficiently process long texts, adapt to dynamic label systems, and reduce dependence on labeled data.

[0006] A long text classification method in an open domain based on topic analysis, the method comprising:

[0007] Constructing a domain adaptive word list, including a general stop word list, a domain stop word list, and a domain keyword list;

[0008] Pretreating the original long text, including noise character removal, text segmentation, and stop word filtering based on the domain adaptive word list, to obtain a purified text;

[0009] The purified text corpus is unsupervised trained using a latent Dirichlet allocation model to output a topic-term probability distribution, and a latent topic space is constructed based on the topic-term probability distribution;

[0010] For a single long text to be classified, the sentences thereof are encoded into semantic vectors by a pre-trained language model, and a clustering algorithm is used to group the sentence vectors, and a representative sentence is selected from each cluster to generate a text summary;

[0011] The text summary is mapped to the latent topic space, and a document-topic probability distribution of the text summary is inferred based on the topic-term probability distribution, and a topic with the highest probability value is taken as a topic identifier;

[0012] The topic identifier is converted into a final business classification label by querying a pre-constructed and dynamically maintained topic-label mapping table.

[0013] In one of the embodiments, the construction of the domain-adaptive word list includes that the general stop word list is used to filter common meaningless words, the domain stop word list is constructed based on high-frequency low-information words of a specific vertical domain, and the domain keyword list is constructed based on core concept words of the vertical domain.

[0014] In one of the embodiments, the method further includes: constructing a text dictionary based on the purified text corpus; converting a document set into a document-term matrix; performing model training by using Gibbs sampling or variational inference algorithm; and outputting a latent topic space; the latent topic space includes a topic-term probability distribution Φ and a document-topic probability distribution Θ.

[0015] In one of the embodiments, the method further includes: segmenting a single long text to be classified into a sentence sequence according to sentence boundaries; encoding each sentence in the sentence sequence into a high-dimensional semantic vector by using a BERT model; applying a K-means clustering algorithm to cluster the sentence vectors, and a preset number of clusters k is used; calculating the Euclidean distance between each sentence and the cluster center to which the sentence belongs, and selecting a sentence with the smallest distance as a representative sentence of the cluster; and combining the representative sentences of the clusters in the original text order to form a text summary.

[0016] In one of the embodiments, the method further includes: randomly initializing k cluster centers; assigning each sentence vector to the cluster to which the nearest cluster center belongs; recalculating the cluster center position based on all vectors in the current cluster; and iteratively performing the assignment and updating steps until the cluster center changes less than a threshold value or a maximum number of iterations is reached.

[0017] In one of the embodiments, the method further includes: calculating a relevance score based on the document-topic probability distribution; wherein, for a text summary , the relevance score Score(k) of the text summary belonging to a topic k is expressed as:

[0018]

[0019] wherein is the document-topic distribution vector of the text summary in the latent topic space.

[0020] In one embodiment, the topic-label mapping table is constructed by selecting the top N words with the highest probability as the representative words of each topic based on the trained topic-term distribution, and mapping the representative words to the predefined business classification labels by domain experts.

[0021] An open domain long text classification device based on topic analysis, the device comprising:

[0022] a vocabulary construction module for constructing a domain adaptive vocabulary; including a general stop word list, a domain stop word list and a domain key word list;

[0023] a text preprocessing module for preprocessing the original long text; including noise character removal, text segmentation and stop word filtering based on the domain adaptive vocabulary, to obtain a purified text;

[0024] a topic space construction module for unsupervised training of the purified text corpus using a latent Dirichlet allocation model, outputting a topic-term probability distribution, and constructing a latent topic space therefrom;

[0025] an abstract generation module for a single long text to be classified, encoding its sentences into semantic vectors by a pre-trained language model, grouping the sentence vectors using a clustering algorithm, and selecting representative sentences from each cluster to generate a text summary;

[0026] a topic identification module for mapping the text summary to the latent topic space, inferring the document-topic probability distribution of the text summary based on the topic-term probability distribution, and taking the topic with the highest probability value as the topic identification;

[0027] a label mapping module for converting the topic identification into the final business classification label by querying a pre-constructed and dynamically maintained topic-label mapping table.

[0028] A computer device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the following steps when executing the computer program:

[0029] constructing a domain adaptive vocabulary; including a general stop word list, a domain stop word list and a domain key word list;

[0030] Preprocessing the original long text; including noise character removal, text segmentation and stop word filtering based on the field adaptive word list, to obtain purified text;

[0031] Using the latent Dirichlet allocation model to unsupervised train the purified text corpus, output the topic-term probability distribution, and construct the latent topic space therefrom;

[0032] For a single long text to be classified, encoding its sentences into semantic vectors through a pre-trained language model, grouping the sentence vectors using a clustering algorithm, and generating a text summary from representative sentences in each cluster;

[0033] Mapping the text summary to the latent topic space, inferring the document-topic probability distribution of the text summary based on the topic-term probability distribution, and taking the topic with the highest probability value as the topic identification;

[0034] Converting the topic identification into the final business classification label by querying a pre-constructed dynamically maintainable topic-label mapping table.

[0035] A computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the following steps:

[0036] Constructing a field adaptive word list; including a general stop word list, a field stop word list and a field keyword list;

[0037] Preprocessing the original long text; including noise character removal, text segmentation and stop word filtering based on the field adaptive word list, to obtain purified text;

[0038] Using the latent Dirichlet allocation model to unsupervised train the purified text corpus, output the topic-term probability distribution, and construct the latent topic space therefrom;

[0039] For a single long text to be classified, encoding its sentences into semantic vectors through a pre-trained language model, grouping the sentence vectors using a clustering algorithm, and generating a text summary from representative sentences in each cluster;

[0040] Mapping the text summary to the latent topic space, inferring the document-topic probability distribution of the text summary based on the topic-term probability distribution, and taking the topic with the highest probability value as the topic identification;

[0041] Converting the topic identification into the final business classification label by querying a pre-constructed dynamically maintainable topic-label mapping table.

[0042] The aforementioned open-domain long text classification method and apparatus based on topic analysis improves the domain-specificity of text preprocessing by constructing a domain-adaptive vocabulary, overcomes the redundancy of long text information by extracting core content through semantic clustering, reduces the dependence on labeled data by unsupervised topic modeling, and enables flexible expansion of the classification system through topic-label dynamic mapping. Thus, it effectively solves the problems of semantic complexity and dynamic scalability in open-domain long text classification. Attached Figure Description

[0043] Figure 1 This is a diagram illustrating an application scenario of an open-domain long text classification method based on topic analysis in one embodiment.

[0044] Figure 2 This is a flowchart illustrating an open-domain long text classification method based on topic analysis in one embodiment;

[0045] Figure 3 This is a structural block diagram of an open-domain long text classification device based on topic analysis in one embodiment;

[0046] Figure 4 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0048] The topic-based open-domain long text classification method provided in this application can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. Terminal 102 sends long text data to server 104, and server 104 executes the classification method of this application and returns the classification result. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices, and server 104 can be a standalone server or a server cluster consisting of multiple servers.

[0049] In one embodiment, such as Figure 2 As shown, an open-domain long text classification method based on topic analysis is presented, and this method is applied to... Figure 1 Taking the server in the example, the following steps are included:

[0050] Step 202: Construct a domain-adaptive lexicon.

[0051] According to the semantic characteristics of specific vertical fields, a special language knowledge base is constructed. The glossary adopts a three-layer structure: a general stop word list, a self-defined stop word list, and a keyword list. The general stop word list is used to filter punctuation marks, common meaningless Chinese and English function words, and special characters; the self-defined stop word list and the keyword list are manually constructed and configured according to the semantic characteristics of specific vertical fields (such as finance, medical treatment, etc.).

[0052] In specific implementation, the self-defined stop word list constructed according to the training data contains about 300 field-related high-frequency low-information words, and the keyword list contains about 50 core field concept words. These word lists jointly act on the subsequent text preprocessing stage to improve the quality and representativeness of the theme words.

[0053] The field stop word list is used to filter high-frequency low-information words in a specific field, and the field keyword list is used to strengthen the core concepts of the field. The special word lists jointly act on the subsequent text preprocessing to improve the field specificity and purity of the theme word extraction.

[0054] Step 204, preprocessing the original long text.

[0055] The original long text is preprocessed by performing the following operations in sequence: first, remove spaces and invisible control characters in the text to form a continuous text stream; then use an efficient word segmentation tool (such as Jieba) to accurately segment the text to obtain a word segmentation list; finally, comprehensively use the multi-level word lists constructed above to filter the general and field stop words in the word segmentation results, and retain the word units containing key semantics, and output the purified text string.

[0056] The noise character removal includes removing spaces and invisible control characters; the text segmentation uses a dictionary-based word segmentation tool; and the stop word filtering is a comprehensive use of the general stop word list and the field stop word list to filter the word segmentation results.

[0057] Step 206, using a latent Dirichlet distribution model to unsupervisedly train the purified text corpus, outputting a theme-word item probability distribution, and constructing a latent theme space therefrom.

[0058] An unsupervised latent Dirichlet distribution (LDA) model is used for theme modeling. The specific implementation process includes: first, based on the preprocessed corpus, a text dictionary is constructed, and each unique word is assigned an index; then, the document set is converted into a document-word frequency matrix as the input of the LDA model; finally, the Gibbs sampling or variational inference algorithm is used to train the model, and finally the theme-word item probability distribution and the document-theme probability distribution are output, thereby establishing the latent theme space of the entire corpus.

[0059] Specifically, as a generative probabilistic model, the LDA topic model can be formally described for the document generation process as follows:

[0060] 1) Symbol definition: Suppose the entire document collection contains K topics; suppose the vocabulary size is V, and each document d in the document collection is considered a word sequence ( ),in This represents the i-th word; each topic k corresponds to a multinomial distribution on the vocabulary. That is, the "topic-term" distribution, which satisfies The document d corresponds to a multinomial distribution on the topic. That is, the "document-topic" distribution, which satisfies .

[0061] 2) Generation process:

[0062] For each topic k: from the Dirichlet prior distribution Mid-sampling to generate topic-term distribution .

[0063] For each document d: from the Dirichlet prior distribution Mid-sampling, generating document-topic distribution .

[0064] For each word in the document :from Sampling a topic ;from Sampling a word .

[0065] The goal of model training is to infer all hidden variables from the observed document set using variational EM algorithm or Gibbs sampling. and The posterior distribution. Finally, the document-topic distribution. This is the vector representation of document d in the latent topic space.

[0066] Step 208: For a single long text to be classified, the sentences are encoded into semantic vectors using a pre-trained language model. A clustering algorithm is then used to group the sentence vectors, and representative sentences are selected from each cluster to generate a text summary.

[0067] Step 210: Map the text summary to the latent topic space, infer the document-topic probability distribution of the text summary based on the topic-term probability distribution, and use the topic with the highest probability value as the topic identifier.

[0068] Step 212, convert the topic identification into the final business classification label by querying the pre-constructed dynamically maintainable topic-label mapping table.

[0069] The above open domain long text classification method based on topic analysis improves the domain-specificity of text preprocessing through the construction of a domain adaptive word list, overcomes the redundancy of long text information through semantic clustering to extract core content, reduces the dependence on labeled data through unsupervised topic modeling, and realizes the flexible expansion of the classification system through dynamic topic-label mapping, thereby effectively solving the problems of semantic complexity and dynamic expansion in open domain long text classification.

[0070] In one of the embodiments, the construction of the domain adaptive word list includes: a general stop word list for filtering common meaningless words; a domain stop word list constructed based on high-frequency low-information words of a specific vertical domain; and a domain keyword list constructed based on core concept words of the vertical domain.

[0071] In one of the embodiments, the construction of the latent topic space using the latent Dirichlet allocation model includes: constructing a text dictionary based on the purified text corpus; converting the document set into a document-term matrix; performing model training using Gibbs sampling or variational inference algorithm; and outputting the latent topic space; the latent topic space includes: topic-term probability distribution Φ and document-topic probability distribution Θ.

[0072] In one of the embodiments, the step of generating a text summary includes: dividing a single long text to be classified into a sentence sequence according to sentence boundaries; using a BERT model to encode each sentence in the sentence sequence into a high-dimensional semantic vector; applying a K-means clustering algorithm to cluster the sentence vectors, with a preset number of clusters k; calculating the Euclidean distance between each sentence and the cluster center to which it belongs, and selecting the sentence with the smallest distance as the representative sentence of the cluster; and combining the representative sentences of each cluster in the original text order to form a text summary.

[0073] Specifically, the execution process of the K-means clustering algorithm includes:

[0074] S1, randomly initialize k centroids.

[0075] S2, assign each sentence vector to the cluster to which the nearest centroid belongs.

[0076] S3, recalculate the centroid positions based on all vectors in the current cluster.

[0077] S4, iteratively perform the assignment and update steps until the centroid changes are less than a threshold or the maximum number of iterations is reached.

[0078] The clustering and summary extraction process can be defined as follows:

[0079] 1) Symbol definition: Let long text D be divided into m sentences ; Each sentence is mapped into n-dimensional semantic vector by BERT model ; The preset number of clusters is k, and k clusters are obtained after clustering , and the corresponding centroids . .

[0080] 2) Core sentence selection: For each cluster , select the sentence with the smallest Euclidean distance to the centroid as the representative sentence of the cluster; The final generated summary is composed of the representative sentences of all clusters in the original text order.

[0081] In one embodiment, the relevance score is calculated based on the document-topic probability distribution; wherein, for a text summary , the relevance score Score(k) of the topic k is expressed as:

[0082]

[0083] Where is the document-topic distribution vector of the text summary in the latent topic space.

[0084] The calculation of the relevance score is based on the trained LDA model, and for a text summary , the relevance score Score(k) of the topic k is the component of the "document-topic" distribution in the kth dimension inferred by the model:

[0085]

[0086] The final output topic identification is:

[0087]

[0088] In one embodiment, the topic-label mapping table is constructed as follows: based on the trained topic-term distribution, the top N words with the highest probability are selected as the representative words of each topic, and the domain expert maps the representative words to the predefined business classification label according to the representative words.

[0089] Specifically, the maintainable topic-label mapping table defines the correspondence between each latent topic identification discovered by the LDA model and the business predefined classification label. In actual classification, the system converts the topic identification into the final business label by querying this mapping table.

[0090] The topic-tag mapping table defines dynamic association rules between the potential topic identification discovered by the model and the user-defined business classification tags; the mapping table supports user maintenance and update according to actual application requirements.

[0091] The business classification tag system is open and extensible; when the classification system changes, the new tag can be activated by updating the topic-tag mapping table without re-executing the model training step, thereby realizing dynamic expansion of the classification system.

[0092] It should be understood that, although Figure 2 The steps in the flowchart of the model training method are displayed in sequence according to the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover, Figure 2 At least part of the steps in the flowchart of the model training method can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or sub-steps or stages of other steps.

[0093] In one embodiment, as shown in Figure 3 An open-domain long text classification device based on topic analysis is provided, comprising:

[0094] A word table construction module 302 is configured to construct a domain adaptive word table; the word table construction module 302 includes a general stop word table, a domain stop word table, and a domain keyword table;

[0095] A text preprocessing module 304 is configured to preprocess the original long text; the text preprocessing module 304 includes noise character removal, text segmentation, and stop word filtering based on the domain adaptive word table, to obtain a purified text;

[0096] A topic space construction module 306 is configured to use a latent Dirichlet allocation model to perform unsupervised training on the purified text corpus, to output a topic-term probability distribution, and to construct a latent topic space therefrom;

[0097] An abstract generation module 308 is configured to encode sentences of a single long text to be classified into semantic vectors by using a pre-trained language model, to group the sentence vectors by using a clustering algorithm, and to select representative sentences from each cluster to generate a text abstract;

[0098] A topic identification module 310 is configured to map the text abstract to the latent topic space, to infer a document-topic probability distribution of the text abstract based on the topic-term probability distribution, and to take a topic with the highest probability value as a topic identification.

[0099] a tag mapping module 312 configured to convert the topic identification into a final business classification tag by querying a pre-constructed topic-tag mapping table that can be dynamically maintained.

[0100] In one of the embodiments, the general stopword list in the vocabulary construction module 302 is used to filter common meaningless words; the domain stopword list is constructed based on high-frequency low-information words in a specific vertical domain; and the domain keyword list is constructed based on core concept words in the vertical domain.

[0101] In one of the embodiments, the topic space construction module 306 is further configured to construct a text dictionary based on the purified text corpus; convert the document set into a document-term matrix; perform model training using Gibbs sampling or variational inference algorithm; and output a latent topic space; the latent topic space includes a topic-term probability distribution Φ and a document-topic probability distribution Θ.

[0102] In one of the embodiments, the abstract generation module 308 is further configured to divide a single long text to be classified into a sentence sequence according to sentence boundaries; encode each sentence in the sentence sequence into a high-dimensional semantic vector using a BERT model; apply a K-means clustering algorithm to cluster the sentence vectors, with a preset number of clusters k; calculate the Euclidean distance between each sentence and the centroid of the cluster to which the sentence belongs, and select the sentence with the smallest distance as the representative sentence of the cluster; and combine the representative sentences of the clusters in the original text order to form a text abstract.

[0103] In one of the embodiments, the abstract generation module 308 is further configured to randomly initialize k centroids; assign each sentence vector to the cluster to which the nearest centroid belongs; recalculate the centroid positions based on all vectors in the current cluster; and iteratively perform the assignment and update steps until the centroid changes are less than a threshold or the maximum number of iterations is reached.

[0104] In one of the embodiments, the topic identification module 310 is further configured to calculate a relevance score based on the document-topic probability distribution; wherein the relevance score Score(k) of a text abstract belonging to topic k is expressed as:

[0105]

[0106] wherein is the document-topic distribution vector of the text abstract in the latent topic space.

[0107] In one of the embodiments, the topic-tag mapping table is constructed by: selecting the top N words with the highest probability as topic representative words for each topic based on the trained topic-term distribution; and mapping the topic representative words to predefined business classification tags by domain experts according to the topic representative words.​

[0108] The specific limitation of the open domain long text classification device based on topic analysis can refer to the limitation of the open domain long text classification method based on topic analysis in the above, which will not be repeated here. Each module in the above open domain long text classification device based on topic analysis can be realized by software, hardware and combination thereof in whole or in part. The above each module can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to call and execute the operation corresponding to each module by the processor.

[0109] In one embodiment, a computer device, which can be a server, is provided, and an internal structure diagram thereof can be as shown in Figure 4 The computer device includes a processor, a memory, a network interface and a database connected through a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as domain adaptive vocabulary, topic-term distribution, document-topic distribution and topic-label mapping table. The network interface of the computer device is used to communicate with the external terminal through the network connection. The computer program is executed by the processor to implement an open domain long text classification method based on topic analysis.

[0110] Those skilled in the art can understand that Figure 4 The structure shown in the above is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or less components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0111] In one embodiment, a computer device is provided, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the method in the above embodiment.

[0112] In one embodiment, a computer readable storage medium is provided, which stores a computer program. The computer program is executed by the processor to implement the steps of the method in the above embodiment.

[0113] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, storage, databases, or other media in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0114] The technical features of the above embodiments can be combined in any way. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, but as long as the combination of the technical features does not exist, it should be considered as the scope of the present application.

[0115] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. An open-domain long text classification method based on topic analysis, characterized in that, The method includes: Construct a domain-adaptive thesaurus; including a general stop word list, a domain stop word list, and a domain keyword list; The original long text is preprocessed, including noise character removal, text segmentation, and stop word filtering based on the domain-adaptive vocabulary, to obtain cleaned text. The latent Dirichlet distribution model is used to train the cleaned text corpus in an unsupervised manner, outputting the topic-term probability distribution, and thus constructing the latent topic space; For a single long text to be classified, the sentences are encoded into semantic vectors using a pre-trained language model. A clustering algorithm is then used to group the sentence vectors, and representative sentences are selected from each cluster to generate a text summary. The text summary is mapped to the latent topic space, and the document-topic probability distribution of the text summary is inferred based on the topic-term probability distribution, with the topic with the highest probability value being used as the topic identifier. By querying a pre-built, dynamically maintainable topic-tag mapping table, the topic identifier is converted into the final business classification tag. The topic-tag mapping table is constructed in the following way: based on the topic-term distribution obtained through training, the top N words with the highest probability for each topic are selected as topic representative words, and domain experts map them to predefined business classification tags according to the topic representative words.

2. The method according to claim 1, characterized in that, The construction of the domain-adaptive lexicon includes: The general stop word list is used to filter common meaningless words; The domain stop word list is constructed based on high-frequency, low-information-content words in a specific vertical domain. The domain keyword list is constructed based on the core concept vocabulary of the vertical domain.

3. The method according to claim 1, characterized in that, The construction of the latent topic space using the latent Dirichlet distribution model includes: Constructing a text dictionary based on purified text corpus; Convert the document collection into a document-term matrix; Model training is performed using Gibbs sampling or variational inference algorithms; Output the latent topic space; the latent topic space includes: topic-term probability distribution Φ and document-topic probability distribution Θ.

4. The method according to claim 1, characterized in that, For a single long text to be classified, its sentences are encoded into semantic vectors using a pre-trained language model. A clustering algorithm is then used to group the sentence vectors, and representative sentences are selected from each cluster to generate a text summary, including: The single long text to be classified is divided into sentence sequences according to sentence boundaries; The BERT model is used to encode each sentence in the sentence sequence into a high-dimensional semantic vector. The K-means algorithm is used to cluster sentence vectors, with a preset number of clusters k; Calculate the Euclidean distance between each sentence and the centroid of its cluster, and select the sentence with the smallest distance as the representative sentence of the cluster; The representative sentences of each cluster are combined in the original text order to form a text summary.

5. The method according to claim 4, characterized in that, The execution process of the K-means clustering algorithm includes: Randomly initialize k centroids; Assign each sentence vector to the cluster to which the nearest centroid belongs; Recalculate the centroid position based on all vectors within the current cluster; Iteratively execute the allocation and update steps until the centroid change is less than the threshold or the maximum number of iterations is reached.

6. The method according to claim 1, characterized in that, The process of calculating the relevance score is based on the document-topic probability distribution, for summary documents. The score Score(k) belonging to topic k is expressed as: in It is the document-topic distribution vector of the summary document in the latent topic space.

7. An open-domain long text classification device based on topic analysis, characterized in that, The device includes: The vocabulary building module is used to build domain-adaptive vocabularies, including a general stop word list, a domain stop word list, and a domain keyword list. The text preprocessing module is used to preprocess the original long text, including noise character removal, text segmentation, and stop word filtering based on the domain-adaptive vocabulary, to obtain cleaned text. The topic space construction module is used to perform unsupervised training on the cleaned text corpus using the latent Dirichlet distribution model, output the topic-term probability distribution, and thereby construct the latent topic space; The summary generation module is used to encode sentences into semantic vectors from a pre-trained language model for a single long text to be classified, and to group the sentence vectors using a clustering algorithm, and select representative sentences from each cluster to generate a text summary; The topic recognition module is used to map the text summary to the latent topic space, infer the document-topic probability distribution of the text summary based on the topic-term probability distribution, and use the topic with the highest probability value as the topic identifier. The tag mapping module is used to convert the topic identifier into the final business classification tag by querying a pre-built, dynamically maintainable topic-tag mapping table. The topic-tag mapping table is constructed in the following way: based on the topic-term distribution obtained through training, the top N words with the highest probability for each topic are selected as topic representative words, and domain experts map them to predefined business classification tags according to the topic representative words.

Images