Topic detection method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a topic detection method and device, computer equipment, a readable storage medium and a program product. The method comprises the following steps: under the condition that to-be-detected incremental text data is received, carrying out abstract extraction on the incremental text data to obtain abstract description information of the incremental text data, and if multiple pieces of incremental text data exist, carrying out first clustering processing on the abstract description information of the multiple pieces of incremental text data to obtain at least one first cluster; and performing second clustering processing on each first cluster and a plurality of topics in a pre-acquired clustering table, determining a target topic of the first cluster, and endowing the target topic to at least one piece of incremental text data in the first cluster, thereby realizing topic detection of the incremental text data.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a topic detection method, apparatus, computer equipment, readable storage medium and program product. Background Art

[0002] With the advent of the information age, the Internet and various communication technologies have flourished, and the amount of data has exploded. People are facing the problem of information overload. In the vast amount of information, how to quickly and accurately obtain valuable content has become an urgent need, which has also prompted the emergence of hot topic analysis.

[0003] In traditional technology, the process of hot topic analysis generally includes: segmenting the original text, obtaining data features with the help of methods such as TF-IDF (term frequency-inverse document frequency, a common weighting technology used in information retrieval and data mining), and then clustering the data using k-means (a clustering method), density clustering, etc., and then determining the topic of the data based on the clustered clusters.

[0004] However, in actual application scenarios, text will continue to increase over time, and traditional technology can only analyze existing stock data but not the incremental data. Summary of the invention

[0005] Based on this, it is necessary to provide a topic detection method, apparatus, computer device, computer-readable storage medium and computer program product that can perform topic analysis on incremental data in response to the above technical problems.

[0006] In a first aspect, the present application provides a topic detection method, comprising:

[0007] When receiving the incremental text data to be detected, extracting a summary of the incremental text data to obtain summary description information of the incremental text data;

[0008] If there are multiple incremental text data, perform a first clustering process on the summary description information of the multiple incremental text data to obtain at least one first cluster, each first cluster including summary description information of at least one incremental text data whose clustering distance is less than a first distance threshold;

[0009] Performing a second clustering process on each first cluster and a plurality of topics in a pre-acquired clustering table, and if there is a topic in the clustering table whose clustering distance with the first cluster is less than a second distance threshold, determining a target topic of the first cluster based on the topic;

[0010] The target topic is assigned to at least one incremental text data in the first cluster.

[0011] In one of the embodiments, before performing a second clustering process on each first cluster and a plurality of topics in a pre-acquired clustering table, the method further includes: performing summary extraction on a plurality of stock text data to be detected to obtain summary description information of each stock text data; performing a first clustering process on the summary description information of the plurality of stock text data to obtain at least one second cluster, each second cluster including summary description information of at least one stock text data having a clustering distance less than a first distance threshold; determining a topic corresponding to the second cluster using a pre-trained artificial intelligence model based on the summary description information of at least one stock text data in each of the second clusters; and generating a clustering table based on each of the second clusters and the corresponding topic, and the summary description information of at least one stock text data in each of the second clusters.

[0012] In one of the embodiments, the method further includes: if there is no topic in the clustering table whose clustering distance with the first cluster is less than a second distance threshold, adding clustering information of the first cluster to the clustering table, the clustering information of the first cluster including summary description information of at least one incremental text data in the first cluster; determining the topic corresponding to the first cluster using a pre-trained artificial intelligence model based on the summary description information of at least one incremental text data in the first cluster; and adding the topic corresponding to the first cluster to the clustering table.

[0013] In one of the embodiments, if there is a topic in the clustering table whose clustering distance with the first cluster is less than a second distance threshold, determining the target topic of the first cluster based on the topic includes: if there is a topic in the clustering table whose clustering distance with the first cluster is less than the second distance threshold, determining the topic with the smallest clustering distance as the target topic of the first cluster.

[0014] In one embodiment, the summary extraction includes: performing data segmentation on the text data to be extracted based on a preset data length to obtain at least one data block and a sequence number of each data block; using a pre-trained artificial intelligence model to obtain summary information of each data block; merging the summary information of each data block according to the sequence number of each data block to obtain merged summary data; when it is determined that the length of the merged summary data is less than or equal to the preset data length, using the merged summary data as summary description information of the text data to be extracted; when it is determined that the length of the merged summary data is greater than the preset data length, using the merged summary data as the text data to be extracted, and returning to execute the step of performing data segmentation on the text data to be extracted based on the preset data length.

[0015] In one embodiment, the first clustering process adopts a union-find based clustering algorithm; the second clustering process adopts a FAISS based vector similarity matching algorithm.

[0016] In a second aspect, the present application provides a topic detection device, the device comprising:

[0017] A summary extraction module is used to extract the summary of the incremental text data when receiving the incremental text data to be detected, so as to obtain summary description information of the incremental text data;

[0018] A first clustering module is used for performing a first clustering process on the summary description information of the plurality of incremental text data if there are a plurality of the incremental text data to obtain at least one first cluster, each of the first clusters including the summary description information of at least one incremental text data whose clustering distance is less than a first distance threshold;

[0019] A second clustering module, configured to perform a second clustering process on each first cluster and a plurality of topics in a pre-acquired clustering table, and if there is a topic in the clustering table whose clustering distance with the first cluster is less than a second distance threshold, determine a target topic of the first cluster based on the topic;

[0020] The topic determination module is used to assign the target topic to at least one incremental text data in the first cluster.

[0021] In a third aspect, the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method described in the first aspect when executing the computer program.

[0022] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method described in the first aspect above.

[0023] In a fifth aspect, the present application further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the method described in the first aspect above.

[0024] The above-mentioned topic detection method, apparatus, computer equipment, computer-readable storage medium and computer program product, when the terminal receives the incremental text data to be detected, extracts the summary of the incremental text data and obtains the summary description information of the incremental text data. If there are multiple incremental text data, the summary description information of the multiple incremental text data is subjected to a first clustering process to obtain at least one first cluster, and each first cluster is subjected to a second clustering process with multiple topics in a pre-acquired clustering table to determine the target topic of the first cluster, and the target topic is assigned to at least one incremental text data in the first cluster, thereby realizing topic detection of the incremental text data. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments of the present application or the related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0026] Figure 1 is an application environment diagram of a topic detection method in one embodiment;

[0027] Figure 2 is a flowchart of a topic detection method in one embodiment;

[0028] Figure 3 A schematic diagram of a process of generating a clustering table in one embodiment;

[0029] Figure 4 A schematic diagram of a process flow of a summary extraction step in an embodiment;

[0030] Figure 5 is a structural block diagram of a topic detection device in one embodiment;

[0031] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0033] The rise of social media has enabled people to express their opinions more freely and widely. At the same time, a large number of topics have spread rapidly in a short period of time and attracted widespread attention. In order to better understand the direction of public opinion and the focus of public interest, it is necessary to analyze these hot topics. For example, the intensification of commercial competition requires companies to grasp market dynamics and consumer demand more accurately. Hot topic analysis can help companies discover potential market opportunities and risks in a timely manner, so as to formulate more effective marketing strategies and product planning. In addition, in order to better formulate policies, provide public services and respond to social issues, relevant departments can also understand the demands of the people and the hot concerns of society through hot topic analysis.

[0034] Traditional hot topic analysis can only analyze existing stock data, but cannot analyze incremental data. Secondly, in the traditional hot topic analysis process, when the number of clusters is too large, it is difficult to determine the hyperparameters of the clustering method. For example, k-means needs to determine the size of k, and density clustering needs to determine the neighborhood radius and the minimum number of samples. Hyperparameter tuning methods such as the elbow method take too long to apply when the amount of data is extremely large. Finally, the capabilities of some traditional deep learning models used for hot topic analysis are also limited by the quality and quantity of labeled data, and data labeling requires a lot of time and manpower, resulting in high usage costs.

[0035] Based on this, the present application provides a topic detection method that can dynamically and quickly analyze massive data to achieve topic detection and extract hot topics within a certain time period.

[0036] The topic detection method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The head-mounted device can be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. The server 104 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services. The method of this embodiment can be applied to a terminal, a server, or a system including a terminal and a server, and is implemented through the interaction between the terminal and the server.

[0037] In one embodiment, Figure 2 As shown, a topic detection method is provided. In this embodiment, the method is applied to a terminal for example, and specifically may include the following steps:

[0038] Step 202: When the incremental text data to be detected is received, a summary is extracted from the incremental text data to obtain summary description information of the incremental text data.

[0039] The incremental text data to be detected may be real-time data that needs to be subject to topic detection, such as public opinion data, product evaluation data, service feedback data, etc. crawled by web crawlers in real time. Specifically, the incremental text data may include specific data content, such as product evaluation data, specific evaluation content, and service feedback data, specific feedback content. Summary extraction refers to the process of extracting important information from the original data. Summary description information refers to the important information obtained by summarizing and extracting from the incremental text data.

[0040] In this embodiment, when the terminal receives the incremental text data to be detected, it can extract the summary of the incremental text data to obtain the summary description information in the incremental text data, and perform topic detection based on subsequent steps.

[0041] Step 204: If there are multiple incremental text data, perform a first clustering process on the summary description information of the multiple incremental text data to obtain at least one first cluster.

[0042] Among them, the first clustering process is the process of grouping several data objects into multiple classes, classifying similar data objects into one class, and data objects between different classes are dissimilar, so that data objects in the same group are more similar than data objects between different groups. In this embodiment, the first clustering process can adopt a clustering algorithm based on a union-find set, which is a method of implementing clustering using a union-find set data structure. The union-find set is a tree structure that can be used to process the merging and querying problems of some disjoint sets. In this clustering algorithm, it can be used to merge similar data points to form different clusters.

[0043] The first cluster is a class obtained after the first clustering process, and each first cluster includes summary description information of at least one incremental text data whose clustering distance is less than the first distance threshold. The first distance threshold can be a pre-configured basis for clustering, i.e., grouping. Specifically, the first distance threshold can be determined experimentally through samples. In this embodiment, the first distance threshold can be set to 0.95.

[0044] In this embodiment, when the terminal receives multiple incremental text data to be detected, in order to avoid dividing the data belonging to the same cluster into different clusters or repeating the matching calculation for the same data when directly matching it with the pre-acquired clustering table, the summary description information of the multiple incremental text data can be first clustered. Specifically, the summary description information of each incremental text data can be taken as a separate set, and based on each two sets, if their distance is less than or equal to the first distance threshold, they are gradually merged until they can no longer be merged (that is, the condition that the distance is less than or equal to the first distance threshold cannot be met), and at least one first cluster after clustering is obtained. Among them, the pre-acquired clustering table can be generated after clustering and topic analysis based on a large amount of stock data, which will be introduced in detail later, and this embodiment will not be repeated.

[0045] Step 206 , performing a second clustering process on each first cluster and a plurality of topics in the pre-acquired clustering table to determine a target topic of the first cluster.

[0046] The target topic is a topic of the first cluster determined based on the second clustering process. The second clustering process can use a vector similarity matching algorithm based on FAISS (Facebook AI Similarity Search, an efficient similarity search method). The core function of FAISS is to quickly find the most similar vector in the database with a given query vector.

[0047] In this embodiment, the terminal performs a second clustering process on each first cluster and a plurality of topics in a pre-acquired clustering table. If there is a topic in the clustering table whose clustering distance from the first cluster is less than a second distance threshold, the target topic of the first cluster is determined based on the topics in the clustering table. The second distance threshold may be pre-configured for performing clustering to determine the boundaries of the search results. Specifically, the second distance threshold may also be determined experimentally through samples. In this embodiment, the second distance threshold may be set to 0.26.

[0048] The pre-acquired clustering table may be generated after clustering and topic analysis based on a large amount of stock data, and includes multiple classes and topics corresponding to each class. In this embodiment, the topics in the clustering table may be converted into vector representations through BGE-Large-zh (a Chinese text representation model) and imported into the FAISS database, and FAISS will index these vectors for quick retrieval.

[0049] Then, the summary description information of each incremental text data in each first cluster is converted into a vector representation, which is input into FAISS as a query vector for query. FAISS performs clustering by calculating the distance between the query vector and each vector in the database. When the distance is less than the second distance threshold, it can be considered similar.

[0050] Specifically, FAISS calculates the distance between the query vector and each vector in the database. Since each first cluster includes summary description information of at least one incremental text data, the query vector corresponding to each first cluster includes at least one vector of summary description information. Based on this, the distance between each vector of summary description information in the first cluster and each vector in the database can be calculated to determine the vectors in the database whose distance is less than the second distance threshold. If there is only one vector in the database whose distance is less than the second distance threshold, the topic corresponding to the vector is used as the target topic of the first cluster. If there are multiple vectors in the database whose distance is less than the second distance threshold, the topic corresponding to the vector with the smallest distance is selected as the target topic of the first cluster.

[0051] Step 208: assigning a target topic to at least one incremental text data in the first cluster.

[0052] Since the first cluster includes summary description information of at least one incremental text data, after determining the target topic of the first cluster based on the above steps, the target topic can be assigned to each incremental text data in the first cluster, thereby realizing topic detection for the incremental text data.

[0053] In the above-mentioned topic detection method, when the terminal receives the incremental text data to be detected, it extracts a summary of the incremental text data to obtain summary description information of the incremental text data. If there are multiple incremental text data, the summary description information of the multiple incremental text data is subjected to a first clustering process to obtain at least one first cluster. Each first cluster is subjected to a second clustering process with multiple topics in a pre-acquired clustering table to determine a target topic of the first cluster, and the target topic is assigned to at least one incremental text data in the first cluster, thereby realizing topic detection of the incremental text data.

[0054] In one embodiment, in step 206, before performing the second clustering process on each first cluster and the plurality of topics in the pre-acquired clustering table, a clustering table generation process is also included, such as Figure 3 As shown, the following steps may be specifically included:

[0055] Step 302: extract summary of several stock text data to be detected, and obtain summary description information of each stock text data.

[0056] The stock text data may be all existing historical data that need to be subject to topic detection. The incremental text data is new data that appears over time. In this embodiment, the terminal may perform topic detection on a number of stock text data to generate a corresponding topic clustering table, and then perform topic detection on the incremental text data through the clustering table.

[0057] Specifically, the terminal may perform summary extraction on a number of stock text data to extract important information therefrom, obtain summary description information of each stock text data, and generate a clustering table based on subsequent steps.

[0058] In an exemplary embodiment, Figure 4 As shown, the above summary extraction process may specifically include the following steps:

[0059] Step 402, segmenting the text data to be extracted based on a preset data length to obtain at least one data block and a sequence number for each data block.

[0060] It is understandable that the summary extraction in this embodiment is not only applicable to the summary extraction of the stock text data, but also to the summary extraction of the incremental text data. Therefore, the text data to be extracted can be the incremental text data mentioned above or the stock text data. In this embodiment, for ease of understanding, the text data to be extracted is taken as the stock text data for explanation.

[0061] Specifically, in order to improve the quality of data and conduct more effective data analysis and mining, the text data to be extracted, i.e., the stock text data, may be preprocessed before extracting the summary. Specifically, a corresponding data table such as Table 1 may be generated based on a number of stock text data, and the column name of the corresponding data of the stock text data may be set as "event description". In order to uniquely identify each piece of data, an "event number" column may also be set, for example, a natural number starting from 1 may be assigned to each event description.

[0062] Since the length of the text data to be extracted varies, in order to make the importance of different text data in the final topic analysis basically consistent, the different text data can be summarized first, that is, the length of the text data can be controlled within one sentence. Since the quality of the extracted summary description information will directly affect the quality of the final topic, this embodiment adopts a hierarchical summary scheme, which mainly includes three parts: segmentation, merging and extraction.

[0063] Specifically, the text data to be extracted can be segmented based on a preset data length to obtain at least one data block and a sequence number of each data block. In this embodiment, the original text can be segmented according to the preset data length with the help of a text segmenter of LangChain (a programming framework that helps use large language models (LLM) in applications) to obtain at least one data block and a sequence number of each data block. The preset data length can be a window of 256 characters.

[0064] Step 404, using a pre-trained artificial intelligence model to obtain summary information of each data block.

[0065] Among them, the artificial intelligence model can be a large language model (also called a large model) trained with a large amount of corpus, which is used to extract the summary information of the data block. Specifically, for each data block obtained by the above segmentation, it can be randomly shuffled and input into the artificial intelligence model to avoid the position deviation problem of the model, so as to obtain the summary information of each data block. For example, the purpose of extracting the summary information of the data block can be achieved by designing a prompt (prompt word). Exemplarily, the design of the prompt can be: "Please summarize the content described in the following text in one sentence, without including any time information. \n". Specifically, when extracting the summary information of the data block, the text input into the large model can be: prompt + data block content. Based on this, the summary information of each data block can be obtained.

[0066] Step 406: merge the summary information of each data block according to the sequence number of each data block to obtain merged summary data.

[0067] After the summary information of each data block is obtained based on the above steps, the summary information of each data block can be merged according to the sequence number of each data block to obtain merged summary data.

[0068] Step 408: determine whether the length of the combined summary data is less than or equal to a preset data length.

[0069] If it is determined that the length of the combined summary data is less than or equal to the preset data length, step 410 is executed; if it is determined that the length of the combined summary data is greater than the preset data length, step 412 is executed.

[0070] Step 410: Use the merged summary data as summary description information of the text data to be extracted.

[0071] Specifically, when it is determined that the length of the combined summary data is less than or equal to the preset data length, the combined summary data can be used as the summary description information of the text data to be extracted. A "summary description" column is added to the data table, and the extracted summary description information is stored in the corresponding row of the column.

[0072] Step 412: Use the merged summary data as text data to be extracted.

[0073] Specifically, when it is determined that the length of the merged summary data is greater than the preset data length, the merged summary data can be used as the text data to be extracted, and the process returns to step 402 to repeat the above segmentation, merging and extraction process until the summary information is finally obtained that is less than or equal to the preset data length.

[0074] Based on this, summary extraction is performed on several stock text data to be detected, and summary description information of each stock text data can be obtained.

[0075] Step 304: Perform a first clustering process on the summary description information of the plurality of stock text data to obtain at least one second cluster.

[0076] The second cluster is a cluster obtained after the first clustering process, and each second cluster includes summary description information of at least one stock text data whose clustering distance is less than the first distance threshold.

[0077] Specifically, the summary description information of each stock text data can be first taken as a separate set, and based on every two sets, if their distance is less than or equal to the first distance threshold, they are gradually merged until they can no longer be merged (that is, the condition that the distance is less than or equal to the first distance threshold cannot be met), thereby obtaining at least one second cluster after clustering.

[0078] In this embodiment, in order to preserve the clustering structure, a new clustering table such as Table 2 may be created, wherein the first column may be a cluster number starting from 1, namely, a "cluster number", which is used to uniquely identify each second cluster. The second column may be a "summary description", which indicates the collection of all summary description information belonging to the same cluster, and different summary description information may be separated by line breaks. In addition, after the clustering operation in this step, a new column "cluster number" may be added to the data table to indicate the cluster to which the corresponding summary description belongs, and the number may be used to associate the data table with the clustering table.

[0079] Step 306 , based on the summary description information of at least one stock text data in each second cluster, a pre-trained artificial intelligence model is used to determine the topic corresponding to the second cluster.

[0080] In this embodiment, the topic corresponding to each second cluster can be determined based on the summary description information of at least one stock text data in each second cluster using a pre-trained artificial intelligence model.

[0081] Specifically, the purpose of extracting topics from summary description information can also be achieved by designing a prompt. For example, the prompt in this embodiment can be designed as: "You are an expert in topic summarization. Each line in the input is a summary description. Please summarize a hot topic. The topic is a sentence and does not contain any punctuation marks and time information. Please answer in the json format. \n". When only the topic needs to be obtained, the text input into the large model can be: prompt+summary description. Among them, the summary description includes the summary description information of each stock text data in the second cluster. The return value of the large model is the topic corresponding to the summary description, that is, the topic corresponding to the second cluster, and is stored in the third column "Topic" of the clustering table.

[0082] In one scenario, if you want to extract personalized information such as the subject and amount involved in the summary description, you can also input the large model in the following format: prompt+"json.dump({"input": summary description, "schema":[{"event_type":"event information", "arguments":["subject involved", "amount", "topic"]}]}, ensure_ascii=False)". Before that, you can also prepare about 200 fine-tuning data in this format to fine-tune the large model to control the output format. By processing the return value of the large model, the topic of each second cluster is obtained and stored in the third column "topic" of the clustering table. If other information is extracted, it is stored in the fourth column, fifth column, and so on of the clustering table.

[0083] Step 308: Generate a clustering table based on each second cluster and the corresponding topic, and summary description information of at least one stock text data in each second cluster.

[0084] Specifically, the clustering table may be as shown in Table 2 below, and the data table may be as shown in Table 1 below.

[0085] The data table Table 1 example is as follows:

[0086]

[0087] An example of clustering table 2 is as follows:

[0088]

[0089] In one scenario, a new topic column can be added to the data table according to the cluster numbers in the above data table and cluster table, and the topics in the data table can be assigned values ​​based on the topics in the cluster table, thereby determining the topic of each stock text data in the data table.

[0090] It is understandable that a data table based on the incremental text data can also be generated when the incremental text data is abstracted. The difference is that the cluster number in the data table of the incremental text data is a temporary cluster number. Specifically, by performing a second clustering process on each first cluster of the incremental text data and multiple topics in the above clustering table, if there are topics in the clustering table whose clustering distance with the first cluster is less than the second distance threshold, the target topic of the first cluster is determined based on the topics in the clustering table whose clustering distance is less than the second distance threshold, and the cluster number in the clustering table corresponding to the target topic is assigned to the first cluster, that is, the temporary cluster number of the first cluster is modified to the cluster number corresponding to the target topic.

[0091] In one scenario, during the second clustering process of each first cluster of incremental text data and multiple topics in the above clustering table, if there is no topic in the clustering table whose clustering distance with the first cluster is less than the second distance threshold, the clustering information of the first cluster is added to the clustering table, that is, the first cluster is regarded as a new cluster. Specifically, the clustering information of the first cluster may include summary description information of at least one incremental text data in the first cluster. It is understandable that the clustering table may also generate a cluster number of the clustering information of the first cluster.

[0092] Then, the topic corresponding to the first cluster can be determined using a pre-trained artificial intelligence model based on the summary description information of at least one incremental text data in the first cluster. The process of determining a specific topic is similar to step 306 above, and this embodiment will not be repeated. Then, the topic corresponding to the first cluster can be added to the clustering table.

[0093] After the above operations, the topics and cluster numbers in the cluster table can also be used to assign values ​​to the data table of the incremental text data, that is, the topic corresponding to the first cluster is assigned to at least one incremental text data of the corresponding cluster in the data table of the incremental text data, and the temporary cluster number of the first cluster is modified to the cluster number of the corresponding topic.

[0094] In this embodiment, in order to avoid spending too much time on tuning the hyperparameters of the clustering algorithm, a stock data clustering algorithm based on a union-find set is adopted. In order to avoid dependence on labeled data, a large model trained with a large amount of corpus is used. Since the large model has strong language understanding and generation capabilities, it does not need fine-tuning. In order to support the dynamic increase of data in hot topic analysis and avoid dependence on labeled data, an incremental topic matching algorithm based on FAISS is also designed. Thus, a hot topic analysis method based on a large model is realized, which reduces the difficulty of selecting hyperparameters through a clustering algorithm based on a union-find set; by making full use of the summary ability of the large model with the help of prompt, the dependence on labeled data is avoided, and the influence of position deviation on the summary effect can be reduced by designing a disordered hierarchical summary algorithm. By designing an incremental text clustering algorithm based on FAISS, it can support the dynamic expansion of text, thereby realizing topic detection for any text data.

[0095] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0096] Based on the same inventive concept, the embodiment of the present application also provides a topic detection device for implementing the topic detection method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more topic detection device embodiments provided below can refer to the limitations on the topic detection method above, and will not be repeated here.

[0097] In an exemplary embodiment, Figure 5As shown, a topic detection device is provided, comprising: a summary extraction module 502, a first clustering module 504, a second clustering module 506 and a topic determination module 508, wherein:

[0098] A summary extraction module 502 is used to extract a summary of the incremental text data to be detected and obtain summary description information of the incremental text data when the incremental text data to be detected is received;

[0099] A first clustering module 504 is used for performing a first clustering process on the summary description information of the plurality of incremental text data if there are a plurality of the incremental text data to obtain at least one first cluster, each of which includes the summary description information of at least one incremental text data whose clustering distance is less than a first distance threshold;

[0100] A second clustering module 506 is used to perform a second clustering process on each first cluster and a plurality of topics in a pre-acquired clustering table, and if there is a topic in the clustering table whose clustering distance with the first cluster is less than a second distance threshold, determine a target topic of the first cluster based on the topic;

[0101] The topic determination module 508 is configured to assign the target topic to at least one incremental text data in the first cluster.

[0102] In an exemplary embodiment, it also includes a clustering table generation module, which is used to: extract summaries of several stock text data to be detected to obtain summary description information of each stock text data; perform a first clustering process on the summary description information of several stock text data to obtain at least one second cluster, each second cluster including summary description information of at least one stock text data whose clustering distance is less than a first distance threshold; determine the topic corresponding to the second cluster based on the summary description information of at least one stock text data in each of the second clusters using a pre-trained artificial intelligence model; and generate a clustering table based on each of the second clusters and the corresponding topic, as well as the summary description information of at least one stock text data in each of the second clusters.

[0103] In an exemplary embodiment, the topic determination module is also used to: if there is no topic in the clustering table whose clustering distance with the first cluster is less than a second distance threshold, add the clustering information of the first cluster to the clustering table, the clustering information of the first cluster including summary description information of at least one incremental text data in the first cluster; determine the topic corresponding to the first cluster using a pre-trained artificial intelligence model based on the summary description information of at least one incremental text data in the first cluster; and add the topic corresponding to the first cluster to the clustering table.

[0104] In an exemplary embodiment, the topic determination module is further configured to: if there is a topic in the clustering table whose clustering distance with the first cluster is less than a second distance threshold, determine the topic with the smallest clustering distance as the target topic of the first cluster.

[0105] In an exemplary embodiment, the summary extraction module is also used to: perform data segmentation on the text data to be extracted based on a preset data length to obtain at least one data block and a sequence number of each data block; use a pre-trained artificial intelligence model to obtain summary information of each data block; merge the summary information of each data block according to the sequence number of each data block to obtain merged summary data; when it is determined that the length of the merged summary data is less than or equal to the preset data length, use the merged summary data as the summary description information of the text data to be extracted; when it is determined that the length of the merged summary data is greater than the preset data length, use the merged summary data as the text data to be extracted, and return to execute the step of performing data segmentation on the text data to be extracted based on the preset data length.

[0106] In an exemplary embodiment, the first clustering process adopts a clustering algorithm based on union-find; and the second clustering process adopts a vector similarity matching algorithm based on FAISS.

[0107] Each module in the above topic detection device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0108] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal or a server. The internal structure diagram thereof may be as follows: Figure 6As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a topic detection method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0109] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0110] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0111] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0112] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0113] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0114] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0115] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0116] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A topic detection method, characterized in that: The method comprises: When receiving the incremental text data to be detected, extracting a summary of the incremental text data to obtain summary description information of the incremental text data; If there are multiple incremental text data, perform a first clustering process on the summary description information of the multiple incremental text data to obtain at least one first cluster, each first cluster including summary description information of at least one incremental text data whose clustering distance is less than a first distance threshold; Performing a second clustering process on each first cluster and a plurality of topics in a pre-acquired clustering table, and if there is a topic in the clustering table whose clustering distance with the first cluster is less than a second distance threshold, determining a target topic of the first cluster based on the topic; The target topic is assigned to at least one incremental text data in the first cluster.

2. The method according to claim 1, characterized in that Before performing a second clustering process on each first cluster and a plurality of topics in the pre-acquired clustering table, the method further includes: Extract summary of several stock text data to be detected, and obtain summary description information of each stock text data; Performing a first clustering process on the summary description information of the plurality of stock text data to obtain at least one second cluster, each second cluster including summary description information of at least one stock text data whose clustering distance is less than a first distance threshold; Determine a topic corresponding to the second cluster using a pre-trained artificial intelligence model according to summary description information of at least one stock text data in each of the second clusters; A clustering table is generated based on each of the second clusters and the corresponding topic, and summary description information of at least one stock text data in each of the second clusters.

3. The method according to claim 2, characterized in that The method further comprises: If there is no topic in the clustering table whose clustering distance from the first cluster is less than a second distance threshold, adding clustering information of the first cluster to the clustering table, wherein the clustering information of the first cluster includes summary description information of at least one incremental text data in the first cluster; Determine a topic corresponding to the first cluster using a pre-trained artificial intelligence model according to summary description information of at least one incremental text data in the first cluster; Add the topic corresponding to the first cluster to the clustering table.

4. The method according to claim 1, characterized in that If there is a topic in the clustering table whose clustering distance from the first cluster is less than a second distance threshold, determining a target topic of the first cluster based on the topic includes: If there is a topic in the clustering table whose clustering distance with the first cluster is less than a second distance threshold, the topic with the smallest clustering distance is determined as the target topic of the first cluster.

5. The method according to any one of claims 1 to 4, characterized in that: The abstract extraction includes: Based on a preset data length, the text data to be extracted is segmented to obtain at least one data block and a sequence number of each data block; Use pre-trained AI models to obtain summary information for each data block; Merging the summary information of each data block according to the sequence number of each data block to obtain merged summary data; In the case where it is determined that the length of the combined summary data is less than or equal to the preset data length, using the combined summary data as summary description information of the text data to be extracted; When it is determined that the length of the combined summary data is greater than the preset data length, the combined summary data is used as the text data to be extracted, and the step of segmenting the text data to be extracted based on the preset data length is returned to be executed.

6. The method according to any one of claims 1 to 4, characterized in that: The first clustering process adopts a clustering algorithm based on union-find; the second clustering process adopts a vector similarity matching algorithm based on FAISS.

7. A topic detection device, characterized in that: The device comprises: A summary extraction module is used to extract the summary of the incremental text data when receiving the incremental text data to be detected, so as to obtain summary description information of the incremental text data; A first clustering module is used for performing a first clustering process on the summary description information of the plurality of incremental text data if there are a plurality of the incremental text data to obtain at least one first cluster, each of the first clusters including the summary description information of at least one incremental text data whose clustering distance is less than a first distance threshold; A second clustering module, configured to perform a second clustering process on each first cluster and a plurality of topics in a pre-acquired clustering table, and if there is a topic in the clustering table whose clustering distance with the first cluster is less than a second distance threshold, determine a target topic of the first cluster based on the topic; The topic determination module is used to assign the target topic to at least one incremental text data in the first cluster.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images