Text data processing method, apparatus, device, storage medium, and program product

Abstract

This disclosure provides a text data processing method, apparatus, device, storage medium, and program product, relating to the field of big data processing. The method includes: acquiring incremental text data and historical clustering data; the incremental text data includes at least one text data item; performing incremental clustering processing on the text data items based on historical clusters in the historical clustering data, matching historical clusters or creating new clusters for each text data item, and generating corresponding cluster topics for clusters that meet preset conditions, thereby obtaining incremental clustering results; and updating historical clustering data according to the incremental clustering results. This method can effectively integrate newly added text data, dynamically adjust the clustering results as data is updated, avoid the high overhead of full re-clustering, and enhance the interpretability of clustering results by automatically generating cluster topics. It achieves efficient dynamic clustering of incremental text data, improving the automation and intelligence level of data management.

Description

Technical Field

[0001] This disclosure relates to the field of big data processing, and in particular to a text data processing method, apparatus, device, storage medium, and program product. Background Technology

[0002] With the development of computer technology and natural language processing technology, the Internet continues to generate massive amounts of text data, and hot topic mining has become an important research direction in the fields of natural language processing and information retrieval. Among them, the accuracy of topic clustering and the generation of hot topics based on clustering results affect the practicality of topic clusters.

[0003] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0004] The purpose of this disclosure is to provide a text data processing method, apparatus, device, storage medium, and program product.

[0005] According to a first aspect of the present disclosure, a text data processing method is provided, comprising: acquiring incremental text data and historical clustering data; the incremental text data including at least one text data item; performing incremental clustering processing on the text data item based on historical clusters in the historical clustering data, matching historical clusters or creating new clusters for each text data item, and generating corresponding cluster topics for clusters that meet preset conditions, thereby obtaining incremental clustering results; and updating the historical clustering data according to the incremental clustering results.

[0006] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects: This disclosure can reuse historical clustering results, dynamically match existing clusters or create new clusters for each new text, and automatically generate semantically clear cluster topics for mature clusters that meet the conditions, continuously optimizing and enriching the knowledge structure of the clustering results. Therefore, this method can effectively integrate new text data, dynamically adjust the clustering results as the data is updated, avoid the high overhead of full re-clustering, and enhance the interpretability of the clustering results by automatically generating cluster topics. It achieves efficient dynamic clustering of incremental text data, improving the automation and intelligence level of data management.

[0007] In some implementations, incremental clustering is performed on the text data items based on historical clusters in the historical clustering data to match historical clusters or create new clusters for each text data item. This includes: determining the cluster features of the historical clusters and converting each text data item into a corresponding text feature; determining the similarity between the text feature and the cluster feature; and matching historical clusters or creating new clusters for each text data item based on the relationship between the similarity and a similarity threshold.

[0008] In the above implementation, the similarity between the text and the historical cluster features can be calculated, and the cluster to which each newly added text belongs can be determined based on the threshold. This provides a clear matching logic and a new cluster generation mechanism for the incremental clustering process, improving the standardization and operability of clustering.

[0009] In some implementations, determining the similarity between the text feature and the cluster feature includes: mapping the text feature and each cluster feature to a hash bucket based on a locality-sensitive hashing algorithm; determining candidate historical clusters based on the hash bucket where the text feature is located; and determining the similarity between the text feature and the cluster features of each candidate historical cluster.

[0010] In the above implementation, locality-sensitive hashing can be used to quickly map multiple historical clusters and the currently added text to their respective hash buckets, thereby quickly filtering out a small number of candidate clusters for the currently added text from a massive number of historical clusters for similarity calculation, which greatly improves the efficiency of similarity calculation while ensuring approximate accuracy.

[0011] In some implementations, matching historical clusters or creating new clusters for each text data item based on the relationship between the similarity and the similarity threshold includes: matching the text data item with the historical cluster with the highest similarity in response to the existence of a similarity exceeding the similarity threshold; and creating a new cluster for the text data item in response to the absence of a similarity exceeding the similarity threshold.

[0012] In the above implementation, new text can be matched with historical clusters with similarity higher than a threshold to ensure clustering consistency, and new clusters can be created when there is no similarity higher than the threshold, thus ensuring the stability and scalability of the clustering structure.

[0013] In some implementations, the incremental text data is divided into at least one batch for incremental clustering processing; the method further includes: after completing the incremental clustering processing of the text data items in the current batch, determining the proportion of data items in the text data items in the current batch that match historical clusters; adjusting the similarity threshold according to the proportion of data items, and using the adjusted similarity threshold for the incremental clustering processing of the next batch of text data items.

[0014] In the above implementation, incremental text data can be processed in batches, and the similarity threshold can be dynamically adjusted according to the successful matching ratio between the previous batch of data and historical clusters, so that the system can adapt to changes in data distribution, optimize the clustering process, and improve the stability of long-term clustering results.

[0015] In some implementations, adjusting the similarity threshold based on the proportion of the data items includes: increasing the similarity threshold in response to the proportion of the data items being higher than a first proportion threshold; decreasing the similarity threshold in response to the proportion of the data items being lower than a second proportion threshold; wherein the second proportion threshold is lower than the first proportion threshold.

[0016] In the above implementation, a similarity threshold adjustment method is provided. When the data matching ratio is too high, the similarity threshold can be increased to promote the formation of new clusters and prevent over-clustering from causing themes to mix within the clusters. When the data matching ratio is too low, the similarity threshold can be decreased to promote the clustering of data into historical clusters and avoid cluster fragmentation. This achieves dynamic adaptive adjustment of the similarity threshold and a sensitive balance between clustering accuracy.

[0017] In some implementations, the preset conditions include: the number of text data items within a cluster reaches a threshold, and the cluster is not associated with a cluster topic.

[0018] In the above implementation, the triggering conditions for cluster topic generation can be clearly defined. Topic generation is triggered only when the number of data items in a cluster reaches a threshold and there is no associated topic, thus avoiding redundant calculations and ensuring that the cluster topics have statistical significance and practical value.

[0019] In some implementations, generating a corresponding cluster topic for a cluster that meets preset conditions includes: generating a corresponding cluster topic based on the text data item content within the cluster that meets the preset conditions using a semantic generation model.

[0020] In the above implementation, a semantic generation model can be used to automatically extract topics based on intra-cluster text, thereby enhancing the quality of clustering results and facilitating user understanding and subsequent analysis.

[0021] In some implementations, generating a corresponding cluster topic based on the text data item content within the cluster that meets the preset conditions through a semantic generation model includes: constructing a prompt word, the prompt word including a system instruction for defining cluster topic generation rules and a user instruction containing the text data item content within the cluster; constructing a topic generation request based on the prompt word and sending the topic generation request to the server where the semantic generation model is located; and determining the text content returned by the server as the cluster topic of the cluster that meets the preset conditions.

[0022] In the above implementation, the accuracy and consistency of cluster topics can be improved by constructing prompt words that include system instructions and user content to call a large model, standardizing the topic generation format and quality.

[0023] According to a second aspect of the present disclosure, a text data processing apparatus is provided, comprising: an acquisition unit, configured to acquire incremental text data and historical clustering data; the incremental text data including at least one text data item; a clustering unit, configured to perform incremental clustering processing on the text data item based on historical clusters in the historical clustering data, matching historical clusters or creating new clusters for each text data item, and generating corresponding cluster topics for clusters that meet preset conditions, thereby obtaining incremental clustering results; and an update unit, configured to update the historical clustering data according to the incremental clustering results.

[0024] In some implementations, the clustering unit performs incremental clustering processing on the text data items based on historical clusters in the historical clustering data, matching historical clusters or creating new clusters for each text data item, including: determining the cluster features of the historical clusters and converting each text data item into corresponding text features; determining the similarity between the text features and the cluster features; and matching historical clusters or creating new clusters for each text data item based on the relationship between the similarity and a similarity threshold.

[0025] In some implementations, the clustering unit determines the similarity between the text features and the cluster features by: mapping the text features and each cluster feature to a hash bucket based on a locality-sensitive hashing algorithm; determining candidate historical clusters based on the hash bucket where the text features are located; and determining the similarity between the text features and the cluster features of each candidate historical cluster.

[0026] In some implementations, the clustering unit matches historical clusters or creates new clusters for each text data item based on the relationship between the similarity and the similarity threshold, including: matching the text data item with the historical cluster with the highest similarity in response to the existence of a similarity exceeding the similarity threshold; and creating a new cluster for the text data item in response to the absence of a similarity exceeding the similarity threshold.

[0027] In some implementations, the incremental text data is divided into at least one batch for incremental clustering processing; the clustering unit is further configured to: after completing the incremental clustering processing of the text data items in the current batch, determine the proportion of data items in the text data items in the current batch that match historical clusters; adjust the similarity threshold according to the proportion of data items, and use the adjusted similarity threshold for the incremental clustering processing of the next batch of text data items.

[0028] In some implementations, the clustering unit adjusts the similarity threshold based on the proportion of the data items, including: increasing the similarity threshold in response to the proportion of the data items being higher than a first proportion threshold; decreasing the similarity threshold in response to the proportion of the data items being lower than a second proportion threshold; wherein the second proportion threshold is lower than the first proportion threshold.

[0029] In some implementations, the preset conditions include: the number of text data items within a cluster reaches a threshold, and the cluster is not associated with a cluster topic.

[0030] In some implementations, the clustering unit generates corresponding cluster topics for clusters that meet preset conditions, including: generating corresponding cluster topics based on the text data item content within the clusters that meet the preset conditions through a semantic generation model.

[0031] In some implementations, the clustering unit generates corresponding cluster topics based on the text data items within the clusters that meet preset conditions using a semantic generation model. This includes: constructing a prompt word, which includes system instructions for defining cluster topic generation rules and user instructions containing the text data items within the clusters; constructing a topic generation request based on the prompt word and sending the topic generation request to the server where the semantic generation model is located; and determining the text content returned by the server as the cluster topic of the clusters that meet the preset conditions.

[0032] According to a third aspect of the present disclosure, an electronic device is provided, characterized in that it includes: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the text data processing method described above.

[0033] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, which, when the instructions in the storage medium are executed by a processor of a mobile terminal, enables the mobile terminal to perform the text data processing method described above.

[0034] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the text data processing method described above.

[0035] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0036] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0037] Figure 1This is a flowchart illustrating a text data processing method according to some embodiments of the present disclosure.

[0038] Figure 2 This is a flowchart illustrating incremental clustering processing in a text data processing method according to some embodiments of the present disclosure.

[0039] Figure 3 This is a schematic diagram illustrating an incremental clustering process in a text data processing method according to some embodiments of the present disclosure.

[0040] Figure 4 This is a flowchart illustrating the generation of cluster topics in a text data processing method according to some embodiments of the present disclosure.

[0041] Figure 5 This is a schematic diagram of the system architecture of a text data processing method according to some embodiments of the present disclosure.

[0042] Figure 6 This is a flowchart illustrating a text data processing method according to some embodiments of the present disclosure.

[0043] Figure 7 This is a block diagram illustrating a text data processing apparatus according to some embodiments of the present disclosure.

[0044] Figure 8 This is a block diagram illustrating an apparatus for text data processing according to some embodiments of the present disclosure. Detailed Implementation

[0045] Exemplary embodiments of this disclosure will be described in detail herein, examples of which are illustrated in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings denote the same or similar elements unless otherwise indicated. Various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent upon understanding this disclosure. For example, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein, but can be changed as will become apparent upon understanding this disclosure, except for operations that must be performed in a particular order. Furthermore, for clarity and brevity, descriptions of features known in the art may be omitted.

[0046] The flowchart shown in the attached diagram is merely an illustrative example and does not necessarily include all content and steps, nor does it necessarily have to be executed in the described order or in the order of the step numbers. For example, some steps can be broken down, while others can be combined or partially combined, and multiple steps can have their order interchanged or be executed simultaneously. Therefore, the actual execution order may change depending on the actual situation.

[0047] The embodiments described below, which are examples of some of the embodiments of this disclosure, do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0048] In some embodiments of this disclosure, the acquisition of data and information, as well as the collection, updating, analysis, processing, use, transmission, and storage of related user personal information, may comply with the laws and regulations of the country where the location is situated.

[0049] In some embodiments of this disclosure, data or information may be obtained or collected after obtaining the user's consent.

[0050] The specific implementation methods of the embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0051] Figure 1 This is a flowchart illustrating a text data processing method according to some embodiments of the present disclosure, such as... Figure 1 As shown, the text data processing method can be applied to electronic devices, including but not limited to terminal devices such as smartphones, smart tablets, wearable devices, desktop computers, laptops, and smart speakers. It can also include server-side components such as local servers and cloud servers, which can be deployed on a single computer or a computer cluster consisting of multiple computers. The text data processing method may include the following steps.

[0052] In step S110, incremental text data and historical clustering data are obtained; the incremental text data includes at least one text data item.

[0053] In this embodiment, newly added text data and existing clustering data can be obtained from the storage system related to the data platform. The data platform may be, for example, an instant messaging topic group, a social media platform, a news website, or a user comment system. The aforementioned user-related data is obtained with the user's consent.

[0054] Incremental text data can be a collection of newly arrived texts within a preset time period in a data platform that have not yet undergone clustering. The preset time period could be, for example, 24 hours, 12 hours, or 6 hours from the current time. Text data items in incremental text data can include instant messaging group chat messages, social media posts or comments, news headlines or paragraphs, user reviews of products or services, etc.

[0055] Historical clustering data can be existing clustering results. For example, in the first run scenario, historical clusters may not exist in the historical clustering data; in subsequent run scenarios, historical clustering data can include the results of previous clustering processes, such as one or more historical clusters, each of which can represent a set of texts with similar semantic features.

[0056] In an exemplary embodiment, historical clusters in historical clustering data can be associated with corresponding original texts, text source identifiers (such as the user identifier that published the text), cluster feature vectors, cluster topics, etc.

[0057] For example, for a data platform like instant messaging topic groups, incremental text data can be a set of chat statements added on the same day, and a text data item can be a text record to be clustered, which can include one or more chat statements; historical clustering data can be topic clusters that have been formed in the past.

[0058] In step S120, incremental clustering is performed on the text data items based on the historical clusters in the historical clustering data. Historical clusters are matched or new clusters are created for each text data item, and corresponding cluster topics are generated for clusters that meet preset conditions, so as to obtain incremental clustering results.

[0059] In this embodiment of the disclosure, during the first run, if historical clusters in the historical clustering data are missing, a new cluster can be created for the text data item. In subsequent runs, newly added text data items can be classified based on existing historical clustering results. Each text data item can be compared with historical clusters, and through similarity calculation or clustering algorithms, each text data item can be assigned to the most similar historical cluster or a new cluster can be created to accommodate data items that cannot be matched.

[0060] For example, if there are existing historical clusters such as "sports news" and "technology news", and a new text data item about a sports event is added, if the text data item has a high similarity to the "sports news" cluster, it can be assigned to that cluster; if a new text data item about biotechnology news is added, and it is found that it has a low similarity to the existing clusters, a new cluster can be created to accommodate the biotechnology news.

[0061] In this embodiment of the disclosure, the preset condition can be a condition that triggers the generation of cluster topics. After processing the incremental text data, it can be determined whether the existing clusters meet specific preset conditions. If they do, cluster topics describing their content can be generated for the clusters that meet the preset conditions. Cluster topics can be, for example, one or more topic tags or keywords, thereby obtaining the clustering results for this incremental text data.

[0062] The incremental clustering results can include newly formed clusters, updated historical clusters, and their topic information.

[0063] In an exemplary embodiment, cluster topics can be used as hot words for text clusters in downstream systems.

[0064] In some embodiments of this disclosure, the preset conditions include: the number of text data items within a cluster reaches a threshold, and the cluster is not associated with a cluster topic.

[0065] In this embodiment of the disclosure, after clustering is completed, if the number of text data items within a cluster reaches a preset threshold, and the cluster currently has no associated cluster topics, then the cluster can be considered to meet the preset conditions. The threshold can be a preset value (such as 3, 5, 10, etc.), or a value dynamically adjusted based on the data scale of the data platform.

[0066] For example, if the quantity threshold is 10, after clustering a batch of incremental text data, a new cluster about "biotechnology news" is formed, with 20 data items (such as 20 articles). Since this cluster is newly generated in this clustering, it does not yet have a theme. In this case, theme generation can be automatically triggered to extract the corresponding keywords as the cluster theme.

[0067] For example, if the threshold is 10, and a new cluster with 5 data items is generated after the first clustering, topic generation for that cluster will not be triggered after the first clustering. A cluster identifier "a" can be assigned to it. Then, during the second clustering, the cluster corresponding to identifier "a" can be used as a historical cluster to attempt matching with the incremental text data from the second clustering. If, after the second clustering, it is found that the cluster corresponding to identifier "a" has 15 data items, then topic generation for that cluster can be triggered.

[0068] Through the embodiments of this disclosure, the triggering conditions for cluster topic generation can be clearly defined. Topic generation is triggered only when the number of data items within a cluster reaches a threshold and there is no associated topic, thus avoiding redundant calculations and ensuring that cluster topics have statistical significance and practical value.

[0069] In an exemplary embodiment, the preset conditions may also be time-related, such as determining whether the interval between the topic update time of a cluster and the current time reaches a time period threshold (such as one month, two months, etc.). If it does, the topic of the cluster can be regenerated.

[0070] In an exemplary embodiment, the preset conditions may also include a combination of time and data increment within a cluster. For example, it may be determined whether the interval between the topic update time of a cluster and the current time reaches a time period threshold, and whether the number of newly added data items within the cluster during the interval reaches a new addition threshold. If both are determined to be yes, the topic of the cluster may be regenerated.

[0071] In step S130, the historical clustering data is updated based on the incremental clustering results.

[0072] In this embodiment of the disclosure, after the incremental clustering process is completed, the new clustering results can be integrated into the historical clustering data to maintain data integrity and timeliness. The update operation may include adding the newly created clusters to the historical cluster set, updating the relevant information of existing clusters (such as updating the number of texts within a cluster, updating the cluster topic, merging incremental data and historical data, etc.). The updated historical clustering data will contain the newly generated clusters and related information for use in the next incremental data processing.

[0073] For example, the "Biotechnology News" cluster created in the aforementioned embodiments can be added to the historical clustering data, and its relevant information can be updated.

[0074] In an exemplary embodiment, incremental text data and historical text data can be merged to form a complete dataset, and the merged data can be written into a new time partition of the storage space where the historical cluster data is located.

[0075] In an exemplary embodiment, the historical clustering data, once updated, can be used for querying or application in downstream systems. For example, cluster topics can be used to display trending topics lists, and cluster popularity and growth rates can be statistically analyzed. Clusters of topics that have seen a short-term increase in popularity can be considered topics of interest to users. They can also be used for problem governance, merging new issues into existing problem clusters for unified management. Furthermore, they can be used for knowledge accumulation within topic groups, using cluster topics as a tagging system to support user retrieval, system recommendations, and automated responses within topic groups.

[0076] As can be seen from the above steps, the text data processing method provided in this disclosure can reuse historical clustering results, dynamically match existing clusters or create new clusters for each new text, and automatically generate semantically clear cluster topics for mature clusters that meet the conditions, continuously optimizing and enriching the knowledge structure of the clustering results. Therefore, this method can effectively integrate new text data, dynamically adjust the clustering results as the data is updated, avoid the high overhead of full re-clustering, and enhance the interpretability of the clustering results by automatically generating cluster topics. It achieves efficient dynamic clustering of incremental text data, improving the automation and intelligence level of data management.

[0077] In an exemplary embodiment, in response to the existence of incremental text data, historical clustering data can be retrieved, and then incremental clustering processing can be performed on the incremental text data. If no incremental text data exists, clustering and data updates can be omitted, and the processing time of historical clustering data can be updated to indicate "calculation completed, no new data added," ensuring the continuity of system operation.

[0078] In an exemplary embodiment, in the absence of incremental text data, the historical original data corresponding to the historical clustering data can be written into a new time partition in the storage space where the historical clustering data is located, so as to realize the processing time of updating the historical clustering data.

[0079] Figure 2 This is a flowchart illustrating incremental clustering processing in a text data processing method according to some embodiments of the present disclosure.

[0080] like Figure 2 As shown, in some embodiments of this disclosure, the process of performing incremental clustering on the text data items based on historical clusters in the historical clustering data, and matching historical clusters or creating new clusters for each text data item, may include the following steps.

[0081] Step S210: Determine the cluster features of the historical clusters and convert each text data item into the corresponding text features.

[0082] In this embodiment, the cluster features of historical clusters can be obtained by comprehensively analyzing the features of all text data items within a cluster. Specifically, the cluster features of each cluster can be calculated and stored after the previous clustering process and directly retrieved for use in the current clustering process, or the cluster features can be calculated and used during each clustering process. Simultaneously, each incremental text data item to be processed can be individually converted into structured text features. In an exemplary embodiment, keyword or topic vectors can be extracted as cluster features using methods such as word frequency statistics and semantic processing models, and text data items can be converted into corresponding text features using the same method, so that cluster features and text features are mapped to the same vector space.

[0083] Step S220: Determine the similarity between the text features and the cluster features.

[0084] In this embodiment of the disclosure, after obtaining the text features and cluster features, the similarity between the text features and the cluster features of historical clusters can be quantified for each text data item's text features based on a preset strategy and / or a preset similarity calculation function. The similarity calculation function can be, for example, cosine similarity, Euclidean distance, Jaccard similarity coefficient, etc. A similarity closer to 1 indicates a higher correlation between the text data item and the cluster.

[0085] Step S230: Based on the relationship between the similarity and the similarity threshold, match historical clusters or create new clusters for each text data item.

[0086] In this embodiment of the disclosure, the similarity threshold is a value used in the current clustering process to determine whether the similarity between a text data item and a historical cluster is high enough. If the similarity between a text feature and the cluster feature of a certain historical cluster is greater than or equal to the similarity threshold, it indicates that the text data item has a high similarity to this historical cluster, and the text data item can be matched into this historical cluster; if the similarity between a text feature and the cluster feature of the compared historical cluster is less than the similarity threshold, it indicates that the text data item is not similar enough to the existing historical clusters, and a new cluster can be created for it.

[0087] For example, assuming the current similarity threshold is 0.5, for a chat text related to a basketball game, if its similarity to the "volleyball", "basketball", and "football" clusters is calculated to be greater than 0.5, and the chat text has the highest similarity to the "basketball" cluster, then the text can be matched with the historical cluster with the highest similarity, that is, classified into the "basketball" cluster; if the similarity between the chat text and the calculated historical clusters is less than 0.5, then a new cluster can be created for it.

[0088] Through the embodiments of this disclosure, the similarity between text and historical cluster features can be calculated, and the cluster to which each newly added text belongs can be determined based on a threshold. This provides a clear matching logic and a new cluster generation mechanism for the incremental clustering process, improving the standardization and operability of clustering.

[0089] In some embodiments of this disclosure, determining the similarity between the text feature and the cluster feature includes: mapping the text feature and each cluster feature to a hash bucket based on the locality-sensitive hashing algorithm; determining candidate historical clusters based on the hash bucket where the text feature is located; and determining the similarity between the text feature and the cluster features of each candidate historical cluster.

[0090] In this embodiment, the cluster features of each historical cluster and the text features of newly added text data items can be input into the same set of locality-sensitive hash functions to generate a hash code for each cluster feature. This hash code can directly correspond to a hash bucket. In the application of the locality-sensitive hash algorithm, objects with similar feature vectors can be mapped to the same or adjacent hash buckets with a high probability.

[0091] After mapping is complete, the hash bucket to which the current text feature belongs can be located, and all historical clusters that were also mapped to that bucket or mapped to that bucket and adjacent buckets can be filtered out. These filtered historical clusters can be used as candidate historical clusters. In this way, the number of historical clusters that need to be precisely calculated for similarity with the current text feature can be significantly reduced, transforming the global search into a local search.

[0092] A similarity calculation function can be used to calculate the similarity value between the current text features and the cluster features of each candidate historical cluster. This value can then be used to match candidate historical clusters or create new clusters for the current text data item. This ensures that efficiency is improved while maintaining the accuracy of similarity calculations within a critical range.

[0093] In an exemplary embodiment, a local sensitive hash index (such as the form of "cluster vector → hash bucket") can be established for each historical cluster vector. Then, the current text features are mapped to the corresponding bucket, and clusters in the same bucket and neighboring buckets are recalled as candidate historical clusters. Finally, a precise similarity calculation is performed.

[0094] In an exemplary embodiment, parallel computation can be used to calculate the similarity between each candidate historical cluster and the current text features. Parallel computation can significantly improve computational efficiency and greatly shorten the processing time for similarity calculation.

[0095] Through the embodiments of this disclosure, multiple historical clusters and the currently added text can be quickly mapped to their respective hash buckets using locality-sensitive hashing. This allows for the rapid selection of a small number of candidate clusters for the currently added text from a massive number of historical clusters for similarity calculation, thereby significantly improving the efficiency of similarity calculation while ensuring approximate accuracy.

[0096] In some embodiments of this disclosure, the step of matching historical clusters or creating new clusters for each text data item based on the relationship between the similarity and the similarity threshold includes: matching the text data item with the historical cluster with the highest similarity in response to the existence of a similarity exceeding the similarity threshold; and creating a new cluster for the text data item in response to the absence of a similarity exceeding the similarity threshold.

[0097] In this embodiment of the disclosure, if there is a similarity score exceeding the similarity threshold, it means that the current text data item has a certain similarity to some historical clusters, and the current text data item can be classified into a certain historical cluster. The maximum similarity score exceeding the similarity threshold can be found from the calculated similarity scores, and the text data item can be matched into the historical cluster corresponding to this maximum similarity score.

[0098] If none of the calculated similarity values ​​exceed the similarity threshold, it means that the text data item is not similar enough to any existing historical cluster, and therefore cannot be reasonably assigned to any existing historical cluster. In this case, a completely new cluster can be created for this text data item for subsequent management and analysis. For example, a completely new cluster can be initialized using this text data item as the seed or core.

[0099] The calculated similarity can be calculated by comparing the text features of the text data item with the cluster features of each candidate historical cluster.

[0100] Through the embodiments of this disclosure, new text can be matched with historical clusters with similarity higher than a threshold to ensure clustering consistency, and new clusters can be created when there is no similarity higher than the threshold, thus ensuring the stability and scalability of the clustering structure.

[0101] In some embodiments of this disclosure, the incremental text data is divided into at least one batch for incremental clustering processing; the method further includes: after completing the incremental clustering processing of the text data items in the current batch, determining the proportion of data items in the text data items in the current batch that match historical clusters; adjusting the similarity threshold according to the proportion of data items, and using the adjusted similarity threshold for the incremental clustering processing of the next batch of text data items.

[0102] In this embodiment of the disclosure, when processing incremental text data, the entire incremental text data can be divided into at least one batch according to a preset strategy. The preset strategy may be, for example, dividing by a fixed number, dividing by the time the data was generated, or dividing by the channel type of the data generation (such as different topic groups). Each batch may contain a certain number of text data items, and then incremental clustering processing can be performed on each batch in sequence.

[0103] For example, assuming the incremental text data has 5,000 text data items, and we set each batch to process 500 text data items, then the incremental text data can be divided into 10 batches, and then clustered batch by batch.

[0104] After performing incremental clustering (such as matching historical clusters or creating new clusters) on all text data items in a batch, the number of text data items that successfully matched historical clusters in that batch can be counted, and then the proportion of this number to the total number of text data items in the current batch can be calculated. This proportion can reflect the degree of association between the newly added text data in the current batch and historical clusters, or the degree of integration between the newly added text data in the current batch and the existing knowledge structure. For example, if the current batch has 500 text data items, and 300 of them successfully matched historical clusters, then the proportion of data items that matched historical clusters is 60%.

[0105] Next, the similarity threshold can be adjusted according to predefined adjustment rules (such as table lookup, formula calculation, etc.). The new threshold generated by the adjustment can be applied to the next batch of incremental clustering processing, realizing adaptive dynamic optimization of the similarity threshold.

[0106] In an exemplary embodiment, the incremental text data can be divided into multiple batches, and the similarity threshold can be adjusted as in this embodiment during the processing of the incremental text data. Alternatively, the incremental text data can be divided into a single batch, in which case the similarity threshold can be adjusted based on the processing of the current incremental text data, and the similarity threshold can be used in the next batch of incremental text data.

[0107] Through the embodiments of this disclosure, incremental text data can be processed in batches, and the similarity threshold can be dynamically adjusted according to the successful matching ratio between the previous batch of data and historical clusters, so that the system can adapt to changes in data distribution, optimize the clustering process, and improve the stability of long-term clustering results.

[0108] In some embodiments of this disclosure, adjusting the similarity threshold based on the proportion of the data items includes: increasing the similarity threshold in response to the proportion of the data items being higher than a first proportion threshold; decreasing the similarity threshold in response to the proportion of the data items being lower than a second proportion threshold; wherein the second proportion threshold is lower than the first proportion threshold.

[0109] In this embodiment of the disclosure, the first ratio threshold can be a pre-set upper limit constant used to determine whether the matching ratio is too high. When the ratio of matching data items exceeds this value, an adjustment operation to increase the similarity threshold can be triggered. The first ratio threshold can maintain the purity of clusters and promote the discovery of new clusters.

[0110] The second ratio threshold can be a pre-set lower limit constant used to determine whether the matching ratio is too low. When the ratio of matching data items falls below this value, an adjustment operation to lower the similarity threshold can be triggered. The second ratio threshold can ensure the continuity of historical clusters and the efficiency of knowledge accumulation.

[0111] In an exemplary embodiment, the first proportional threshold may be a value such as 0.7 or 0.8, and the second proportional threshold may be a value such as 0.3 or 0.4.

[0112] In this embodiment of the disclosure, the similarity threshold can be increased or decreased according to a preset adjustment step size, such as 0.01, 0.02, etc. For example, assuming the similarity threshold is 0.7, the first proportion threshold is set to 0.8, and the adjustment step size is 0.02; if the proportion of data items matching historical clusters in the current batch is 0.85, since 0.85 > 0.8, the similarity threshold can be increased from 0.7 to 0.72, and 0.72 can be used as the similarity threshold for clustering the next batch of incremental text data.

[0113] The present invention provides a method for adjusting the similarity threshold. When the data matching ratio is too high, the similarity threshold can be increased to promote the formation of new clusters and prevent over-clustering that leads to mixed topics within the cluster. When the data matching ratio is too low, the similarity threshold can be decreased to promote the clustering of data into historical clusters and avoid cluster fragmentation. This achieves dynamic adaptive adjustment of the similarity threshold and a sensitive balance between clustering accuracy.

[0114] Figure 3This is a schematic diagram illustrating an incremental clustering process in a text data processing method according to some embodiments of the present disclosure.

[0115] like Figure 3 As shown, in some embodiments of this disclosure, the process of incrementally clustering text data items may include the following.

[0116] (1) Input check: Incremental text data can be checked first to ensure the integrity and validity of the data.

[0117] (2) Clustering information preprocessing: Clustering information in historical clustering data can be prepared, such as organizing and mapping information such as the characteristics, cluster identifiers, and cluster themes of historical clusters. Among them, a memory mapping structure such as "cluster identifier → features / statistics" can be read and organized.

[0118] (3) Traversing input topics: The system can process each topic statement (i.e., text data item) one by one. A topic statement may include one or more chat statements.

[0119] (4) Check if there are clusters in the historical data: First, it can be determined whether there are historical clusters in the historical clustering data. If there are no historical clusters, a new cluster can be initialized for the text data item being processed; if there are historical clusters, the system will perform subsequent similarity calculation and matching steps.

[0120] (5) Use LSH (Locality-Sensitive Hashing) to find candidate clusters: When there are historical clusters in the historical clustering data, the system can use LSH technology to find candidate clusters. By mapping the data to a low-dimensional space, clusters that may be similar to the current text data items can be quickly filtered out, thereby reducing the amount of computation.

[0121] (6) Calculate the similarity between the current topic and the candidate clusters: The system can use parallel computing to calculate the similarity between each candidate cluster and the current topic (i.e., the current text data item). Parallel computing can significantly improve computational efficiency, for example, it can greatly shorten the processing time when processing large amounts of data.

[0122] (7) Determine if the similarity is greater than the threshold: The calculated similarity can be compared with the similarity threshold. If the similarity is greater than the similarity threshold, the current text data item can be assigned to the candidate cluster with the highest similarity to the current topic. If the similarity is lower than the similarity threshold, it means that there is no cluster similar to the current text data item in the candidate cluster, and a new cluster can be created.

[0123] (8) Update clustering results: Update clustering information based on similarity calculation results to reflect the latest clustering status. Updating clustering information may include: storing the correspondence between data items and clusters, updating cluster statistics (such as counting cluster size), and updating cluster features (aggregate update).

[0124] (9) Dynamically Adjusting the Threshold: To address the impact of changes in data distribution, the clustering threshold can be dynamically adjusted after each batch of incremental text data has been traversed. If most data can match historical clusters, it indicates dense clusters, and the threshold can be increased to avoid erroneous aggregation. If most data creates new clusters, it indicates sparse clusters, and the threshold can be decreased to prevent over-splitting.

[0125] Dynamically adjusting the threshold helps improve the flexibility and accuracy of the clustering process, enabling it to respond to different changes in the data.

[0126] The adjustment method could be as follows: if more than 70% of the data matches a historical cluster (i.e., the proportion of data items is greater than 0.7), the threshold increases by 0.01; if less than 30% of the data matches a historical cluster (i.e., the proportion of data items is less than 0.3), the threshold decreases by 0.01; if the proportion of data items matching a historical cluster is between 0.3 and 0.7, the threshold remains unchanged. The threshold T can be calculated as follows: .

[0127] (10) Continue traversal: After processing the current topic (i.e., text data item), the system continues to process the next topic until all incremental text data has been traversed.

[0128] (11) Return clustering results: After all topics (i.e. text data items) have been processed, the system can return the final clustering results for subsequent analysis and use.

[0129] In some embodiments of this disclosure, generating a corresponding cluster topic for a cluster that meets preset conditions includes: generating a corresponding cluster topic based on the text data item content within the cluster that meets the preset conditions using a semantic generation model.

[0130] In this embodiment of the disclosure, the content of all text data items within a cluster that meet preset conditions can be collected as input information, and then this text content can be input into a pre-trained semantic generation model. Based on its deep semantic understanding and summarization capabilities of the input text, the semantic generation model can output one or more natural language descriptions that can summarize the core theme of the cluster, i.e., the cluster topic.

[0131] The semantic generation model can be an artificial intelligence model trained based on a deep learning architecture, capable of understanding the semantics of natural language and generating readable text. For example, a semantic generation model can be a large, trained language model.

[0132] Through the embodiments of this disclosure, a semantic generation model can be used to automatically extract topics based on intra-cluster text, thereby enhancing the quality of clustering results and facilitating user understanding and subsequent analysis.

[0133] Figure 4 This is a flowchart illustrating the generation of cluster topics in a text data processing method according to some embodiments of the present disclosure.

[0134] like Figure 4 As shown, in some embodiments of this disclosure, the process of generating a corresponding cluster topic based on the text data item content within the cluster that meets the preset conditions through a semantic generation model may include the following steps.

[0135] Step S410: Construct a prompt word, the prompt word including system instructions for defining cluster topic generation rules and user instructions containing the content of text data items within the cluster.

[0136] In this embodiment of the disclosure, structured prompt words can be created to drive a semantic generation model, and the prompt words can consist of system instructions and user instructions.

[0137] The system instruction section allows setting global tasks and output constraints for the model, clearly defining the rules and requirements that the generated cluster topics should follow, such as specifying the topic format, length limits, and style characteristics. The user instruction section provides the content input to the model, which can incorporate text data items within clusters that meet preset conditions, allowing the semantic generation model to understand which specific text content to base the topic generation.

[0138] In an exemplary embodiment, the content of the intra-cluster text data item used to construct the user instruction can be several texts, representative texts, or all texts within the cluster. Representative texts can be randomly selected, or selection can be based on a preset mechanism. The preset mechanism could be, for example, selecting multiple texts with a length greater than a length threshold, selecting multiple texts with an evenly distributed generation time, selecting multiple texts from different source channels, and so on.

[0139] In an exemplary embodiment, the system instructions may include the following aspects: (1) Input format: The text cluster consists of multiple semantically related sentences, from which the core meaning needs to be extracted. (2) Output format: The output format is a concise and refined theme that accurately expresses the main content of the cluster. (3) Generation rules: The cluster theme needs to summarize the core content of the cluster, maintain fluency and semantic accuracy. (4) Priority principle: The cluster theme should be concise, abstract, consistently applicable, and neutral. (5) Restrictions: The cluster theme is extracted from the text, with no more than 10 words, ensuring semantic accuracy and close relevance to the cluster content.

[0140] In an exemplary embodiment, the user instruction may include the following aspects: (1) Input structure: the provided text from which a topic is to be generated; (2) Output structure: the format of the output topic, which should concisely and accurately summarize the core content of the text.

[0141] Step S420: Construct a topic generation request based on the prompt words and send the topic generation request to the server where the semantic generation model is located.

[0142] In this embodiment of the disclosure, The complete prompt words constructed in the previous step are encapsulated according to the communication protocol (such as HTTP or gRPC) and data structure (usually JSON format) agreed upon with the server to form a formal "topic generation request". Subsequently, this request is sent over the network to the backend server (server) hosting the semantic generation model to request it to perform inference calculations.

[0143] The constructed system instructions and user instructions can be encapsulated according to a pre-defined communication protocol (such as HTTP) and data structure to form a topic generation request. This request can then be sent over the network to the backend server hosting the semantic generation model to invoke the model for inference calculation.

[0144] In an exemplary embodiment, system instructions and user instructions can be encapsulated in the request body of an HTTP POST request. The request body may include parameters such as the following: (1) Model: The name of the large model to be called. In this embodiment, the large model is a replaceable component, and different large models can be selected. Different models can be adapted to be selected based on factors such as cost, latency, language coverage, and industry corpus. (2) Message structure: Contains the text content of the user role and the instructions of the system role. (3) Generation parameters: Set the temperature and probability filtering parameters (top_p) to ensure the stability and quality of the generated results.

[0145] Step S430: Determine the text content returned by the server as the cluster topic of the cluster that meets the preset conditions.

[0146] In this embodiment, the system can receive and parse the server's response to the request to obtain natural language text content generated by the semantic generation model based on prompt words. This text content can then be directly adopted and bound as a cluster topic for a cluster that meets preset conditions.

[0147] In an exemplary embodiment, if the request fails, the server can return an exception and output an error message. The request can then be reconstructed using a backup model for cluster topic generation, or a cluster topic can be generated based on other semantic extraction rules, or the server can be re-requested at a preset time to generate a cluster topic.

[0148] Through the embodiments of this disclosure, a large model can be constructed by invoking prompt words that include system instructions and user content, thereby standardizing the topic generation format and quality and improving the accuracy and consistency of cluster topics.

[0149] Figure 5 This is a schematic diagram of the system architecture of a text data processing method according to some embodiments of the present disclosure.

[0150] like Figure 5 As shown, the system architecture of the text data processing method may include: a scheduling and monitoring layer, a data processing layer, and a data storage layer.

[0151] The scheduling and monitoring layer manages the timed scheduling of data processing tasks and the recording of the entire process logs, facilitating monitoring and subsequent troubleshooting. The data processing layer is responsible for data acquisition (including incremental text data and historical clustering data), cluster processing, initialization and updating of cluster information, data merging, and cluster topic generation. The data storage layer can use a database to store data to be processed, historical data, and clustering information.

[0152] Figure 5 Other aspects of the embodiments can be found in the other embodiments described above.

[0153] Figure 6 This is a flowchart illustrating a text data processing method according to some embodiments of this disclosure. Figure 6 As shown, the process of a text data processing method may include the following.

[0154] (1) Data acquisition.

[0155] Obtaining incremental data: Incremental text data to be processed can be obtained from table 1 in the database. For example, data from day T-1 can be selected for processing on day T. Table 1 can be the source table for incremental data, containing incremental data provided by the business side, which may include the original text of the topic and text identifiers.

[0156] Obtaining historical clustering data: Previously processed historical data, i.e., historical clustering data, can be obtained from Table 2 of the database. Table 2 can be a historical results table, which may contain the original text of the topic, text identifiers, cluster feature vectors, and clustering results, etc.

[0157] (2) Data inspection.

[0158] The extracted incremental data can be checked. If the incremental data is found to be empty, subsequent clustering and update steps can be skipped, and only the processing time can be updated to ensure the continuity of system operation.

[0159] (3) Query the clustering information of historical data and initialize it.

[0160] You can query or generate clustering information for historical clusters. For example, you can query the number of current historical clusters, the number of topics contained in each cluster, cluster characteristics, etc., from data table 2 in the database.

[0161] Then initialization can be performed. Each cluster can be traversed, and information such as the feature vector and the current cluster topic of each cluster can be extracted in turn (for example, the topic and embedding fields in Table 2 can be read directly), and this information can be stored in a data structure in memory (such as a dictionary or other mapping structure) for use in subsequent single clustering processes.

[0162] (4) Single clustering.

[0163] The Single-Pass Clustering algorithm can be invoked to process text data items. The algorithm can match new data (i.e., the current text data item) with candidate historical clusters based on the vector information of existing clusters, and calculate the similarity between the new data and the candidate historical clusters.

[0164] Then the clustering results can be updated. For example, the cluster identifier and similarity to cluster features of each text data item can be recorded in the data record.

[0165] (5) Cluster topic update.

[0166] Statistical data item count: This allows you to count the number of data items contained in each cluster and update the count.

[0167] Conditional topic generation: For clusters containing data items that reach a preset threshold (such as 3 items, 5 items, etc.) and have no topic, a cluster topic can be generated for them.

[0168] Clusters without a topic that have not reached the preset threshold can be designated as "clusters awaiting maturity," retaining their cluster identifier and characteristics. If new data items are subsequently added, causing the cluster size to reach the threshold, topic generation for that cluster can be triggered. Before any topic is generated for a "cluster awaiting maturity," its characteristics (i.e., the information corresponding to the embedding field) can be extracted during initialization, and its topic field (i.e., the topic field) can be left empty.

[0169] Topic synchronization update: Newly generated cluster topics can be updated in the currently processed data, and can also be updated in historical data records to ensure the consistency of cluster topic information.

[0170] (6) Data update and merging.

[0171] Update field information: Cluster features and their similarity to individual text data items, chat IDs (such as text source identifiers), processing times, and other fields can be updated in data records to ensure that each processed text data item carries the latest processing status. The updated similarity can be applied to downstream business applications, such as using similarity to sort topics and perform statistical analysis. Chat IDs can be used for source tracing.

[0172] Data merging: Processed incremental data can be merged with historical data to form a complete record dataset.

[0173] Output Write: The merged data can be written to the output table or a specified database, ensuring persistent data storage and support for subsequent queries. For example, historical and incremental data can be written to a new time partition of Table 2.

[0174] Figure 6 Other aspects of the embodiments can be found in the other embodiments described above.

[0175] It should be noted that the above figures are merely illustrative representations of the processes included in methods according to some embodiments of this disclosure, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0176] The following are embodiments of the apparatus disclosed herein, which can be used to execute embodiments of the method disclosed herein. For details not disclosed in the apparatus embodiments of this disclosure, please refer to the embodiments of the method disclosed herein.

[0177] Figure 7 This is a block diagram illustrating a text data processing apparatus according to some embodiments of the present disclosure. (Refer to...) Figure 7 The device includes: an acquisition unit 701, a clustering unit 702, and an update unit 703.

[0178] The acquisition unit 701 is used to acquire incremental text data and historical clustering data; the incremental text data includes at least one text data item; the clustering unit 702 is used to perform incremental clustering processing on the text data item based on the historical clusters in the historical clustering data, match historical clusters for each text data item or create new clusters, and generate corresponding cluster topics for clusters that meet preset conditions, so as to obtain incremental clustering results; the update unit 703 is used to update the historical clustering data according to the incremental clustering results.

[0179] In some embodiments of this disclosure, the clustering unit 702 performs incremental clustering processing on the text data items based on historical clusters in the historical clustering data, matching historical clusters or creating new clusters for each text data item, including: determining the cluster features of the historical clusters and converting each text data item into corresponding text features; determining the similarity between the text features and the cluster features; and matching historical clusters or creating new clusters for each text data item based on the relationship between the similarity and a similarity threshold.

[0180] In some embodiments of this disclosure, the clustering unit 702 determines the similarity between the text features and the cluster features by: mapping the text features and each cluster feature to a hash bucket based on the locality-sensitive hashing algorithm; determining candidate historical clusters based on the hash bucket where the text features are located; and determining the similarity between the text features and the cluster features of each candidate historical cluster.

[0181] In some embodiments of this disclosure, the clustering unit 702 matches historical clusters or creates new clusters for each text data item based on the relationship between the similarity and the similarity threshold, including: matching the text data item with the historical cluster with the highest similarity in response to the existence of a similarity exceeding the similarity threshold; and creating a new cluster for the text data item in response to the absence of a similarity exceeding the similarity threshold.

[0182] In some embodiments of this disclosure, the incremental text data is divided into at least one batch for incremental clustering processing; the clustering unit 702 is further configured to: after completing the incremental clustering processing of the text data items in the current batch, determine the proportion of data items in the text data items of the current batch that match historical clusters; adjust the similarity threshold according to the proportion of data items, and use the adjusted similarity threshold for the incremental clustering processing of the next batch of text data items.

[0183] In some embodiments of this disclosure, the clustering unit 702 adjusts the similarity threshold according to the proportion of the data items, including: increasing the similarity threshold in response to the proportion of the data items being higher than a first proportion threshold; decreasing the similarity threshold in response to the proportion of the data items being lower than a second proportion threshold; wherein the second proportion threshold is lower than the first proportion threshold.

[0184] In some embodiments of this disclosure, the preset conditions include: the number of text data items within a cluster reaches a threshold, and the cluster is not associated with a cluster topic.

[0185] In some embodiments of this disclosure, the clustering unit 702 generates corresponding cluster topics for clusters that meet preset conditions, including: generating corresponding cluster topics based on the text data item content within the clusters that meet the preset conditions through a semantic generation model.

[0186] In some embodiments of this disclosure, the clustering unit 702 generates corresponding cluster topics based on the text data item content within the clusters that meet preset conditions through a semantic generation model, including: constructing a prompt word, the prompt word including a system instruction for defining cluster topic generation rules and a user instruction containing the text data item content within the clusters; constructing a topic generation request based on the prompt word and sending the topic generation request to the server where the semantic generation model is located; and determining the text content returned by the server as the cluster topic of the clusters that meet the preset conditions.

[0187] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0188] Figure 8 This is a block diagram illustrating an apparatus 800 for text data processing according to some embodiments of the present disclosure. For example, apparatus 800 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0189] Reference Figure 8 The device 800 may include one or more of the following components: a processing component 802, a memory 804, a power component 806, a multimedia component 808, an audio component 810, an input / output (I / O) interface 812, a sensor component 814, and a communication component 816.

[0190] Processing component 802 typically controls the overall operation of device 800, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 802 may include one or more processors 820 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 802 may include one or more modules to facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.

[0191] Memory 804 is configured to store various types of data to support the operation of device 800. Examples of this data include instructions for any application or method operating on device 800, contact data, phonebook data, messages, pictures, videos, etc. Memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0192] The power supply component 806 provides power to the various components of the device 800. The power supply component 806 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device 800.

[0193] Multimedia component 808 includes a screen that provides an output interface between the device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 808 includes a front-facing camera and / or a rear-facing camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0194] Audio component 810 is configured to output and / or input audio signals. For example, audio component 810 includes a microphone (MIC) configured to receive external audio signals when device 800 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 804 or transmitted via communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

[0195] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0196] Sensor assembly 814 includes one or more sensors for providing status assessments of various aspects of device 800. For example, sensor assembly 814 may detect the on / off state of device 800, the relative positioning of components such as the display and keypad of device 800, changes in the position of device 800 or a component of device 800, the presence or absence of user contact with device 800, the orientation or acceleration / deceleration of device 800, and temperature changes of device 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 814 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0197] Communication component 816 is configured to facilitate wired or wireless communication between device 800 and other devices. Device 800 can access wireless networks based on communication standards, such as WiFi, 3G, 4G, 5G, other communication standards, or combinations thereof. In some embodiments of this disclosure, communication component 816 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In some embodiments of this disclosure, communication component 816 further includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0198] In some embodiments of this disclosure, the apparatus 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0199] In some embodiments of this disclosure, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions that can be executed by a processor 820 of device 800 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0200] A non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by the processor of a mobile terminal, enables the mobile terminal to perform the above-described text data processing method.

[0201] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0202] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method of processing text data, characterized by, include: Acquire incremental text data and historical clustering data; the incremental text data includes at least one text data item. Incremental clustering is performed on the text data items based on the historical clusters in the historical clustering data. Each text data item is matched with a historical cluster or a new cluster is created. A corresponding cluster topic is generated for the clusters that meet the preset conditions to obtain the incremental clustering results. The historical clustering data is updated based on the incremental clustering results.

2. The method of claim 1, wherein, Incremental clustering is performed on the text data items based on historical clusters in the historical clustering data, matching historical clusters or creating new clusters for each text data item, including: Determine the cluster features of the historical clusters and convert each text data item into its corresponding text features; Determine the similarity between the text features and the cluster features; Based on the relationship between the similarity and the similarity threshold, historical clusters are matched or new clusters are created for each text data item.

3. The method according to claim 2, characterized in that, Determining the similarity between the text features and the cluster features includes: Based on the locality-sensitive hashing algorithm, the text features and each cluster feature are mapped to hash buckets respectively; Candidate historical clusters are determined based on the hash bucket containing the text features; Determine the similarity between the text features and the cluster features of each candidate historical cluster.

4. The method of claim 2, wherein, The process of matching historical clusters or creating new clusters for each text data item based on the relationship between the similarity and the similarity threshold includes: In response to the existence of a similarity exceeding the aforementioned similarity threshold, the text data item is matched with the historical cluster with the highest similarity. In response to the absence of similarity exceeding the stated similarity threshold, a new cluster is created for the text data item.

5. The method of claim 2, wherein, The incremental text data is divided into at least one batch for incremental clustering processing; the method further includes: After performing incremental clustering on the text data items of the current batch, the proportion of data items in the text data items of the current batch that match the historical clusters is determined; The similarity threshold is adjusted according to the proportion of the data items, and the adjusted similarity threshold is used for incremental clustering processing of the next batch of text data items.

6. The method according to claim 5, characterized in that, The step of adjusting the similarity threshold according to the proportion of the data items includes: In response to the fact that the proportion of the data items is higher than a first proportion threshold, the similarity threshold is increased; In response to the data item ratio being lower than a second ratio threshold, the similarity threshold is reduced; the second ratio threshold is lower than the first ratio threshold.

7. The method according to claim 1, characterized in that, The preset conditions include: the number of text data items within a cluster reaches a certain threshold, and the cluster is not associated with a cluster topic.

8. The method according to claim 1 or 7, characterized in that, The process of generating corresponding cluster topics for clusters that meet preset conditions includes: Based on the text data items within the cluster that meet the preset conditions, a corresponding cluster topic is generated through a semantic generation model.

9. The method according to claim 8, characterized in that, The step of generating corresponding cluster topics based on the text data item content within the cluster that meets the preset conditions through a semantic generation model includes: Construct prompt words, the prompt words including system instructions for defining cluster topic generation rules and user instructions containing text data items within the cluster; A topic generation request is constructed based on the prompt words, and the topic generation request is sent to the server where the semantic generation model is located; The text content returned by the server is determined as the cluster topic of the cluster that meets the preset conditions.

10. A text data processing apparatus, characterized in that, include: An acquisition unit is used to acquire incremental text data and historical clustering data; the incremental text data includes at least one text data item. The clustering unit is used to perform incremental clustering processing on the text data items based on the historical clusters in the historical clustering data, match historical clusters for each text data item or create new clusters, and generate corresponding cluster topics for clusters that meet preset conditions, so as to obtain incremental clustering results. An update unit is used to update the historical clustering data based on the incremental clustering results.

11. The apparatus according to claim 10, characterized in that, The clustering unit performs incremental clustering processing on the text data items based on historical clusters in the historical clustering data, matching historical clusters or creating new clusters for each text data item, including: Determine the cluster features of the historical clusters and convert each text data item into its corresponding text features; Determine the similarity between the text features and the cluster features; Based on the relationship between the similarity and the similarity threshold, historical clusters are matched or new clusters are created for each text data item.

12. The apparatus of claim 11, wherein, The clustering unit determines the similarity between the text features and the cluster features, including: Based on the locality-sensitive hashing algorithm, the text features and each cluster feature are mapped to hash buckets respectively; Candidate historical clusters are determined based on the hash bucket containing the text features; Determine the similarity between the text features and the cluster features of each candidate historical cluster.

13. The apparatus of claim 11, wherein, The incremental text data is divided into at least one batch for incremental clustering processing; the clustering unit is also used for: After performing incremental clustering on the text data items of the current batch, the proportion of data items in the text data items of the current batch that match the historical clusters is determined; The similarity threshold is adjusted according to the proportion of the data items, and the adjusted similarity threshold is used for incremental clustering processing of the next batch of text data items.

14. The apparatus according to claim 10, characterized in that, The clustering unit generates corresponding cluster topics for clusters that meet preset conditions, including: Based on the text data items within the cluster that meet the preset conditions, a corresponding cluster topic is generated through a semantic generation model.

15. An electronic device, comprising: include: processor; Memory used to store processor-executable instructions; The processor is configured to implement the steps of the method according to any one of claims 1-9.

16. A non-transitory computer-readable storage medium, wherein when instructions in the storage medium are executed by a processor of a mobile terminal, the mobile terminal is enabled to perform the steps of the method according to any one of claims 1-9.

17. A computer program product comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1-9.

Images