Massive short text distributed KNN classification method and system

Abstract

The application relates to a mass short text distributed KNN classification method and system, aiming at the problems of mass short text semantic sparsity, feature diversity and insufficient classification confidence, adopts a distributed computing architecture combined with a KNN (K nearest neighbor) algorithm, optimizes distance calculation and category voting process of the KNN through data fragmentation and parallel computation, solves the efficiency bottleneck of traditional KNN under mass data, and realizes fast and accurate text classification. Dynamic feedback is realized, the scheme effectively enhances short text semantic information, improves the reliability of the classification result, and is convenient for decision tracing and model self-updating.

Description

Technical Field

[0001] This invention relates to the field of "distributed short text intelligent classification and semantic feature fusion technology", and in particular to a distributed KNN classification method and system for massive short texts. Background Technology

[0002] Distributed short text classification and semantic feature fusion have become key areas of focus in the field of natural language processing in recent years. In particular, in applications involving massive amounts of short text such as social networks, news summaries, intelligent customer service, and public opinion monitoring, there is a great demand for efficient, accurate, and scalable automatic classification technology.

[0003] However, current distributed K-Nearest Neighbors (KNN) short text classification methods generally suffer from the following technical shortcomings: First, because short texts are inherently sparse in semantic expression and have a limited number of characters, the feature density of a single sample is low. In the high-dimensional semantic space, the distribution of neighborhood samples is sparse or even fragmented, causing the standard KNN class voting results to fluctuate easily due to the influence of the number and weight of neighboring samples. Especially in scenarios with ambiguous semantic boundaries or low class discrimination, the KNN voting results with limited neighborhoods are often unstable due to insufficient sample size, affecting the overall classification accuracy. Second, there is still a lack of confidence tuning mechanisms in low-density embedding spaces, making it difficult to achieve quantitative gains in classification confidence for dynamic semantic scenarios. Summary of the Invention

[0004] This application provides a distributed KNN classification method for massive short texts, aiming to solve one of the problems or issues of the existing technology mentioned in the background section above.

[0005] This application provides a distributed KNN classification method for massive short texts, specifically including:

[0006] S1: Perform standardized preprocessing on the massive short text data to be classified, including text segmentation, stop word filtering, and semantic normalization, to generate a standardized short text input data set.

[0007] S2: Based on a standardized short text input dataset, perform multi-perspective semantic completion processing, including context window prediction, external knowledge graph association mapping, and subspace clustering analysis, to form a multi-source semantic completion view.

[0008] S3: For each short text within a semantic completion view, extract the aggregated multidimensional semantic view representation vector through feature embedding, and label the semantic view according to different completion sources to achieve feature diversity modeling.

[0009] S4: For the semantic view representation vector of each tag, perform distributed K-nearest neighbor retrieval in the high-dimensional feature space, select a specified number of nearest neighbor samples, and label the category distribution and similarity statistics of different view sources.

[0010] S5: Based on the category distribution and similarity statistics obtained from distributed K-nearest neighbor retrieval, perform independent category voting on each semantic view and output multiple sets of category voting distribution data.

[0011] S6: For multiple sets of category voting distribution data, a dynamic confidence gain model is adopted. Through softmax normalization, Bayesian confidence correction and historical decision stability determination, the category confidence scores of each semantic view are integrated to obtain the intermediate classification results with adaptive confidence labeling.

[0012] S7: Determine whether the intermediate classification results of the adaptive confidence label have reached the preset confidence threshold. If not, automatically trigger the neighborhood incremental sampling and semantic induction completion operation to add supplementary semantic views to the short text in the low-density space and regenerate the category voting distribution.

[0013] S8: Finally, the confidence level, category voting distribution and semantic inductive evidence will be integrated to output the short text classification results with confidence labels, and the basis for multi-source voting and feature completion will be stored to support subsequent decision tracing and model self-updating process.

[0014] This application also provides a distributed KNN classification system for massive short texts, which uses the above-mentioned distributed KNN classification method for massive short texts to classify massive short text data.

[0015] This application provides a distributed KNN classification method and system for massive short texts, which has the following beneficial effects:

[0016] (1) This invention overcomes the problem of semantic information loss caused by the reliance on surface features and narrow neighborhoods in traditional KNN methods by using multi-perspective semantic completion (including context-aware completion, external knowledge graph inference, and subspace clustering induction). By completing the latent semantic neighborhood, the system can mine more discriminative evidence in the originally extremely sparse short text space, making the basis of KNN retrieval and category voting richer and significantly improving the classification accuracy of low-density regions.

[0017] (2) To address the problem of unstable confidence in class voting when there is insufficient support from a few nearest neighbor samples, this invention designs a gain fusion algorithm that combines Softmax normalization, Bayesian confidence correction, and historical decision stability features. This algorithm can analyze the voting distribution, similarity variance, and uncertainty entropy under each semantic view in real time, and adaptively improve or correct the confidence of the class based on dynamic weights. This mechanism eliminates unreliable decisions caused by overly dispersed voting distribution, significantly reduces the misjudgment rate, and enables the system to remain stable even under conditions of sparse data and uneven class distribution.

[0018] (3) When the system detects that the confidence level of the current category judgment is lower than the set threshold, it can automatically trigger neighborhood sampling and semantic induction completion, add new views and continuously iterate category voting and confidence correction until the confidence level is met. This mechanism breaks the blind spot of traditional single-judgment, effectively prevents misjudgment of boundary samples and difficult-to-distinguish categories, and significantly improves the robustness and anti-interference performance of the system's judgment.

[0019] (4) The entire process of the invention is based on a distributed storage and computing architecture. From raw data sharding, distributed word segmentation, semantic normalization, to distributed KNN retrieval and voting, efficient parallel scheduling and load balancing strategies are adopted. In the scenario of real-time streaming classification of millions of short texts, the invention can reduce data access latency to the second level and significantly improve concurrent processing capabilities, achieving fast and accurate text classification.

[0020] (5) This invention stores the multi-source category voting process, confidence evolution trajectory, and feature completion evidence in a structured form in a knowledge base, and adds confidence labels and evidence chains to each classification result, which greatly improves the interpretability and traceability of the model results and supports subsequent automatic model tuning and anomaly analysis. Compared with the existing "black box decision-making" mode, this mechanism helps enterprises review results and accumulate knowledge in compliance scenarios.

[0021] In summary, this invention achieves an original breakthrough in traditional distributed KNN short text classification technology through multi-view dynamic semantic completion and credibility enhancement methods. It addresses the efficiency bottleneck of traditional KNN under massive data, enabling fast and accurate text classification. This effectively solves key bottleneck problems in existing technologies such as sparse semantic expression in short texts, unstable class voting decisions in low semantic density spaces, and insufficient confidence. Attached Figure Description

[0022] Figure 1 This is the main flowchart of a distributed KNN classification method for massive short texts.

[0023] Figure 2 This is a sub-flowchart of a distributed KNN classification method for massive short texts.

[0024] Figure 3 This is another sub-flowchart of a distributed KNN classification method for massive short texts. Detailed Implementation

[0025] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0026] As attached Figure 1 As shown, this application provides a distributed KNN classification method for massive short texts, specifically including:

[0027] S1: Perform standardized preprocessing on the massive short text data to be classified, including text segmentation, stop word filtering, and semantic normalization, to generate a standardized short text input data set.

[0028] S2: Based on a standardized short text input dataset, perform multi-perspective semantic completion processing, including context window prediction, external knowledge graph association mapping, and subspace clustering analysis, to form a multi-source semantic completion view.

[0029] S3: For each short text within a semantic completion view, extract the aggregated multidimensional semantic view representation vector through feature embedding, and label the semantic view according to different completion sources to achieve feature diversity modeling.

[0030] S4: For the semantic view representation vector of each tag, perform distributed K-nearest neighbor retrieval in the high-dimensional feature space, select a specified number of nearest neighbor samples, and label the category distribution and similarity statistics of different view sources.

[0031] S5: Based on the category distribution and similarity statistics obtained from distributed K-nearest neighbor retrieval, perform independent category voting on each semantic view and output multiple sets of category voting distribution data.

[0032] S6: For multiple sets of category voting distribution data, a dynamic confidence gain model is adopted. Through softmax normalization, Bayesian confidence correction and historical decision stability determination, the category confidence scores of each semantic view are integrated to obtain the intermediate classification results with adaptive confidence labeling.

[0033] S7: Determine whether the intermediate classification results of the adaptive confidence label have reached the preset confidence threshold. If not, automatically trigger the neighborhood incremental sampling and semantic induction completion operation to add supplementary semantic views to the short text in the low-density space and regenerate the category voting distribution.

[0034] S8: Finally, the confidence level, category voting distribution and semantic inductive evidence will be integrated to output the short text classification results with confidence labels, and the basis for multi-source voting and feature completion will be stored to support subsequent decision tracing and model self-updating process.

[0035] Step S1: Standardize the massive short text data to be classified by preprocessing, including text segmentation, stop word filtering, and semantic normalization, to generate a standardized short text input data set. Specifically, this includes:

[0036] S1.1: Obtain massive amounts of raw short text data to be classified, and implement the fragmentation management of raw text data based on a distributed storage system (such as HDFS or distributed NoSQL database) to achieve high-concurrency access and data partitioning of raw short text data, so that the data source has parallel scheduling capabilities.

[0037] Using massive amounts of raw short text data as input, and employing a distributed data access interface (adapted to HDFS API or high-concurrency read / write protocols of distributed NoSQL databases), the system enables batch import of short text data and initial data aggregation.

[0038] An index sharding algorithm (with parameters including sharding granularity settings, load balancing strategies, and primary key or hash sharding rules) is used to automatically shard the imported original short text collection, dividing the original data into several logical shards. Each shard has an independent data key and index identifier, and can be distributed and scheduled.

[0039] Furthermore, through a distributed storage scheduling mechanism (such as a sharding allocation strategy based on MapReduce or Spark schedulers), the load balancing distribution of short text shards across distributed storage nodes is achieved, with each node responsible for the data management and reliable access of the specified shards.

[0040] Furthermore, for each sharded dataset, a distributed replica management strategy (parameters include the number of replicas, data consistency level, and automatic failover threshold) is adopted to establish multiple replica redundancy for key short text shards, thereby improving the high availability and fault tolerance of the original data and ensuring the secure storage of massive amounts of data in a distributed environment.

[0041] For access requests to fragmented data, a distributed access control and concurrent read-write lock mechanism (parameters include access token, concurrency control, and write conflict detection) is adopted to achieve high-concurrency access scheduling for short text fragments, and data consistency verification is used to prevent fragmented data from being distorted or lost during the access process.

[0042] By using distributed sharding management and high-concurrency access scheduling, the original short text data is transformed into a set of data partitions that support parallel computing, providing an efficient and reliable input data foundation for subsequent short text segmentation, feature extraction, and distributed KNN classification processing, achieving highly scalable and stable raw data management.

[0043] For example, in a social commenting scenario, raw data containing 100 million short text comments is imported into a distributed NoSQL database. A primary key hash sharding algorithm is used, with a sharding granularity of one shard per 10,000 texts. The load balancing strategy is round-robin allocation, the number of replicas is set to 3, and the data consistency level is set to strong consistency. The distributed system automatically divides the data into 10,000 shards and distributes them across 20 storage nodes. Each node establishes multiple replicas for its local shards for redundancy and enables an automatic failover mechanism. The actual concurrent access pressure is 2000 queries per second, and the reading and writing of sharded data is scheduled through a distributed concurrent read-write lock mechanism. This achieves second-level access response for the raw short text sharded data, providing a highly reliable, low-latency data reading channel for subsequent Spark NLP word segmentation, stop word filtering, and semantic normalization modules. In actual testing, all data access consistency checks of the distributed sharded storage passed, the data loss rate was zero, and the overall system scalability supports dynamic incremental access of millions of text data and horizontal scaling of nodes.

[0044] S1.2: For the original short text data after segmentation, a distributed word segmentation algorithm (such as the word segmentation function based on Spark NLP) is used to perform word segmentation on the short text content to generate short text segments after word segmentation. Each segment retains a structured word sequence for subsequent feature processing.

[0045] Using the original short text data after being segmented as input, a distributed word segmentation algorithm (parameter settings: Spark NLP word segmentation function, number of cluster nodes, segment batch size, text language model selection, etc.) is adopted to achieve parallel word segmentation processing of short text content and regulate the distributed execution efficiency of word segmentation tasks.

[0046] After each data segment is distributed to a distributed computing node, the Spark DataFrame parallel mapping mechanism is used to call a specified word segmentation model (such as Chinese jieba or English BERT word segmenter) to perform batch word segmentation on the content of each short text segment, splitting the original sentence into words or sub-word structures while retaining the segmentation context association identifier.

[0047] Furthermore, a segmented task pipeline scheduling algorithm (parameters include segmentation priority, queue scheduling rules, and node load monitoring) is adopted to monitor the throughput and latency of each node's word segmentation processing and dynamically adjust the load of segmented tasks to achieve a stable and efficient parallel processing flow for word segmentation tasks.

[0048] For the word segmentation results, a distributed word order structured storage algorithm (parameters: word order serialization format, word index mapping strategy) is used to structure and encode the word sequence generated by each short text segment, mapping it into an indexable set of word sequences, providing standardized input data for subsequent feature processing and vocabulary selection.

[0049] A distributed data integrity verification method is adopted to perform batch consistency checks on the word sequence set after word segmentation, detecting whether there are omissions, missegments, encoding conflicts, or data loss. The verification results are fed back to the word segmentation module through a scheduling notification mechanism to achieve real-time correction and compensation, ensuring the high reliability and integrity of the word segmentation data.

[0050] By using a distributed word segmentation algorithm chain, the original short text data after segmentation is transformed into a set of structured word sequences, realizing standardized word segmentation of short text segments, and providing an efficient and reliable data foundation for the next step of distributed stop word filtering and semantic normalization operations.

[0051] For example, in a real-world scenario processing millions of social text comments, every 10,000 short text fragments are allocated to their respective Spark distributed nodes. The English BERT word segmentation model is used for segmentation, with a fragment batch size of 20,000 fragments and a node count of 40. The segmentation tasks are scheduled according to priority-based fragment queues, achieving an average throughput of 1500 texts per second per node. The segmentation results are structured using JSON word order array format and checked by a distributed consistency verification module, finding a missegmentation rate of less than 0.05%. The system ultimately outputs 400,000 segmented short text fragments, each containing a structured word sequence, which can be seamlessly integrated into subsequent stop word filtering and semantic normalization pipelines.

[0052] S1.3: Based on the short text segments after word segmentation, a distributed stop word filtering mechanism (such as a distributed stop word dictionary matching algorithm) is called to remove redundant stop words from the structured word sequences of each segment, thereby achieving effective information vocabulary filtering and outputting the short text segment structure after stop word filtering.

[0053] S1.4: For the short text fragment structure after stop word filtering, a semantic normalization algorithm is adopted, such as stemming, word form restoration and domain vocabulary mapping, to uniformly normalize synonyms, near-synonyms and word form variation items to obtain a word sequence list with standardized semantic structure.

[0054] S1.5: The word sequence list of standardized semantic structure is summarized into a standardized short text input data set, and the data set is stored in the input channel of the subsequent semantic completion module through an efficient distributed batch processing interface, so as to achieve multi-module data format compatibility and efficient connection of processing pipeline.

[0055] Step S2: Based on the standardized short text input data set, perform multi-view semantic completion processing, including context window prediction, external knowledge graph association mapping, and subspace clustering analysis, to form a multi-source semantic completion view. Specifically, this includes:

[0056] S2.1: Perform context window sampling on the standardized short text input data set, and use in-window context-aware prediction algorithms (such as sliding window joint probability modeling) to complete missing or ambiguous context semantic fragments to generate expanded context semantic vector expressions, ensuring that potential related information can be mined in the case of single-sentence short texts.

[0057] S2.2: Based on contextual semantic vector representation, call the external knowledge graph reasoning module, use entity alignment and semantic linking algorithms to map keywords or phrases in short text to structured entity nodes in the knowledge graph, and infer related attributes and associated nodes to obtain the enhanced semantic features of the completed knowledge graph, thereby enhancing the structure and relevance of the original semantic information.

[0058] The input conditions include the short text semantic vector representation expanded by context-aware prediction, which serves as the processing object of the external knowledge graph reasoning module.

[0059] An entity recognition and phrase extraction algorithm (parameters: custom domain NER model, maximum entropy phrase segmentation algorithm) is used to achieve structured recognition of keywords and high-frequency phrases in the semantic vector of short texts.

[0060] Furthermore, through an entity alignment method (parameters: polysemous unification rule, context-related weighting, knowledge graph entity dictionary), the identified keywords are mapped to structured entity nodes in the external knowledge graph. A dual verification mechanism of cosine similarity and Levenshtein distance is adopted to ensure the uniqueness and accuracy of entity correspondence.

[0061] Furthermore, a semantic linking algorithm (parameters: entity semantic relevance threshold, attribute link transition probability, knowledge base relationship layer depth) is employed to establish multi-hop relationships between short texts and knowledge graph nodes, automatically inferring entity-related attribute nodes and their semantically closely related adjacent nodes in the knowledge graph. The attribute link transition probability can be represented by the following transition matrix. Modeling:

[0062]

[0063] in, To be from the entity Transfer to attribute The probability, For knowledge graphs arrive Relationship weights Let i be the sum of the weights of all outgoing edges of entity i.

[0064] Furthermore, based on the entity alignment and attribute inference results, knowledge graph ontology constraint rules and semantic completion algorithms (parameters: relation type priority table, inference threshold, completion layer number) are applied to automatically generate enhanced semantic features of the completed knowledge graph. These features can be numerically processed using embedding methods (such as TransE / DistMult) and encoded as structured high-dimensional vectors.

[0065] Through the above reasoning chain, the original sparse semantic information of the short text is transformed into a knowledge graph feature with high information content, including entities, attributes and multi-level semantic relationships, thereby realizing the structured enhancement and multi-dimensional expansion of the original semantic expression.

[0066] For example, in a short text classification task for social news summaries, given a context-complete semantic vector of "earthquake in a certain area, casualties unknown," the BERT-NER model is used for entity recognition. In practice, the key entities "earthquake" and "casualties" are identified. From the collected Chinese knowledge graph (such as CN-DBpedia), through polysemous normalization and context-based weighting algorithms, the single-meaning "earthquake" is associated with event-related entities, and the query cosine similarity is [value missing]. The Levenshtein distance is This meets the threshold setting for unique entity mapping. Through attribute establishment, the relevant attributes "location of occurrence" and "number of people affected," along with their historical nodes, are retrieved. The transition probability is determined using knowledge base weights. Set as "Number of people affected" node "Location of Occurrence" node The corresponding attribute transition probabilities are respectively , In practical applications, the TransE embedding algorithm embeds this relationship path into a 64-dimensional knowledge graph enhancement vector, ultimately outputting structured features.

[0067] S2.3: Using knowledge graph-enhanced semantic features as input, high-dimensional semantic representation is performed using the embedding vector space projection method. For low-density embedding subspaces, mini-batch K-means or DBSCAN subspace clustering analysis is applied to extract local candidate neighborhoods of short texts in the semantic vector space and generate subspace clustering labels to reveal potential semantic attribution relationships.

[0068] S2.4: For the contextual semantic vector representation, knowledge graph enhanced semantic features and subspace clustering labels obtained by the above three completion methods, a multi-source semantic completion view is generated, and a unique semantic view label is assigned to each view to realize the structured management of multi-perspective semantic information and ensure that feature diversity modeling can be carried out according to different observation perspectives in the subsequent processing stage.

[0069] S2.5: Perform consistency verification on the multi-source semantic completion view, and use semantic overlap measurement and inter-view correlation evaluation algorithm to select semantic views with high information gain and strong complementarity as the final multi-view semantic completion result output, and provide basic input data for subsequent feature embedding and distributed KNN retrieval, thereby improving the reliability and generalization ability of downstream category determination.

[0070] Step S3: For each short text within a semantic completion view, extract the aggregated multidimensional semantic view representation vector through feature embedding, and label the semantic view according to different completion sources to achieve feature diversity modeling. For example... Figure 2 As shown, it specifically includes:

[0071] S3.1: For each short text in the semantic completion view, perform semantic encoding processing based on a general or domain-specific word vector model (such as BERT, word2vec, etc.) to obtain an initial multidimensional feature embedding vector, ensuring that the semantic features of the original short text are fully quantized.

[0072] S3.2: Using the initial multidimensional feature embedding vector as input, feature fusion algorithms (such as feature average pooling, attention weighted fusion, or feature concatenation) are used to aggregate features of each short text in the same view, so as to realize the joint induction of local semantic enhancement features generated by different semantic completion methods and output a weighted aggregated semantic expression vector.

[0073] Feature fusion processing is performed on the initial multidimensional feature embedding vector to aggregate the semantic features of short texts within different semantic completion views.

[0074] A feature-average pooling method (parameter: pooling window size is determined based on the actual number of completion results in each group) is used to average multiple short text embedding vectors within the same semantic completion view to obtain a representative aggregated feature vector that expresses a typical semantic distribution trend. Furthermore, an attention-weighted fusion algorithm (parameter: attention weights are dynamically allocated from training set mutual information or view confidence) is used to assign weighting coefficients to each embedding vector, achieving weighted aggregation using the following formula:

[0075]

[0076] in, This is the final aggregated semantic representation vector. For the first An embedding vector of a short text, These are the corresponding attention weights.

[0077] Furthermore, through a feature concatenation algorithm (parameter: feature dimension order is preset based on completion source priority), the embedding vectors of multiple short texts are directly concatenated into a long string vector, achieving feature fusion with maximum information content. Further, through feature dimensionality reduction algorithms (such as principal component analysis, PCA, parameter: target dimension)... To ensure that over 95% of the information is retained, the high-dimensional vectors generated after concatenation or weighted aggregation undergo dimensionality reduction to preserve core semantic features while reducing downstream computational complexity. Furthermore, batch consistency checks are performed on all aggregated semantic expression vectors, using a cosine similarity index to filter redundant information, ensuring that the output features possess diversity and stability in terms of expressive power and discriminative power. Through the aforementioned multi-level fusion and induction algorithms, the initial set of embedded vectors is transformed into weighted aggregated semantic expression vectors, achieving joint induction and multi-dimensional representation of semantic information within the completed view.

[0078] For example, in a single processing task, the average pooling window is set to [value] for the initial short text embedding vector encoded by the BERT model. For each sample, a weighted aggregated feature mean vector is obtained; in the attention-weighted fusion stage, the attention weights are set according to the confidence level of the training set. The actual aggregate vector is calculated as follows:

[0079]

[0080] The above will then be By combining it with context completion features, knowledge graph enhancement features, and subspace clustering features, a three-dimensional fusion feature set is formed, with a splicing dimension reaching [number missing]. Dimensionality reduction via PCA Dimensions, information retention rate reaches The above, through cosine similarity filtering, ensures that the minimum similarity threshold between each aggregated vector is less than [value missing]. The final output is a set of quantitative weighted aggregate semantic expression vectors.

[0081] S3.3: For each aggregated semantic expression vector, assign a corresponding semantic view label based on its semantic completion source (such as context prediction, external knowledge graph, subspace clustering analysis) to achieve the distinguishing label for feature expressions generated by different semantic completion methods.

[0082] The input conditions include a set of multidimensional semantic expression vectors that have undergone feature aggregation processing, and their corresponding semantic completion source identifiers (such as context prediction, external knowledge graphs, and subspace clustering analysis).

[0083] A structured mapping method based on semantic view label allocation (parameters: view source mapping table, label generation rules, feature metadata) is adopted to automatically identify the semantic completion source of each aggregated semantic expression vector. Furthermore, a semantic view label generation algorithm (parameters: label prefix, view category encoding, completion method identifier) ​​embeds feature metadata into each expression vector to generate a unique semantic view label, achieving structured marking of feature sources. Further, a label allocation algorithm performs a one-to-one mapping process between aggregated semantic expression vectors and their completion sources, binding label metadata to the expression vector's data structure to distinguish feature sources across multiple perspectives. Further, a label consistency verification method (parameters: label rule base, mapping conflict detection mechanism) is used to check for conflicts and confirm the integrity of all allocated labels, ensuring that each aggregated semantic expression vector has a unique and correct semantic view label. Finally, a semantic view label archiving method (parameters: database structure, label index field) is called to archive and store the allocated labels and aggregated semantic expression vectors in a field-based manner, generating a feature expression dataset with semantic view labels.

[0084] Through the aforementioned label assignment and verification algorithm, the aggregated semantic expression vector is structurally distinguished and bound to its semantic completion source, thereby achieving the distinguishable labeling of feature expressions generated by different semantic completion methods and providing diverse feature perspectives to support subsequent distributed KNN retrieval and category voting.

[0085] For example, in a real-world massive short text classification application scenario, for the 96-dimensional aggregated semantic representation vector after BERT encoding and average pooling, the following label generation rules are adopted based on the completion source (context prediction, external knowledge graph, subspace clustering analysis): the context prediction view label is set to "CTX-001", the external knowledge graph view label is set to "KG-002", and the subspace clustering analysis view label is set to "CLUSTER-003". A semantic view label allocation algorithm is applied to bind the above labels to the "source_tag" field of the aggregated vector data structure, and the data is automatically stored in a distributed feature library through an archiving system. From a multi-source perspective, after label verification, the entire feature set is uniquely labeled, with no label conflicts. When using distributed KNN retrieval subsequently, the system can automatically identify the view source and group the data, improving the robustness of category determination in a sparse semantic space.

[0086] S3.4: For the aggregated semantic expression vector that has been assigned semantic view labels, further feature regularization processing (such as L2 normalization, mean calibration, etc.) is performed to eliminate the problem of inconsistent feature scales between different completion sources, and obtain multi-source aggregated semantic expression vectors with consistent distribution and comparable performance, so as to provide a basis for unifying the subsequent K-nearest neighbor retrieval standard.

[0087] S3.5: The regularized multi-source aggregated semantic expression vector is structurally bound to its semantic view label to construct a set of structured feature expressions, which serves as the input for the downstream distributed K-nearest neighbor retrieval stage, realizing feature diversity modeling and providing multi-view support for category voting.

[0088] Step S4: For the semantic view representation vector of each tag, perform distributed K-nearest neighbor retrieval in the high-dimensional feature space, select a specified number of nearest neighbor samples, and label the category distribution and similarity statistics of different view sources. For example... Figure 3 As shown, it specifically includes:

[0089] S4.1: For the semantic expression vector set of the labeled semantic view labels, based on the high-dimensional embedding space distribution, use distributed vector retrieval algorithms (such as Approximate Nearest Neighbor recall or distributed Faiss technology) to perform efficient K-nearest neighbor sample query to obtain the neighborhood sample candidate set corresponding to each semantic view label.

[0090] The input condition is a set of multidimensional aggregated semantic expression vectors that have been assigned semantic view labels. The data has undergone feature embedding, aggregation, label allocation and regularization, and has a high-dimensional structured representation, which serves as the technical input for distributed K-nearest neighbor retrieval.

[0091] A distributed vector retrieval algorithm is adopted (parameters: distributed computing platform such as Spark cluster, using Approximate Nearest Neighbor (ANN) or distributed Faiss library, parallel retrieval on each node, retrieval vector dimension of 256-1024, index type set to a mixture of IVF and PQ, data sharding granularity dynamically set according to data volume) to realize high-dimensional neighborhood candidate sample query for the semantic expression vector set under each semantic view label.

[0092] Furthermore, a distributed ANN indexing algorithm is used (parameter: number of IVF codebook cluster centers n). centroids =128, PQ block count m=32), index initialization of the vector dataset, partitioning and database building, to achieve global fast location and retrieval of semantic expression vectors.

[0093] Furthermore, a distributed Faiss vector recall algorithm (parameter: n) is used. probe =24, TopN recall K=100), for each labeled semantic representation vector, a high-dimensional neighborhood approximation query is initiated sequentially to obtain the candidate neighborhood sample set with the highest similarity. The number of candidate sets can be dynamically adjusted according to the system business load.

[0094] Furthermore, by using a candidate neighborhood aggregation filtering algorithm (parameters: samples conform to semantic view label consistency, distance threshold set between 0.75 and 0.85), the recalled candidate sample set is screened for label consistency and judged for distance threshold, eliminating samples with low relevance or heterogeneous features, thereby improving retrieval accuracy.

[0095] Furthermore, by using a batch result archiving method (parameters: distributed storage interface, structured batch import), the neighborhood sample candidate sets of each semantic view label are assigned to a unified data result set, enabling fast archiving and retrieval of partitions and sub-views.

[0096] By using distributed ANN and Faiss multi-level high-dimensional vector retrieval algorithms, the structured semantic expression vector is mapped to a set of semantic neighborhood candidate samples, which effectively improves the neighborhood discovery efficiency of short texts in sparse semantic space and provides high-quality candidate data for subsequent category distribution statistics and voting.

[0097] For example, in a real-world news summary short text classification scenario, the input is a normalized 256-dimensional BERT aggregated semantic representation vector set, involving three types of semantic view labels (CTX-001, KG-002, CLUSTER-003). The distributed vector library uses a Faiss-CPU cluster, with each node indexing 128 IVF codebook cluster centers and 32 PQ blocks. Each representation vector is processed using n... probe =24 initiated a Top 100 approximate neighbor retrieval, resulting in an initial pool of 100 candidate samples after recall. Label consistency was used to select 68 samples that met the label matching criteria and had an Euclidean distance less than 0.8. In the batch archiving phase, after partitioning, mapping, and efficient storage, 68 candidate neighborhood datasets were ultimately formed under each semantic view label.

[0098] S4.2: For the candidate set of neighborhood samples of each semantic view label, according to the preset K value and the constraints of the business scenario on the number of nearest neighbors, use the vector space distance measurement criteria (such as cosine similarity, Euclidean distance) to filter and obtain a specified number of high similarity nearest neighbor samples, forming a K nearest neighbor sample set belonging to the specific semantic view label.

[0099] For the neighborhood sample candidate set obtained by distributed vector retrieval, the input conditions include the aggregate semantic expression vector set with labeled semantic view tags, the neighborhood sample candidate index, and the corresponding high-dimensional embedding space distance matrix.

[0100] A vector space distance metric algorithm (parameters: cosine similarity or Euclidean distance as the similarity criterion, with dimensionality consistent with the feature embedding model) is used to calculate the distance between each candidate neighboring sample and the target short text expression vector. Further, a distance sorting algorithm (parameters: ascending order by distance index or descending order by cosine value, sorting range is all samples in the candidate set) is used to filter high-similarity samples. Further, a sample quantity constraint algorithm (parameters: preset K value, typically set to 5-50 depending on business size and semantic space scale; this K value limits the number of samples in the final nearest neighbor set) is used to extract the top K samples after distance sorting to obtain a specified number of high-similarity neighboring samples. Further, an attribute label binding algorithm (parameters: view label, sample identifier, source index) is used to semantically view bind and archive the filtering results, forming a K-nearest neighbor sample set belonging to a specific semantic view label. Finally, a sample set integrity verification algorithm (parameters: K-value tolerance threshold, missing data completion strategy) is used to ensure that the K-nearest neighbor set under each semantic view label has data integrity and distribution consistency.

[0101] Through the aforementioned distance measurement, sorting and filtering, quantity constraints, and tag archiving chain derivation, the distributed vector retrieval results are transformed into a semantic view tag-specific K-nearest neighbor sample set with comparability and discriminative power, realizing a multi-source semantically driven nearest neighbor structured expression in a sparse feature space.

[0102] For example, in a short news text classification scenario, the input for the context-complete semantic view "CTX-075" is a 256-dimensional regularized aggregate vector and 25 candidate sample indices. The cosine similarity formula is used.

[0103]

[0104] in, This is the aggregated vector of the short text to be retrieved. The embedding vectors for candidate samples are generated. The similarity scores are calculated for each of the 25 candidate samples, and the top 10 samples with the highest similarity are ranked. A K=10 nearest neighbor set is formed under the "CTX-075" view. For the subspace clustering view "CLUSTER-021", the Euclidean distance formula is used.

[0105]

[0106] For 32 candidate samples in the subspace, the 15 samples with the smallest distance are selected to form a K=15 nearest neighbor sample set. The entire process outputs the nearest neighbor sample index, original category, embedded features, and view label bindings, with no data loss detected during data integrity checks. In practical applications, through the above chain-like selection, the mean semantic similarity between candidate samples increased by 0.12, and the variance of the nearest neighbor category distribution decreased to 0.09, significantly enhancing the semantic discriminative power and category voting effectiveness of the nearest neighbor structure.

[0107] S4.3: For each K-nearest neighbor sample set under a semantic view label, based on its original category label, historical distribution statistics and multi-source semantic source markers, the distributed category classification statistics module is called to perform classification counting and weight labeling on the distribution of each category in the sample set, thereby generating multi-source category distribution statistics.

[0108] S4.4: For the K-nearest neighbor sample set of each semantic view label, a vector space similarity aggregation algorithm (such as similarity mean, weighted variance) is used to perform statistical analysis on the distance distribution between each neighbor sample and the target short text, and output the overall similarity mean, variance and distribution feature parameters of the neighbor samples to supplement the semantic discriminative evidence of the category distribution.

[0109] S4.5: Integrate the generated multi-source category distribution statistics and similarity distribution feature parameters, and archive and label the category distribution and similarity statistics of the K nearest neighbor samples under each semantic view label according to the archiving standard, so as to generate a distributed basic decision dataset that can be used for subsequent category voting, and realize the data closure of the multi-view semantic inductive decision chain.

[0110] Step S5: Based on the category distribution and similarity statistics obtained from distributed K-nearest neighbor retrieval, perform independent category voting on each semantic view and output multiple sets of category voting distribution data. Specifically, this includes:

[0111] S5.1: Based on the semantic view representation vector in the high-dimensional feature space, retrieve the nearest sample set and its corresponding category distribution from the distributed K-nearest neighbor retrieval output as the original input data for category voting.

[0112] S5.2: For the semantic view representation vector under each semantic completion view, a weighted voting algorithm is used to perform category counting for each category based on the category labels of the nearest samples, and the similarity weight is adjusted according to the similarity statistics to help improve the discriminative power of the voting effect in sparse space.

[0113] The input conditions are the K-nearest neighbor sample sets under the semantic completion view labels of the distributed K-nearest neighbor retrieval output, the original category labels of the samples, and the corresponding similarity statistics.

[0114] A category-weighted counting algorithm (parameters: category label set C={c1,c2,...,cn}, nearest neighbor sample set N={x1,x2,...,xK}) is used to count the frequency of each category in the nearest neighbor samples. Each nearest neighbor sample xi corresponds to a unique category label ci and is included in the initial count for that category.

[0115] Furthermore, a similarity weight correction algorithm is used (parameters: the similarity score si for each sample, and the correction function f(s) can be set to linear or non-linear weighting, such as...). This allows for a weighted adjustment of the number of votes in each category. The weighted number of votes in each category can be calculated using the following formula:

[0116]

[0117] in, This is an indicator function; if sample i belongs to category c, the value is 1, otherwise it is 0. The similarity weight assigned to sample i.

[0118] Furthermore, through a standardized weight distribution algorithm (parameter: sum of all weights) The weighted votes are normalized to obtain a standardized voting distribution for each category:

[0119]

[0120] in, The standardized voting score for category c reflects the weighted discriminative power of the actual category.

[0121] Furthermore, a weighted variance and entropy analysis algorithm (parameter: category-weighted vote distribution) is used. This enables an auxiliary evaluation of the discriminative power of voting distributions in sparse spaces, outputting the variance and uncertainty entropy indices of the class distribution.

[0122]

[0123]

[0124] in, The mean of the category distribution. It reflects the degree of dispersion of category scores. Describe the degree of uncertainty in the category.

[0125] Through the above chain derivation, the original category distribution and similarity information of neighboring samples are processed into a weighted and corrected category voting distribution, thereby enhancing the category discrimination power of short texts in the high-dimensional sparse semantic space.

[0126] For example, in a news summary short text classification scenario, the semantic completion view label is set to "KG-002", the input is a nearest neighbor sample set of K=20, the category label of each sample is between C={"politics", "economics", "technology"}, and the sample cosine similarity score si is distributed between 0.56 and 0.92. Weighted correction function. For the 8 samples in the "politics" category, the average weight Weighted votes The "Economy" category has 7 samples with an average weight. Weighted votes Five samples were collected from the "Technology" category, with an average weight. Weighted votes Standardized voting scores are obtained by normalizing according to the formula. , , categorical distribution variance ,entropy After applying this weighted algorithm, the category distribution becomes more concentrated, effectively enhancing the credibility and discriminative power of the category votes. The final output is a weighted category vote distribution array as the voting result for this view.

[0127] S5.3: Apply parallel data stream scheduling to distribute and aggregate the weighted and corrected category count results from each semantic completion view to generate multiple independent category voting distribution arrays, ensuring efficient output of results from each independent perspective.

[0128] S5.4: For each group of category voting distribution results, based on statistics such as the similarity variance and uncertainty entropy of the nearest neighbor samples, further output the category distribution reliability label to provide data basis for the subsequent confidence gain model.

[0129] S5.5: Integrate the category voting distribution arrays and corresponding reliability statistics from the output of each semantic completion view, and use them as multi-view category voting distribution data with high information content to provide a unified interface for the next weighted decision and confidence output of the dynamic confidence gain fusion module.

[0130] Step S6: For multiple sets of category voting distribution data, a dynamic confidence gain model is used. Through softmax normalization, Bayesian confidence correction, and historical decision stability determination, the category confidence scores of each semantic view are fused to obtain intermediate classification results with adaptive confidence labeling. Specifically, this includes:

[0131] S6.1: Based on multiple sets of category voting distribution data output by the distributed K-nearest neighbor independent category voting module, softmax normalization is performed on each set of category distribution data to obtain standardized category confidence scores for each semantic view. The standardized category confidence scores serve as the basic parameters for fusion between semantic completion views, enabling unified numerical scaling and comparison of the original category voting probability values.

[0132] S6.2: Based on the standardized category confidence scores and combined with the historical confidence records of the category distribution under the current semantic view markers, a Bayesian confidence correction algorithm is executed to obtain the corrected category confidence criterion. The Bayesian confidence correction process utilizes the prior probability of the category distribution and the likelihood function of the current distribution to achieve posterior probability inference, thereby improving the stability of the confidence of category determination under low sample density views.

[0133] S6.3: Based on the Bayesian-corrected category confidence criterion and standardized category confidence scores, calculate the category distribution variance and uncertainty entropy index corresponding to each semantic view label, and obtain historical decision stability feature parameters. These historical decision stability feature parameters are used to quantify the consistency of long-term judgments for the same category under different semantic views, supporting the dynamic weight adjustment of the subsequent confidence gain function.

[0134] S6.4: Using the modified category confidence criterion, standardized category confidence score, and historical decision stability feature parameters under each semantic view label as input, an adaptive confidence gain coefficient is calculated through a dynamic weighted gain model, and the final multi-view category confidence score is output. The dynamic weighted gain model includes softmax distribution-based normalized weighted fusion and adaptive weight allocation using a Bayesian model of the confidence layer, forming a global category confidence score result for multi-source semantic fusion.

[0135] S6.5: Perform adaptive thresholding on the multi-view category confidence scores of the fused output to generate an intermediate classification result dataset with adaptive confidence labels. The intermediate classification results include professional terminology parameters such as confidence scores for each category, contribution weights of each semantic view, and decision stability criteria, providing highly reliable input for subsequent progressive confidence determination processing and final classification result integration.

[0136] The multi-view category confidence scores from the fusion output are used as input data for adaptive threshold determination. For all category confidence scores with standardized confidence labels, an adaptive confidence threshold determination algorithm is employed (parameter: fused category confidence score). Preset reliability threshold This enables the screening of the credibility of classification results.

[0137] Using the confidence judgment function This involves threshold mapping of the fusion category confidence score for each short text, determining whether it meets the set confidence requirements, and outputting the label. (Satisfied / Not satisfied).

[0138] Furthermore, a structured multi-parameter data generation method is employed (parameter: category confidence score). Semantic view contribution weight Criteria for the stability of historical decisions This enables the collection of all parameters for intermediate classification results. Specifically, for samples that meet the confidence requirements, the system summarizes their class confidence scores, the distribution of contribution weights for each semantic view, and the historical decision stability criteria to generate a structured intermediate classification result dataset. .

[0139] Furthermore, through the archiving algorithm (parameter: judgment tag) This enables the classification and archiving of datasets, assigning intermediate results that meet the confidence criteria to the decision output channel and assigning results that do not meet the confidence criteria to the progressive supplementary processing channel, thus ensuring downstream intelligent diversion and data closed-loop structure.

[0140] Through the above adaptive threshold determination process, the multi-view category confidence scores are standardized and transformed into a classification intermediate result dataset with confidence labels, including category confidence scores, contribution weights and decision criteria, to achieve reliability identification and subsequent intelligent triage.

[0141] For example, in a social media comment classification scenario, batch configuration of fused category confidence scores is used. Preset confidence threshold For each short text, calculate the confidence score function: , , , Judgment Result ,in and The sample enters the confidence output channel. and The samples are then fed into the semantic supplementation channel. The intermediate classification result dataset is generated, and structured data is output. This processing result effectively improved the reliability of classification results and the efficiency of hierarchical data flow management in subsequent judgment and automatic supplementation processes.

[0142] Step S7: Determine whether the intermediate classification result of the adaptive confidence labeling reaches the preset confidence threshold. If not, automatically trigger neighborhood incremental sampling and semantic induction completion operations to add supplementary semantic views to the short text in the low-density space and regenerate the category voting distribution. Specifically, this includes:

[0143] S7.1: Perform confidence threshold determination on the intermediate classification results with adaptive confidence labeling to determine whether the stability of the current category voting distribution meets the preset decision credibility requirements. The input is the intermediate classification results, the confidence score in the dynamic credibility gain model is used, the determination is made through the confidence threshold algorithm, and the output is the determination label (satisfied / not satisfied).

[0144] For classification intermediate result datasets with adaptive confidence labels, this study relies on a set of structured parameters such as multi-view category confidence scores, contribution weights, and decision stability criteria to achieve the fundamental goal of quantitative screening and decision diversion of classification decision credibility.

[0145] A segmented confidence threshold-based decision algorithm is adopted (including parameters: preset confidence threshold). and fusion category confidence score The intermediate classification result dataset of all short texts to be judged is processed sequentially in batch input form, forming the following progressive data processing chain:

[0146] First, a dynamic confidence gain model is used to automatically aggregate the multi-view category confidence scores of each short text, and then extract the fused category confidence score. Furthermore, each short text will... With preset threshold A rigorous comparison is performed, and a decision discriminant function is used to determine the label. Binarization labeling: satisfies If the condition is met, output the label "Satisfied"; otherwise, output the label "Not Satisfied".

[0147] Subsequently, for the input number set All short texts are processed iteratively in this way, and the output is a judgment tag array. This enables multi-dimensional verification and clear archiving of intermediate classification results to ensure credibility. For short texts identified as "not satisfied" labels, the system automatically archives them to the neighborhood incremental sampling and semantic induction completion processing channel through a decision-making and diversion mechanism, ensuring the independence and efficiency of the supplementary judgment link.

[0148] Through the above data screening and transfer judgment process, all intermediate classification results are divided into an output queue with sufficient confidence and a progressive processing queue to be supplemented, forming a clear and effective multi-level decision-making path, laying a data foundation for downstream category voting enhancement, semantic expansion and system robustness improvement.

[0149] For example, for three short texts input in a single batch, their fusion category confidence scores are configured sequentially as follows: , , Preset confidence threshold The system determines, through the above discrimination algorithm, that when hour, Output "Satisfied"; while and If all are below the preset threshold, then and All three results output "Not satisfied". Based on their judgment labels, they are respectively archived to the confidence output channel and the further semantic completion processing queue. This processing chain effectively ensures automatic identification of decision credibility during classification, intelligent progressive supplementation of low-confidence data, and the system's end-to-end data hierarchical processing capability.

[0150] S7.2: For short texts where the confidence level is not satisfied, perform incremental neighborhood sampling based on the existing distributed K-nearest neighbor retrieval results. The input is the feature representation vector of the short text in low-density space. The neighborhood expansion sampling algorithm is used to dynamically select additional semantic nearest neighbor samples to obtain an enhanced semantic neighborhood feature set.

[0151] S7.3: The enhanced semantic neighborhood feature set obtained by incremental sampling of the neighborhood is expanded into a multi-source semantic view using a semantic inductive completion algorithm. The input is the enhanced semantic neighborhood feature set and the multi-source completion module (context completion, external knowledge graph, clustering subspace). Through semantic adaptive inductive processing, an additional semantic completion view is generated, which increases the amount of feature information.

[0152] S7.4: Based on the expanded semantic completion view, re-execute the distributed KNN category voting distribution generation. The input is the newly generated semantic completion view set, and the distributed K-nearest neighbor retrieval and category voting algorithm are used to output the enhanced category voting distribution results.

[0153] S7.5: Apply the dynamic confidence gain model to the enhanced category voting distribution results again for confidence labeling. Through confidence normalization and Bayesian confidence correction algorithm, obtain the latest adaptive confidence labeling intermediate classification results to support subsequent decision-making and output.

[0154] Step S8: The final fused confidence score, category voting distribution, and semantic inductive evidence are output to obtain a short text classification result with a confidence label, and the multi-source voting and feature completion basis are stored to support subsequent decision tracing and model self-updating processes. Specifically, this includes:

[0155] S8.1: Structure the collection of category confidence scores from all perspectives after dynamic confidence gain fusion to obtain multiple sets of category voting distributions and corresponding confidence labels, thus preparing data for subsequent classification results generation.

[0156] S8.2: Based on the fusion category confidence score, perform the final category assignment determination for each short text, output the category label with the highest confidence as the main category, and generate a confidence quantification label to form a classification result entity with confidence label.

[0157] The input is a dataset of fused category confidence scores collected by the dynamic confidence gain fusion module. The final category assignment is determined for each short text to be classified.

[0158] The maximum confidence class assignment algorithm is used (parameter: fused class confidence score vector). Category tag set ), enabling the processing of each short text from The category with the highest confidence score among the available categories is selected as the primary classification label.

[0159] Furthermore, the label generation algorithm is quantified using confidence metrics (parameter: main classification label). and corresponding confidence scores This allows for the numerical labeling of the confidence score output of the main classification, and the mapping of the confidence score value to a displayable label, thus enabling an explicit expression of the confidence level of the classification results.

[0160] Furthermore, structured result entity generation processing is employed (parameter: main category label). Confidence labels and multi-view contribution weight This generates classification result entities with diverse information such as category labels, confidence labels, view weights, and decision stability indicators, facilitating subsequent storage, retrieval, and traceability logic integration.

[0161] By normalizing the classification results (methods: confidence score standardization mapping, data entity format standardization), consistent archiving of classification results under the distributed data interface is achieved, providing standardized input for downstream knowledge base storage and case database indexing.

[0162] By adopting the above-mentioned chain processing method of maximum confidence determination and quantitative label generation, the dynamic confidence gain is fused and output to complete the final classification decision summary, thereby realizing the structured annotation of the short text main classification results and ensuring the classification effectiveness.

[0163] For example, in the specific implementation process, for each short text, its fusion category confidence score vector is configured as follows: The category labels are respectively The maximum confidence class assignment formula is used:

[0164]

[0165] in, For the final main category label, For the first Confidence score for the category.

[0166] The short text was analyzed and classified as "Finance". Confidence level label is .

[0167] Furthermore, Normalize to a confidence level display format (such as the commonly used "high confidence", "medium confidence", and "low confidence" mapping ranges), and integrate multi-view weights into the classification result entities, for example... These represent the contributions from different semantic completion perspectives. The final generated classification output entity includes the main category label "Finance", the confidence label "0.89 - High Confidence", the contribution weight array, and the decision stability parameter.

[0168] Through the above chain processing, the system achieves the clarity of the main classification output, the universality of high-confidence annotation, and the ability to trace the results after the aggregation of multi-source information in batch short text classification applications, effectively supporting the final decision output and subsequent model self-update of the distributed KNN text classification system.

[0169] S8.3: For all categories of voting distribution and confidence labeling results, combine multi-source semantic inductive evidence and reference feature supplementation sources, store them using a unified distributed data recording interface, and form a traceable classification auxiliary knowledge base to support the chain audit of subsequent classification decision-making processes.

[0170] S8.4: In the storage process, the multi-source voting process, feature completion integration, and final category determination results are processed by metadata indexing to build an automatically searchable case database, thus establishing a knowledge deposition foundation for the self-updating of the distributed KNN model and the backtracking analysis of abnormal cases.

[0171] S8.5: Based on the archived multi-source category voting distribution, confidence labels, and related semantic completion evidence, it generates classification decision tracking reports in real time and opens up a model self-update interface, providing a dynamic feedback basis with an evidence chain for parameter tuning and subsequent semantic feature expansion.

[0172] For those skilled in the art, various other corresponding changes and modifications can be made based on the technical solutions and concepts described above, and all such changes and modifications should fall within the protection scope of the claims of this invention.

[0173] Unless otherwise defined, the technical or scientific terms used herein shall have the ordinary meaning as understood by one of ordinary skill in the art to which this application pertains. The terms “first,” “second,” “third,” and similar terms used in this patent application specification and claims do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, the terms “an” or “a” and similar terms do not indicate a quantity limitation, but rather indicate the presence of at least one. The terms “comprising” or “including” and similar terms mean that the elements or objects preceding “comprising” or “including” encompass the elements or objects listed following “comprising” or “including” and their equivalents, and do not exclude other elements or objects. The “multiple” mentioned in the embodiments of this application refers to two or more. A and / or B indicate three possibilities: A; B; and A and B.

[0174] The above description is merely an exemplary embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and such modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A distributed KNN classification method for massive short texts, specifically including the following steps: S1: Perform standardization preprocessing on the massive short text data to be classified to generate a standardized short text input data set; S2: Based on a standardized short text input data set, perform multi-view semantic completion processing to form a multi-source semantic completion view; Step S2 specifically includes: Context window sampling is performed on a standardized short text input dataset to complete missing or ambiguous context semantic fragments, thereby generating an expanded context semantic vector representation; Based on contextual semantic vector representation, an external knowledge graph reasoning module is invoked. Entity alignment and semantic linking algorithms are used to map keywords or phrases in short texts to structured entity nodes in the knowledge graph, and related attributes and associated nodes are inferred to obtain enhanced semantic features of the completed knowledge graph. Using knowledge graph-enhanced semantic features as input, for low-density embedding subspaces, local candidate neighborhoods of short texts in the semantic vector space are extracted, and subspace clustering labels are generated. Based on the obtained contextual semantic vector representation, knowledge graph enhanced semantic features, and subspace clustering labels, a multi-source semantic completion view is generated, and a unique semantic view label is assigned to each view. Consistency checks are performed on the multi-source semantic completion views, and semantic views with high information gain and strong complementarity are selected as the final multi-view semantic completion results. S3: For each short text within a semantic completion view, extract the aggregated multidimensional semantic view representation vector through feature embedding, and label the semantic view according to different completion sources; S4: For the semantic view representation vector of each tag, perform distributed K-nearest neighbor retrieval in the high-dimensional feature space, select a specified number of nearest neighbor samples, and label the category distribution and similarity statistics of different view sources; S5: Based on the obtained category distribution and similarity statistics, perform independent category voting on each semantic view and output multiple sets of category voting distribution data; S6: Based on multiple sets of category voting distribution data, obtain intermediate classification results with adaptive confidence labels; S7: Determine whether the intermediate classification results of the adaptive confidence label have reached the preset confidence threshold. If not, automatically trigger the neighborhood incremental sampling and semantic induction completion operation to add a supplementary semantic view to the short text in the low-density space and regenerate the category voting distribution. S8: Finally, the confidence level, category voting distribution, and semantic inductive evidence will be integrated to obtain short text classification results with confidence labels.

2. The distributed KNN classification method for massive short texts according to claim 1, characterized in that, Step S8 specifically includes: storing the basis for multi-source voting and feature completion to support subsequent decision tracing and model self-updating processes.

3. The distributed KNN classification method for massive short texts according to claim 1, characterized in that, The standardization preprocessing in step S1 includes text segmentation, stop word filtering, and semantic normalization.

4. The distributed KNN classification method for massive short texts according to claim 1, characterized in that, The multi-perspective semantic completion processing in step S2 includes context window prediction, external knowledge graph association mapping, and subspace clustering analysis.

5. The distributed KNN classification method for massive short texts according to claim 1, characterized in that, Step S1 specifically includes: To acquire the massive amount of raw short text data to be classified, a distributed storage system is used to manage the raw text data in segments, enabling high-concurrency access and data partitioning of the raw short text data, and giving the data source the ability to be scheduled in parallel.

6. The distributed KNN classification method for massive short texts according to claim 3, characterized in that: The semantic normalization algorithm includes stemming, word form restoration, and domain vocabulary mapping, and performs unified normalization and standardization processing on synonyms, near-synonyms, and word form variation terms.

7. The distributed KNN classification method for massive short texts according to claim 1, characterized in that, Step S4 specifically includes: For the semantic expression vector set of the labeled semantic view labels, based on the high-dimensional embedding space distribution, a distributed vector retrieval algorithm is used to perform efficient K-nearest neighbor sample query to obtain the candidate set of neighborhood samples corresponding to each semantic view label. For each semantic view label's neighborhood sample candidate set, according to the preset K value and the business scenario's constraint on the number of nearest neighbors, a specified number of highly similar nearest neighbor samples are selected and obtained to form a K nearest neighbor sample set belonging to the specific semantic view label; For each K-nearest neighbor sample set under a semantic view label, based on its original category label, historical distribution statistics and multi-source semantic source tags, the distributed category classification statistics module is called to perform classification counting and weight labeling on the distribution of each category in the sample set. For the K-nearest neighbor sample set of each semantic view tag, perform statistical analysis on the distance distribution between each neighbor sample and the target short text, and output the mean, variance and distribution characteristic parameters of the overall similarity of the neighbor samples. By combining the generated multi-source category distribution statistics and similarity distribution feature parameters, the category distribution and similarity statistics of the K nearest neighbor samples under each semantic view label are archived and marked according to the archiving criteria.

8. The distributed KNN classification method for massive short texts according to claim 1, characterized in that, For each semantic view representation vector under a semantic completion view, a weighted voting algorithm is used to perform category counting for each category based on the category labels of the nearest samples.

9. A distributed KNN classification system for massive short texts, characterized in that, The massive short text data is classified using the distributed KNN classification method for massive short texts as described in any one of claims 1-8.

Images