Short Text Intelligent Analysis and Classification Optimization Methods

Abstract

This invention relates to the field of natural language processing technology using artificial intelligence, and particularly to a method for intelligent analysis and classification optimization of short texts, comprising the following steps: S1: Preliminary processing of short text information to obtain preliminarily processed short text information; S2: Introducing a load-balanced data sharding algorithm to shard the preliminarily processed short text information and store it on multiple nodes k, using a distributed processing algorithm to achieve parallel data processing; S3: Introducing a contextual deep understanding algorithm to optimize the accuracy of short text information classification; S4: Each node k uses the optimized natural language processing algorithm to intelligently analyze the sharded short text data and determine its category; S5: Summarizing and analyzing the classification results on all nodes k to obtain the final analysis and classification results of the short text information. This method improves the efficiency and accuracy of short text information processing, analysis, and classification.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence in natural language processing, and in particular to a method for intelligent analysis and classification optimization of short texts. Background Technology

[0002] In the digital communication era, short text has become one of the important means of daily communication. With the widespread use of mobile devices, the amount of short text has grown rapidly, covering multiple fields such as advertising, personal communication, and business notifications. In this context, how to efficiently and accurately perform intelligent analysis and classification of large amounts of short text has become a pressing technical problem to be solved.

[0003] With the development of Natural Language Processing (NLP) technology, some machine learning-based methods have begun to be applied to text processing tasks. In particular, the rise of deep learning technology has provided new solutions for the intelligent analysis and classification of short texts. By learning from large amounts of text data, deep learning models can automatically extract complex text features, achieving a deep semantic understanding of short text content. However, deep learning models have high computational resource requirements, especially when processing large-scale short text data. How to improve processing efficiency while ensuring analytical accuracy has become a challenge. Meanwhile, short text data typically uses short words or concise sentences to convey information, resulting in less information and problems such as data sparsity. In large-scale data processing, each text requires significant computational resources, and processing such sparse short text data involves a large feature space, further reducing overall computational efficiency.

[0004] Chinese invention patent CN114840677B discloses a short text classification and intelligent analysis method for multi-granularity needs, including the following steps: S1, data preprocessing using integration, cleaning, statistics, annotation, replacement, and normalization; S2, data augmentation and expansion of labeled data using thesaurus, text similarity calculation, and word embedding techniques; S3, coarse-grained text classification using a pre-trained BERT model; S4, fine-grained short text classification by integrating textual information and externally introduced knowledge; S5, extraction of interpersonal relationships using named entity recognition and relation extraction techniques; S6, generation of multi-granularity text classification labels and intelligent analysis of the classified texts. While the above-mentioned patent addresses the problems of low efficiency and non-standardization in current multi-granularity classification of SMS text to some extent, it does not fundamentally solve the problem that each text requires high computational resources during large-scale data processing, leading to overall low computational efficiency. Summary of the Invention

[0005] Technical objective: In order to overcome the shortcomings of existing technologies, this invention provides a method for intelligent analysis and classification optimization of short texts.

[0006] Technical Solution: To achieve the above objectives, this invention discloses a short text intelligent analysis and classification optimization method, comprising the following steps:

[0007] S1: Perform preliminary processing on the short text information to obtain the pre-processed short text information;

[0008] S2: Introduce a load-balanced data sharding algorithm to shard the pre-processed short text information and store it on multiple nodes k. Use a distributed processing algorithm to achieve parallel data processing.

[0009] S3: Introduce a contextual deep understanding algorithm to optimize the accuracy of short text information classification;

[0010] S4: Each node k uses an optimized natural language processing algorithm to intelligently analyze the short text fragments and determine their category.

[0011] S5: Summarize and analyze the classification results on all nodes k to obtain the final analysis and classification results of the short text information.

[0012] Furthermore, the specific steps of the load balancing data sharding algorithm in step S2 are as follows:

[0013] S2.1: Evaluate the size and complexity of the dataset D containing short text information, and determine the number of records N and the average number of features M for each record in the dataset D;

[0014] S2.2: Evaluate the computational power C of each node k used to process the data. k ;

[0015] S2.3: Based on the number of records N, average number of features M, and computing power C k Develop a sharding strategy and determine the number S of records S allocated to each node k. k ;

[0016] S2.4: According to the recorded number S k The result is that the short text fragments are actually allocated to each node k;

[0017] S2.5: Execute steps S3, S4 and S5 in sequence to obtain the final analysis and classification results of the short text information.

[0018] Furthermore, the computing power C k The calculation formula is:

[0019]

[0020] Among them, C k For the computing power of node k, CPU cores，k Let k be the number of CPU cores. speed，k For the computing speed of each core, MEM available，k Let k be the total available memory and I / O. speed，k For the data read / write speed of node k, NET bandwidth，k The network bandwidth of node k, These are weighting coefficients used to adjust the relative importance of each factor;

[0021] The number of records S k The calculation formula is:

[0022]

[0023] Among them, S k The number of records to be assigned to node k, where K is the total number of nodes, Num is the total number of records in dataset D, and E is the number of records to be assigned to node k. k λ is the base value of the computing power of node k, λ is an adjustment factor used to balance the relationship between computing overhead and communication overhead, and O is the average communication overhead.

[0024] Furthermore, the specific steps of introducing the contextual deep understanding algorithm in step S3 are as follows:

[0025] S3.1: Construct a text concatenation representation for the pre-processed short text fragments, and construct a context-enhanced embedding representation C(w) for each word that is associated with its context. i );

[0026] S3.2: To capture the deep semantics of words in specific contexts, a feature extraction function F is defined, which embeds the context-enhanced representation C(w) into the context. i Transform into feature vector f i ;

[0027] S3.3: The obtained feature vector f i Steps S4 and S5 are executed sequentially for subsequent intelligent analysis and classification.

[0028] Furthermore, the context-enhanced embedding representation C(w) i The calculation formula is:

[0029]

[0030] Among them, C(w) i ) represents the context-enhanced embedding representation, and σ is the non-linear activation function; w i For the current target word, v j For the word wj The vector representation of the captured word w j Basic semantic information, where n is the size of the context window, α j For the weights adjusted based on distance, POS(w) j ) for word w j The part-of-speech tagging vector, β is an adjustment factor used to balance the word vector v j and part-of-speech vector POS(w j The contribution of D(w) to context embedding, i w j ) is the distance function, representing w i with w j The distance between them is a function, with γ as an adjustment factor used to control D(w) i w j The strength of the influence of context embedding, S(w) i ) for w i Additional semantic features, where δ is a weight parameter used to adjust S(w) i The extent of the impact on the final context embedding.

[0031] Furthermore, the feature vector f i The calculation formula is:

[0032] f i =F(C(w) i ))=ReLU(W f2 ·ReLU(W f1 ·C(w i )+b f1 )+b f2 ),

[0033] Among them, f i To reflect the deep semantic information of words, W f1 and W f2 W represents the weight matrices for two different layers. f1 Used for the first-level transformation, W f2 Used for the second-level transformation, b f1 and b f2 For the corresponding W f1 and W f2 The bias term is denoted by ReLU, which is a non-linear activation function.

[0034] Furthermore, the optimized natural language processing algorithm includes RoBERTa pre-trained model transformation and a recurrent neural network for intelligent analysis and classification of vectorized short text data. The recurrent neural network includes an RNN layer, a fully connected layer, a softmax layer, and a threshold judgment layer. The specific steps in step S4 for intelligently analyzing the segmented data and determining its category using the optimized natural language processing algorithm are as follows:

[0035] S4.1: The pre-processed short text segment data consists of words w1, w2, ..., w N Composition, where N represents the number of words, and each word w is composed using the RoBERTa model in optimized natural language processing. N The data is converted into vector representations to obtain vectorized short text fragments. Markers for marking the start and end of the text are added at the beginning and end of the short text fragments, respectively.

[0036] S4.2: Use the RoBERTa to extract short text vectors with text start identifiers from the vectorized short text fragment data, and input them into the fully connected layer;

[0037] S4.3: In the fully connected layer, the vector with the text start identifier is mapped to the score vector z to obtain the score of the category represented by each element in the score vector z, and the score vector z is sent to the Softmax layer;

[0038] S4.4: In the Softmax layer, the Softmax function transforms the score vector z into class probability values ​​P(class... q This is used to indicate the probability distribution of short text fragments across different categories;

[0039] S4.5: The class probability value P(classq) is fed into the threshold determination layer, which has a threshold vector Θ, Θ = {θ1, θ2, ..., θ}. M}, where M represents the number of thresholds, when the class probability value P(class q When the value is between the two thresholds, the short text is classified into that category.

[0040] Furthermore, the class probability value P(class) q The formula for calculating ) is:

[0041]

[0042] Where, P(class) q () represents the class probability value of class q after processing with the Softmax function. The score for category q is derived from the output of the fully connected layer.

[0043] Furthermore, during the parallel processing of short text fragment data by each node k, a data synchronization optimization algorithm is introduced to reduce redundant transmission between nodes k during data synchronization. The specific steps of the data synchronization optimization algorithm are as follows:

[0044] S10.1: Draw the network topology diagram and determine the optimal data transmission path between nodes k;

[0045] S10.2: Introduce a dependency analysis algorithm to determine the total dependency score I for each data point. i ;

[0046] S10.3: Based on the total dependency score I i The data priority levels are divided into high priority group, medium priority group, and low priority group;

[0047] S10.4: Based on data priority levels, formulate a data synchronization plan so that high-priority groups, medium-priority groups, and low-priority groups can synchronize in sequence.

[0048] Furthermore, the total dependency score I i The calculation formula is:

[0049] I i =∑ j≠i F ij ,

[0050] Among them, I i F represents the total dependency score for data i. ij The dependency score between data i and data j;

[0051] The F ij The calculation formula is:

[0052]

[0053] Among them, F ij A is the dependency score between data i and data j. it and A jt Let be the number of times node k accesses data i and data j within time t, respectively. δ is the time decay factor, t is the decay parameter. k It is the current time R il , and R jl Let θ represent the roles of data i and data j in the first-level dependency. l λ1, λ2, and λ3 are the weights of the first-level dependency, λ1, λ2, and λ3 are the coefficients L that adjust for each influencing factor, L is the number of the dependency level, and T is the total time period.

[0054] The beneficial effects of this invention are:

[0055] 1. By sharding short text information and distributing it across multiple computing nodes for parallel processing, the load on individual nodes is reduced, the overall processing time is shortened, and processing efficiency is significantly improved. The introduction of a load-balanced data sharding algorithm aims to balance the data volume and processing capacity of each shard, ensuring that each computing node can efficiently utilize its resources. This balanced allocation maximizes the utilization of system resources and avoids waste. The distributed processing architecture reduces the risk of single points of failure and enhances system stability.

[0056] 2. By constructing an embedding representation for each word that is associated with its context, the understanding of polysemous words and context-dependent words is improved. Deep contextual understanding helps to accurately capture semantic changes, enabling a more accurate understanding of the true meaning of short texts containing polysemous words or specific context-dependent words. By defining a feature extraction function, feature vectors reflecting the deep semantics of words in a specific context can be transformed and extracted from the context embeddings, improving the expressiveness of the features and enhancing the model's ability to capture complex semantic relationships, providing richer and more accurate feature representations for subsequent classification. Classification using context-related features enables decisions based on a more refined semantic understanding, thereby significantly improving the accuracy of short text classification.

[0057] 3. Preliminary processing of short text information can effectively remove interference information, ensuring the quality and accuracy of subsequent analysis. Using the RoBERTa model to process vectorized short text data can effectively handle the temporal information in the text sequence and improve the performance of classification tasks. By introducing a threshold judgment layer after the fully connected layer and the Softmax layer, more granular control over the classification results can be allowed. By setting the threshold vector, the sensitivity of classification can be adjusted according to actual needs, so that the model can achieve optimal performance in different application scenarios. Attached Figure Description

[0058] Figure 1 This is a flowchart illustrating the overall process of the short text intelligent analysis and classification optimization method in this invention.

[0059] Figure 2 This is a flowchart of the load balancing data sharding algorithm in this invention;

[0060] Figure 3 This is a flowchart of the contextual deep understanding algorithm in this invention;

[0061] Figure 4 This is a flowchart of the optimized natural language processing algorithm in this invention. Detailed Implementation

[0062] The following is in conjunction with the appendix Figure 1 To be continued Figure 4The principles and features of the present invention are described, and the examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0063] A method for intelligent analysis and classification optimization of short texts includes the following steps:

[0064] S1: Perform preliminary processing on the short text information to obtain the pre-processed short text information.

[0065] In this embodiment, the preliminary processing of short text information is as follows: First, the acquired short text information is translated using a language model pre-trained with an empirical method to achieve cross-language conversion, resulting in short text information in a single language. Next, the short text information in the single language is cleaned by removing irrelevant characters using regular expressions and text cleaning algorithms. Then, text standardization rules, such as vocabulary unification and abbreviation processing, are applied to standardize the cleaned short text information to ensure consistency and accuracy, resulting in standardized short text information. Finally, NLP tools are used to perform part-of-speech tagging on the standardized short text information, converting it into a format suitable for further analysis, thus achieving preliminary processing of the short text information. This preliminary processing, including text cleaning and standardization, effectively removes interfering information, ensuring the quality and accuracy of subsequent analysis. In this embodiment, the technical means used in the preliminary processing of short text information are well-known to those skilled in the art and will not be elaborated upon here.

[0066] S2: Introduce a load-balanced data sharding algorithm to shard the pre-processed short text information and store it on multiple nodes k. Balance the data volume and processing capacity of each shard to optimize the overall processing efficiency. Use a distributed processing algorithm to achieve parallel data processing.

[0067] Typically, using the RoBERTa algorithm for classification consumes significant computational resources. Furthermore, when processing large-scale short text datasets, the processing power, storage capacity, and real-time response of a single computing unit are insufficient to effectively handle the entire dataset. To improve computation time, optimize resource utilization, and increase work efficiency, a distributed processing algorithm is introduced to optimize the efficiency of intelligent analysis and classification of short texts. This algorithm schedules resources from different nodes to achieve load balancing and classify short texts. In this embodiment, the distributed processing algorithm distributes the tasks of intelligent analysis and classification of short texts, thereby optimizing the efficiency of this process.

[0068] The specific steps of the load balancing data sharding algorithm in step S2 are as follows:

[0069] S2.1: Data volume assessment. Evaluate the size and complexity of the short text information dataset D. Through statistical calculations of the dataset, determine the number of records N in dataset D and the average number of features M for each record.

[0070] S2.2: Computational power assessment, evaluating the computational power C of each node k used to process the data. k .

[0071] Computational ability C k The calculation formula is:

[0072]

[0073] Among them, C k For the computing power of node k, CPU cores，k Let k be the number of CPU cores. speed，k For the computing speed of each core, MEM available，k Let k be the total available memory and I / O. speed，k For the data read / write speed of node k, NET bandwidth，k The network bandwidth of node k, These are weighting coefficients used to adjust the relative importance of each factor; all the above variables have been standardized to be converted to the same magnitude and range.

[0074] S2.3: Based on the number of records N, average number of features M, and computing power C k Develop a sharding strategy and determine the number S of records S allocated to each node k. k This achieves balanced load across all nodes, while also avoiding excessively long sharding processing times due to data homogeneity.

[0075] Number of records S k The calculation formula is:

[0076]

[0077] Among them, S k The number of records to be allocated to node k, where K is the total number of nodes, Num is the total number of records in dataset D, and E represents the amount of data to be processed. k λ is the base value of the computing power of node k, λ is an adjustment factor used to balance the relationship between computing overhead and communication overhead, and O is the average communication overhead.

[0078] S2.4: According to the recorded number S k The result is that the dataset DD is sorted according to S k The number of records S is determined k The data is then segmented, and the short text fragments are actually allocated to each node k.

[0079] S2.5: Execute steps S3, S4 and S5 in sequence to obtain the final analysis and classification results of the short text information.

[0080] S3: Each node k introduces a contextual deep understanding algorithm to optimize the accuracy of short text information classification.

[0081] In the process of analyzing and classifying short text information, we often encounter situations involving polysemous words and sparse text. Polysemous words are a common phenomenon in natural language, and they can easily lead to misinterpretations and misunderstandings in semantic analysis in different contexts. When short text is sparse, it can easily cause semantic ambiguity, resulting in inaccurate classification of short text information. In order to improve the accuracy of short text classification, we introduce a deep contextual understanding algorithm.

[0082] The specific steps for introducing the contextual deep understanding algorithm in step S3 are as follows:

[0083] S3.1: Construct text cohesion representations for the pre-processed short text fragments. Based on the characteristics of polysemous words and context-dependent words, construct a context-enhanced embedding representation C(w) for each word that is associated with its context. i ).

[0084] Context-enhanced embedding representation C(w) i The calculation formula is:

[0085]

[0086] Among them, C(w) i ) represents the context-enhanced embedding representation, σ is a nonlinear activation function used to introduce nonlinearity and increase the expressive power of the model, and w i For the current target word, v j For the word w j The vector representation of the captured word w j The basic semantic information, where n is the size of the context window, determines the calculation of the current target word w. i When embedding context, how far should the range of neighboring words be considered? j The weights are adjusted based on distance to balance the impact of words at different distances on the current target word w. i The effect of contextual embedding, POS(w j ) for word w j The part-of-speech tag vectors represent grammatical roles such as nouns and verbs, with β being a moderating factor used to balance the word vectors v. j and part-of-speech vector POS(w j The contribution of D(w) to context embedding, i w j ) is the distance function, representing w i with wj The output is a function of the distance between them, and decreases as the distance increases. γ is an adjustment factor used to control D(w). i w j The strength of the influence of context embedding, S(w) i ) for w i Additional semantic features, such as semantic category information of synonyms, antonyms, or words, are used to adjust S(w). δ is a weight parameter used to adjust S(w). i The extent of the impact on the final context embedding.

[0087] S3.2: To capture the deeper semantics of words in specific contexts, further context-related feature extraction is performed. A feature extraction function F is defined, and the context-enhanced embedding representation C(w) is embedded. i Transform into feature vector f i .

[0088] eigenvector f i The calculation formula is:

[0089] f i =F(C(w) i ))=ReLU(W f2 ·ReLU(W f1 ·C(w i )+b f1 )+b f2 ),

[0090] Among them, f i For feature vectors, for word w i High-level representations within a given context can reflect the deeper semantic information of words; W f1 and W f2 W represents two weight matrices from different layers, used to transform the context embedding vector into a feature vector. f1 Used for the first-level transformation, W f2 Used for the second-layer transformation, it learns more complex feature representations from the context embeddings by adjusting the weights of each layer; b f1 and b f2 For the corresponding W f1 and W f2 The bias term is used to add a bias to the nonlinear transformation of each layer, ensuring that some neurons are activated even when the input is zero; ReLU is a nonlinear activation function used to increase the nonlinearity of the model and capture more complex features.

[0091] S3.3: The final eigenvector f i Steps S4 and S5 are executed sequentially. After the model is converted into vectors by the RoBERTa pre-trained model, subsequent intelligent analysis and classification are performed.

[0092] By constructing a context-enhanced embedding representation C(w) for each word that is associated with its context. i This improves the ability to understand polysemous words and context-dependent vocabulary; deep context understanding helps to accurately capture semantic changes, enabling a more accurate understanding of the true meaning of short texts containing polysemous words or specific context-dependent vocabulary; by defining a feature extraction function F, it is possible to transform and extract feature vectors f reflecting the deep semantics of words in specific contexts from context embeddings. i This improves the expressiveness of features, enhances the model's ability to capture complex semantic relationships, and provides richer and more accurate feature representations for subsequent classification. Classification using context-related features enables decision-making based on more refined semantic understanding, thereby significantly improving the accuracy of short text classification.

[0093] S4: Each node k uses an optimized natural language processing algorithm to intelligently analyze the short text fragments and determine their category.

[0094] The optimized natural language processing algorithm includes RoBERTa pre-trained model transformation and a recurrent neural network for intelligent analysis and classification of vectorized short text data. The recurrent neural network includes RNN layers, fully connected layers, softmax layers, and threshold judgment layers. The specific steps in step S4 for using the optimized natural language processing algorithm to intelligently analyze the segmented data and determine its category are as follows:

[0095] S4.1: The pre-processed short text segment data consists of words w1, w2, ..., w N Composition, where N represents the number of words, using the optimized RoBERTa model in natural language processing to process each word w N The data is converted into vector representations to obtain vectorized short text fragments. Markers for marking the start and end of the text are added to the beginning and end of the short text fragments, respectively.

[0096] RoBERTa can accept complete sentences as input and output high-dimensional vector representations of each text unit in the sentence, with the vectors containing contextual semantic information of the words.

[0097] S4.2: Utilize RoBERTa to extract short text vectors with text start identifiers from the vectorized short text segments and input them into a fully connected layer. These short text vectors, as a comprehensive semantic representation of the entire text, contain contextual information about the entire sentence, making them suitable for classification tasks.

[0098] By using RoBERTa to process vectorized short text fragments, it is possible to effectively handle temporal information in text sequences and improve the performance of classification tasks.

[0099] S4.3: In the fully connected layer, the vector with the text start identifier is mapped to the score vector z to obtain the score of the category represented by each element in the score vector z, and the score vector z is sent to the Softmax layer; in this embodiment, the score refers to the probability of the text belonging to each category.

[0100] S4.4: In the Softmax layer, the Softmax function converts the score vector z into class probability values ​​P(classq), which are used to indicate the probability distribution of short text fragments in each class;

[0101] Class probability value P(class) q The formula for calculating ) is:

[0102]

[0103] Where, P(class) q ) is the category probability value of category q after processing by the Softmax function, which is used to represent the probability that the short text belongs to category q. Softmax ensures that the sum of the probability values ​​of all categories is equal to 1;

[0104] The score for category q is the output of the fully connected layer, which is converted into a probability in the Softmax layer.

[0105] S4.5: Calculate the probability value P(class) for each class. q The output of the Softmax layer is fed into the thresholding layer. In the thresholding layer, a threshold vector Θ, Θ = {θ1, θ2, ..., θ...} is defined based on expert experience. M}, where M represents the number of thresholds, and only when the probability value of the class P(class) is... q The short text is only classified into the category when its value falls between two thresholds, and all class probabilities P(class) are calculated. q After threshold judgment, the category of short text information is obtained. In this embodiment, the category is set as: negative, neutral, positive.

[0106] A threshold layer is introduced after the fully connected layer and the Softmax layer to provide more detailed control over the classification results. By setting a threshold vector, the sensitivity of classification can be adjusted according to actual needs, so that the model can achieve the best performance in different application scenarios.

[0107] S5: Summarize and analyze the classification results on all nodes k to obtain the final analysis and classification results of the short text information. In this embodiment, the technical solution for summarizing and analyzing the classification results adopts technical means well known to those skilled in the art, and will not be described in detail here.

[0108] To optimize the data synchronization process between nodes and reduce data redundancy and transmission latency, a data synchronization optimization algorithm is introduced. The goal of this algorithm is to optimize the data synchronization process in a distributed environment, reduce redundant transmissions, minimize synchronization latency, and ensure data consistency without sacrificing overall performance. The specific steps of the data synchronization optimization algorithm are as follows:

[0109] S10.1: Initialization and node evaluation; using the computational power C of node k k The computing power of node k is evaluated to lay the foundation for subsequent data synchronization. At the same time, detailed information on the node's resources and network status is obtained using expert methods, including CPU capacity, memory capacity, network bandwidth, etc. A network topology diagram is drawn to determine the optimal data transmission path between nodes k. The drawing of the network topology diagram and the determination of the optimal data transmission path adopt techniques well known to those skilled in the art, which will not be elaborated here.

[0110] S10.2: Data Dependency and Access Frequency Analysis; Introducing a dependency analysis algorithm to analyze the dependencies between data and the access frequency of each node to the data, and determining the total dependency score I for each data. i ;

[0111] Total Dependency Score I i The calculation formula is:

[0112] I i =∑ j≠i F ij ,

[0113] Among them, I i The total dependency score for data i is calculated by summing the dependency scores F between data i and all other data j. ij Obtain; F ij The dependency score between data i and data j.

[0114] F ij The calculation formula is:

[0115]

[0116] Among them, F ij A is the dependency score between data i and data j. it and A jt Let be the number of times node k accesses data i and data j within time t, respectively. The time decay factor considers the effect of time distance on the dependency, where δ is the decay parameter and t is the time decay factor. k This is the current time. R il and R jl Let θ represent the roles of data i and data j in the first-level dependency. lλ represents the weight of the l-th layer dependency, and λ1, λ2, and λ3 are coefficients that adjust for the various influencing factors. L is the number of layers of dependency, and T is the total time period.

[0117] The above formulas increase the depth and breadth of dependency analysis by incorporating time series analysis and multi-level dependencies; the time decay factor allows the model to give greater weight to recent access behavior, thus more accurately reflecting the current dynamic dependencies between data. Multi-level dependency analysis provides a comprehensive consideration of not only direct relationships between data but also indirect relationships, enabling synchronization strategies to more comprehensively take into account the complex interaction patterns between data.

[0118] In this way, dependency analysis algorithms can not only more accurately identify the relationships between data, but also provide more detailed information for data synchronization plans, thereby optimizing the performance and efficiency of the entire distributed processing.

[0119] S10.3: Based on the total dependency score I i The data priority levels are divided into high priority group, medium priority group, and low priority group, with high-scoring data belonging to the high priority group.

[0120] In this embodiment, the data priority level division strategy is as follows:

[0121] High priority group: the top 25% of the data;

[0122] Medium priority group: the middle 50% of the data;

[0123] Low priority group: the bottom 25% of the data.

[0124] S10.4: Based on data priority levels, formulate a data synchronization plan to ensure that high-priority groups, medium-priority groups, and low-priority groups are synchronized in sequence to ensure the consistency and availability of critical data.

[0125] In this embodiment, during data synchronization, the synchronization order for different priority levels is as follows:

[0126] High-priority group data: Ensures that this data is updated first in any given synchronization cycle.

[0127] Medium priority group data: Synchronize after high priority data synchronization is completed.

[0128] Low-priority group data: Synchronize when the load is low, or when it is certain that it will not affect the synchronization of high and medium-priority data.

[0129] After implementing data synchronization, continuously monitor system performance and data integrity, and adjust data priorities and synchronization plans as needed. If high-priority data synchronization is found to impact system performance, consider adjusting the synchronization frequency or optimizing the data processing flow. Periodically recalculate the dependency score F. ij This is to reflect any changes in the dependencies between data.

[0130] This method reduces the load on individual nodes and shortens the overall processing time by sharding short text information and distributing it across multiple computing nodes for parallel processing, thus significantly improving processing efficiency. The introduction of a load-balanced data sharding algorithm aims to balance the data volume and processing capacity of each shard, ensuring that each computing node can efficiently utilize its resources. This balanced allocation maximizes system resource utilization and avoids resource waste. The distributed processing architecture reduces the risk of single points of failure and enhances system stability.

[0131] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A short text intelligent analysis and classification optimization method, characterized in that: The method comprises the following steps: S1: performing preliminary processing on the short text information to obtain preliminary processed short text information; S2: introducing a balanced load data sharding algorithm to perform data sharding on the preliminary processed short text information and store the short text information to multiple nodes k, and using a distributed dispersion processing algorithm to realize data parallel processing; S3: introducing a context depth understanding algorithm to optimize the accuracy of short text information classification; The specific steps of introducing the context depth understanding algorithm in step S3 are as follows: S3.1: Constructing text coherence representations for the short text shard data after preliminary processing, constructing for each word a context enhanced embedding representation associated with its context ; S3.2: To capture the deep semantics of a word in a particular context, define a feature extraction function Contextual augmented embedding representation Convert to feature vector ; S3.3: the resulting feature vector is sent to the classifier The steps S4 and S5 are executed sequentially for subsequent intelligent analysis and classification; The context-enhanced embedding represents The calculation formula is: ， wherein, is a contextual augmented embedding representation, is a non-linear activation function; is a current target word, is a word vector representation, capturing the basic semantic information of the word , is the size of the context window, is a distance-adjusted weight, is a part-of-speech tagging vector of the word , is a tuning factor for balancing the contribution of the word vector and the part-of-speech vector to the contextual embedding, is a distance function representing the function of the distance between , is a tuning factor for controlling the strength of the influence of on the contextual embedding, is an additional semantic feature for , is a weight parameter for adjusting the influence of on the final contextual embedding; S4: each node k uses an optimized natural language processing algorithm to intelligently analyze the obtained short text data and determine the category to which the short text data belongs; The optimized natural language processing algorithm comprises a RoBERTa pre-trained model conversion and a recurrent neural network for intelligent analysis and classification of vectorized short text data, and the recurrent neural network comprises an RNN layer, a full connection layer, a Softmax layer, and a threshold judgment layer; S5: aggregating and analyzing the classification results on all nodes k to obtain the final analysis and classification results of the short text information.

2. The short-text intelligent analysis and classification optimization method of claim 1, wherein, The specific steps of the balanced load data sharding algorithm in step S2 are as follows: S2.1: Assessing a dataset of short text information the size and complexity of the dataset the number of records of the dataset and the average number of features per record ; S2.2: Assessing the computing power of each node k for processing data ; S2.3: Based on the number of records N, the average feature number M and the computing power , make a fragmentation strategy to determine the number of records allocated to each node k ; S2.4: According to the record number As a result, the short text fragment data is actually allocated to each node k; S2.5: sequentially performing steps S3, S4 and S5 to obtain the final analysis and classification results of the short text information.

3. The short-text intelligent analysis and classification optimization method of claim 2, wherein, The computing capability The computing formula is: ， wherein, is the computing power of node k, is the number of CPU cores of node k, is the computing speed of each core, is the total amount of available memory of node k, is the data read-write speed of node k, is the network bandwidth of node k, is a weight coefficient for adjusting the relative importance of each factor; The number of records The calculation formula is: ； wherein, is the number of records assigned to node k, K is the total number of nodes, is the total number of records in the dataset is the total number of records in the dataset is the base value of the computing power of node k, is the adjustment factor used to balance the relationship between the computing overhead and the communication overhead; is the average communication overhead.

4. The short-text intelligent analysis and classification optimization method of claim 1, wherein, The feature vector The calculation formula is: ， wherein, is a feature vector reflecting the deep semantic information of a word, and are weight matrices of two different layers, is used for the first layer conversion, is used for the second layer conversion, and is a bias term corresponding to and is a bias term corresponding to is a nonlinear activation function.

5. The short-text intelligent analysis and classification optimization method of claim 1, wherein, The specific steps of intelligently analyzing the sharded data and determining the category to which the sharded data belongs in step S4 are as follows: S4.1: The short text fragment data after preliminary processing consists of words Composition, in which The number of words is represented by the RoBERTa model in optimized natural language processing, which is used to represent each word. The data is converted into vector representations to obtain vectorized short text fragments. Markers for marking the start and end of the text are added at the beginning and end of the short text fragments, respectively. S4.2: using the RoBERTa to extract a short text vector with a text start identifier from the vectorized short text sharded data, and inputting the short text vector into a full connection layer; S4.3: At the fully connected layer, the vector with the text start identifier is mapped to a score vector , obtaining a score vector , where each element represents the score of the class, and the score vector is fed into a Softmax layer; S4.4: At the Softmax layer, the Softmax function converts the score vector into class probability values indicating the probability distribution of the short text snippet data over the various classes. S4.5: the category probability value a threshold value judgment layer, the threshold value judgment layer being provided with a threshold value vector , wherein denotes the number of threshold values, when the category probability value is located between two threshold values, the short text is classified into the category.

6. The short-text intelligent analysis and classification optimization method according to claim 5, characterized in that, The category probability value The calculation formula is: ， wherein, is the class probability value of class q after being processed by the Softmax function, is the score of class q from the output of the fully connected layer.

7. The short-text intelligent analysis and classification optimization method of claim 1, wherein, In the process of parallel processing of short text sharded data by each node k, in order to reduce redundant transmission between nodes k during data synchronization, a data synchronization optimization algorithm is introduced, and the specific steps of the data synchronization optimization algorithm are as follows: S10.1: drawing a network topology diagram to determine the optimal data transmission path between nodes k; S10.2: Introduce dependency analysis algorithm to determine total dependency score for each data ; S10.3: Based on the total dependency score The data priority level is divided into a high priority group, a medium priority group, and a low priority group. S10.4: according to the data priority level, making a data synchronization plan to make the high priority group, the medium priority group, and the low priority group realize synchronization in turn.

8. The short-text intelligent analysis and classification optimization method of claim 7, wherein, the total dependency score The formula for the calculation is: ， wherein, is the total dependency score for data , is the dependency score between data and data . The The calculation formula is: ， wherein, is the data and data , is the dependency score between data and data , , is the number of visits of node k to data and data at time t, is the time decay factor, is the decay parameter, is the current time, is the role of data and data in the l-th layer of dependency, , is the weight of the l-th layer of dependency, is the number of layers of dependency, is the total time period.

Images