Two-stage classification method and system embedding semantic feature encoding of technical text

Abstract

The application discloses a two-stage classification method and system embedded with technical text semantic feature coding, relates to the technical field of natural language processing, and comprises the following steps: extracting local and global topic features to construct topic semantic features, and performing hierarchical aggregation based on word item dependency relations to construct structural semantic features; embedding the two into LLM coding layers to generate a fusion feature sequence; extracting classification features through a decoder and performing binary classification judgment to obtain a value sentence sequence; constructing a category feature template and calculating sub-class semantic features; and performing multi-classification through a semantic perception routing mechanism and a hybrid expert model. The application solves the technical problems that existing text classification cannot fully utilize deep semantic information, resulting in insufficient classification accuracy, and performs poorly in class imbalance and fine-grained classification, and achieves the technical effects of significantly improving classification accuracy and class imbalance and fine-grained classification ability by introducing double semantic feature coding and a hybrid expert model.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, specifically to a two-stage classification method and system for embedding semantic feature encoding of scientific and technological texts. Background Technology

[0002] Text classification is a fundamental task in natural language processing, widely used in information retrieval, sentiment analysis, and topic classification. Traditional text classification methods mainly rely on statistical features such as the bag-of-words model and TF-IDF, but these methods have limitations when dealing with texts that are structurally complex or semantically deep. Especially in scientific and technical texts, the text structure is complex, the terminology is specialized, and the context is rich, making it difficult for traditional methods to fully extract the deep semantic information. In recent years, deep learning, especially Large Language Models (LLMs), has made significant progress in text classification, but it still faces some challenges, such as insufficient utilization of text semantic features, performance degradation due to class imbalance, and insufficient fine-grained classification capabilities. Most existing methods rely on a unified classification framework, which cannot effectively handle fine-grained semantic differences. Summary of the Invention

[0003] This application provides a two-stage classification method and system for embedding semantic feature encoding of scientific and technological texts, which is used to address the technical problem that existing text classification cannot fully utilize deep semantic information, resulting in insufficient classification accuracy and poor performance in class imbalance and fine-grained classification.

[0004] Firstly, this application provides a two-stage classification method for embedding semantic feature encoding of scientific and technological texts. The method includes: inputting a source text sequence; extracting local topic features and calculating global topic relevance in the source text sequence to construct topic semantic features; extracting basic feature representations for each term in the source text sequence, and performing hierarchical feature aggregation on the basic feature representations based on term dependency relationships to obtain an aggregated hierarchical structure; representing inter-sentence relationships based on the aggregated hierarchical structure to construct structural semantic features; embedding the topic semantic features and the structural semantic features into the encoding layer of LLMs; replacing the CLS feature representation with the topic semantic features; and constructing structural semantic features. The SEP feature representation is replaced to generate a fused feature sequence. The fused feature sequence is then processed by an LLM decoder to extract classification features, outputting a classification feature vector. This vector is then used for binary classification based on a classifier to obtain a set of value sentence sequences. The set of value sentence sequences and their corresponding category labels are input to construct a category feature template. Based on the category prototype features of the template, subclass semantic features are calculated. Based on these subclass semantic features, a semantic-aware routing mechanism and a hybrid expert model are designed. The semantic-aware routing mechanism dynamically allocates the set of value sentence sequences to the hybrid expert model for two-stage multi-class classification, outputting the text classification result.

[0005] Secondly, this application provides a two-stage classification system embedding semantic feature encoding of scientific and technological texts. The system includes: a topic semantic feature construction module, used to input a source text sequence, extract local topic features and calculate global topic relevance to the source text sequence, and construct topic semantic features; a hierarchical feature aggregation module, used to extract the basic feature representation of each term in the source text sequence, and perform hierarchical feature aggregation on the basic feature representation based on term dependency relationships to obtain an aggregated hierarchical structure, and represent inter-sentence relationships based on the aggregated hierarchical structure to construct structural semantic features; and a feature fusion module, used to embed the topic semantic features and the structural semantic features into the encoding layer of LLMs, replacing the CLS feature representation with the topic semantic features, and the structural semantic features... The system replaces the SEP feature representation to generate a fused feature sequence; a binary classification judgment module is used to extract classification features from the fused feature sequence through the decoder of LLMs, output classification feature vectors, and perform binary classification judgment on the classification feature vectors based on the classifier to obtain a set of value sentence sequences; a subclass semantic feature calculation module is used to input the set of value sentence sequences and the corresponding category labels, construct a category feature template, and calculate the subclass semantic features based on the category prototype features of the category feature template; a two-stage multi-classification module is used to design a semantic-aware routing mechanism and a hybrid expert model based on the subclass semantic features, dynamically allocate the set of value sentence sequences to the hybrid expert model for two-stage multi-classification through the semantic-aware routing mechanism, and output the text classification result.

[0006] One or more technical solutions provided in this application have at least the following technical effects or advantages:

[0007] The two-stage classification method and system for embedded semantic feature encoding of scientific and technological text provided in this application relates to the field of natural language processing technology. By combining the extraction and fusion of topic semantic features and structural semantic features, text classification is optimized. The first stage constructs semantic features through local and global topic feature extraction and hierarchical feature aggregation. The second stage achieves dynamic expert selection and fine classification through a semantic-aware routing mechanism and a hybrid expert model, thereby improving the accuracy and adaptability of text classification. It performs particularly well in complex text and class imbalance situations. It solves the technical problem that existing text classification cannot fully utilize deep semantic information, resulting in insufficient classification accuracy and poor performance in class imbalance and fine-grained classification. It achieves the technical effect of significantly improving classification accuracy and class imbalance and fine-grained classification capabilities by introducing dual semantic feature encoding and a hybrid expert model. Attached Figure Description

[0008] Figure 1 A schematic diagram of the overall framework of the two-stage text multi-classification method for embedding semantic feature encoding of scientific and technological texts provided in this application;

[0009] Figure 2 This application provides a flowchart illustrating the extraction of topic semantic features in a two-stage classification method that embeds semantic feature encoding of scientific and technological texts.

[0010] Figure 3 This application provides a flowchart illustrating the semantic feature encoding embedding mechanism in a two-stage classification method that embeds semantic features of scientific and technological texts.

[0011] Figure 4 This application provides a flowchart illustrating the semantic-aware routing mechanism in a two-stage classification method that embeds semantic features of scientific and technological texts.

[0012] Figure labeling: Topic semantic feature construction module 11, hierarchical feature aggregation module 12, feature fusion module 13, binary classification judgment module 14, subclass semantic feature calculation module 15, two-stage multi-classification module 16. Detailed Implementation

[0013] This application provides a two-stage classification method and system that embeds semantic feature encoding of scientific and technological texts to solve the technical problem that existing text classification methods cannot fully utilize deep semantic information, resulting in insufficient classification accuracy and poor performance in class imbalance and fine-grained classification.

[0014] Example 1, as Figure 1 As shown, this application provides a two-stage classification method that embeds semantic feature encoding of scientific and technological texts. The method includes:

[0015] P10: Input the source text sequence, extract local topic features and calculate global topic relevance to construct topic semantic features.

[0016] Furthermore, such as Figure 2 As shown, step P10 in this embodiment further includes:

[0017] P11: A sliding window is used to scan the source text sequence, and the topic aggregation representation of the terms within the window is calculated to extract local topic features. Where σ(·) is the activation function, w is the sliding window size, and e j The vector representation of the j-th term, α j The corresponding attention weights are calculated as follows: α j =softmax(e j ·W q ·e i ), where W q e is a learnable query matrix parameter iP12: The local topic features are pooled to obtain the global topic representation. (The vector representation of the central term is used, and softmax is used to normalize the attention score.) Where Pool(·) represents the pooling operation; P13: Concatenate the local topic features and the global topic representation to construct the topic semantic feature v. topic (i)=tanh(W t ·[h local (i); h global ]+b t ), where W t and b t Here, represents the transformation matrix and the bias term, respectively; [;] denotes the feature concatenation operation, and tanh is the hyperbolic tangent activation function.

[0018] It should be understood that in-depth semantic analysis of the input source text sequence is crucial for constructing topical semantic features that accurately reflect the text's subject matter. This step is key to achieving efficient text classification tasks, and its implementation includes two main stages: local topic feature extraction and global topic relevance calculation.

[0019] First, in the local topic feature extraction stage, this method employs a sliding window technique to scan the source text sequence. The size of the sliding window is defined by the parameter w, and this window slides across the text sequence, calculating the topic aggregation representation for each term within the window. Specifically, for each term e within the window... j Its vector representation and corresponding attention weight α j Together, they determine the contribution of the term to the local topic features. As mentioned above, the calculation formula for attention weights is an activation function, specifically the softmax function, which is used to calculate the attention weights between terms, ensuring the importance of the focused term in the local context. The softmax function obtains a normalized attention score by calculating the influence of each term on other terms in the window. Where W... q e is a learnable query matrix parameter i For the vector representation of the central term, softmax normalizes these weights so that their sum is 1.

[0020] After extracting local topic features, a pooling operation (Pool(·)) is performed to obtain a global topic representation. The purpose of pooling is to aggregate local features into a single global feature, capturing the global topic information of the entire text. This step simplifies information and enhances global understanding by pooling local features. Specifically, the pooling operation integrates local topic features to obtain a global topic representation for each text sequence:

[0021] Finally, the local topic features and the global topic representation are concatenated to construct the final topic semantic features. This process includes concatenating the local and global features using a specific transformation matrix W. t and bias term b t A linear transformation is performed to obtain a merged representation. Finally, this feature is activated using the hyperbolic tangent activation function (tanh) to generate the desired topic semantic feature representation. The mathematical expression for feature concatenation is as follows: v topic (i)=tanh(W t ·[h local (i); h global ]+b t In this process, [;] represents the feature concatenation operation, which combines local and global features into a long vector, and then uses W... t and b t After performing a linear transformation, the tanh function is used for nonlinear activation.

[0022] Through the steps described above, this application can effectively extract semantically rich thematic features from source text sequences, laying a solid foundation for subsequent text classification tasks. This method can not only capture local details of the text but also understand the overall structure and theme, thereby achieving higher accuracy and better generalization ability in multi-classification tasks.

[0023] P20: Extract the basic feature representation of each term in the source text sequence, and perform hierarchical feature aggregation on the basic feature representation based on term dependency relations to obtain the aggregated hierarchical structure. Based on the aggregated hierarchical structure, perform inter-sentence relation representation to construct structural semantic features.

[0024] Furthermore, step P20 in this embodiment of the application also includes:

[0025] P21: Extract the basic feature representation h of each term in the source text sequence. init (i)=FFN([e i ;pos i ]), where e i For the vector representation of a term, pos i For positional encoding, FFN is a feedforward neural network, and [;] represents feature concatenation operation; P22: Based on term dependency relations, hierarchical feature aggregation is performed on the basic feature representation to obtain the aggregation hierarchical structure h. l (i)=σ(∑ j∈N(i) w j ·h l-1 (j)), where σ(·) is the activation function, N(i) represents the set of neighboring terms of term i, and w jFor aggregation weights, l represents the layer number, h l-1 (j) represents the feature representation of the (l-1)th layer; P23: Based on the aggregated hierarchical structure, represent the inter-sentence relationships and construct the structural semantic features. Where Pool(·) is the pooling operation, h L (i) represents the term representation of the last level, v struct This represents the final structural features.

[0026] Optionally, structural semantic features can be extracted from the source text sequence. These features can reflect the dependency relationships between terms and the logical structure between sentences in the text, which is crucial for understanding the deeper meaning of the text.

[0027] First, the basic feature representation of each term in the source text sequence is extracted. The basic feature representation of each term is its vector representation e. i This vector captures the semantic information of the term. To help the model understand the order of terms in the text, the positional encoding pos... i It is added to the vector representation of each word to ensure that the model can perceive the relative position of the words.

[0028] Next, the feedforward neural network (FFN) concatenates the vector representation of each term with its positional encoding, and through a series of transformations, generates the final basic feature representation h for each term. init (i)=FFN([e i ;pos i The key to this process is that the FFN network, through learning, can process and transform the basic features of terms, thereby enhancing their expressive power.

[0029] After extracting the basic features of each term, a higher-level feature representation is obtained through hierarchical feature aggregation based on term dependency relationships. This process constructs the relationships between terms layer by layer by aggregating information from each term and its neighboring terms.

[0030] Specifically, a feature aggregation structure is constructed based on the dependency relationships between terms. In this structure, the feature representation h of each term i is... l (i) is updated to be its relation to the adjacent term h l-1 The aggregation result of (j). The aggregation process uses a weighted sum, where the aggregation weight w j This represents the strength of the dependency relationship between term i and term j. That is, for each term i, its feature representation h at layer l. l (i) is obtained by using the feature representation h of the terms in its neighboring term set N(i). l-1 (j) is obtained by weighted summation.

[0031] After hierarchical feature aggregation, pooling operations are used to merge the features of each term into a global representation, constructing inter-sentence relationship features. Pooling operations effectively aggregate information while preserving the most critical structural semantic features of the text. Specifically, pooling operations aggregate the features of all terms, for example, using max pooling or average pooling methods, to obtain a global representation of the entire sentence or paragraph. These structural features can capture the logical structure within a sentence and the relationships between sentences. The formula is expressed as follows:

[0032] Through the above steps, this application can effectively extract structural semantic features from the source text sequence. These features can not only reflect the dependency relationship between terms, but also reveal the deep structural information of the sentence.

[0033] P30: As Figure 3 As shown, the topic semantic features and the structural semantic features are embedded into the encoding layer of LLMs, the topic semantic features replace the CLS feature representation, and the structural semantic features replace the SEP feature representation to generate a fused feature sequence.

[0034] Specifically, the topic semantic features and structural semantic features extracted in the preceding steps are embedded into the encoding layer of Large Language Models (LLMs). Specifically, topic semantic features replace the [CLS] marker in LLMs, while structural semantic features replace the [SEP] marker. This feature substitution operation helps the model more effectively capture the topic information and inter-sentence structural information of the text.

[0035] In traditional large language models, the [CLS] tag is used to represent global features of the entire text, typically for subsequent text classification tasks. The [SEP] tag, on the other hand, is used to separate different sentences or paragraphs within the text. In this approach, topic semantic features represent the core content of the text and effectively help the model understand the main information; therefore, they are used to replace the [CLS] tag. Simultaneously, structural semantic features reflect the syntactic and structural relationships in the text, including dependencies between sentences, and are therefore used to replace the [SEP] tag.

[0036] By embedding these two types of features into the encoding layer of LLMs, a fused feature sequence is generated. This fused feature sequence not only contains thematic information of the text but also incorporates structural information, enabling the model to more accurately capture the deep semantics of the text when performing subsequent classification or inference tasks. The text sequence after this feature fusion will be used for further processing and analysis, such as text classification or sentiment analysis.

[0037] This process enhances the semantic representation of the input text, ensuring that the model can fully utilize the contextual and structural information in the text, thereby improving the performance of subsequent tasks.

[0038] P40: The fused feature sequence is subjected to classification feature extraction by the decoder of LLMs, the classification feature vector is output, and the classification feature vector is subjected to binary classification judgment based on the classifier to obtain the set of value sentence sequences.

[0039] Optionally, after the preceding feature fusion processing, the resulting fused feature sequence will be input into the LLMs decoder for further processing. The decoder will extract a classification feature vector from this fused feature sequence. The classification feature vector represents the important feature information of the text in a specific task, and it will be used in subsequent classification tasks.

[0040] Specifically, the decoder in LLMs identifies information in the text that effectively distinguishes different categories by decoding the fused feature sequence. These features are integrated into a classification feature vector, which is a highly abstract and condensed version of the text's semantic and structural information. It contains the key information needed for text classification tasks and can serve as a high-level semantic representation of the text, facilitating classification judgment.

[0041] Next, a classifier is used to perform binary classification on the extracted feature vectors. This classifier can be a simple linear classifier, such as logistic regression, or a more complex neural network structure. Binary classification tasks typically involve dividing text into two classes, such as determining whether a sentence is a "value sentence" or a "non-value sentence." Through this process, the model can effectively filter out a set of value sentences that meet specific criteria. This set contains all sentences identified as meeting the criteria for value sentences. These sentences will serve as the foundational data for subsequent tasks (such as text summarization or value analysis), used for further analysis, or as input for other applications, such as text summarization, information retrieval, or knowledge graph construction. This process not only improves the accuracy of text classification but also provides valuable information for subsequent text processing tasks.

[0042] P50: Input the set of value sentence sequences and the corresponding category labels, construct a category feature template, and calculate the subclass semantic features based on the category prototype features of the category feature template.

[0043] Furthermore, in constructing the category feature template, step P50 of this embodiment also includes:

[0044] P51: Input the set of value sentence sequences X = <X1,X2…,X n > and the corresponding category label Y, construct the category feature template T. c=Pool({v topic (x i )|y i =c}), where c represents the value sentence category, v topic (x i ) represents the semantic features of the topic.

[0045] Optionally, a category feature template can be constructed by inputting a set of value sentence sequences and their corresponding category labels, and then the subclass semantic features can be calculated. This step is crucial for achieving fine-grained text classification because it allows the model to learn the unique semantic features of different categories of text.

[0046] Specifically, the input is first a set of value sentence sequences X, which contains multiple value sentences X1, X2, ..., X... n And their corresponding category labels Y. Each value sentence has been processed in the previous steps, and its topic semantic features v have been extracted. topic (x i These thematic semantic features contain the core thematic information of the text and form the basis for constructing category feature templates.

[0047] Next, construct the categorical feature template T. c This template is obtained by aggregating the topic semantic features of all value sentences under the same category c. Specifically, the pooling operation Pool(·) is used to aggregate the topic semantic features of all value sentences under category c, constructing the category feature template: T c =Pool({v topic (x i )|y i =c}), where y i Value sentence x i The category label is 'c', where 'c' represents a specific value category. Pooling is a common feature aggregation method that can extract global information from local features, providing a more comprehensive and abstract feature representation for subsequent classification tasks.

[0048] After obtaining the category feature template, the next step is to calculate subclass semantic features. Subclass semantic features represent the model's deep understanding of text within a specific category, revealing the unique semantic patterns of that category. In this application, subclass semantic features are calculated by interacting the topic semantic features of the input sentence with the category feature template. These subclass semantic features accurately capture the unique semantic information of each category, helping the model perform more detailed classification. To achieve this, a template can be constructed based on the category labels, and within each category, the topic semantic features v are used... topic (x i The core features of a category are represented by a label, thus ensuring that the semantic features of each category can be matched with its label.

[0049] Through the above process, the category feature template generates a unique set of subclass semantic features for each category, enabling the model to perform more precise fine-grained classification and improve classification accuracy. This process not only enhances the model's ability to distinguish between various features but also allows the model to optimize based on the features of different categories in multi-class scenarios.

[0050] Furthermore, in calculating the semantic features of the subclass, step P50 of this embodiment also includes:

[0051] P52: Based on the category prototype features of the category feature template, the subclass semantic feature v is calculated. sub (i)=σ(W c ·[v topic (x i ); P c ]), where W c The transformation matrix is ​​a category-specific matrix, σ(·) is the activation function, and v sub (i) represents the semantic features of the i-th subclass.

[0052] It should be understood that the calculation process for subclass semantic features can be further refined. For example, the subclass semantic features are first calculated based on the category prototype features of the category feature template. Category feature template T c It is obtained by aggregating the topic semantic features of all value sentences under the same category, representing the common semantic features of text in that category. Then, for each input sentence x i Calculate the semantic features v of its subclasses sub (i) This process involves taking the topic semantic features v of the input sentence topic (x i ) and category feature template P c Interact with the target audience and apply a category-specific transformation matrix W. c And the activation function σ(·), combined with the formula v sub (i)=σ(W c ·[v topic (x i ); P c ]) is used for calculation, where [v topic (x i ); P c ] indicates that the topic semantic features and category feature templates of the input sentence are concatenated. W c σ is a category-specific transformation matrix used to map the concatenated features to the semantic feature space of the subclass. σ(·) is an activation function used to introduce nonlinearity and enhance the expressive power of the model.

[0053] In this way, a subclass semantic feature vector can be computed for each input sentence. This vector not only contains the sentence's local semantic information but also incorporates common semantic features of the categories, thus providing a richer and more accurate feature representation for subsequent classification tasks. This feature representation can better capture the subtle differences between different categories of text, thereby improving the accuracy and robustness of classification.

[0054] Furthermore, the calculation of subclass semantic features involves the dynamic updating of category feature templates to adapt to constantly changing text data and category distributions. This dynamic update mechanism ensures that the model maintains high classification performance when facing new categories or new samples.

[0055] P60: As Figure 4 As shown, based on the semantic features of the subclass, a semantic-aware routing mechanism and a hybrid expert model are designed. The semantic-aware routing mechanism dynamically allocates the set of value sentence sequences to the hybrid expert model for two-stage multi-classification and outputs the text classification results.

[0056] Furthermore, step P60 in this embodiment of the application also includes:

[0057] P61: Each expert model in the hybrid expert model adopts a two-layer feedforward network structure E j (x)=FFN2 j (FFN1 j (x)), where FFN1 j and FFN2 j Each layer is a two-layer feedforward network for the j-th expert, with different parameter sizes to adapt to tasks of varying complexity; P62: Based on the semantic-aware routing mechanism, the topic semantic features are interacted with the routing parameters to construct the routing feature r(x) = W. r •Topicsemantic, where W r Let r(x) be the routing weight matrix, Topicemantic be the topic semantic feature, and r(x) be the routing feature; P63: Based on the routing features, calculate the expert selection probability p(j|x) = softmax(r(x)·h) of the sample in the value sentence sequence set and each expert model. j ), where h j P64: Based on the expert selection probability, dynamically allocate Top-k relevant FFN expert models for two-stage multi-class classification, and output the text classification result y = ∑(p(j|x)·FFNj(x)), where k is the preset number of experts and y is the final text classification result.

[0058] Optionally, a semantically aware routing mechanism and a hybrid expert model can be designed using subclass semantic features to achieve dynamic allocation of the value sentence sequence set and two-stage multi-classification, ultimately outputting the text classification result. This step is crucial for achieving fine-grained text classification, allowing the model to dynamically select the most suitable classification expert based on the text's semantic features, thereby improving classification accuracy and efficiency.

[0059] First, in the hybrid expert model, each expert model employs a two-layer feedforward network structure. The network structure of each expert can be configured with different parameter sizes based on the task complexity to adapt to different types of classification tasks. For example, the first layer network FFN1... j Second layer network FFN2 j Each of these forward propagation networks corresponds to a different expert model. These expert networks are responsible for handling different tasks and adapt to the complexity of the tasks through parameter adjustments.

[0060] Next, based on the semantic-aware routing mechanism, topic semantic features are interacted with routing parameters to construct routing features. This process involves integrating topic semantic features with the routing weight matrix W. r Interact with the system to obtain routing features. Here, the routing weight matrix W... r The weights in the routing process are controlled. Topic semantics represents the topic semantic features extracted from previous steps, and r(x) represents the routing features. The routing features are used to guide the model on how to assign the input text to the most suitable expert model.

[0061] Next, based on the constructed routing features, the expert selection probability p(j|x) = softmax(r(x)·h) for each sample in the value sentence sequence set and each expert model is calculated. j ), where h j Let represent the semantic representation vector of the j-th FFN expert. This step uses the softmax function to transform the dot product between the routing features and the semantic representation vector of the expert model into a probability distribution, thereby determining which expert models each sample is most likely to be assigned to.

[0062] Finally, based on the calculated expert selection probabilities, the top-k relevant FFN expert models are dynamically assigned for two-stage multi-class classification, outputting the final text classification result y = ∑(p(j|x)·FFNj(x)). Here, k is the preset number of experts, representing the selection of the k expert models with the highest probabilities to classify the input text. In this way, the model can dynamically select the most suitable classification experts based on the semantic features of the text, thereby improving the accuracy and efficiency of classification.

[0063] Through the steps described above, the model can dynamically allocate text samples to multiple expert models based on a semantically aware routing mechanism. These models then perform two-stage multi-class classification based on the expert selection probabilities. This method can distribute tasks among different expert models, thereby improving classification accuracy and effectively addressing the class imbalance problem.

[0064] Furthermore, step P64 in this embodiment of the application also includes:

[0065] P64-1: Based on the expert selection probability, a dynamic threshold expert selection strategy is adopted, specifically S={j|p(j|x)>τ j}, τ j =β·avg(p(j|X)), where S is the selected set of experts, τ j The threshold is dynamic, β is an adjustable coefficient, and avg(p(j|X)) is the average selection probability of the expert; P64-2: Using the expert selection strategy, dynamically allocate the Top-k selected FFN expert models; P64-3: Based on the selected FFN expert models, perform weighted combination collaboration to output the text classification result y=∑(w j ·E j (x)) / |S|, where w j The expert weight is calculated using the following formula: w j =softmax(q) j ·k x ), where q j For expert-specific query vectors, k x The key vector of the input sample.

[0066] Specifically, we can further refine how to dynamically allocate expert models based on expert selection probabilities and perform weighted combination collaboration to output text classification results.

[0067] In this process, a dynamic threshold expert selection strategy is first adopted based on the selection probability of each expert. Specifically, the expert selection strategy involves setting a dynamic threshold τ. j The threshold for determining whether to select an expert is adjusted by the average selection probability avg(p(j|X)). This ensures that only expert models with high relevance to the current sample are selected. The set of selected experts S is defined as satisfying p(j|x)>τ. j All expert models.

[0068] After expert selection, the top-k expert models are dynamically assigned. Here, "Top-k" represents selecting the k most relevant experts and assigning them to the input text samples for classification tasks. By dynamically adjusting the number and weights of the selected experts, the model can optimize its decision-making process based on task complexity and the characteristics of the input samples.

[0069] Finally, on page 64-3, a weighted combination of the selected FFN expert models is performed to output the text classification result. Specifically, each selected expert model will have a certain influence on the final classification result, and this influence is determined by the expert weight w. j The decision is made. The formula for calculating expert weights is w. j =softmax(q) j ·k x ), where q j For expert-specific query vectors, k x Let be the key vector of the input sample. This ensures that the contribution of each expert model is proportional to its match to the current sample. The final text classification result y is a weighted sum of the outputs of all selected expert models, i.e., y = Σ(w j ·E j (x)) / |S|, where |S| represents the size of the selected expert set.

[0070] In this way, this application can dynamically select the expert model best suited for the current sample and combine their outputs in a weighted manner based on the contribution of each model, thereby achieving efficient and accurate text classification. This method not only improves classification accuracy but also enhances the model's deeper understanding of text semantics, providing new possibilities for natural language processing tasks.

[0071] In summary, the embodiments of this application have at least the following technical effects:

[0072] This application improves the accuracy of text classification, especially when dealing with complex scientific and technical texts, and can effectively capture the topic and structural information of the text; it improves the adaptability of the model through two-stage classification and fine-grained subclass semantic feature extraction, especially in the case of class imbalance; it optimizes the expert selection process through semantic-aware routing mechanism and hybrid expert model, and improves the recognition ability of minority class samples; it enhances the flexibility and generalization ability of the model, and can perform well in different types of text classification tasks.

[0073] The technology has achieved significant improvements in classification accuracy, class imbalance, and fine-grained classification capabilities by introducing dual semantic feature encoding and a hybrid expert model.

[0074] Example 2: Based on the same inventive concept as the two-stage classification method embedding semantic feature encoding of scientific and technological texts in the foregoing examples, this application provides a two-stage classification system embedding semantic feature encoding of scientific and technological texts. The system and method embodiments in this application are based on the same inventive concept. The system includes:

[0075] The topic semantic feature construction module 11 is used to input the source text sequence, extract local topic features and calculate global topic relevance of the source text sequence, and construct topic semantic features.

[0076] The hierarchical feature aggregation module 12 is used to extract the basic feature representation of each term in the source text sequence, and perform hierarchical feature aggregation on the basic feature representation based on the term dependency relationship to obtain the aggregated hierarchical structure. Based on the aggregated hierarchical structure, the inter-sentence relationship representation is performed to construct structural semantic features.

[0077] The feature fusion module 13 is used to embed the topic semantic features and the structural semantic features into the encoding layer of LLMs, replace the CLS feature representation with the topic semantic features, replace the SEP feature representation with the structural semantic features, and generate a fused feature sequence.

[0078] The binary classification judgment module 14 is used to extract classification features from the fused feature sequence through the decoder of LLMs, output classification feature vectors, and perform binary classification judgment on the classification feature vectors based on the classifier to obtain a set of value sentence sequences.

[0079] The subclass semantic feature calculation module 15 is used to input the set of value sentence sequences and the corresponding category labels, construct a category feature template, and calculate the subclass semantic features based on the category prototype features of the category feature template.

[0080] The two-stage multi-classification module 16 is used to design a semantic-aware routing mechanism and a hybrid expert model based on the semantic features of the subclasses. The semantic-aware routing mechanism dynamically allocates the set of value sentence sequences to the hybrid expert model for two-stage multi-classification and outputs the text classification results.

[0081] Furthermore, the topic semantic feature construction module 11 is also used to perform the following steps:

[0082] The source text sequence is scanned using a sliding window, and the topic aggregation representation of terms within the window is calculated to extract local topic features. Where σ(·) is the activation function, w is the sliding window size, and e j The vector representation of the j-th term, α j The corresponding attention weights are calculated as follows: α j=softmax(e j ·W q ·e i ), where W q e is a learnable query matrix parameter i The vector representation of the central term is used, and softmax is used to normalize the attention score; pooling is performed on the local topic features to obtain the global topic representation. Where Pool(·) represents the pooling operation; the local topic features and the global topic representation are concatenated to construct the topic semantic feature v. topic (i)=tanh(W t ·[h local (i); h global ]+b t ), where W t and b t Here, represents the transformation matrix and the bias term, respectively; [;] denotes the feature concatenation operation, and tanh is the hyperbolic tangent activation function.

[0083] Furthermore, the hierarchical feature aggregation module 12 is also used to perform the following steps:

[0084] Extract the basic feature representation h of each term in the source text sequence. init (i)=FFN([e i ;pos i ]), where e i For the vector representation of a term, pos i For positional encoding, FFN is a feedforward neural network, and [;] represents a feature concatenation operation; based on term dependency relations, hierarchical feature aggregation is performed on the basic feature representation to obtain an aggregated hierarchical structure h. l (i)=σ(∑ j∈N(i) w j ·h l-1 (j)), where σ(·) is the activation function, N(i) represents the set of neighboring terms of term i, and w j For aggregation weights, l represents the layer number, h l-1 (j) represents the feature representation of the (l-1)th layer; the inter-sentence relationship representation is performed based on the aggregated hierarchical structure, and the structural semantic features are constructed. Where Pool(·) is the pooling operation, h L (i) represents the term representation of the last level, v struct This represents the final structural features.

[0085] Furthermore, the subclass semantic feature calculation module 15 is also used to perform the following steps:

[0086] Input the set of value sentence sequences X = <X1,X2…,X n > and the corresponding category label Y, construct the category feature template T. c =Pool({v topic (x i )|y i =c}), where c represents the value sentence category, v topic (x i ) represents the semantic features of the topic.

[0087] Furthermore, the subclass semantic feature calculation module 15 is also used to perform the following steps:

[0088] Based on the category prototype features of the category feature template, the subclass semantic features v are calculated. sub (i)=σ(W c ·[v topic (x i ); P c ]), where W c The transformation matrix is ​​a category-specific matrix, σ(·) is the activation function, and v sub (i) represents the semantic features of the i-th subclass.

[0089] Furthermore, the two-stage multi-classification module 16 is also used to perform the following steps:

[0090] Each expert model in the hybrid expert model adopts a two-layer feedforward network structure E j (x)=FFN2 j (FFN1 j (x)), where FFN1 j and FFN2 j Each layer is a two-layer feedforward network for the j-th expert, with different parameter sizes to adapt to tasks of varying complexity. The topic semantic features are interacted with the routing parameters according to the semantic-aware routing mechanism to construct the routing feature r(x) = W. r • Topic semantics, where W r Let r(x) be the routing weight matrix, r(x) be the topic semantic features, and r(x) be the routing features. Based on the routing features, calculate the expert selection probability p(j|x) = softmax(r(x).h) for each expert model among the samples in the value sentence sequence set. j ), where h jLet represent the semantic representation vector of the j-th FFN expert; based on the expert selection probability, dynamically allocate Top-k relevant FFN expert models for two-stage multi-class classification, and output the text classification result y = ∑(p(j|x)·FFNj(x)), where k is the preset number of experts and y is the final text classification result.

[0091] Furthermore, the two-stage multi-classification module 16 is also used to perform the following steps:

[0092] Based on the expert selection probability, a dynamic threshold expert selection strategy is adopted, specifically S={j|p(j|x)>τ j}, τ j =β·avg(p(j|X)), where S is the selected set of experts, τ j The threshold is dynamic, β is an adjustable coefficient, and avg(p(j|X)) is the average selection probability of the expert. Using the expert selection strategy, the Top-k selected FFN expert models are dynamically allocated. Based on the selected FFN expert models, a weighted combination and collaborative approach is performed to output the text classification result y = ∑(w j ·E j (x)) / |S|, where w j The expert weight is calculated using the following formula: w j =softmax(q) j ·k x ), where q j For expert-specific query vectors, k x The key vector of the input sample.

[0093] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, the above description focuses on specific embodiments of this specification. Additionally, the processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired results. In some implementations, multitasking and parallel processing are possible or may be advantageous.

[0094] The above description is only a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0095] This specification and accompanying drawings are merely illustrative examples of this application and are intended to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from its scope. Therefore, if such modifications and variations fall within the scope of this application and its equivalents, this application intends to include such modifications and variations.

Claims

1. A two-stage classification method embedding semantic feature encoding of scientific and technological texts, characterized in that, The method includes: Input a source text sequence, extract local topic features and calculate global topic relevance to construct topic semantic features; The basic feature representation of each term in the source text sequence is extracted, and the basic feature representation is hierarchically aggregated based on the term dependency relationship to obtain the aggregated hierarchical structure. The inter-sentence relationship is represented according to the aggregated hierarchical structure to construct structural semantic features. The topic semantic features and the structural semantic features are embedded into the encoding layer of LLMs, the topic semantic features replace the CLS feature representation, and the structural semantic features replace the SEP feature representation to generate a fused feature sequence; The fused feature sequence is subjected to classification feature extraction by the decoder of LLMs, and a classification feature vector is output. The classification feature vector is then subjected to binary classification judgment based on the classifier to obtain a set of value sentence sequences. Input the set of value sentence sequences and the corresponding category labels, construct a category feature template, and calculate the subclass semantic features based on the category prototype features of the category feature template; Based on the semantic features of the subclasses, a semantic-aware routing mechanism and a hybrid expert model are designed. The semantic-aware routing mechanism dynamically allocates the set of value sentence sequences to the hybrid expert model for two-stage multi-classification and outputs the text classification results.

2. The two-stage classification method for embedding semantic feature encoding of scientific and technological text as described in claim 1, characterized in that, The extraction of local topic features from the source text sequence includes: The source text sequence is scanned using a sliding window, and the topic aggregation representation of terms within the window is calculated to extract local topic features. ; Where σ(·) is the activation function, and w is the sliding window size. The vector representation of the j-th term. The corresponding attention weights are calculated as follows: ,in, For learnable query matrix parameters, The vector representation of the central term is used, and softmax is used to normalize the attention score; Pooling is performed on the local topic features to obtain the global topic representation. Where Pool(·) represents the pooling operation; The local topic features and the global topic representation are concatenated to construct the topic semantic features. ,in, and These represent the transformation matrix and the bias term, respectively. [;] indicates the feature concatenation operation, and tanh is the hyperbolic tangent activation function.

3. The two-stage classification method for embedding semantic feature encoding of scientific and technological text as described in claim 1, characterized in that, The construction of structural semantic features includes: Extract the basic feature representation of each term in the source text sequence. ,in, For the vector representation of terms, For positional encoding, FFN is a feedforward neural network. Indicates feature concatenation operation; Based on term dependency relationships, hierarchical feature aggregation is performed on the basic feature representation to obtain an aggregated hierarchical structure. ,in, For activation function, Describes the set of adjacent terms of term i. The aggregation weights are represented by l, which indicates the layer number. This represents the feature representation of the (l-1)th layer; Based on the aggregated hierarchical structure, inter-sentence relationships are represented, and the structural semantic features are constructed. ,in, For pooling operations, For the term representation of the last level, This represents the final structural features.

4. The two-stage classification method for embedding semantic feature encoding of scientific and technological text as described in claim 1, characterized in that, The construction of the category feature template includes: Input the set of value sentence sequences X= <X1,X2…,X n > and the corresponding category label Y, construct the category feature template. Where c represents the value sentence category, These are the semantic features of the topic.

5. The two-stage classification method for embedding semantic feature encoding of scientific and technological text as described in claim 1, characterized in that, The calculation yields subclass semantic features, including: The subclass semantic features are calculated based on the category prototype features of the category feature template. ,in, The transformation matrix is ​​a category-specific matrix, and σ(·) is the activation function. Let i be the semantic feature representation of the i-th subclass. For the topic semantic features, P c The category prototype feature is the category feature template.

6. The two-stage classification method for embedding semantic feature encoding of scientific and technological text as described in claim 1, characterized in that, The output text classification results include: Each expert model in the hybrid expert model adopts a two-layer feedforward network structure. ,in, and Each of the two feedforward networks for the j-th expert has a different parameter scale to adapt to tasks of varying complexity. The semantically aware routing mechanism interacts the topic semantic features with routing parameters to construct routing features. ,in, Here, the routing weight matrix is ​​used, and Topic semantics represents the topic semantic features. For routing characteristics; Based on the routing features, calculate the expert selection probability of the samples in the value sentence sequence set and each expert model. ,in, Let represent the semantic representation vector of the j-th FFN expert. Indicates the probability of expert choice; Based on the expert selection probabilities, the top-k relevant FFN expert models are dynamically assigned for two-stage multi-class classification, outputting the text classification results. , where k is the preset number of experts, and y is the final text classification result.

7. The two-stage classification method for embedding semantic feature encoding of scientific and technological text as described in claim 6, characterized in that, The process involves dynamically assigning Top-k relevant FFN expert models based on the expert selection probability to perform two-stage multi-class classification, and outputting the text classification result, including: Based on the expert selection probability, a dynamic threshold expert selection strategy is adopted, wherein the expert selection strategy is specifically as follows: , Where S is the selected set of experts, For dynamic thresholds, It is an adjustable coefficient. This represents the expert's average selection probability; Using the aforementioned expert selection strategy, the Top-k selected FFN expert models are dynamically allocated; Based on the selected FFN expert model, a weighted combination and collaborative approach is performed to output the text classification result. ,in, The expert weight is calculated using the following formula: ,in, Query vectors specific to experts The key vector of the input sample.

8. A two-stage classification system embedding semantic feature encoding of scientific and technological texts, characterized in that, The system includes: The topic semantic feature construction module is used to input a source text sequence, extract local topic features and calculate global topic relevance of the source text sequence, and construct topic semantic features; The hierarchical feature aggregation module is used to extract the basic feature representation of each term in the source text sequence, and perform hierarchical feature aggregation on the basic feature representation based on the term dependency relationship to obtain the aggregated hierarchical structure. Based on the aggregated hierarchical structure, the inter-sentence relationship representation is performed to construct structural semantic features. The feature fusion module is used to embed the topic semantic features and the structural semantic features into the encoding layer of LLMs, replace the CLS feature representation with the topic semantic features, replace the SEP feature representation with the structural semantic features, and generate a fused feature sequence. The binary classification judgment module is used to extract classification features from the fused feature sequence through the decoder of LLMs, output classification feature vectors, and perform binary classification judgment on the classification feature vectors based on the classifier to obtain a set of value sentence sequences. The subclass semantic feature calculation module is used to input the set of value sentence sequences and the corresponding category labels, construct a category feature template, and calculate the subclass semantic features based on the category prototype features of the category feature template. The two-stage multi-classification module is used to design a semantic-aware routing mechanism and a hybrid expert model based on the semantic features of the subclasses. The semantic-aware routing mechanism dynamically allocates the set of value sentence sequences to the hybrid expert model for two-stage multi-classification and outputs the text classification results.

Images