A text classification method and system based on dynamic multi-layer semantic perception machine

Abstract

The application provides a text classification method and system based on a dynamic multi-layer semantic perception machine, comprising: obtaining a text to be classified; obtaining a category to which the text to be classified belongs by using a dynamic multi-layer semantic perception machine model; wherein the dynamic multi-layer semantic perception machine model comprises a word embedding layer, a plurality of layers of weighted feature learners connected in sequence and a dynamic depth controller, each layer of the weighted feature learners is composed of a semantic perception machine and a base classifier connected in sequence; the output of the word embedding layer is used as the input of the semantic perception machine of all the weighted feature learners, the output of the base classifier of each layer of the weighted feature learners is input into the dynamic depth controller, and the output of the semantic perception machine of each layer of the weighted feature learners is used as the input of the semantic perception machine of the next layer of the weighted feature learners. Not only the feature learning quality is improved, the time complexity and the space complexity are reduced, but also the model parameter quantity is reduced, and the method is suitable for the case that the training sample is insufficient.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing, and in particular relates to a text classification method and system based on a dynamic multi-layer semantic perception machine. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] In the field of Natural Language Processing (NLP), the self-attention mechanism (SA) is used to represent the connection between different words in the same text and is a feature weight representation method. Models using Multi-Headed Attention (MHA), such as Transformer and BERT, have shown excellent performance in various machine learning tasks and are widely used in various fields such as text classification, machine translation, named entity recognition, and image classification. However, the Transformer model using MHA also has obvious shortcomings: (1) it relies on a large number of samples, and when there are insufficient training samples, the feature learning effect is poor; (2) the model is complex and has a large number of parameters. Many studies have shown the substitutability of MHA, but there are problems such as the performance degradation of the model after removing MHA.

[0004] When training samples are insufficient, the primary method for improving the Transformer model's feature extraction capabilities is data augmentation. Generative Adversarial Networks (GANs) are a representative data augmentation method. This method uses a generator to generate pseudo samples and a discriminator to score these pseudo samples. However, embedding the GAN model into feature learning can make the Transformer model more complex. Word / Sentence Mixup is another representative data augmentation method. This method interpolates two samples and their corresponding labels to generate a new sample and its pseudo label. However, the pseudo labels generated by these methods may not belong to any known class. Simple Data Augmentation (EDA) methods, such as random synonym replacement, require calculating the similarity of each word, increasing the algorithm's time complexity. However, these data augmentation methods can lead to more complex models or introduce unknown class labels, making Transformer feature learning more difficult.

[0005] To address the Transformer model's complexity and large number of parameters, ResMLP replaces the Transformer's Multi-Layer Perceptron (MHA) with two globally shared Multi-Layer Perceptrons (MLPs). MLP-Mixer, similar to ResMLP, replaces the MHA with a channel-specific MLP, reducing the Transformer's space complexity. ResMLP and MLP-Mixer achieve good performance in image classification. FFNet replaces the Transformer's MHA with a cascaded feedforward layer, but its performance is inferior to ResMLP and MLP-Mixer. FNet replaces the MHA with a Fast Fourier Transform (FFT), reducing the Transformer's time complexity, but its model performance is poor. External Attention reduces the Transformer model's complexity by replacing the KV matrix (the K matrix is ​​generated from the key tensor, and the V matrix is ​​generated from the value tensor) with two globally shared external memory units. These methods improve the model's temporal and spatial performance, achieving excellent results in image classification. However, they perform poorly in text classification, particularly when training samples are insufficient. If MHA is removed from the Transformer, the text classification performance will drop sharply. This is because although removing MHA can reduce the number of model parameters, the text feature learning effect will be worse, especially when there are insufficient training samples. Summary of the Invention

[0006] In order to solve the technical problems existing in the above-mentioned background technology, the present invention provides a text classification method and system based on a dynamic multi-layer semantic perceptron, which adopts a multi-layer semantic perceptron as a feature learner, improves the quality of feature learning, reduces the time complexity and space complexity, and adopts a dynamic depth controller to autonomously optimize the model depth, reducing the number of model parameters. It is suitable for solving the problem of deep learning's dependence on large-scale training samples and the problem of the model being too large and complex, and provides a feasible method for miniaturization of deep learning models, which is convenient for broadening the practical application of deep learning.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] A first aspect of the present invention provides a text classification method based on a dynamic multi-layer semantic perceptron, comprising:

[0009] Get the text to be classified;

[0010] Using a dynamic multi-layer semantic perception model to obtain the category to which the text to be classified belongs;

[0011] Among them, the dynamic multi-layer semantic perception machine model includes a word embedding layer, a dynamic depth controller and several layers of weighted feature learners connected in sequence, and each layer of weighted feature learners is composed of a semantic perception machine and a base classifier connected in sequence; the output of the word embedding layer serves as the input of the semantic perception machine of all weighted feature learners, and the output of the semantic perception machine of each layer of weighted feature learners serves as the input of the semantic perception machine of the next layer of weighted feature learners, and the output of the base classifier of each layer of weighted feature learners is input into the dynamic depth controller.

[0012] Furthermore, the semantic perception machine of each layer of weighted feature learner obtains the output text features based on the text features output by the semantic perception machine of the previous layer, the initial text features after random sequence conversion, and the text features output by the semantic perception machine of the previous layer after processing by the multi-layer perception machine, through addition and normalization operations.

[0013] Furthermore, the base classifier of each layer of weighted feature learner obtains the prediction result output by the weighted feature learner based on the text features output by the semantic perception machine of the weighted feature learner of the same layer.

[0014] Furthermore, the dynamic depth controller performs weighted averaging on the prediction results output by the base classifiers of each layer of the weighted feature learner to obtain the probability that the text belongs to each category, and takes the category with the highest probability as the category to which the text belongs.

[0015] Furthermore, the training process of the dynamic multi-layer semantic perception model is as follows:

[0016] Initialize the number of iterations and model parameters, and calculate the initial features of the text in the training set;

[0017] Based on the initial features and model parameters, the prediction results obtained by the weighted feature learner of each layer are calculated and input into the dynamic depth controller to optimize the model depth, and the base classifier weights obtained by the weighted feature learner of each layer are calculated. The text category probabilities obtained by the weighted feature learner of each layer are integrated to obtain an integrated prediction result, and the model parameters are optimized in combination with the real labels in the training set;

[0018] Determine whether the end condition is met. If so, output the model parameters as the optimal model parameters to obtain a trained dynamic multi-layer semantic perceptron model; otherwise, increase the number of iterations and continue to optimize the model parameters.

[0019] Furthermore, the specific method for optimizing the model depth is: based on the prediction results obtained by the weighted feature learner of each layer, the accuracy of the base classifier of the weighted feature learner of each layer is calculated, and based on the accuracy and threshold, it is determined whether the conditions for adjusting the model depth are met. If so, the model depth is adjusted; otherwise, the model depth is kept unchanged.

[0020] A second aspect of the present invention provides a text classification system based on a dynamic multi-layer semantic perceptron, comprising:

[0021] The text acquisition module is configured to: acquire the text to be classified;

[0022] A classification module is configured to: use a dynamic multi-layer semantic perception model to obtain the category to which the text to be classified belongs;

[0023] Among them, the dynamic multi-layer semantic perception machine model includes a word embedding layer, a dynamic depth controller and several layers of weighted feature learners connected in sequence, and each layer of weighted feature learners is composed of a semantic perception machine and a base classifier connected in sequence; the output of the word embedding layer serves as the input of the semantic perception machine of all weighted feature learners, and the output of the semantic perception machine of each layer of weighted feature learners serves as the input of the semantic perception machine of the next layer of weighted feature learners, and the output of the base classifier of each layer of weighted feature learners is input into the dynamic depth controller.

[0024] Furthermore, a training module is included, which is configured to:

[0025] Initialize the number of iterations and model parameters, and calculate the initial features of the text in the training set;

[0026] Based on the initial features and model parameters, the prediction results obtained by the weighted feature learner of each layer are calculated and input into the dynamic depth controller to optimize the model depth, and the base classifier weights obtained by the weighted feature learner of each layer are calculated. The text category probabilities obtained by the weighted feature learner of each layer are integrated to obtain an integrated prediction result, and the model parameters are optimized in combination with the real labels in the training set;

[0027] Determine whether the end condition is met. If so, output the model parameters as the optimal model parameters to obtain a trained dynamic multi-layer semantic perceptron model; otherwise, increase the number of iterations and continue to optimize the model parameters.

[0028] A third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the above-mentioned text classification method based on a dynamic multi-layer semantic perceptron.

[0029] The fourth aspect of the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the text classification method based on a dynamic multi-layer semantic perceptron as described above are implemented.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] The present invention provides a text classification method based on a dynamic multi-layer semantic perceptron. This method uses a multi-layer semantic perceptron without an attention mechanism as a feature learner, improving feature learning quality and reducing time and space complexity. By autonomously optimizing model depth and reducing the number of model parameters, the model's text classification performance is improved. This method is suitable for addressing the problem of deep learning's dependence on large-scale training samples and the problem of overly large and complex models, miniaturizing deep learning models, and facilitating their practical application. The present invention provides a text classification method based on a dynamic multi-layer semantic perceptron. This method uses a sequence transformation method to replace the multi-head attention in the encoder, reducing model complexity, enriching the sample word order semantics, obtaining better feature semantic expression, and alleviating overfitting.

[0032] The present invention provides a text classification method based on a dynamic multi-layer semantic perceptron. The DMSP model proposed by the method can find the optimal balance between accuracy and complexity. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0034] Figure 1 This is a structural diagram of a dynamic multi-layer semantic perception model according to the first embodiment of the present invention;

[0035] Figure 2 This is a structural diagram of a Transformer model for text classification tasks according to the first embodiment of the present invention;

[0036] FIG3( a ) is a structural diagram of each SA in the MHA according to the first embodiment of the present invention;

[0037] FIG3( b ) is a diagram of the MLP structure of each encoder in the first embodiment of the present invention;

[0038] Figure 4 This is a structural diagram of the MSP model according to the first embodiment of the present invention;

[0039] Figure 5 This is a structural diagram of the dynamic depth control DDCF framework according to the first embodiment of the present invention;

[0040] Figure 6 This is a comparison diagram of the Accuracy of the SMSP model according to the first embodiment of the present invention as the depth increases;

[0041] Figure 7 3. This is a comparison diagram of the Macro-Recall of the SMSP model according to the first embodiment of the present invention as the depth increases;

[0042] Figure 8 3. This is a comparison diagram of the change of Macro-Precision of the SMSP model with increasing depth according to the first embodiment of the present invention;

[0043] Figure 9 This is a comparison diagram of the change of Macro-F1 of the SMSP model with increasing depth according to the first embodiment of the present invention. DETAILED DESCRIPTION

[0044] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0045] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0046] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0047] Example 1

[0048] This embodiment provides a text classification method based on a dynamic multi-layer semantic perception machine, which specifically includes the following steps:

[0049] Step 1: Get the training set x i represents the i-th sample, n is the number of samples, m is the word length, C is the number of categories, the training set contains several texts (samples), each text corresponds to a true label y i (True category).

[0050] A text may be a sentence or an article.

[0051] In this embodiment, due to the problem of insufficient training samples in news text classification, each text is a news article, and the categories y include "politics", "economy", "military", "sports", and "entertainment", which are represented by numbers 1 to 5 respectively, so C=5.

[0052] Step 2: Use the training set to train the Dynamic Multi-Layer Semantics Perceptron (DMSP) model to obtain a trained Dynamic Multi-Layer Semantics Perceptron model. Figure 1 As shown in Figure 2, the dynamic multi-layer semantic perception model consists of a word embedding layer, several layers of weighted feature extractors (WFE) connected in sequence, and a dynamic depth controller (DDC). Each layer of weighted feature extractors consists of a semantic perception machine (SP) and a base classifier connected in sequence. The output of the word embedding layer serves as the input to the semantic perception machines of all weighted feature learners. The output of the semantic perception machine of each weighted feature learner layer serves as the input to the semantic perception machine of the next weighted feature learner layer. The output of the base classifier of each weighted feature learner layer is input to the dynamic depth controller. Using the semantic perception machine as a feature learner, DMSP improves feature learning quality and reduces time and space complexity by removing MHA. The training process of the DMSP model is as follows: initialize the number of iterations and model parameters; input the text in the training set into the word embedding layer to obtain the initial features; based on the initial features and model parameters, calculate the text category probability obtained by each layer of weighted feature learner, and input it into the dynamic depth controller to optimize the model depth, and calculate the base classifier weights obtained by each layer of weighted feature learner, integrate the text category probabilities obtained by each layer of weighted feature learner according to the base classifier weights to obtain the prediction results, and optimize the model parameters in combination with the real labels in the training set; determine whether the end conditions are met, and if so, output the model parameters as the optimal model parameters to obtain a trained dynamic multi-layer semantic perception model; otherwise, increase the number of iterations and continue to optimize the model parameters. Specifically, it includes the following steps:

[0053] Step 201: Set the accuracy threshold ξ, the initial model depth M, and the maximum number of iterations R; initialize the model depth γ=M, the number of iterations r=0, and the model parameters θ and ω.

[0054] Step 202: x each text i Input word embedding layer, and get the initial features of each text after word embedding representation

[0055] Step 203: The semantic perception machine of each layer of WFE is based on the initial features of each text. And the features of each text output by the semantic perception machine of the previous WFE Calculate the features of each text output by the semantic perception machine of the WFE layer The input of the first layer WFE is the initial features of each text.

[0056] The feature learner f of each WFE layer j (·) Using semantic perceptron as feature learning method. The semantic perceptron of each layer of weighted feature learner is based on the text features output by the semantic perceptron of the previous layer. Initial features of text after random sequence conversion And the text features output by the semantic perception machine on the previous layer After addition and normalization operations (Add&Norm), the output text features are obtained. Given a sample x i , after the j-th layer feature learner f j (·), that is, the learning of the semantic perceptron, whose output The calculation is as follows:

[0057]

[0058] Among them, f j (·) represents the feature learning function of the j-th layer (j-th weighted feature learner WFE), is the feature of the i-th text output by the j-1-th layer SP, is the initial feature of the i-th text, Norm(·) represents normalization, represents the feature representation after MLP learning in the j-th layer SP, θ j is the parameter of the j-th MLP, is the result of sequence conversion, sequence conversion Used for Perform random sequence conversion and obtain the sequence conversion result as follows:

[0059]

[0060]

[0061] in, Represents m words (w1, w2, ..., w m ), (w j1 , w j2 ,…,w jm )for A random permutation of is the initial feature of the i-th text The corresponding j-th word order conversion result.

[0062] Step 204: Base classifier of each layer of WFE The features of each text output by the semantic perception machine based on the WFE layer Get the category of each text output by the WFE of this layer

[0063] In the weighted feature learner WFE of each layer, in addition to the feature learning function f j (·), adding a base classifier At each layer, feature learning f j (·) After that, the base classifier is trained All training samples are tested, and their classification accuracy reflects the effect of feature learning at each layer and serves as the decision basis for depth control.

[0064] For the sample x i , in the jth layer WFE, through f j (·) After learning, it is expressed as Then the j-th base classifier right The classification prediction result is recorded as

[0065]

[0066] in, Indicates the category of the i-th text output by the j-th layer WFE.

[0067] Get the prediction results for all samples Input to the dynamic depth controller DDC for depth control.

[0068] Step 205: The base classifier of each layer of WFE obtains the classification prediction results of each text and inputs them into the dynamic depth controller (DDC), which is used to calculate the weight of the base classifier of each layer of WFE and autonomously control the model depth γ. The DDC adjusts the penalty coefficient μ of the base classifier according to the set accuracy threshold ξ∈(0,1). j and weight β j , divided into three situations, dynamically and autonomously control the model depth.

[0069] The dynamic depth controller DDC monitors the base classifiers of each layer, measures the feature learning ability of the model at the current depth, and dynamically adjusts the base classifiers of each layer. The weight β j , thereby controlling the optimal depth of the model. Let ξ∈(0,1) be the accuracy threshold of the base classifier, then the penalty coefficient μ of the base classifier is j and weight β j The calculations are as follows: the penalty coefficient calculation formula and the base classifier weight calculation formula are:

[0070]

[0071]

[0072] Among them, α j is the accuracy of the j-th layer base classifier, γ is the model depth dynamically learned at the current number of iterations; g is the number of base classifiers with a classification accuracy greater than ξ at the current depth, and s is the number of base classifiers with a classification accuracy less than ξ.

[0073] Based on the text category probability obtained by each layer of weighted feature learners, the accuracy of the base classifier of each layer of weighted feature learners is calculated. Based on the accuracy and threshold, it is determined whether the conditions for adjusting the model depth are met. If so, the model depth (the number of layers of weighted feature learners in the multi-layer semantic perceptron model) is adjusted; otherwise, the model depth remains unchanged. Specifically, the DDC depth optimization strategy is handled according to the following three situations:

[0074] (1) When α j >ξ: μ j =1, do not delete the weighted feature learner WFE;

[0075] (2) When α j ≤ξ and g≥s+1: set all α j The penalty coefficient of the base classifier ≤ξ is set to μ j =-∞, according to the weight calculation formula of the base classifier, its weight β j When it approaches 0, DDC will delete the γth weight feature learner WFE, that is, the last WFE, and update the model depth γ;

[0076] (3) When α j ≤ξ and g<s+1: set all α j Base classifier penalty coefficient μ ≤ξ j Set to -1, according to the base classifier weight calculation formula, its weight β j It will become smaller, that is, DDC retains the WFE of this layer, but increases the penalty for the base classifier of this layer.

[0077] Therefore, according to the effect of feature learning and classification performance of each layer, DDC dynamically adjusts the penalty coefficient μ of the base classifier j and weight β j , to achieve dynamic optimization of model depth. The base classifier weight vector β of the model obtained under DDC control is β = {β 1 , β 2 ,…,β γ}, where β j is the weight of the j-th base classifier.

[0078] It should be noted that in the DDCF framework, when the feature learner f j(·) are not of the same structure, the dynamic depth controller DDC can remove one or several layers of WFE according to actual requirements. j When (·) has the same structure, if the jth (1 <= j < γ)th WFE layer is deleted, the parameters of the WFEs from j+1 to γth layers need to be retrained in the next iteration. However, if the last WFE is deleted, the parameters of other sub-layers do not need to be retrained. Therefore, when g ≥ s + 1, DDC can accelerate model convergence by removing the last WFE, while also controlling model depth.

[0079] The DDCF framework differs from other models in that it autonomously optimizes model depth through DDC, eliminating the need for manual optimization. Models like ResNet improve accuracy with increasing depth, but this limited improvement in accuracy comes at the expense of significantly increased model complexity. In some models, such as VDCNN, accuracy is not proportional to depth; instead, accuracy decreases when model depth exceeds a certain value. DDC detects a decrease in the extraction capability of a feature learner and automatically discards the corresponding weighted feature learner (WFE), thereby reducing model complexity and improving classification performance.

[0080] Step 206: For each text x i , DDC integrates the classification prediction results output by the base classifier of each layer of WFE based on the calculated weight of the base classifier of each layer of WFE as each text x i The prediction result p i :

[0081]

[0082] Among them, γ is the model depth dynamically learned at the current iteration number, β j represents the weight of the j-th base classifier, For the j-th classifier on sample x i predictions.

[0083] Step 207: Based on each text x in the training set i The true label y i , each text x output by DDC i The predicted classification result p i , optimize the model parameters ω j ,θ j The DMSP model uses the cross entropy loss function, the objective function is shown below, and the Adam algorithm is used to optimize the model parameters.

[0084]

[0085]

[0086] Among them, n is the number of training samples, C is the number of categories, For sample x i The true label of (belonging to the kth category), For sample x i The final feature representation under the DMSP model with a depth of γ, is the model depth dynamically learned by the DMSP model under the current number of iterations, For the model, the sample x i The probability of predicting the kth class, ω, θ are the base classifiers and feature learner f γ (·) parameters.

[0087] Step 208: Determine whether the end condition (whether the number of iterations r reaches the maximum number of iterations) R is met. If so, output the model parameter ω * ,θ * As the optimal model parameters, a trained DMSP model is obtained; otherwise, the number of iterations r is increased by 1, and the process returns to step 203.

[0088] Step 3: Get the text to be classified, input the trained DMSP model, and obtain the category of the text to be classified.

[0089] DMSP model loading optimal model parameter ω * ,θ * , treat the classification text x′ i After the initial features of each text are obtained through word embedding representation, sequence conversion is used Convert word order to get According to the feature learner f of each layer WFE j (·) Each layer of semantic perception machine SP is used for learning and output And the corresponding base classifier right Classify; finally, DDC performs weighted average of the classification prediction results output by the base classifier of each layer of WFE according to the optimal base classifier weight β to obtain the sample x′ i The probability of belonging to each category; the category with the highest probability is taken as the text to be classified x′ i The category to which it belongs.

[0090] The construction idea of ​​the dynamic multi-layer semantic perceptron model is as follows: first, a multi-layer semantic perceptron (MSP) is proposed. This method uses a sequence transformation method to replace the multi-head attention (MHA) in the encoder, and this encoder is called a semantic perceptron (SP); the semantic perceptron SP is used to remove semantic ambiguity caused by word order neglect and obtain better feature representation; then, a dynamic depth control framework (DDCF) is proposed to dynamically optimize the depth of the neural model and reduce the model complexity; based on the MSP and DDCF frameworks, a dynamic multi-layer semantic perceptron (DMSP) is proposed, which can improve the model performance and reduce the model complexity; comparative experimental results on multiple text datasets show that the DMSP model can greatly reduce the model complexity while ensuring improved accuracy.

[0091] Deep Convolutional Neural Network (DCNN), Multi-Layer Perceptron (MLP) and Transformer network are three feature learners that are currently widely used. Among them, Transformer has good feature representation capabilities due to the use of Multi-Head Attention (MHA) and Positional Encoder (PE). However, Transformer is completely dependent on MHA, and its complexity is much higher than that of DCNN and MLP of the same depth. If MHA is deleted from Transformer, the model complexity can be reduced, but the model accuracy is significantly reduced. To this end, the present invention attempts to reduce the model complexity and improve the feature extraction performance, and proposes a multi-layer semantic perceptron MSP method.

[0092] Usually Transformer is an encoder-decoder structure. When used for text classification tasks, Transformer only uses the encoder. Its structure is as follows Figure 2 shown.

[0093] like Figure 2As shown in Figure 3, the Transformer architecture for text classification includes a word embedding layer, a position encoder (PE), M encoders, and a linear classifier (Linear & Softmax). Each encoder consists of a multi-head attention (MHA), a feature fusion layer (Add & Norm), and a multi-layer perceptron (MLP). The MHA consists of multiple self-attention (SA) layers. Figure 3(a) shows an SA structure, while Figure 3(b) shows an MLP structure.

[0094] Given a dataset Where n is the number of samples, m is the word length, C is the number of categories, and M is the maximum model depth. Let x i Represents the i-th sample, which is obtained after word embedding and position encoding As the input of the first Encoder. Input them into the three linear sublayers of each SA of MHA respectively, construct Q, K, V matrices respectively, calculate the similarity matrix of Q and K matrices, and then dot product with V matrix, and add and normalize them. After addition and normalization, the MHA output is obtained. The output of MHA is input to MLP for feature learning, and the output of MLP is obtained after two layers of Linear, two layers of activation function and one layer of normalization. The output of MLP and the output of MHA are added and normalized by Add&Norm to obtain the first Encoder encoding output. And as the input of the next Encoder. Initial features After M layers of Encoder encoding output That is the sample x i The final feature representation of . Finally, Input the linear classifier Linear&Softmax and output the predicted category label.

[0095] Although Transformer has good feature representation capabilities, due to the use of a large number of linear sublayers in its multi-head attention (MHA), its parameter count is much higher than other feature learning models of the same depth (such as Deep Convolutional Neural Network (DCNN) and MLP, etc.). If the MHA of Transformer is deleted, the model complexity can be reduced, but the feature extraction quality will decrease and the model classification accuracy will be significantly worse. In addition, when the number of training samples is small, the performance of the Transformer model is difficult to train to achieve the best. For this reason, the present invention attempts to reduce the model complexity and improve the feature extraction performance, and proposes a Multi-Layer Semantics Perceptron (MSP).

[0096] MSP removes the position encoder (PE) of the Transformer and proposes a new Encoder structure. It uses a sequence conversion method to replace the multi-head attention mechanism MHA and calls the Encoder a Semantics Perceptron (SP). When the number of semantic perceptron layers is fixed, the method is called a Multi-Layer Semantics Perceptron (MSP) method with a de-attention mechanism. This method uses multi-layer semantic perceptrons to remove semantic ambiguity caused by ignoring word order in the sequence, aiming to better perceive semantic features. The MSP model structure is as follows: Figure 4 shown.

[0097] like Figure 4 As shown in the figure, the MSP model includes a word embedding layer and a multi-layer semantic perceptron MSP. Among them, the MSP is composed of M semantic perceptrons SP stacked together, each SP includes a sequence conversion It consists of feature learning MLP and feature fusion Add&Norm.

[0098] Let x i Represents the i-th sample, which is obtained after word embedding As the initial input feature representation of the first layer SP, m is the sample x i The number of words contained, d is the embedding dimension of the word. and the previous layer SP output Input the jth layer SP, learn to remove the word order semantic ambiguity caused by word order neglect, and the output is The output of the M-th layer SP is That is the sample x i The final feature representation of . The specific description is as follows:

[0099] In Transformer, MHA uses a large number of fully connected layers, which results in a large amount of matrix operations. Transform the word order of the sentence to enrich the sample’s multiple word order representations. Given a sample x i =(x1, x2, ..., x m ), Represents sample x i The initial feature word order of m words is represented by the semantic perception machine SP of each layer. right Perform random sequence transformation as shown below:

[0100]

[0101]

[0102] in, Represents m words (w1, w2, ..., w m ), (w j1 , w j2 ,…,w jm )for A random permutation of for The j-th word order conversion result.

[0103] Sequence conversion While enriching the sample's representation of multiple word orders, it may also introduce semantic ambiguity caused by word order changes. Here, we take the sentence "An apple was placed on the end table." as an example. The semantic ambiguity analysis after word order conversion is shown in Table 1.

[0104] Table 1. Semantic ambiguity analysis after sentence word order conversion

[0105]

[0106] As can be seen from Table 1, some After the word order is changed, the semantics of the sentence becomes ambiguous, which needs to be learned and removed in the subsequent multi-layer semantic perceptron MSP.

[0107] Sequence conversion It can perceive the position of words, but it may cause semantic ambiguity in word order. To this end, a multi-semantic perceptron (MSP) is proposed to learn to remove this semantic ambiguity and obtain better feature semantic representation.

[0108] like Figure 4As shown, the multi-layer semantic perceptron (MSP) is composed of a stack of SPs (Semantics Perceptrons), where each layer of the semantic perceptron SP is converted by a sequence The feature learning MLP and feature fusion Add&Norm layer are composed. The feature learning MLP in the semantic perception machine SP has the same structure as the MLP in the Transformer. Its structure is shown in Figure 3(b). Its parameters θ j The MLP learns features from the output of the previous SP layer, without sharing with other MLPs. The first linear sublayer of the MLP maps features from a low-dimensional feature space to a high-dimensional feature space, the activation sublayer performs feature filtering, and the second linear sublayer reduces the feature dimension, maintaining the consistency of input and output dimensions. Represents sample x i The feature representation after learning by the 1st to Mth SP, where the jth layer SP outputs As shown below:

[0109]

[0110] Among them, Norm(·) represents normalization, is the sequence conversion result, is the initial moment characteristic, is the j-1th layer SP output, Representation characteristics The feature representation after MLP learning in the j-th layer SP, θ j are the parameters of the j-th MLP.

[0111] Include The SP semantic perception machine can perceive the position information of words in the sample and remove the semantic ambiguity of word order. As shown in the above formula, each time Transformation, sample x i The word order will change, and the position of the same word will be different, that is, the sample x i The word vectors of are different, which enriches the feature expression of the sample, but may also cause word order semantic ambiguity. Therefore, in the above formula, Add a minus sign before learning reduction. Then compare it with the SP output of the previous layer. The MLP output of layer j Add and finally normalize the output is the feature representation of the learning of the j-th layer SP. After the perception learning of the M-layer SP, the final output is For sample x i feature representation.

[0112] Therefore, after removing the MHA and PE, a multi-layer semantic perceptron (MSP) composed of multiple layers of SPs can perceive word position information and remove word order semantic ambiguity. Here, M takes a fixed value, so it is called a static multi-layer semantic perceptron (SMSP). The contribution of this invention is that the MSP method can also be applied to other models and generalized to other downstream tasks.

[0113] contrast Figure 4 From Figure 3, we can see the difference between the MSP model and the transformer. The core SP of MSP uses Sequence conversion function, and the core Encoder of Transformer corresponds to MHA, that is, multiple Self-Attention. Therefore, compared The sequence transformation function and multi-head attention MHA can be used to analyze the advantages and disadvantages of their time complexity and space complexity.

[0114] Given a sample x i , whose initial characteristics are where m is the number of samples x i The number of words included, d is the embedding dimension of the word. And assume that the number of Self-Attention in MHA is t. Let's compare And the time complexity and space complexity of MHA.

[0115] It is a linear transformation function with simple calculation, time complexity of O(m) and space complexity of O(d).

[0116] However, MHA uses a large number of fully connected layers. Each Self-Attention in MHA requires the construction of Q, K, and V matrices, and the amount of matrix operations is large. The calculation of a single Self-Attention in MHA is shown in the following formula:

[0117]

[0118] Among them, the size of Q, K and V matrices is m×d, so the additional space required for t Self-Attention in MHA is O(tmd). In a Self-Attention, similarity needs to be calculated by the dot product of matrices Q and K, and after Softmax normalization, the dot product is calculated with the matrix V. MHA contains t Self-Attentions, so its time complexity is O(tm 2 d).

[0119] The comparison of time complexity and space complexity with MHA is shown in Table 2.

[0120] Table 2 Complexity comparison with MHA

[0121]

[0122] As can be seen from Table 2, from the analysis of time complexity and space complexity, It is obviously better than MHA. Therefore, the SMSP model proposed in this invention has better time performance and spatial performance than the transformer.

[0123] The performance of deep neural models generally improves with increasing depth. However, there is a limit to how much accuracy can be achieved, and as the classification accuracy limit is approached, model complexity increases exponentially. The optimal depth for most models is often artificially set and fixed. When the model is too large and the number of training samples is insufficient, overfitting can be severe. Finding the optimal balance between model performance and complexity is crucial.

[0124] To this end, this paper proposes a dynamic depth control framework (DDCF) to autonomously optimize the model depth. Figure 5 As shown in Figure 2. By dynamically controlling the depth of the MSP using DDCF, we further propose a dynamic multi-layer semantic perceptron (DMSP) to achieve the optimal balance between performance and complexity. To distinguish it from the previously mentioned MSP model, which has a fixed depth, we refer to it as the static multi-layer semantic perceptron (SMSP) without the attention mechanism.

[0125] Dynamic Depth Control Framework DDCF Figure 5 As shown in Figure 1, it consists of a word embedding layer, a weighted feature extractor (WFE) and a dynamic depth controller (DDC). WFE consists of a feature learner f j (·) and base classifier Composition. Sample x i Initial features are obtained through the embedding layer As the input of the first weighted feature learner WFE, the feature learner f in the jth WFE j The output of (·) is used as the input of the j+1th WFE. Each WFE uses f j (·) After extracting features, the base classifier For sample x iThe prediction is made and the prediction results are input into the dynamic depth controller DDC. DDC takes a weighted average of the prediction results and dynamically adjusts the model depth.

[0126] Feature learning function f in DDCF j (·) is not limited to using a specific feature learning method, but can be the semantic perceptron SP proposed in this paper, or other feature learning methods.

[0127] Given a sample x i ,make Denoted as the jth feature learner f j Output of (·):

[0128]

[0129] in, is the j-1th layer f j-1 (·) feature learning results, f j (·) represents the feature learning function in the j-th weighted feature learner WFE, which can be the SP of the present invention or other methods.

[0130] Based on MSP, DDCF is used to dynamically control the model depth and a dynamic multi-layer semantic perceptron model (DMSP) is proposed. Figure 1 In other words, the DDCF framework (such as Figure 5 f in the figure) j (·) , using semantic perceptrons (SP) as feature learners. In this way, DMSP improves feature learning quality and reduces time and space complexity by removing MHA. Furthermore, DDCF autonomously optimizes model depth, reduces model complexity, and improves model classification performance.

[0131] The DMSP model uses the DDCF framework to autonomously control depth, reducing model complexity. Here, we evaluate the complexity of the DMSP model by analyzing the number of DMSP model parameters and the number of floating-point operations.

[0132] Assume that the depth of the DMSP model is γ, the classification threshold is ξ, and the structure of the MLP in SP is two cascaded fully connected layers, and the input and output dimensions of the fully connected layers are the same. It is a fully connected layer and a softmax function, which completes the mapping from 0 to C classes. Given a sample x i The initial features are Where m is the length of the sample word and d is the word embedding dimension. Then, the number of parameters required for a weighted feature learner WFE in the DMSP model is denoted as r:

[0133] τ=2m×d 3 +d+m×d 2 ×C+C

[0134] In the rth iteration, the DMSP model parameter consisting of γ WFEs is denoted as ψ r ,for:

[0135] ψ r =γτ

[0136] After the rth iteration, if there is a condition α j ≤ξ, g<s+1, which belongs to the second case of DDC. Then, the last layer of weighted feature learner WFE is deleted according to DDC control. Then the number of parameters and floating-point operations of the r+1th round DMSP model are reduced by τ, that is:

[0137] ψ r+1 =ψ r -τ

[0138] If it is the other two cases of DDC, then

[0139] ψ r+1 =γτ

[0140] The comparison of the number of parameters and floating-point operations between the rth round and the r+1th round under DDC control is shown in Table 3.

[0141] Table 3. Comparison of model parameters and floating-point operations between the rth round and the r+1th round under DDC control

[0142]

[0143] As can be seen from Table 3, in the rth iteration, the parameter of the DMSP model is ψ r =γτ. After the rth round of iteration, if the DDC control condition ② is met, the last layer of weighted feature learner WFE is deleted, that is, the model depth γ = γ-1, then in the r+1th round, the number of parameters is reduced to ψ r+1 =ψ r Compared to round r, the number of parameters and floating-point operations in round r+1 is reduced by τ. If DDC control conditions ① and ③ are met, the depth γ remains unchanged, and the number of parameters and floating-point operations remains unchanged. This demonstrates that the DMSP model under the DDCF framework can dynamically control model depth, thereby reducing the number of parameters and accelerating model training.

[0144] In summary, DMSP uses the multi-layer semantic perceptron (MSP) without MHA and PE to improve feature learning quality and reduce time and space complexity. In addition, DDCF is used to optimize model depth, finding the best balance between performance and complexity, reducing model complexity while improving model classification performance.

[0145] This example selects four datasets: AGNews, Amazon, Sogou, and 20news groups. Each dataset retains four classes, abbreviated as AGN.4, Ama.4, So.4, and 20ngp.4. The number of samples in each class is shown in Table 4. 20ngp.4 is an unbalanced sample set, and the sample sizes of the four datasets are relatively small.

[0146] Table 4. Experimental dataset and number of samples of each type

[0147]

[0148] To verify the effectiveness of the SMSP method, DDCF framework, and DMSP model proposed in this paper, comparative experiments were conducted with various methods. The model descriptions of the comparative methods are shown in Table 5.

[0149] Table 5. Model description of the comparison methods

[0150]

[0151]

[0152] (1) Transformer method: This is the model proposed in the article “Attention Is All You Need” published in the journal “the 31st International Conference on Neural Information Processing Systems, vol. 30, 2017” in 2017.

[0153] (2) MLP method: Use MLP to learn features, and the last layer is connected to a Softmax classifier, which is called an MLP model.

[0154] (3) SMSP method: It uses the MSP method to learn features, and the last layer is connected to a Softmax classifier. It does not use DDFC to optimize the depth, and is called a static multi-layer semantic perceptron.

[0155] (4) DC method: Use the MLP method to learn features and adopt the dynamic depth control framework DDCF model, which is called the dynamic MLP classification model.

[0156] (5) SMSP-E method: It uses the MSP method for feature extraction, connects a Softmax classifier to each layer, and finally integrates the classification, which is called static integrated multi-layer semantic perceptron. It does not adopt the dynamic depth control framework DDCF.

[0157] (6)DMSP method: Dynamic multi-layer semantic perceptron using the MSP method and DDCF framework.

[0158] Methods (3) to (6) are the methods and their variations proposed in this paper, and are intended for targeted comparative experiments. The classification evaluation metrics used are accuracy, Macro-F1, Macro-Precision, and Macro-Recall, while the model complexity metric uses the number of model parameters. Table 6 lists the hyperparameters and settings of the models used in the experiment.

[0159] Table 6. Model hyperparameter settings in the model

[0160]

[0161]

[0162] To prevent overfitting, an early stopping mechanism was added during model training, and the error rate was used as the early stopping indicator. A single Tesla T4 GPU was used in the experiment.

[0163] To verify the effectiveness of the attention-free Multi-Layer Semantic Perceptron (SMSP), a comparative experiment was conducted between the SMSP model and the MLP model. As shown in Table 5, the SMSP model's core approach is to learn features using the MSP method without the attention mechanism, while the classic MLP model uses the MLP method. Table 7 shows a comparison of the accuracy and model parameter count of the two models on the Ama.4, AGN.4, So.4, and 20ngp.4 datasets.

[0164] Table 7. Comparison of accuracy and parameter count between SMSP and MLP models at different depths

[0165]

[0166] As shown in Table 7, when the depth is 2 to 4 layers, the SMSP model's accuracy is slightly lower than that of an MLP model with the same number of layers. However, as the number of layers increases, after 4 layers, the SMSP model's accuracy begins to improve and surpasses the MLP model. For example, when the depth is 6, the SMSP model's accuracy improves by 10.63%, 8.67%, and 1.84% on AGN.4, So.4, and 20ngp.4, respectively, compared to the MLP model. It only slightly decreases on Ama.4. At a depth of 8, the SMSP model's accuracy surpasses the MLP model across the board.

[0167] Analysis shows that when the depth is 2 to 4 layers, the MLP model has fewer parameters, the number of samples is sufficient to support its training, and the model accuracy has not yet reached the accuracy saturation point. After 4 layers, the MLP model tends to saturate the model accuracy and overfitting occurs, resulting in a decrease in model accuracy. The SMSP model uses word order transformation to enrich the word order semantic information of the sample, and eliminates word order semantic ambiguity through multi-layer semantic perception MSP learning. As the depth increases, it can alleviate overfitting, so its generalization ability gradually increases. This feature is caused by Figure 6 、 Figure 7 、 Figure 8 and Figure 9 The experiments can be further verified.

[0168] like Figure 6 、 Figure 7 、 Figure 8 and Figure 9 Figure 2 shows the changes in the SMSP model's accuracy, macro-recall, macro-precision, and macro-F1 across four datasets as the depth increases. When the model depth exceeds 4, the various SMSP model metrics improve with increasing depth. This demonstrates that the SMSP method can enrich word order semantic information, and its effectiveness becomes more pronounced with increasing depth.

[0169] To verify the effectiveness of the dynamic depth control framework DDCF, the following comparative experiments were conducted on four datasets.

[0170] As shown in Table 5, both the DC model and the MLP model use the MLP method to learn features, but the difference is that the DC model uses the DDCF framework to optimize the model depth, while the MLP model does not.

[0171] In the experiment, given the initial depth IDepth, the DC model autonomously optimizes the depth FDepth and compares its classification results with the best classification results of the MLP model to verify the effectiveness of the DDCF method. The comparative experimental results on four datasets are shown in Table 8.

[0172] Table 8. Comparison of DC’s optimization depth and classification performance with the best case of the MLP model

[0173]

[0174] As shown in Table 8, the final depth FDepth of the DC model is dynamically optimized by DDCF. As can be seen from the data in Table 7, compared to the best case of MLP, except for So.4 where the classification result of the DC model is slightly worse, the classification results of the DC model on the other three datasets are significantly better than the MLP model.

[0175] For example, on Ama.4, the DC model, which initially had a depth of 5, achieved a final depth of 4 after DDCF optimization. Compared to the best classification result of the MLP model on this dataset, DC achieved Accuracy, Macro-Recall, and Macro-F1 values ​​of 92.05%, 92.85%, and 91.93%, respectively, representing improvements of 3.55%, 4.47%, and 3.49%. Similar comparisons were made on the AGN.4 and 20ngp.4 datasets. Overall, the DC model uses DDCF to optimize model depth and integrate classification results, effectively improving classification performance while reducing the number of parameters.

[0176] As shown in Table 5, the DMSP model and the SMSP-E model are similar in that they both use the MSP method to learn features, with a softmax classifier at each layer, followed by ensemble classification. The difference between the two models is that the DMSP model uses a depth-dependent controller (DDC) to optimize model depth, while the SMSP-E model does not. This experimental comparison further validates that the DDFC framework improves model classification performance due to the effectiveness of the DDC method, not simply ensemble classification.

[0177] The best classification results of the DMSP model are compared with those of SMSP-E on the four datasets, as shown in Table 9.

[0178] Table 9. Comparison of DMSP model depth optimization and classification results with the best classification results of SMSP-E

[0179]

[0180]

[0181] The SMSP-E model is given an initial depth IDepth with a fixed depth, while the DMSP model uses DDC and its final depth FDepth can be dynamically optimized.

[0182] Table 9 shows that the DMSP model significantly outperforms the SMSP-E model in classification on all four datasets. For example, on the Ama.4 dataset, compared to SMSP-E, DMSP's three classification metrics improve by 2.19%, 2.22%, and 2.29%, respectively. On the So.4 dataset, DMSP's three classification metrics reach 91.64%, 91.47%, and 91.75%, respectively, representing improvements of 3.97%, 4.49%, and 4.23%. Similar results are found on the other two datasets.

[0183] Analysis reveals that both DMSP and SMSP-E utilize MSP for feature learning and integrate the classification results at each layer. The difference lies in the use of DDC to dynamically control model depth, while SMSP-E does not. This suggests that the effectiveness of the DDFC framework stems not simply from integrated classification but from its use of DDC to optimize model depth, a key factor in its improved generalization capabilities.

[0184] In addition, the DDCF framework can optimize the model depth, thereby reducing the complexity of the model.

[0185] The DMSP model, which adopts the MSP method to learn features and uses the DDCF framework to control depth, is the final model proposed in this paper. To verify its effectiveness, comparative experiments were conducted with the Transformer model and the MLP model. The experimental results on four datasets are shown in Tables 10, 11, 12, and 13.

[0186] Table 10. Comparison of classification results among DMSP, MLP, and Transformer models (Ama.4)

[0187]

[0188] Table 11. Comparison of classification results among DMSP, MLP, and Transformer models (AGN.4)

[0189]

[0190] Table 12. Comparison of classification results among DMSP, MLP, and Transformer models (So.4)

[0191]

[0192]

[0193] Table 13. Comparison of classification results of DMSP, MLP and Transformer models (20ngp.4)

[0194]

[0195] The Transformer model and MLP model are given an initial depth IDepth, and the model depth is fixed. The DMSP model uses DDC, and its final depth FDepth can be dynamically optimized.

[0196] As shown in Table 10, on the Ama.4 dataset, the DMSP model achieves a final depth of 5, compared to the best-performing MLP and Transformer models (i.e., an MLP with a fixed depth of 4 and a Transformer with a fixed depth of 1). Its accuracy is 1.60% higher than that of the MLP and 6.90% higher than that of the Transformer. Macro-Recall and Macro-F1 also significantly improve over those of the MLP and Transformer. Table 11 shows that on the AGN.4 dataset, the DMSP model optimizes from an initial depth of 5 to a final depth of 3, achieving superior classification metrics such as Accuracy, Macro-Recall, and Macro-F1 compared to the fixed-depth Transformer and MLP models. Table 12 shows that on the So.4 dataset, the DMSP model optimizes to a final depth of 5, achieving Accuracy, Macro-Recall, and Macro-F1 of 91.47%, 91.47%, and 91.75%, respectively, significantly outperforming Transformer and MLP models of varying fixed depths. On the imbalanced 20ngp.4, DMSP optimized its three classification metrics from an initial depth of 5 to a depth of 2, reaching 84.86%, 84.67%, and 84.87%, respectively. These performances are all higher than those of the fixed-depth Transformer and MLP models. Compared with the best performance of the Transformer and MLP models, DMSP's Accuracy improved by 11.77% and 7.87%, Macro-Recall improved by 13.92% and 7.84%, and Macro-F1 improved by 11.76% and 6.76%, respectively.

[0197] This shows that the DMSP model outperforms the Transformer and MLP models. Analysis reveals that the MLP model does not consider word order semantics. When the training sample size is insufficient, overfitting occurs more frequently than the DMSP model as the model depth increases. Although the Transformer considers word position information and employs multi-head attention (MHA), it also tends to overfit when training samples are insufficient, and the classification results worsen with increasing depth. The DMSP model, on the other hand, uses a sequence transformation function to enrich sample features and employs a multi-layer semantic perceptron (MSP) approach to eliminate word order semantic ambiguity. On small sample sets, it can mitigate overfitting compared to the Transformer and MLP models. Furthermore, it uses DDCF to optimize model depth. Therefore, the DMSP model's generalization ability is superior to that of the Transformer and MLP models.

[0198] Various models were comprehensively evaluated from the aspects of model complexity and classification performance. The experimental results are shown in Tables 14, 15, 16 and 17.

[0199] Table 14. Comparison of the complexity and classification results of various models on the Ama.4 dataset

[0200]

[0201] Table 15. Comparison of the complexity and classification results of various models on the AGN.4 dataset

[0202]

[0203] Table 16. Comparison of the complexity and classification results of various models on the So.4 dataset

[0204]

[0205] Table 17. Comparison of the complexity and classification results of various models on the 20ngp.4 dataset

[0206]

[0207]

[0208] As shown in Tables 14, 15, 16, and 17, the Transformer, MLP, SMSP, and SMSP-E models have fixed model depth, while the DC and DMSP models use DDC to optimize model depth, with FDepth representing the final optimized depth. Comparing classification results alone, the SMSP, DC, and DMSP models significantly outperform the Transformer and MLP models. Comparing model complexity alone, the Transformer, MLP, DC, and DMSP models outperform the SMSP and SMSP-E models. Taking into account both classification results and model complexity, the DMSP model is optimal. The detailed comparative analysis is as follows:

[0209] Tables 14, 15, 16, and 17 show that SMSP's single-layer parameter count is approximately 35% of that of the Transformer. Comparing Table 6 with Tables 10 through 13, SMSP's accuracy is higher than that of the Transformer at the same depth. For example, on Ama.4, the SMSP model with a depth of 2 has an accuracy of 88.56%, while the Transformer's is 82.81%. SMSP's accuracy is 5.75% higher than the Transformer's, but the SMSP model has only 35% of the Transformer's parameters. Similar results are also seen in comparisons of the same dataset at other depths and other datasets at the same depth. This shows that SMSP has better classification performance and lower model complexity than the Transformer, and also verifies the effectiveness of the MSP method for removing MHA.

[0210] As shown in Tables 8, 14, 15, 16, and 17, in the comparison between DC and MLP, DC outperforms the MLP model in classification, with the exception of So.4. For example, on Ama.4, with an initial depth of 5 and a final depth of 4 after DDC optimization, the DC model achieves 3.17% and 3.18% improvements in Accuracy and Macro-F1, respectively, over the MLP with a depth of 4, despite having the same number of parameters. Similar results are also observed on AGN.4 and 20ngp.4. This shows that DC outperforms MLP in terms of both complexity and classification performance, validating the effectiveness of the DDCF approach.

[0211] As can be seen from Tables 14, 15, 16, and 17, SMSP achieves the best classification results in most cases, with the highest model complexity. However, due to the use of DDCF for depth optimization, the DC and DMSP models, while performing slightly worse, have significantly lower model complexity than the SMSP model. As shown in Table 15, on AGN.4, the DC model achieves a Macro-F1 of 80.39%, only 0.94% lower than the SMSP, yet its parameter count is only 33% of that of the SMSP. Compared to the SMSP, the DMSP achieves a Macro-F1 of 1.55% lower, yet its model parameter count is only 25% of that of the SMSP. Similar results were achieved on the other three datasets. In other words, DC and DMSP outperform SMSP in terms of both complexity and classification performance, validating the effectiveness of the DDCF approach.

[0212] As shown in Table 9, DMSP outperforms SMSP-E in all classification results. Considering both model parameter count and classification results, Tables 14 to 17 show that, with the exception of So.4, DMSP outperforms SMSP-E on Ama.4, AGN.4, and 20ngp.4. As shown in Table 15, on Ama.4, DMSP's Accuracy and Macro-F1 improve by 2.19% and 2.29%, respectively, compared to SMSP-E, yet DMSP's parameter count is only 80% of that of SMSP-E. On So.4, DMSP's classification results significantly improve over SMSP-E, yet the two models have comparable parameter counts. Similar results are found on 20ngp.4. This indicates that, considering both complexity and classification performance, DMSP outperforms SMSP-E, and the effectiveness of the DDCF method lies not solely in ensemble classification but in the effectiveness of the DDC deep optimization method.

[0213] Tables 14, 15, 16, and 17 show that DMSP performs best when compared to the Transformer and MLP, respectively, taking into account both model parameter count and classification performance. As shown in Table 17, on the 20ngp.4 dataset, DMSP utilizes only 70% of the Transformer's parameters, yet its Accuracy and Macro-F1 scores are 11.77% and 11.76% higher, respectively. Tables 10 to 13 show that the Transformer's classification results decrease with increasing depth. This is due to the increasing overfitting of the Transformer with increasing depth on small training sets. While DMSP and MLP have comparable parameter counts, DMSP significantly outperforms MLP in classification performance. Similar results are found on the other three datasets.

[0214] The Transformer model has achieved excellent results on large-scale datasets and has been widely used. However, when training samples are insufficient, the Transformer can overfit and its generalization ability declines. Shallow MLP models outperform the Transformer, but overfitting also occurs as depth increases. In other words, considering both complexity and classification performance, the DMSP model outperforms both the Transformer and MLP models on small training sets.

[0215] In summary, the comparative experimental results on various datasets demonstrate the effectiveness of the DMSP model. The multi-layer semantic perceptron (MSP) method proposed in this paper, which removes the attention mechanism, can enrich the word order semantics of the samples, obtain better feature representation, alleviate overfitting, and improve the generalization ability of the model. The DDCF framework is used to optimize the model depth and reduce the model complexity, thus finding an optimal balance between model complexity and model accuracy.

[0216] The proposed method, MSP, replaces the multi-head attention (MHA) in the encoder with a sequence transformation method, reducing model complexity and achieving better feature representation. The proposed method, DDCF, can find an optimal balance between model accuracy and complexity by autonomously optimizing model depth and reducing the number of model parameters. Based on the MSP method and the DDCF framework, the dynamic multi-layer semantic perceptron (DMSP) is proposed. Comparative experimental results on various text datasets demonstrate that the DMSP model can significantly reduce model complexity while ensuring improved accuracy. Comprehensively evaluating complexity and classification performance, the DMSP model outperforms the Transformer and MLP models on small-scale training sample sets.

[0217] Example 2

[0218] This embodiment provides a text classification system based on a dynamic multi-layer semantic perception machine, which specifically includes the following modules:

[0219] The text acquisition module is configured to: acquire the text to be classified;

[0220] A classification module is configured to: use a dynamic multi-layer semantic perception model to obtain the category to which the text to be classified belongs;

[0221] Among them, the dynamic multi-layer semantic perception machine model includes a word embedding layer, a dynamic depth controller and several layers of weighted feature learners connected in sequence, and each layer of weighted feature learners is composed of a semantic perception machine and a base classifier connected in sequence; the output of the word embedding layer serves as the input of the semantic perception machine of all weighted feature learners, and the output of the semantic perception machine of each layer of weighted feature learners serves as the input of the semantic perception machine of the next layer of weighted feature learners, and the output of the base classifier of each layer of weighted feature learners is input into the dynamic depth controller.

[0222] The training module is configured to:

[0223] Initialize the number of iterations and model parameters, and calculate the initial features of the text in the training set;

[0224] Based on the initial features and model parameters, the prediction results obtained by the weighted feature learner of each layer are calculated and input into the dynamic depth controller to optimize the model depth, and the base classifier weights obtained by the weighted feature learner of each layer are calculated. The text category probabilities obtained by the weighted feature learner of each layer are integrated to obtain an integrated prediction result, and the model parameters are optimized in combination with the real labels in the training set;

[0225] Determine whether the end condition is met. If so, output the model parameters as the optimal model parameters to obtain a trained dynamic multi-layer semantic perceptron model; otherwise, increase the number of iterations and continue to optimize the model parameters.

[0226] It should be noted here that the various modules in this embodiment correspond one-to-one to the various steps in Example 1, and the specific implementation processes are the same, which will not be repeated here.

[0227] Example 3

[0228] This embodiment provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the steps of the text classification method based on a dynamic multi-layer semantic perceptron as described in the first embodiment above are implemented.

[0229] Example 4

[0230] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of the text classification method based on a dynamic multi-layer semantic perceptron as described in the first embodiment above are implemented.

[0231] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage and optical storage, etc.) containing computer-usable program code.

[0232] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0233] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0234] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0235] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware through a computer program. The program can be stored in a computer-readable storage medium, and when executed, the program can include the processes in the above-described method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0236] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A text classification method based on dynamic multi-layer semantic perception, characterized in that: include: Get the text to be classified; Using a dynamic multi-layer semantic perception model to obtain the category to which the text to be classified belongs; The dynamic multi-layer semantic perception model includes a word embedding layer, a dynamic depth controller, and several layers of sequentially connected weighted feature learners. Each layer of weighted feature learners is composed of sequentially connected semantic perception machines and base classifiers. The output of the word embedding layer serves as the input of the semantic perception machines of all weighted feature learners. The output of the semantic perception machine of each layer of weighted feature learners serves as the input of the semantic perception machine of the next layer of weighted feature learners. The output of the base classifier of each layer of weighted feature learners is input into the dynamic depth controller. The semantic perception machine of each layer of weighted feature learner obtains output text features through addition and normalization operations based on the text features output by the semantic perception machine of the previous layer, the initial text features after random sequence conversion, and the text features output by the semantic perception machine of the previous layer after processing by the multi-layer perception machine; The training process of the dynamic multi-layer semantic perception model is as follows: initializing the number of iterations and model parameters, and calculating the initial features of the text in the training set; Based on the initial features and model parameters, the prediction results obtained by the weighted feature learner of each layer are calculated and input into the dynamic depth controller to optimize the model depth, and the base classifier weights obtained by the weighted feature learner of each layer are calculated, and the text category probabilities obtained by the weighted feature learner of each layer are integrated to obtain an integrated prediction result, and the model parameters are optimized in combination with the real labels in the training set; it is determined whether the end condition is met, and if so, the model parameters are output as the optimal model parameters to obtain a trained dynamic multi-layer semantic perceptron model; otherwise, the number of iterations is increased, and the model parameters are continuously optimized; The specific method for optimizing the model depth is: based on the prediction results obtained by the weighted feature learner of each layer, calculate the accuracy of the base classifier of the weighted feature learner of each layer, and based on the accuracy and threshold, determine whether the conditions for adjusting the model depth are met. If so, adjust the model depth; otherwise, keep the model depth unchanged.

2. A text classification method based on a dynamic multi-layer semantic perceptron as claimed in claim 1, characterized in that: The base classifier of each layer of weighted feature learner obtains the prediction result output by the weighted feature learner based on the text features output by the semantic perception machine of the weighted feature learner in the same layer.

3. A text classification method based on a dynamic multi-layer semantic perceptron as claimed in claim 1, characterized in that: The dynamic depth controller performs weighted averaging on the prediction results output by the base classifiers of each layer of the weighted feature learner to obtain the probability that the text belongs to each category, and takes the category with the highest probability as the category to which the text belongs.

4. A text classification system based on dynamic multi-layer semantic perception, characterized in that: include: A training module, configured to: initialize the number of iterations and model parameters, and calculate initial features of the text in the training set; Based on the initial features and model parameters, the prediction results obtained by the weighted feature learner of each layer are calculated and input into the dynamic depth controller to optimize the model depth, and the base classifier weights obtained by the weighted feature learner of each layer are calculated, and the text category probabilities obtained by the weighted feature learner of each layer are integrated to obtain an integrated prediction result, and the model parameters are optimized in combination with the real labels in the training set; it is determined whether the end condition is met, and if so, the model parameters are output as the optimal model parameters to obtain a trained dynamic multi-layer semantic perceptron model; otherwise, the number of iterations is increased, and the model parameters are continuously optimized; The text acquisition module is configured to: acquire the text to be classified; A classification module is configured to: use a dynamic multi-layer semantic perception model to obtain the category to which the text to be classified belongs; The dynamic multi-layer semantic perception model includes a word embedding layer, a dynamic depth controller, and several layers of sequentially connected weighted feature learners. Each layer of weighted feature learners is composed of sequentially connected semantic perception machines and base classifiers. The output of the word embedding layer serves as the input of the semantic perception machines of all weighted feature learners. The output of the base classifiers of each layer of weighted feature learners is input into the dynamic depth controller. The output of the semantic perception machine of each layer of weighted feature learners serves as the input of the semantic perception machine of the next layer of weighted feature learners. The semantic perception machine of each layer of weighted feature learner obtains output text features through addition and normalization operations based on the text features output by the semantic perception machine of the previous layer, the initial text features after random sequence conversion, and the text features output by the semantic perception machine of the previous layer after processing by the multi-layer perception machine; The specific method for optimizing the model depth is: based on the prediction results obtained by the weighted feature learner of each layer, calculate the accuracy of the base classifier of the weighted feature learner of each layer, and based on the accuracy and threshold, determine whether the conditions for adjusting the model depth are met. If so, adjust the model depth; otherwise, keep the model depth unchanged.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the text classification method based on a dynamic multi-layer semantic perceptron as claimed in any one of claims 1 to 3 are implemented.

6. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the text classification method based on a dynamic multi-layer semantic perceptron according to any one of claims 1 to 3 are implemented.

Images