Text classification method, apparatus, device, and storage medium

Abstract

The application provides a text classification method, device and equipment and a storage medium. The method comprises the following steps: obtaining a text to be classified; determining at least two tokens corresponding to the text to be classified; obtaining a classification result of the text to be classified based on the at least two tokens, at least one prefix information and a classification model; the classification model and each prefix information are obtained by inputting at least one sample token corresponding to a sample text and a category token corresponding to sample prefix information into an initial classification model, and training the initial classification model based on the correlation between the category tokens and the correlation between the sample token and the category token; and each prefix information is used for modeling the semantics of each category. The application can improve the accuracy of text classification.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a text classification method, apparatus, device, and storage medium. Background Technology

[0002] Text classification technology is a method that uses machine learning and natural language processing techniques to classify text. Its core idea is to divide text data into different categories or labels in order to better understand and process large-scale text data.

[0003] Existing text classification methods include statistical methods, such as Naive Bayes or Support Vector Machines, deep learning methods such as Convolutional Neural Networks or Recurrent Neural Networks, as well as traditional rule and pattern matching methods.

[0004] However, for the same text, it may involve different topics, such as both sports and entertainment. When classifying texts involving different topics, the above text classification methods are not very accurate. Summary of the Invention

[0005] This invention provides a text classification method, apparatus, device, and storage medium to address the shortcomings of low accuracy in text classification results in the prior art, thereby improving the accuracy of text classification results.

[0006] This invention provides a text classification method, comprising: Get the text to be categorized; Identify at least two tokens corresponding to the text to be classified; Based on the at least two tokens, at least one prefix information, and a classification model, the classification result of the text to be classified is determined; the classification model and each of the prefix information are obtained by inputting at least one sample token corresponding to the sample text and the category token corresponding to the sample prefix information into the initial classification model, and training the initial classification model based on the correlation between the category tokens and the correlation between the sample token and the category token; each of the prefix information is used to model the semantics of each category.

[0007] According to a text classification method provided by the present invention, determining the classification result of the text to be classified based on at least two tokens, at least one prefix information, and a classification model includes: If the number of tokens is greater than a preset number, downsample at least two tokens to obtain at least one feature information; The at least one feature information and the prefix information are input into the classification model to obtain the classification result output by the classification model.

[0008] According to a text classification method provided by the present invention, when the number of tokens is greater than a preset number, downsampling of at least two tokens to obtain at least one feature information includes: The sampling interval is determined based on the number of tokens and the preset number; Based on the sampling interval, at least two tokens are sampled at intervals to obtain at least one feature information.

[0009] According to a text classification method provided by the present invention, when the number of tokens is greater than a preset number, downsampling of at least two tokens to obtain at least one feature information includes: The target quantity is determined based on the number of tokens and the preset quantity; The average value of the embedding vectors of the target number of tokens is determined sequentially to obtain the at least one feature information.

[0010] According to a text classification method provided by the present invention, the classification model includes multiple text classification layers and a downsampling layer disposed between two adjacent text classification layers; The process of determining the classification result of the text to be classified based on the at least two tokens, at least one prefix information, and the classification model includes: The at least two tokens and the at least one prefix information are input into the first text classification layer. The first target feature information corresponding to the at least two tokens output by the first text classification layer is downsampled through a downsampling layer connected to the first text classification layer. The downsampled first target feature information is input into the second text classification layer. The second target feature information output by the second text classification layer is downsampled through a downsampling layer connected to the second text classification layer until the classification result is output.

[0011] According to a text classification method provided by the present invention, the classification model is trained based on the following method: obtaining at least two sample tokens corresponding to the sample text and a category token corresponding to the sample prefix information; The at least two sample tokens and the category token are input into the initial classification model. The correlation between the category tokens and the correlation between the sample tokens and the category tokens are determined through a self-attention mechanism. Based on the correlation between the category tokens and the correlation between the sample tokens and the category tokens, the initial classification model is trained to obtain the classification model and the prefix information.

[0012] According to a text classification method provided by the present invention, the method further includes: If the classification result indicates that the text to be classified does not contain illegal content, the text to be classified is input into the large language model to obtain the response information corresponding to the text to be classified. The response information is identified as a new text to be classified, and a new classification result is determined for the new text to be classified. If the new classification result indicates that the response information does not include illegal content, the response information is output through the large language model.

[0013] The present invention also provides a text classification device, comprising: The acquisition module is used to acquire the text to be classified. The determination module is used to determine at least two tokens corresponding to the text to be classified; An input module is used to determine the classification result of the text to be classified based on the at least two tokens, at least one prefix information, and a classification model. The classification model and each prefix information are obtained by inputting at least one sample token corresponding to the sample text and a category token corresponding to the sample prefix information into an initial classification model, and training the initial classification model based on the correlation between the category tokens and the correlation between the sample tokens and the category tokens. Each prefix information is used to model the semantics of each category.

[0014] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the text classification methods described above.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the text classification method as described above.

[0016] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements any of the text classification methods described above.

[0017] The text classification method, apparatus, device, and storage medium provided by this invention acquire the text to be classified and determine at least two tokens corresponding to the text. Based on these at least two tokens, at least one pre-trained prefix information for modeling the semantics of each category, and a classification model, the classification result of the text to be classified can be determined. Specifically, the classification model and each prefix information are obtained by inputting at least one sample token corresponding to the sample text and a category token corresponding to the sample prefix information into an initial classification model. The initial classification model is trained based on the correlation between category tokens and the correlation between sample tokens and category tokens. Since the classification model and prefix information are trained by establishing the correlation between different category tokens and the correlation between category tokens and each sample token, if there is conceptual overlap between different categories modeled by multiple prefix information after establishing the correlation, the classification model and prefix information trained in the above manner will consider the intersection and correlation between the different categories when performing classification, resulting in higher accuracy of the trained classification model and prefix information. Therefore, when classifying the text to be classified based on this classification model and prefix information, the accuracy of the classification result is also higher. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the text classification method provided in an embodiment of the present invention.

[0020] Figure 2 This is one of the structural schematic diagrams of the downsampling method provided in the embodiments of the present invention.

[0021] Figure 3 This is the second schematic diagram of the downsampling method provided in the embodiment of the present invention.

[0022] Figure 4 The third schematic diagram of the downsampling method provided in the embodiment of the present invention.

[0023] Figure 5 The fourth schematic diagram of the downsampling method provided in the embodiments of the present invention.

[0024] Figure 6 This is a schematic diagram illustrating downsampling via a model, provided as an embodiment of the present invention.

[0025] Figure 7 This is a schematic diagram of the training process of the classification model provided in an embodiment of the present invention.

[0026] Figure 8 This is a schematic diagram of the structure of the text classification device provided in an embodiment of the present invention.

[0027] Figure 9 A schematic diagram of the physical structure of an electronic device is provided. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0029] Text classification technology has wide applications in fields such as sentiment analysis, spam filtering, and news classification. Existing technologies typically employ statistical methods, deep learning-based methods, or traditional rule and pattern matching methods for text classification. These methods offer high accuracy when classifying single-type text. However, in text classification tasks, each text may be assigned multiple labels or categories. A concrete example of multi-label text classification is content classification on social media. Consider a social media platform where users can post content on various topics, such as sports, entertainment, and technology. Each user's post may cover multiple topics, thus requiring multi-label classification. For example, a user's post might cover both sports and technology, such as discussing the impact of artificial intelligence on the sports industry. In this case, multi-label classification requires identifying both the sports and technology labels. Through multi-label text classification, social media platforms can better provide personalized recommendations and content filtering services. For example, they can recommend relevant content based on a user's interest tags or filter out content the user is not interested in. Therefore, accurately classifying text involving different categories is a pressing technical problem that needs to be solved.

[0030] This invention addresses the aforementioned problems by proposing a text classification method. In this method, a classification model can be used to classify the text to be classified. During model training, at least one sample token corresponding to the sample text and a category token corresponding to the sample prefix information are input into an initial classification model. The initial model extracts the correlations between category tokens and the correlations between sample tokens and category tokens before training. Since each category term has semantic meaning, and different category labels are also related, this embodiment can establish correlations between different category tokens and between category tokens and each sample token. Thus, if there is conceptual overlap between different labels of multi-sample prefix information after establishing these correlations, it will be more helpful for multi-label classification. For example, in a large language model, the safety category includes both violence and vulgarity; these two category concepts overlap. Therefore, after establishing the correlations, the accuracy of the trained classification model and prefix information is higher. Consequently, when classifying the text to be classified based on this classification model and prefix information, the accuracy of the classification result is also higher.

[0031] The following is combined Figures 1 to 7 The text classification method provided in this invention is described below. This invention is applicable to text classification in any scenario, particularly in scenarios involving the review of input and output content of large language models. The review of input and output content essentially involves classifying the input and output content. The execution entity of this method can be an electronic device such as a terminal device, computer, server, server cluster, or a specially designed text classification device. It can also be a text classification device installed within such an electronic device, which can be implemented through software, hardware, or a combination of both.

[0032] Figure 1 This is a flowchart illustrating the text classification method provided in an embodiment of the present invention, as shown below. Figure 1 As shown, the method includes: Step 101: Obtain the text to be classified.

[0033] In this embodiment, the text to be classified can be understood as the text that needs to be classified. It can be text entered by the user or text obtained from the Internet through web crawling.

[0034] Step 102: Determine at least two tokens corresponding to the text to be classified.

[0035] In this step, the text to be classified can be decomposed into a token sequence, which includes at least two tokens. A token can be understood as the smallest unit or element that the model can process, such as word segments, phrase fragments, characters, or combinations of n consecutive words or characters in the text to be classified.

[0036] Step 103: Input at least two tokens and at least one prefix information into the classification model to determine the classification result of the text to be classified output by the classification model.

[0037] The classification model and each prefix information are obtained by inputting at least one sample token corresponding to the sample text and the category token corresponding to the sample prefix information into the initial classification model. The initial classification model is trained based on the correlation between category tokens and the correlation between sample tokens and category tokens. Each prefix information is used to represent the semantics of each category.

[0038] In this step, after identifying at least two tokens corresponding to the text to be classified, the obtained at least two tokens and at least one pre-trained prefix information can be input into the classification model to obtain the classification result of the text to be classified. Each prefix information is used to model the semantics of its respective class; that is, after pre-training, each prefix information has the ability to determine the category of the text to be classified.

[0039] When training the classification model and each prefix information, at least one sample token corresponding to the sample text is obtained, and at least one sample prefix information is obtained through initialization, thereby determining the category token corresponding to each sample prefix information. Each sample token and each category token is input into the initial classification model. Through a self-attention mechanism, the correlation between category tokens and the correlation between each category token and each sample token can be established. Based on these correlations, the initial classification model is trained to obtain the trained prefix information and classification model. By establishing the correlation between different category tokens and the correlation between each category token and each sample token, if there is conceptual overlap between different labels of multiple prefix information after establishing the correlation, it becomes more helpful for multi-label classification. Moreover, the trained prefix information is used to model the semantics of each category, and through this prefix information, the different categories of the text to be classified can be identified.

[0040] The text classification method provided in this invention obtains the text to be classified and determines at least two tokens corresponding to the text. Based on these at least two tokens, at least one pre-trained prefix information for modeling the semantics of each category, and a classification model, the classification result of the text to be classified can be determined. Specifically, the classification model and each prefix information are obtained by inputting at least one sample token corresponding to the sample text and a category token corresponding to the sample prefix information into an initial classification model. The initial classification model is trained based on the correlation between category tokens and the correlation between sample tokens and category tokens. Since the classification model and prefix information are trained by establishing the correlation between different category tokens and the correlation between category tokens and each sample token, if there is a conceptual overlap between different categories modeled by multiple prefix information after establishing the correlation, the classification model and prefix information trained in the above manner will consider the intersection and correlation between the different categories when performing classification, resulting in higher accuracy of the trained classification model and prefix information. Therefore, when classifying the text to be classified based on this classification model and prefix information, the accuracy of the classification result is also higher.

[0041] For example, based on the above embodiments, when determining the classification result of the text to be classified by inputting at least two tokens and prefix information into the classification model, the following method can be used: If the number of tokens is greater than the preset number, at least two tokens are downsampled to obtain at least one feature information; at least one feature information and prefix information are input into the classification model to obtain the classification result output by the classification model.

[0042] Specifically, text classification tasks are relatively easy with short texts. However, with the application of large-scale models, various language models can now handle long text information. Text classification tasks typically have strict limitations on computational resources; for texts with hundreds or thousands of tokens, the computational overhead is enormous. For some pre-trained language models, such as BERT or ERNIE, the maximum input length they can accept is pre-defined, such as 256 or 512 tokens. Therefore, how to handle the scope of long texts is a key issue in long text classification.

[0043] In existing technologies, long texts are typically segmented into smaller segments using a sliding window approach, and these resulting segments are then input into a classification model. However, this segmentation method can lead to inconsistencies between the different segments. For example, if a long text contains a contrasting relationship between its parts, simple segmentation can result in inconsistencies between the two segments, leading to incorrect text classification. Similarly, if the text to be classified contains a sarcastic statement, with the first half expressing praise and the second half conveying the opposite sentiment, truncating the text using a sliding window approach will also create inconsistencies between the two segments. Furthermore, increasing the maximum input length of the model significantly increases its computational load.

[0044] To address the aforementioned issues, in this embodiment of the invention, when the number of tokens corresponding to the text to be classified exceeds a preset number, at least two tokens can be downsampled to obtain at least one feature. This downsampled feature, along with the trained prefix information, is then input into the classification model to classify the text. The preset number can be understood as the maximum input length of the model, or it can be a pre-set value.

[0045] If the number of tokens is less than or equal to the preset number, there is no need to downsample at least two tokens. You can directly input at least two tokens and prefix information into the classification model for classification.

[0046] In this embodiment, when the number of tokens exceeds a preset number, at least two tokens can be downsampled. The resulting feature information and prefix information are then input into the classification model to obtain the classification result output by the model. By downsampling at least two tokens, the model's field of view can be increased without truncating the text to be classified, and this does not increase computational power consumption.

[0047] Based on the above embodiments, this invention proposes multiple methods for downsampling at least two tokens, including directly downsampling the input tokens and downsampling the input data through a model.

[0048] In one possible implementation, when the number of tokens is greater than a preset number, at least two tokens are downsampled to obtain at least one feature information. The sampling interval can be determined based on the number of tokens and the preset number, and the at least two tokens are sampled at intervals based on the sampling interval to obtain at least one feature information.

[0049] In the specific implementation, sampling is only performed on the text to be classified when it is a long text, meaning the number of its corresponding tokens exceeds a preset number. Typically, for long texts, there are certain connections and complementary relationships between the contexts; as long as the entire segment after the maximum input length is not truncated, semantics can be recovered to some extent. Furthermore, BERT and Ernie are both models trained using masks, which have strong semantic recovery capabilities. Therefore, in this embodiment, mask dimensionality reduction can be considered for downsampling.

[0050] Specifically, the sampling interval can be determined based on the total number of tokens and a preset number. For example, assuming the maximum input length of the model is 512 (i.e., the preset number is 512), then text with a length within 512*2 will be downsampled using a double mask (i.e., a sampling interval of 2), and text with a length within 512*3 will be downsampled using a triple mask (i.e., a sampling interval of 3). In this way, the length of the sampled text will always remain within the 512-bit range. It should be understood that longer text will be sampled more sparsely, and shorter text will be sampled more densely.

[0051] After determining the sampling interval, at least two tokens can be sampled at intervals based on the sampling interval to obtain at least one feature information.

[0052] Figure 2 This is one of the structural schematic diagrams of the downsampling method provided in the embodiments of the present invention. Figure 3 This is a second schematic diagram of the downsampling method provided in an embodiment of the present invention, as shown below. Figure 2 and Figure 3 As shown, assuming the maximum input length of the BERT model is 512, and the length of the text to be classified is within 512*2, then the sampling interval is 2. Figure 2 or Figure 3 As shown, for at least two tokens corresponding to the text to be classified, one token can be sampled every other token. The length of the feature information obtained after sampling is half the length of at least two tokens. Since the maximum input length of the BERT model is 512, the actual field of view of the BERT model becomes 1024 through the above mask downsampling method. In this embodiment, the mask can be understood as discarding the masked tokens. For example, in... Figure 2 In the process, the 2nd token, 4th token, 6th token, etc., can be discarded. Figure 3 In the process, the first token, the third token, the fifth token, and so on can be discarded.

[0053] For mask downsampling methods with sampling intervals of other values, and Figure 2 or Figure 3 The mask downsampling method with a sampling interval of 2 shown is similar and will not be described again here.

[0054] In this embodiment, the sampling interval can be determined based on the number of tokens and a preset number. Then, at least two tokens can be sampled at intervals based on the sampling interval to obtain at least one feature information. By using the above-mentioned interval sampling downsampling method, not only can the field of view of the classification model be improved and the accuracy of the classification results be increased, but the downsampling method is also relatively simple and improves the efficiency of downsampling.

[0055] In another possible implementation, when the number of tokens is greater than a preset number, at least two tokens are downsampled to obtain at least one feature information. This can be done by determining the target number based on the number of tokens and the preset number, and then determining the average value of the embedding vectors of the target number of tokens in sequence to obtain at least one feature information.

[0056] Specifically, the target number can be determined based on the total number of tokens and a preset number. For example, assuming the maximum input length of the model is 512, i.e., the preset number is 512, then for text with a length within 512*2, the target number is 2; for text with a length within 512*3, the target number is 3. After determining the target number, the average value of the embedding vectors of the target number of tokens can be determined sequentially to obtain at least one feature. In this way, the length of the sampled text can always be kept within the range of 512.

[0057] Figure 4 This is the third schematic diagram of the downsampling method provided in the embodiments of the present invention, as shown below. Figure 4 As shown, assuming the maximum input length of the BERT model is 512, and the length of the text to be classified is within 512*2, then the number of targets is 2. For at least two tokens corresponding to the text to be classified, the embedding vectors of two adjacent tokens can be averaged to obtain at least one feature. The length of the feature obtained after sampling in the above way is half the length of at least two tokens. Since the maximum input length of the BERT model is 512, the actual field of view of the BERT model becomes 1024 through the above mask downsampling method.

[0058] Figure 5 This is the fourth schematic diagram of the downsampling method provided in the embodiments of the present invention, as shown below. Figure 5As shown, assuming the maximum input length of the BERT model is 512, and the length of the text to be classified is within 512*3, then the number of targets is 3. For at least two tokens corresponding to the text to be classified, the embeddings of the three adjacent tokens can be averaged to obtain at least one feature. The length of the feature obtained after sampling in the above way is 1 / 3 of the length of at least two tokens. Since the maximum input length of the BERT model is 512, the actual field of view of the BERT model becomes 1536 through the above mask downsampling method.

[0059] In this embodiment, the target number can be determined based on the number of tokens and a preset number. Then, the average value of the embedding vectors of the target number of tokens can be determined sequentially to obtain at least one feature information. By using the downsampling method described above to determine the average value of the embedding vectors of the target number of tokens, on the one hand, the field of vision of the classification model can be improved and the accuracy of the classification results can be increased. On the other hand, the information of the downsampled tokens can be preserved without increasing the computational load of the model itself.

[0060] It should be understood that both of the above downsampling methods enhance the field of view by performing downsampling at the data level. The following section will explain in detail the method of performing downsampling at the model level without performing downsampling at the data level.

[0061] In another possible implementation, the classification model may include multiple text classification layers and a downsampling layer set between two adjacent text classification layers. When determining the classification result of the text to be classified based on at least two tokens, at least one prefix information and the classification model, at least two tokens and at least one prefix information may be input into the first text classification layer. The first target feature information corresponding to the at least two tokens output by the first text classification layer is downsampled by the downsampling layer connected to the first text classification layer. The downsampled first target feature information is input into the second text classification layer. The second target feature information output by the second text classification layer is downsampled by the downsampling layer connected to the second text classification layer until the classification result is output.

[0062] Specifically, the structure of the classification model can be improved, which includes multiple text classification layers, and a downsampling layer can be set between two adjacent text classification layers. Figure 6 This is a schematic diagram of downsampling through a model provided in an embodiment of the present invention, such as... Figure 6As shown, taking the BERT model as an example, the BERT model includes multiple text classification layers (BERT layers). A downsampling layer (conv1D Subsampling) is set between adjacent text classification layers. After obtaining at least two tokens corresponding to the text to be classified, these at least two tokens and at least one prefix information can be input into the first text classification layer of the model. That is, the first text classification layer receives a relatively long input that has not been downsampled. After processing the at least two tokens, the first text classification layer outputs the first target feature information. This first target feature information is then downsampled by conv1D after passing through the downsampling layer connected to the first text classification layer. Next, the downsampled first target feature information is input into the second text classification layer for processing, outputting the second target feature information. This second target feature information is then downsampled by conv1D after passing through the downsampling layer connected to the second text classification layer, until the classification result is output from the last text classification layer.

[0063] like Figure 6 As shown, during the downsampling process described above, the length of each BERT layer remains unchanged when at least two tokens of full length pass through it. However, by adding a conv1D layer between two BERT layers for downsampling, the length is reduced to half of the original length. The maximum input length of the second BERT layer can then be set to half that of the first BERT layer. After passing through a Conv1D layer, another downsampling is performed, and the maximum input length of the third BERT layer is set to one-quarter of that of the first BERT layer. After two downsampling operations, the maximum input length of the subsequent n-2 BERT layers is one-quarter of the initial input. This allows the model to receive independent semantic representations for each token while maintaining the computational cost of the n-2 layers unchanged.

[0064] In this embodiment, a downsampling layer can be set between two adjacent text classification layers. The downsampling layer downsamples the feature information output by the previous text classification layer. On the one hand, this method can preserve the semantics of each input token. On the other hand, it can expand the scope of the classification model while reducing the amount of computation.

[0065] For example, based on the above embodiments, the classification models described in the foregoing embodiments are trained in the following manner: Obtain at least two sample tokens corresponding to the sample text and a category token corresponding to the sample prefix information. Input the at least two sample tokens and the category token into the initial classification model. Determine the correlation between category tokens and the correlation between sample tokens and category tokens through a self-attention mechanism. Then, train the initial classification model based on the correlation between category tokens and the correlation between sample tokens and category tokens to obtain the classification model and prefix information.

[0066] Specifically, each category in the text has its own semantic meaning. The usual approach is to use BERT or Ernie's [CLS] tokens to map to each category dimension, using category indexes as category representations. However, this approach does not utilize the semantics of the categories, nor does it model the relationships between categories, and it does not model the relationships between categories and all tokens in the input sequence.

[0067] To address the aforementioned issues, in this embodiment of the invention, category semantics can be modeled using sample prefix information, and the correlation between categories and between categories and the correlation between categories and each sample token of the input sample text can be modeled.

[0068] Figure 7 This is a schematic diagram of the training process of the classification model provided in the embodiments of the present invention, as shown below. Figure 7 As shown, the obtained sample text is decomposed to obtain a sample token sequence, which includes at least two sample tokens. Additionally, a class token corresponding to a sample prefix can be initialized. The at least two sample tokens and the class token are input into the initial classification model. After passing through the BERT LAYE layer in the initial classification model, the self-attention mechanism determines the pairwise correlations between class tokens, as well as the correlations between each class token and each sample token. In this embodiment, the [CLS] token is removed; that is, only at least two sample tokens and the class token are input, without the [CLS] token. During the forward computation, the class token establishes the relationships between classes and also models the relationships with the input sample tokens. The class token of each class is used to determine whether that class is included in the input sequence.

[0069] After establishing the correlation between category tokens and the correlation between sample tokens and category tokens, the initial classification model can be trained based on these correlations. For a certain category token, a binary classification prediction can be made to determine whether it belongs to the category corresponding to that category token. The network parameters of the initial classification model can be adjusted based on the prediction results and the category labels corresponding to the sample text, thereby obtaining the trained classification model and prefix information.

[0070] In this embodiment, when training the classification model and prefix information, the correlation between category tokens and the correlation between sample tokens and category tokens can be determined through a self-attention mechanism. The prefix information trained by this correlation can model category semantics. Moreover, since the classification model considers the correlation between category tokens and the correlation between sample tokens and category tokens, it can accurately identify different categories when the same text involves multiple categories. When classifying such text based on prefix information and the classification model, the accuracy of the classification results can be greatly improved.

[0071] For example, based on the above embodiments, when the classification result indicates that the text to be classified does not include illegal content, the text to be classified is input into the large language model to obtain the response information corresponding to the text to be classified, and the response information is determined as the new text to be classified, and the new classification result corresponding to the new text to be classified is determined. When the new classification result indicates that the response information does not include illegal content, the response information is output through the large language model.

[0072] Specifically, when receiving user input, the large language model needs to review the input. Normal and risk-free content is input into the large language model, while risky content is blocked. Therefore, the classification method provided in this embodiment can be used to review the input and output content of the large language model.

[0073] For example, if the text to be classified is the input content to the large language model, the classification result of the text to be classified is obtained through the methods described in the foregoing embodiments. If the classification result indicates that the text to be classified does not contain illegal content, the text to be classified will be normally input into the large language model to determine the response information corresponding to the text to be classified. If the classification result indicates that the text to be classified contains illegal content, the text to be classified will be intercepted, that is, the text to be classified will not be input into the large language model.

[0074] After determining the response information, the classification result of the response information can be determined according to the methods in the aforementioned embodiments, such as determining the classification result based on the token corresponding to the response information, at least one prefix information, and a classification model. If the classification result of the response information indicates that the response information does not contain illegal content, the response information will be output normally through the large language model. If the classification result of the response information indicates that the response information does not contain illegal content, the response information will be output normally through the large language model. If the classification result of the response information indicates that the response information contains illegal content, the response information will be blocked.

[0075] In this embodiment, if the classification result indicates that the text to be classified does not contain illegal content, inputting the text to be classified into the large language model ensures that all text input into the large language model is risk-free and harmless. After the large language model determines the response information, it can also classify the response information. If the classification result corresponding to the response information indicates that the response information does not contain illegal content, the large language model outputs the response information, thereby ensuring that all text output from the large language model is risk-free and harmless, thus improving the security of the large language model.

[0076] The text classification device provided in the embodiments of the present invention is described below. The text classification device described below and the text classification method described above can be referred to each other.

[0077] Figure 8 This is a schematic diagram of the structure of the text classification device provided in an embodiment of the present invention, with reference to... Figure 8 As shown, the text classification device 800 includes: Module 11 is used to acquire the text to be classified; Module 12 is used to determine at least two tokens corresponding to the text to be classified; The determining module 12 is further configured to determine the classification result of the text to be classified based on the at least two tokens, at least one prefix information, and the classification model; the classification model and each of the prefix information are obtained by inputting at least one sample token corresponding to the sample text and the category token corresponding to the sample prefix information into the initial classification model, and training the initial classification model based on the correlation between the category tokens and the correlation between the sample tokens and the category tokens; each of the prefix information is used to model the semantics of each category.

[0078] In one example embodiment, the determining module 12 is specifically used for: If the number of tokens is greater than a preset number, downsample at least two tokens to obtain at least one feature information; The at least one feature information and the prefix information are input into the classification model to obtain the classification result output by the classification model.

[0079] In one example embodiment, the apparatus further includes: a sampling module, wherein: The determining module 12 is further configured to determine the sampling interval based on the number of tokens and the preset number; A sampling module is used to perform interval sampling on the at least two tokens based on the sampling interval to obtain the at least one feature information.

[0080] In one example embodiment, the determining module 12 is further configured to: The target quantity is determined based on the number of tokens and the preset quantity; The average value of the embedding vectors of the target number of tokens is determined sequentially to obtain the at least one feature information.

[0081] In one example embodiment, the classification model includes multiple text classification layers and a downsampling layer disposed between two adjacent text classification layers; Module 12 is specifically used for: The at least two tokens and the at least one prefix information are input into the first text classification layer. The first target feature information corresponding to the at least two tokens output by the first text classification layer is downsampled through a downsampling layer connected to the first text classification layer. The downsampled first target feature information is input into the second text classification layer. The second target feature information output by the second text classification layer is downsampled through a downsampling layer connected to the second text classification layer until the classification result is output.

[0082] In one example embodiment, the classification model is trained in the following manner: Obtain at least two sample tokens corresponding to the sample text and the category token corresponding to the sample prefix information; The at least two sample tokens and the category token are input into the initial classification model. The correlation between the category tokens and the correlation between the sample tokens and the category tokens are determined through a self-attention mechanism. Based on the correlation between the category tokens and the correlation between the sample tokens and the category tokens, the initial classification model is trained to obtain the classification model and the prefix information.

[0083] In one example embodiment, the device further includes an input module and an output module, wherein: The input module is used to input the text to be classified into the large language model when the classification result indicates that the text to be classified does not contain illegal content, so as to obtain the response information corresponding to the text to be classified. The determining module 12 is further configured to determine the response information as a new text to be classified, and to determine the new classification result corresponding to the new text to be classified; An output module is used to output the response information through the large language model when the new classification result indicates that the response information does not include illegal content.

[0084] The apparatus of this embodiment can be used to execute the method of any embodiment in the text classification method side embodiment. Its specific implementation process and technical effects are similar to those in the text classification method side embodiment. For details, please refer to the detailed description in the text classification method side embodiment, which will not be repeated here.

[0085] Figure 9 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 9 As shown, the electronic device may include a processor 910, a communications interface 920, a memory 930, and a communication bus 940, wherein the processor 910, the communications interface 920, and the memory 930 communicate with each other via the communication bus 940. The processor 910 can call logical instructions in the memory 930 to execute a text classification method. This method includes: acquiring text to be classified; determining at least two tokens corresponding to the text to be classified; and determining a classification result for the text to be classified based on the at least two tokens, at least one prefix information, and a classification model. The classification model and each prefix information are obtained by inputting at least one sample token corresponding to the sample text and a category token corresponding to the sample prefix information into an initial classification model, and training the initial classification model based on the correlation between the category tokens and the correlation between the sample tokens and the category tokens. Each prefix information is used to model the semantics of each category.

[0086] Furthermore, the logical instructions in the aforementioned memory 930 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0087] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the text classification method provided by the above methods. The method includes: acquiring text to be classified; determining at least two tokens corresponding to the text to be classified; and determining the classification result of the text to be classified based on the at least two tokens, at least one prefix information, and a classification model. The classification model and each of the prefix information are obtained by inputting at least one sample token corresponding to the sample text and a category token corresponding to the sample prefix information into an initial classification model, and training the initial classification model based on the correlation between the category tokens and the correlation between the sample tokens and the category tokens. Each of the prefix information is used to model the semantics of each category.

[0088] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a text classification method provided by the methods described above. The method includes: acquiring a text to be classified; determining at least two tokens corresponding to the text to be classified; and determining a classification result for the text to be classified based on the at least two tokens, at least one prefix information, and a classification model. The classification model and each prefix information are obtained by inputting at least one sample token corresponding to a sample text and a category token corresponding to the sample prefix information into an initial classification model, and training the initial classification model based on the correlation between the category tokens and the correlation between the sample tokens and the category tokens. Each prefix information is used to model the semantics of each category.

[0089] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0090] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0091] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A text classification method, characterized in that, include: Get the text to be categorized; Identify at least two tokens corresponding to the text to be classified; Based on the at least two tokens, at least one prefix information, and the classification model, determine the classification result of the text to be classified; The classification model and each of the prefix information are obtained by inputting at least one sample token corresponding to the sample text and the category token corresponding to the sample prefix information into the initial classification model, and training the initial classification model based on the correlation between the category tokens and the correlation between the sample token and the category token. Each of the prefix information is used to model the semantics of each category.

2. The text classification method according to claim 1, characterized in that, The process of determining the classification result of the text to be classified based on the at least two tokens, at least one prefix information, and the classification model includes: If the number of tokens is greater than a preset number, downsample at least two tokens to obtain at least one feature information; The at least one feature information and the prefix information are input into the classification model to obtain the classification result output by the classification model.

3. The text classification method according to claim 2, characterized in that, When the number of tokens is greater than a preset number, at least two tokens are downsampled to obtain at least one feature information, including: The sampling interval is determined based on the number of tokens and the preset number; Based on the sampling interval, at least two tokens are sampled at intervals to obtain at least one feature information.

4. The text classification method according to claim 2, characterized in that, When the number of tokens is greater than a preset number, at least two tokens are downsampled to obtain at least one feature information, including: The target quantity is determined based on the number of tokens and the preset quantity; The average value of the embedding vectors of the target number of tokens is determined sequentially to obtain the at least one feature information.

5. The text classification method according to claim 2, characterized in that, The classification model includes multiple text classification layers and a downsampling layer set between two adjacent text classification layers; The process of determining the classification result of the text to be classified based on the at least two tokens, at least one prefix information, and the classification model includes: The at least two tokens and the at least one prefix information are input into the first text classification layer. The first target feature information corresponding to the at least two tokens output by the first text classification layer is downsampled through a downsampling layer connected to the first text classification layer. The downsampled first target feature information is input into the second text classification layer. The second target feature information output by the second text classification layer is downsampled through a downsampling layer connected to the second text classification layer until the classification result is output.

6. The text classification method according to any one of claims 1-5, characterized in that, The classification model was trained in the following manner: Obtain at least two sample tokens corresponding to the sample text and the category token corresponding to the sample prefix information; The at least two sample tokens and the category token are input into the initial classification model, and the correlation between the category tokens and the correlation between the sample tokens and the category tokens are determined through a self-attention mechanism. The initial classification model is trained based on the correlation between the category tokens and the correlation between the sample token and the category token to obtain the classification model and the prefix information.

7. The text classification method according to any one of claims 1-5, characterized in that, The method further includes: If the classification result indicates that the text to be classified does not contain illegal content, the text to be classified is input into the large language model to obtain the response information corresponding to the text to be classified. The response information is identified as a new text to be classified, and a new classification result is determined for the new text to be classified. If the new classification result indicates that the response information does not include illegal content, the response information is output through the large language model.

8. A text classification device, characterized in that, include: The acquisition module is used to acquire the text to be classified. The determination module is used to determine at least two tokens corresponding to the text to be classified; The determining module is used to determine the classification result of the text to be classified based on the at least two tokens, at least one prefix information and the classification model; The classification model and each of the prefix information are obtained by inputting at least one sample token corresponding to the sample text and the category token corresponding to the sample prefix information into the initial classification model, and training the initial classification model based on the correlation between the category tokens and the correlation between the sample token and the category token. Each of the prefix information is used to model the semantics of each category.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the text classification method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the text classification method as described in any one of claims 1 to 7.

Images