Text classification method, device, apparatus and computer readable storage medium

Abstract

Embodiments of the present application provide a text classification method, device, apparatus and computer readable storage medium, comprising: obtaining a text classification request, wherein the text classification request comprises a consultation text; in response to the text classification request, performing text feature extraction and character feature extraction on the consultation text respectively, and correspondingly obtaining a text feature vector of the consultation text and a character feature vector of each character in the consultation text; performing pooling processing on each character feature vector to determine a local feature vector of the consultation text; and classifying the consultation text according to the text feature vector and the local feature vector to obtain a classification result of the consultation text.

Description

Technical Field

[0001] This application relates to the field of computers, and includes, but is not limited to, a text classification method, device, apparatus, and computer-readable storage medium. Background Technology

[0002] Currently, my country's medical resources are unevenly distributed, and the emergence of intelligent medical question-and-answer platforms can effectively reduce this imbalance. On these platforms, patients describe their health conditions by inputting text, and the intelligent medical question-and-answer platform uses algorithms to predict the appropriate department and corresponding condition the patient should consult, then notifies the relevant specialists to respond.

[0003] However, medical question-and-answer platforms in related technologies only make disease judgments based on the overall semantics of the text. When the semantics of the text entered by the user are not rich enough, the question-and-answer platform cannot make an accurate judgment based on the semantics. Summary of the Invention

[0004] In view of the problems in related technologies, embodiments of this application provide a text classification method, device, apparatus and computer-readable storage medium.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides a text classification method, including:

[0007] Obtain a text classification request, wherein the text classification request includes consultation text;

[0008] In response to the text classification request, text feature extraction and character feature extraction are performed on the consultation text respectively, to obtain the text feature vector of the consultation text and the character feature vector of each character in the consultation text.

[0009] Pooling is performed on each character feature vector to determine the local feature vector of the consultation text;

[0010] The consultation text is classified based on the text feature vector and the local feature vector to obtain the classification result of the consultation text.

[0011] This application provides a text classification device, including:

[0012] The acquisition module is used to acquire text classification requests, which include consultation text.

[0013] The feature extraction module is used to perform text feature extraction and character feature extraction on the consultation text in response to the text classification request, so as to obtain the text feature vector of the consultation text and the character feature vector of each character in the consultation text.

[0014] The pooling processing module is used to perform pooling processing on each of the character feature vectors to determine the local feature vectors of the consultation text.

[0015] The classification module is used to classify the consultation text based on the text feature vector and the local feature vector to obtain the classification result of the consultation text.

[0016] This application provides a text classification device, including:

[0017] The memory is used to store executable instructions; the processor is used to implement the above-mentioned text classification method when executing the executable instructions stored in the memory.

[0018] This application provides a computer-readable storage medium storing executable instructions, which, when executed by a processor, implement the above-described text classification method.

[0019] The text classification method, device, apparatus, and computer-readable storage medium provided in this application embodiment extract text features and character features from the consultation text to obtain the text feature vector of the consultation text and the character feature vector of each character in the consultation text. Pooling is then performed on each character feature vector to obtain the local feature vector of the consultation text. Based on the text feature vector and the local feature vector, the classification result of the consultation text is obtained. Thus, this application embodiment combines the text feature vector, which can represent the sentence features of the consultation text, and the local feature vector, which can represent the local features of the consultation text, to classify the consultation text. This allows for classification based on more semantic information, improving the accuracy of classification. Attached Figure Description

[0020] Figure 1 This is a schematic diagram illustrating an application scenario of the text classification method provided in the embodiments of this application;

[0021] Figure 2 This is a flowchart illustrating the text classification method provided in an embodiment of this application;

[0022] Figure 3 This is a flowchart illustrating the text classification method provided in an embodiment of this application;

[0023] Figure 4 This is a flowchart illustrating the text classification method provided in an embodiment of this application;

[0024] Figure 5 This is a flowchart illustrating the text classification method provided in an embodiment of this application;

[0025] Figure 6 This is a schematic diagram of the pre-trained network structure provided in the embodiments of this application;

[0026] Figure 7 This is a flowchart illustrating the text classification method provided in an embodiment of this application;

[0027] Figure 8 This is a schematic diagram of the composition structure of the text classification device provided in the embodiments of this application;

[0028] Figure 9 This is a schematic diagram of the composition structure of the text classification device provided in the embodiments of this application. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0030] In the following description, references to "some embodiments" refer to a subset of all possible embodiments. However, it is understood that "some embodiments" may be the same or different subsets of all possible embodiments and may be combined with each other without conflict. Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art to which the embodiments of this application pertain. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit the application.

[0031] Based on the problems existing in related technologies, this application provides a text classification method. By extracting text features and character features from the consultation text, the text feature vector and the character feature vector of each character in the consultation text are obtained. The character feature vector is then pooled to obtain the local feature vector of the consultation text. Based on the text feature vector and the local feature vector, the classification result of the consultation text is obtained. Thus, this application combines the text feature vector that can represent the sentence features of the consultation text and the local feature vector that can represent the local features of the consultation text to classify the consultation text, so that the consultation text can be classified based on more semantic information, thereby improving the accuracy of classification.

[0032] Figure 1 This is a schematic diagram illustrating an application scenario of the text classification method provided in the embodiments of this application, such as... Figure 1As shown, the text classification system for implementing the text classification method includes a terminal 10, a network 20, and a server 30. The server 30 obtains the text classification request sent by the terminal 10 through the network 20. The text classification request includes consultation text. In response to the text classification request, the server 30 performs text feature extraction and character feature extraction on the consultation text, respectively, to obtain the text feature vector of the consultation text and the character feature vector of each character in the consultation text. The server performs pooling processing on each character feature vector to determine the local feature vector of the consultation text. Based on the text feature vector and the local feature vector, the server classifies the consultation text to obtain the classification result. The server then sends the classification result to the terminal 10 through the network 20. The terminal 10 visualizes the classification result, and the user can directly display the received classification result on the current interface 10-1.

[0033] The text classification method provided in this application will be described below with reference to exemplary applications and implementations of the server provided in the embodiments of this application. See also Figure 2 , Figure 2 This is a flowchart illustrating the text classification method provided in the embodiments of this application, which will be combined with... Figure 2 The steps shown are explained.

[0034] Step S201: Obtain a text classification request, wherein the text classification request includes consultation text.

[0035] In some embodiments, users can initiate text classification requests through medical consultation platforms or other consulting platforms. When initiating a text classification request, users can directly enter a consultation text in the platform's input box, or the consultation text can be the result of the user's selection through the platform's options. For example, when a user initiates a text classification request through a medical consultation platform, they can directly enter text describing their physical problems or feelings to form a consultation text. Alternatively, they can select from the platform's options whether to buy medicine or make an appointment, which part of their body is experiencing a problem, or the symptoms they are experiencing, thus forming the consultation text.

[0036] Step S202: In response to the text classification request, perform text feature extraction and character feature extraction on the consultation text respectively, and obtain the text feature vector of the consultation text and the character feature vector of each character in the consultation text.

[0037] In some embodiments, text feature extraction refers to extracting features from the entire consultation text to obtain a text feature vector, which is used to represent the overall semantics of the consultation text. Character feature extraction refers to extracting features from each character of the consultation text, or from each word after word segmentation of the consultation text, to obtain a character feature vector, which is used to represent the information of each character or each word in the consultation text.

[0038] In some embodiments, feature extraction of the consultation text can be achieved through the feature extraction layer in the RoBERTa-wwm-ext model (a Chinese pre-trained language model). Before inputting the consultation text into the model, a [CLS] identifier can be added to the beginning of the text to represent the feature vector of the entire sentence, and a [SEP] identifier can be added to the end to indicate the end of the text. When the consultation text with the identifiers is input into the model, the global information of the consultation text, i.e., the text feature vector, is output at the [CLS] position, and local information, i.e., the local feature vector, is output at each character of the consultation text except for the identifier.

[0039] It should be noted that the local feature vector of each character can be a two-dimensional vector, which can include the semantic features of each character and the positional information of each character in the consultation text.

[0040] Step S203: Perform pooling processing on each character feature vector to determine the local feature vector of the consultation text.

[0041] In some embodiments, pooling of character feature vectors can be achieved through pooling layers in a Text Convolutional Neural Network (TextCNN). Pooling is essentially a feature extraction process from character feature vectors. That is, in this embodiment, a TextCNN model can be cascaded after the RoBERTa-wwm-ext model. The TextCNN model can effectively capture the local features of the consultation text. This embodiment can employ max-pooling to process the character feature vectors, obtaining local feature vectors of the consultation text. This improves both computational speed and the robustness of the extracted local feature vectors.

[0042] Here, local feature vectors are used to represent the semantic features of certain words or characters in the consultation text. Dimensionality reduction is performed on the character feature vectors during pooling. For example, the number of character feature vectors can be the same as the number of characters in the consultation text. If the number of characters is L, the character feature vectors can be L two-dimensional vectors. After pooling these L two-dimensional vectors, a one-dimensional vector of size P1 is obtained, which is the local feature vector of the consultation text.

[0043] Step S204: Classify the consultation text according to the text feature vector and the local feature vector to obtain the classification result of the consultation text.

[0044] In some embodiments, after obtaining the text feature vector and the local feature vector, the text feature vector at [CLS] and the pooled local feature vector can be fused to obtain a target feature vector that contains both the overall semantic information of the consultation text and the local information. The target feature vector can be put into a preset classifier for prediction and classification, and the classification result can be normalized to obtain the probability of each type. For example, when a user consults a doctor on a medical consultation platform, the medical consultation text undergoes feature extraction and pooling. The text feature vector and local feature vectors corresponding to the medical consultation text are then fused to obtain the target feature vector. This target feature vector can be input into a preset department classifier or disease classifier for prediction and classification. When a hospital has fifty departments or one hundred diseases, the department classifier will output a fifty-dimensional department vector, where each dimension represents a department. After normalizing the department vector, the probability of each dimension in the department vector is obtained. The department with the highest probability is determined as the classification result and displayed on the medical consultation platform. Similarly, the disease classifier will output a one hundred-dimensional disease vector, where each dimension represents a disease. After normalizing the disease vector, the probability of each dimension in the disease vector is obtained. The disease with the highest probability is determined as the classification result and displayed on the medical consultation platform.

[0045] In some embodiments, results with a probability greater than a preset threshold (e.g., 30%) can be identified as classification results, and at least one classification result can be displayed to the user for the user to make their own judgment.

[0046] This application embodiment combines text feature vectors that can represent the characteristics of the consultation text statements and local feature vectors that can represent the local characteristics of the consultation text to classify the consultation text, so that the consultation text can be classified based on more semantic information, thereby improving the accuracy of classification.

[0047] In some embodiments, before performing feature processing on the consultation text, it is also necessary to convert the consultation text into computer language that the server can recognize. Therefore, based on the foregoing embodiments, Figure 3 This is a flowchart illustrating the text classification method provided in the embodiments of this application, such as... Figure 3 As shown, step S202 can be achieved through the following steps:

[0048] Step S301: Perform word embedding processing on the consultation text, mapping the consultation text into a consultation text vector and a character vector for each character in the consultation text.

[0049] In some embodiments, feature extraction of consultation text can be achieved through the feature extraction layer in the RoBERTa-wwm-ext model. This model has an embedding layer before the feature extraction layer. The embedding layer is used to perform word embedding processing on the consultation text, mapping the consultation text to a numerical semantic space, which we can call the word vector space.

[0050] In some embodiments, before performing word embedding processing on the consultation text, a [CLS] identifier can be added to the beginning of the consultation text to represent the feature vector of the entire sentence, and a [SEP] identifier can be added to the end to indicate the end of the consultation text.

[0051] In this embodiment, the embedding layer consists of three parts: 1. Word embedding, used to determine the features of each character; 2. Position embedding, used to determine the positional features of each character; 3. Segment embedding, used to segment different sentences in the consultation text, mainly for determining the sentence relationships in the consultation text. After word embedding processing of the consultation text, the consultation text is mapped into a vector. The consultation text vector is output in [CLS], and the character vector corresponding to each character is output at each character position. The consultation text vector contains the semantics of the entire consultation text, and the character vector contains the semantics of the corresponding character.

[0052] Step S302: Extract text features from the consultation text vector to obtain the text feature vector of the consultation text.

[0053] Step S303: Extract character features from the character vector of each character in the consultation text to obtain the character feature vector of each character.

[0054] In some embodiments, feature extraction of the consultation text can be achieved through the feature extraction layer in the RoBERTa-wwm-ext model. Text feature extraction is performed on the consultation text vector, outputting the global information of the consultation text, i.e., the text feature vector, at [CLS]. Character feature extraction is performed on the character vector of each character in the consultation text, obtaining the local information of each character, i.e., the local feature vector.

[0055] This application embodiment performs word embedding and feature extraction on the consultation text, enabling it to classify the text not only based on its overall semantics but also on the local information of the characters within the text. This enriches the semantic information of the text and makes the classification results more accurate.

[0056] In some embodiments, character feature vectors can be pooled using a neural network, based on the foregoing embodiments. Figure 4 This is a flowchart illustrating the text classification method provided in the embodiments of this application, such as... Figure 4 As shown, step S203 can be achieved through the following steps:

[0057] Step S401: Perform a convolution operation on each character feature vector to determine the feature information of each character feature vector.

[0058] In some embodiments, the TextCNN model includes at least convolutional layers and pooling layers. When performing convolution operations on characters, convolutional kernels of different sizes can slide one-dimensionally on each character feature vector to extract features from the character feature vector and obtain the feature information of the character feature vector.

[0059] Step S402: Perform a first pooling process and a second pooling process on each of the feature information to determine the first local feature vector and the second local feature vector of the consultation text, wherein the Euclidean distance between the first local feature vector and the second local feature vector is less than a preset value.

[0060] In some embodiments, when classifying consultation texts, it is necessary to perform two different classifications of that type. In this case, two TextCNN models can be cascaded after the RoBERTa-wwm-ext model. These two TextCNN models process character feature vectors using a hard data sharing approach. In this embodiment, the hard data sharing approach can be that the parameters of the convolutional layers and pooling layers in the two TextCNN models are the same, and the two TextCNN models use the L2 norm (a regularization method) to reduce differences.

[0061] In this embodiment, performing a first pooling process and a second pooling process on the feature information can be used to extract different types of features from the feature information, thereby obtaining feature vectors with different information. For example, if the feature information is related to the user's physical health, performing a first pooling process on the feature information can be used to extract departmental features, and the resulting first local feature vector is used to represent departmental-related features. Performing a second pooling process on the feature information can be used to extract disease features, and the resulting second local feature vector is used to represent disease-related features.

[0062] It should be noted that, in the embodiments of this application, the Euclidean distance between the first local feature vector and the second local feature vector can be limited to a preset value. In this way, when the difference between the two local feature vectors is small enough, the first local feature vector and the second local feature vector are more universal.

[0063] This application embodiment combines multiple classification tasks using hard parameter sharing by cascading two TextCNN models after the RoBERTa-wwm-ext model, making the acquired local feature information more general and improving the robustness of the model.

[0064] Based on the foregoing embodiments, Figure 5 This is a flowchart illustrating the text classification method provided in the embodiments of this application, such as... Figure 5 As shown, step S204 can be achieved through the following steps:

[0065] Step S501: Perform fusion processing on the text feature vector and the local feature vector to determine the target feature vector corresponding to the consultation text.

[0066] In some embodiments, after determining the text feature vector representing the overall semantics of the consultation text and the local feature vector representing the local information of the consultation text, the two vectors can be concatenated and fused to obtain a target feature vector that contains both the overall semantic information of the consultation text and the local information.

[0067] Step S502: Obtain the project classifier for the target project type.

[0068] In some embodiments, the target item type refers to the type of classification used for the consultation text. If the consultation text is classified into one type, then there is one target item type; if the consultation text is classified into two types, then there are two target item types. For example, when classifying medical consultation text, it is possible to classify the medical text only by department to obtain the corresponding department result, in which case the target item type is department; if it is necessary to classify the medical text by department or by disease to obtain the corresponding department result and disease result, then the target item type is department or disease.

[0069] In this embodiment of the application, each target project type corresponds to a project classifier, the classifier corresponding to a department is a department classifier, and the classifier corresponding to a disease is a disease classifier.

[0070] Step S503: Based on the target feature vector, the consultation text is classified using the project classifier to obtain the classification result of the consultation text corresponding to the target project type.

[0071] In some embodiments, the consultation text is classified according to the project classifier corresponding to it, thus obtaining the classification result of the consultation text corresponding to the target project type. In some embodiments, the classification of consultation text can be achieved through the following steps:

[0072] Step S5031: Using the project classifier, determine the matching degree between the target feature vector and each project in the project library of the target project type.

[0073] In the embodiments of this application, each target project type corresponds to a project library, which contains all the projects of the target project type. For example, when the target project type is a department, the project library contains all the departments of the hospital, such as internal medicine, surgery and pediatrics.

[0074] In some embodiments, the item classifier can match the target feature vector with each item in the item library, and determine the degree of matching between the target feature vector and each vector by using the semantic information carried in the target vector.

[0075] Step S5032: Identify items with a matching degree greater than the threshold as the classification results of the consultation text corresponding to the target item type.

[0076] In some embodiments, items with a matching degree greater than a threshold (e.g., 80%) are presented to the user as classification results corresponding to the target item type in the consultation text. If multiple items have a matching degree exceeding the threshold, all items are presented to the user with their matching degree indicated, allowing the user to make a selection. If no item in the item library has a matching degree exceeding the threshold, the three items with the highest matching degree are presented to the user, allowing the user to choose according to their own situation.

[0077] In some embodiments, the classification results can be normalized to determine the probability of each item in the item library. Therefore, step S503 can also be implemented through the following steps:

[0078] Step S5033: Use the project classifier to perform vector classification processing on the target feature vector to obtain the processed feature vector.

[0079] In this embodiment of the application, the dimension of the feature vector after processing by the classifier can be the same as the total number of items in the item library, where each dimension of the vector corresponds to one item information. For example, if the department item library contains fifty department items, then the dimension of the feature vector after processing by the department classifier is fifty, and each dimension corresponds to one department information.

[0080] Step S5034: Perform probability normalization on the processed feature vector to obtain the probability value of each item in the target item type corresponding to the consultation text.

[0081] Step S5035: The item with the highest probability value is determined as the classification result of the consultation text corresponding to the target item type.

[0082] In some embodiments, the consultation text includes medical consultation text, and the item classifier includes at least a department classifier and a disease classifier. Therefore, step S503 can also be implemented through the following steps:

[0083] Step S5036: Based on the target feature vector, the medical consultation text is classified using the department classifier and the disease classifier respectively, thereby obtaining at least one department category and at least one disease category of the medical consultation text.

[0084] Step S5037: Determine the at least one department category and the at least one disease category as the classification result corresponding to the consultation text.

[0085] In this embodiment, two TextCNN models can be cascaded after the RoBERTa-wwm-ext model. After performing different pooling processes on the feature information through the two different TextCNN models, two different local feature vectors are obtained. These two different local feature vectors are determined as target feature vectors. The medical consultation text is classified using a department classifier and a disease classifier, respectively, to obtain at least one department category and at least one disease category of the medical consultation text. The department category and disease category are then displayed as classification results.

[0086] After determining the target feature vector corresponding to the consultation text, this embodiment classifies the consultation text according to at least one classifier, thereby using a multi-task approach to classify the text and improving classification efficiency.

[0087] In some embodiments, before performing text feature extraction and character feature extraction on the consultation text, data preprocessing is required. Based on the foregoing embodiments, this application embodiment may further include the following steps:

[0088] Step S1: Before performing text feature extraction and character feature extraction on the consultation text, the consultation text is subjected to error correction processing, text length processing and special character processing to obtain the processed consultation text.

[0089] In some embodiments, since users inevitably make spelling mistakes when entering consultation text, it is necessary to first perform error correction processing on the consultation text before extracting text features and character features respectively. In some embodiments, open-source Chinese error correction tools (such as pycorrector) can be used. The error correction tool detects the location of misspelled words in the consultation text based on the language model, and corrects the phonetic similarity features of pinyin, the stroke Wubi editing distance features, and the perplexity features of the language model for phonetic similarity misspellings, shape similarity misspellings, or variant characters in the consultation text.

[0090] Secondly, the consultation text needs to be processed for text length. If the consultation text is too short, semantic information will be lost, and if the text is too long, the text classification time will increase. In this embodiment, the text length can be fixed, for example, 384 can be selected as the sentence length, so as to reduce the classification time while preserving the text information as much as possible.

[0091] Finally, special character processing is required for the consultation text. Due to the special nature of certain fields, a small number of special characters may not appear in the general thesaurus. In this embodiment, words not appearing in the thesaurus are uniformly processed according to the [UNK] identifier. The consultation text after error correction, text length processing, and special character processing is determined as the processed consultation text.

[0092] Correspondingly, step S202 involves extracting text features and character features from the consultation text, including extracting text features and character features from the processed consultation text.

[0093] This application embodiment avoids the problem of incorrect classification results caused by typos and special characters in the consultation text by performing data preprocessing, thereby improving the accuracy of the classification results.

[0094] The following describes an exemplary application of the embodiments of this application in a real-world application scenario.

[0095] In related technologies, since the release of the BERT pre-trained language model, pre-training has become one of the most popular directions in the field of Natural Language Processing (NLP). BERT pre-training mainly includes two tasks: Masked Language Model (MLM) and Next Sentence Prediction (NSP). For the MLM task, the Chinese version of the BERT pre-trained model released by Google randomly masks 15% of the characters in a sentence before making predictions. However, for Chinese, words contain richer semantic information than characters. Therefore, the WWM model, which uses lexical-level masks instead of character-level masks, was proposed, allowing the model to learn lexical information. This application optimizes and improves upon this pre-trained model.

[0096] This application proposes a multi-task medical text classification system based on a pre-trained Chinese RoBERTa-wwm-ext model. The system mainly consists of an input module, a data preprocessing module, a model prediction module, and a result display module. This system is the front-end and core module of the entire medical question-answering platform. Only by accurately and quickly determining the user's corresponding disease can the system recommend the most relevant doctor. Therefore, the accuracy of the disease classification algorithm is crucial.

[0097] This application proposes a multi-task medical question classification algorithm based on a Chinese pre-trained language model. The algorithm is built on the RoBERTa-wwm-ext pre-trained model and consists of two main parts: (1) The collected medical dialogue platform domain data is used to continue training on the publicly available RoBERTa-wwm-ext language model with a small learning rate, so that the language model is closer to the target task data distribution and the subsequent training time is accelerated; a multi-task training method is designed to combine department classification and disease classification tasks, and the parameter hard sharing method is adopted to make the acquired features more universal and improve the robustness of the model.

[0098] Compared with classification models in related technologies, the MT-Bert-CNN classification model proposed in this application has the following advantages:

[0099] 1. Classification algorithms in related technologies treat disease category prediction and department prediction as independent classification tasks. However, in reality, disease types and departments are closely related. Using a multi-task approach can effectively extract more general features, reduce model overfitting, and improve model robustness.

[0100] 2. The original classification method only uses the [CLS] sentence vector, which lacks local information. In this embodiment, a TextCNN model is cascaded after the RoBERTa-wwm-ext model. CNN can effectively capture local features. By fusing the local features extracted by CNN with the sentence features output by RoBERTa-wwm-ext, a more effective semantic representation of sentences can be obtained.

[0101] 3. The RoBERTa-wwm-ext pre-trained model contains richer semantic information than BERT. Therefore, in this embodiment, RoBERTa-wwm-ext is used as the base model for low-level feature extraction.

[0102] The following provides a more detailed description of this application with reference to examples. The embodiments of this application include two parts: data training and task classification. The data classification includes two parts: data preprocessing and model training.

[0103] In the data preprocessing section, firstly, the input text character series C1, C2, ..., C... N-1 C N And the word sequence W1, W2, ..., W after text segmentation. M-1 W M Here, N represents the length of the original text sequence, and M is the length of the text after word segmentation. In the original BERT model, for each character, there is a 15% probability that it needs to be predicted. These 15% of characters to be predicted are divided into three parts: 80% of the characters are replaced with [MASK] characters, 10% are randomly replaced with other words, and 10% are retained as the original words. However, in Chinese, a character contains the smallest semantic unit. Therefore, when a character is selected for prediction, all Chinese characters that make up the same word need to be [MASK], which is called full-word masking.

[0104] Table 1 shows examples of full-word masks. For ease of understanding, Table 1 only considers replacing words with the [MASK] tag.

[0105] Table 1 Examples of Full-Word Masks

[0106]

[0107] In the model training section:

[0108] Step 1: The pre-trained network structure provided in this application embodiment mainly consists of three parts, including: an embedding layer, a feature layer composed of 12 Transformer components, and a final classification layer, as follows. Figure 6 As shown, Figure 6 This is a schematic diagram of the pre-trained network structure provided in the embodiments of this application. The masked text [CLS], w1, w2, [mask], ... [mask], ... [mask], w L-1 w L The [SEP] input is placed into the embedding layer 610 of the pre-trained network structure. The masked text is then positionally encoded in the embedding layer. The encoded vector is then input into the feature layer 620. The vector is then processed by the multi-head attention mechanism layer 621 and the feedforward linear layer 622 in the feature layer 620. The extracted feature vector is then input into the classification layer 630 for prediction. The predicted character at each mask position is obtained, and the cross-entropy loss of each predicted character is also obtained.

[0109] In some embodiments, the embedding layer consists of three parts: word embedding, used to represent the features of each word; position embedding, used to represent the positional features of each word; and fragment embedding, used to segment and represent different sentences, mainly for sentence relationship understanding tasks.

[0110] Step 2, Model Mathematical Definition: In this embodiment of the application, all parameters of the embedding layer 610 and feature layer 620 in the pre-trained model are defined as θ. f The learning parameter in classification layer 630 is θ v ,definition The text feature vector is obtained after passing through the embedding layer 610 and the feature layer 620. L represents the original text length excluding the special characters [CLS] and [SEP], and D represents the feature vector of each character generated by the feature layer. It is the output of the text feature vector after being processed by the classification layer 630. The output is a vector with a size equal to the number of words in the sample vocabulary. We define the dimension of this vector as V and the length as the size of the vocabulary. Each dimension represents the probability of the sample belonging to each word in the vocabulary.

[0111] In some embodiments, the target loss can be defined as the cross-entropy loss, and the sample loss function can be expressed as the following formula (1):

[0112]

[0113] Where P represents the number of Mask characters in the sample, label(i) represents the true value corresponding to the i-th character, and predict(i) represents the vocabulary probability value output by the model at the i-th character.

[0114] Step 3: The pre-trained network structure provided in this application embodiment can use gradient descent to accelerate the learning process. In order to update the parameters of the objective function along the direction of the fastest gradient descent, the updated parameters are as shown in formula (2):

[0115]

[0116] The objective function is updated using formulas (3) and (4):

[0117]

[0118]

[0119] Where α is the learning rate of the pre-trained network structure, typically around 1e -5 ~1e -4 In this application embodiment, it can be set to 2e. -5 .

[0120] In some embodiments, the model training portion includes data preprocessing, a multi-task classification network structure, a mathematical definition of the model, an optimization strategy, and a model prediction component.

[0121] In some embodiments, data preprocessing includes three steps, where step 1 is error correction. To reduce error propagation caused by word segmentation and the problem of an excessively large vocabulary, characters are used as input in the open-source Chinese pre-trained model. Since users inevitably make spelling errors during input, it is necessary to correct the annotated training text. In this embodiment, the open-source Chinese error correction tool pycorrector can be used. pycorrector detects the location of misspelled characters based on a language model and corrects phonetic, visually similar, or variant characters through phonetic similarity features, stroke-based Wubi editing distance features, and language model perplexity features.

[0122] Step 2 is text length processing. In some embodiments, the maximum length of the pre-trained model text input is limited to 512. If the length exceeds 512, it cannot be matched. If the text length is too short, semantic information will be lost, and if it is too long, the training time will increase. In this embodiment, by statistically analyzing the length of the training samples, 384 can be selected as the sentence length. This length covers more than 95% of the training sample lengths, which can reduce the training time while preserving text information as much as possible.

[0123] Step 3 involves handling special characters. Due to the specialized nature of the medical field, a small number of special characters may not appear in the vocabulary of the pre-trained model. Words that do not appear in the vocabulary are uniformly marked with [UNK]. Finally, a [CLS] identifier is added to the beginning of the text to represent the feature vector of the entire sentence, and a [SEP] identifier is added to the end to indicate the end of the sentence.

[0124] The backbone of the multi-task classification network structure provided in this embodiment consists of 12 Transformer layers, referred to as the feature extraction layer. It is initialized using model parameters pre-trained in Data Training 1. The vector output at the [CLS] position (i.e., the text feature vector) contains the semantics of the entire original sentence. This embodiment uses two TextCNN networks to further extract local features (i.e., local feature vectors) for departmental classification and disease type classification, respectively. Then, the obtained local features (i.e., local feature vectors) are concatenated and fused with the global information (i.e., the text feature vector) at the [CLS] position, and the fused vector is fed into the classifiers for each task for classification.

[0125] The embodiments of this application have three advantages: 1. The embodiments of this application use a large amount of medical data to continuously train on publicly available pre-trained models, making the model parameters more consistent with the medical field; 2. In terms of feature extraction, the embodiments of this application combine local features (i.e., local feature vectors) and vectors (i.e., text feature vectors) obtained by TextCNN to further improve the classification effect; 3. Combining department classification and disease classification tasks, since there is a strong correlation between the two, the two can use hard parameter sharing for the underlying features, and for local features, in order to improve the scalability of subsequent tasks, the embodiments of this application adopt a soft-margin approach and use the L2 metric for the two TextCNN networks to reduce the difference, thereby further reducing the overfitting phenomenon.

[0126] In this embodiment of the application, all parameters of the feature extraction layer in the classification model are defined as θ in the mathematical definition of the model. f The parameters in the two TextCNNs are θ. c_1 and θ c_2 The learning parameter in the classification layer is θ y_1 and θ y_2 ,definition This is the text feature vector obtained after the feature extraction layer. and This involves passing the two-dimensional vectors obtained from the feature extraction layer (excluding the [CLS] and [SEP] positions) through two pooling layers of a TextCNN network to obtain local feature vectors. These local feature vectors are one-dimensional vectors with dimensions P_1 and P_2. The D-dimensional vector at the [CLS] position extracted by the feature extraction layer is concatenated with the two local feature vectors to obtain the final classification feature vectors D+P_1 and D+P_2 for the two tasks. These final feature vectors are then fed into the final task classifier for classification.

[0127] Therefore, the target loss in this application embodiment can be divided into three parts: cross-entropy loss for the department prediction task, cross-entropy loss for the disease prediction task, and L2 distance between two local feature vectors. The L2 distance is used to measure the difference between two features. When the difference between the two vectors is small enough, the obtained features are more general, making the model more robust and preventing overfitting caused by a small number of training samples.

[0128] In this embodiment, a sample loss function L(θ) is provided. f ,θ c_1 ,θ c_2 ,θ y_1 ,θ y_2 This can be expressed as shown in formula (5):

[0129]

[0130] The optimization strategies in this application include using gradient descent to accelerate the learning process, and using dropout to deactivate some neurons to avoid overfitting caused by too many parameters. This effectively improves the robustness of the model. Meanwhile, to avoid catastrophic forgetting, this application uses a warm-up strategy during training to prevent the discarding of previously retained general information due to an excessively high initial learning rate.

[0131] The embodiments of this application can perform model prediction (i.e., text classification) according to the following steps, such as... Figure 7 As shown, Figure 7 This is a flowchart illustrating the structure of the text classification method provided in this application embodiment.

[0132] Step 1: Preprocess the original text.

[0133] Step 2: Use the RoBERT-WWM-extra model to obtain the [CLS] sentence features of the original text.

[0134] Step 3: Input the obtained sentence features into the TextCNN model for local semantic acquisition to obtain local features.

[0135] Step 4: Concatenate and fuse the sentence features with the local features to obtain the fused features.

[0136] Step 5: Input the fused features into a classifier for at least one task to perform classification prediction.

[0137] Step 6: Input the classification prediction results into the softmax layer for probability normalization, and determine the result with the highest probability as the final recognition result.

[0138] This application combines the two tasks of department classification and disease prediction using a hard-shared parameter approach, which can effectively extract more general features across multiple tasks and avoid overfitting. Furthermore, it employs a convolutional neural network to obtain local features of the text and concatenates these local features with the overall sentence features, allowing the model to acquire more semantic information to identify the predicted category. The use of WWM masking effectively improves upon BERT's random character-based masking method, defining the smallest mask unit as the word granularity in Chinese. RoBERT-WWM-extra utilizes a larger Chinese corpus, and subsequent expansions based on this corpus achieve higher accuracy than previous BERT methods.

[0139] Figure 8 This is a schematic diagram of the composition structure of the text classification device provided in the embodiments of this application, as shown below. Figure 8 As shown, the text classification device 800 includes:

[0140] The acquisition module 801 is used to acquire a text classification request, which includes consultation text; the feature extraction module 802 is used to extract text features and character features from the consultation text in response to the text classification request, thereby obtaining the text feature vector of the consultation text and the character feature vector of each character in the consultation text; the pooling processing module 803 is used to perform pooling processing on each of the character feature vectors to determine the local feature vector of the consultation text; and the classification module 804 is used to classify the consultation text according to the text feature vector and the local feature vector to obtain the classification result of the consultation text.

[0141] In some embodiments, the feature extraction module 802 is further configured to perform word embedding processing on the consultation text, mapping the consultation text to a consultation text vector and a character vector of each character in the consultation text; perform text feature extraction on the consultation text vector to obtain the text feature vector of the consultation text; and perform character feature extraction on the character vector of each character in the consultation text to obtain the character feature vector of each character.

[0142] In some embodiments, the pooling processing module 803 is further configured to perform a convolution operation on each character feature vector to determine the feature information of each character feature vector; perform a first pooling process and a second pooling process on each feature information to determine a first local feature vector and a second local feature vector of the consultation text, wherein the Euclidean distance between the first local feature vector and the second local feature vector is less than a preset value; and determine the first local feature vector and the second local feature vector as the local feature vector of the consultation text.

[0143] In some embodiments, the classification module 804 is further configured to perform fusion processing on the text feature vector and the local feature vector to determine the target feature vector corresponding to the consultation text; classify the target feature vector to obtain a classification vector with a preset dimension; obtain a project classifier for the target project type; and classify the consultation text using the project classifier based on the target feature vector to obtain a classification result of the consultation text corresponding to the target project type.

[0144] In some embodiments, the classification module 804 is further configured to use the project classifier to determine the matching degree between the target feature vector and each project in the project library of the target project type; and to determine the projects with a matching degree greater than a threshold as the classification result of the consultation text corresponding to the target project type.

[0145] In some embodiments, the classification module 804 is further configured to perform vector classification processing on the target feature vector using the project classifier to obtain a processed feature vector; perform probability normalization processing on the processed feature vector to obtain a probability value of each item in the target project type corresponding to the consultation text; and determine the item with the highest probability value as the classification result of the consultation text corresponding to the target project type.

[0146] In some embodiments, the consultation text includes medical consultation text; the project classifier includes at least a department classifier and a disease classifier; the classification module 804 is further configured to classify the medical consultation text according to the target feature vector using the department classifier and the disease classifier respectively, thereby obtaining at least one department category and at least one disease category of the medical consultation text; and to determine the at least one department category and the at least one disease category as the classification result corresponding to the consultation text.

[0147] In some embodiments, the text classification device 800 further includes: an error correction processing module, a text length processing module, and a special character processing module, which are respectively used to perform error correction processing, text length processing, and special character processing on the consultation text before performing text feature extraction and character feature extraction on the consultation text to obtain the processed consultation text; correspondingly, the step of performing text feature extraction and character feature extraction on the consultation text includes: performing the text feature extraction and character feature extraction on the processed consultation text.

[0148] It should be noted that the description of the apparatus in this application embodiment is similar to the description of the method embodiment described above, and has similar beneficial effects as the method embodiment; therefore, it will not be repeated. For technical details not disclosed in this apparatus embodiment, please refer to the description of the method embodiment of this application for understanding.

[0149] It should be noted that, in the embodiments of this application, if the above-described text classification method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the related technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a terminal to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0150] Correspondingly, embodiments of this application provide a text classification device. Figure 9 This is a schematic diagram of the composition structure of the text classification device provided in the embodiments of this application, as shown below. Figure 9 As shown, the text classification device 900 includes at least a processor 901 and a computer-readable storage medium 902 configured to store executable instructions, wherein the processor 901 generally controls the overall operation of the text classification device. The computer-readable storage medium 902 is configured to store instructions and applications executable by the processor 901, and may also cache data to be processed or processed by various modules in the processor 901 and the text classification device 900, and can be implemented using flash memory or random access memory (RAM).

[0151] This application provides a storage medium storing executable instructions. When these executable instructions are executed by a processor, they cause the processor to perform the method provided in this application, for example... Figure 2 The method shown.

[0152] In some embodiments, the storage medium may be a computer-readable storage medium, such as a ferromagnetic random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic surface memory, optical disc, or a compact disk-read-only memory (CD-ROM); or it may be a device that includes one or any combination of the above-mentioned memories.

[0153] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0154] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts within a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files storing one or more modules, subroutines, or code sections). As an example, executable instructions may be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0155] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

[0156] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0157] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components may be combined, or integrated into another system, or some features may be ignored or not performed.

[0158] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A text classification method, characterized in that, The method includes: Obtain a text classification request, wherein the text classification request includes consultation text; In response to the text classification request, text feature extraction and character feature extraction are performed on the consultation text respectively, to obtain the text feature vector of the consultation text and the character feature vector of each character in the consultation text. Pooling is performed on each character feature vector to determine the local feature vector of the consultation text; Based on the text feature vector and the local feature vector, the consultation text is classified to obtain the classification result of the consultation text; The step of pooling each character feature vector to determine the local feature vector of the consultation text includes: Perform a convolution operation on each of the character feature vectors to determine the feature information of each character feature vector; Each of the aforementioned feature information undergoes a first pooling process and a second pooling process to determine a first local feature vector and a second local feature vector of the consultation text. The first and second pooling processes are used to extract different types of features from each of the aforementioned feature information. The Euclidean distance between the first and second local feature vectors is less than a preset value. The models used in the first and second pooling processes employ a hard data sharing approach to process the character feature vectors. This hard data sharing includes at least the identical parameters of the convolutional and pooling layers in the models. The first local feature vector and the second local feature vector are determined as the local feature vectors of the consultation text; The step of classifying the consultation text based on the text feature vector and the local feature vector to obtain the classification result of the consultation text includes: The text feature vector and the local feature vector are fused to determine the target feature vector corresponding to the consultation text; Obtain a project classifier for the target project type; the target project type is the type of consultation text to be classified, and each target project type corresponds to a project classifier; Based on the target feature vector, the consultation text is classified using the project classifier to obtain the classification result of the consultation text corresponding to the target project type.

2. The method according to claim 1, characterized in that, The step of performing text feature extraction and character feature extraction on the consultation text to obtain the text feature vector of the consultation text and the character feature vector of each character in the consultation text includes: The consultation text is processed by word embedding, and the consultation text is mapped into a consultation text vector and a character vector of each character in the consultation text; Text feature extraction is performed on the consultation text vector to obtain the text feature vector of the consultation text; Character feature extraction is performed on the character vector of each character in the consultation text to obtain the character feature vector of each character.

3. The method according to claim 1, characterized in that, The step of classifying the consultation text using the project classifier based on the target feature vector to obtain a classification result of the consultation text corresponding to the target project type includes: The project classifier is used to determine the matching degree between the target feature vector and each project in the project library of the target project type; Items with a matching degree greater than the threshold are identified as the classification results of the consultation text corresponding to the target item type.

4. The method according to claim 1, characterized in that, The step of classifying the consultation text using the project classifier based on the target feature vector to obtain a classification result of the consultation text corresponding to the target project type includes: The target feature vector is classified using the project classifier to obtain the processed feature vector. The processed feature vector is subjected to probability normalization to obtain the probability value of each item in the target item type corresponding to the consultation text. The item with the highest probability value is identified as the classification result corresponding to the target item type in the consultation text.

5. The method according to claim 1, characterized in that, The consultation text includes medical consultation text; the project classifier includes at least a department classifier and a disease classifier; The step of classifying the consultation text using the project classifier based on the target feature vector to obtain a classification result of the consultation text corresponding to the target project type includes: Based on the target feature vector, the medical consultation text is classified using the department classifier and the disease classifier respectively, resulting in at least one department category and at least one disease category for the medical consultation text. The at least one department category and the at least one disease category are determined as the classification results corresponding to the consultation text.

6. The method according to claim 1, characterized in that, The method further includes: Before performing text feature extraction and character feature extraction on the consultation text, the consultation text is subjected to error correction, text length reduction and special character processing to obtain the processed consultation text. Correspondingly, the step of extracting text features and character features from the consultation text includes: The processed consultation text is subjected to text feature extraction and character feature extraction respectively.

7. A text classification device, characterized in that, include: The acquisition module is used to acquire text classification requests, which include consultation text. The feature extraction module is used to perform text feature extraction and character feature extraction on the consultation text in response to the text classification request, so as to obtain the text feature vector of the consultation text and the character feature vector of each character in the consultation text. The pooling processing module is used to perform pooling processing on each of the character feature vectors to determine the local feature vectors of the consultation text. The classification module is used to classify the consultation text based on the text feature vector and the local feature vector to obtain the classification result of the consultation text; The pooling module is further configured to perform convolution operations on each character feature vector to determine the feature information of each character feature vector; perform first pooling and second pooling operations on each feature information to determine the first local feature vector and the second local feature vector of the consultation text, wherein the first pooling and second pooling operations are used to extract different types of features from each feature information, and the Euclidean distance between the first local feature vector and the second local feature vector is less than a preset value; the model used in the first pooling operation and the model used in the second pooling operation process the character feature vectors using a hard data sharing method, wherein the hard data sharing includes at least the same parameters for the convolutional layer and the pooling layer in the model; and determine the first local feature vector and the second local feature vector as the local feature vectors of the consultation text. The classification module is further configured to fuse the text feature vector and the local feature vector to determine the target feature vector corresponding to the consultation text; obtain a project classifier for the target project type; the target project type is the type for classifying the consultation text, and each target project type corresponds to a project classifier; and classify the consultation text using the project classifier based on the target feature vector to obtain the classification result of the consultation text corresponding to the target project type.

8. A text classification device, characterized in that, include: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the text classification method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The device stores executable instructions for causing a processor to execute the executable instructions to implement the text classification method according to any one of claims 1 to 6.

Images