Text classification method and device, electronic equipment and computer readable storage medium

Abstract

The present disclosure provides a text classification method and device, electronic equipment and computer readable storage medium, and relates to the technical field of natural language processing. The method comprises: obtaining a text to be classified; inputting the text to be classified into a trained text category prediction model to output a text category of the text, wherein the text category comprises an attribute category and / or a domain category, and the text category prediction model comprises a format conversion layer, a vector processing layer, a target information processing layer and a prediction layer, the target information processing layer is used to process a word vector of each word in a text in a predetermined format to obtain a global vector of the text, and the prediction layer is used to predict the attribute category and the domain category of the text according to the global vector and the word vector of each word in the text. The present disclosure can overcome the problem that the related art ignores the domain information of the document and the ability of modeling the global attribute information is weak to a certain extent.

Description

Technical Field

[0001] This disclosure relates to the field of natural language processing technology, and in particular to a text classification method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] Attribute category extraction is a subtask of attribute-level sentiment analysis. This task aims to identify the attribute categories corresponding to attribute words described in a sentence, such as the price of a product or the service. Attribute category extraction is currently a hot research topic in natural language processing and a key technology for realizing strong artificial intelligence and question-answering systems. It has attracted widespread attention not only in academia but also in industry. For example, product providers can use attribute category extraction technology to enable machines to automatically analyze the attribute categories of products that people are interested in, thereby improving the quality of their products and saving significant time and labor costs.

[0003] Attribute category extraction methods can be divided into three categories: rule-based and dictionary-based methods, traditional statistical machine learning-based methods, and deep learning-based methods.

[0004] Rule-based and dictionary-based methods mostly rely on heuristic rules and manually constructed dictionaries to extract attribute categories. Although these methods have achieved certain results, their generalization performance is poor, making them difficult to apply to domains or new scenarios with insufficient dictionary resources.

[0005] Traditional statistical machine learning algorithms, such as LDA (Linear Discriminant Analysis) and SVM (Support Vector Machine), are also widely used in attribute category extraction tasks. Attribute category extraction methods based on traditional statistical machine learning rely on manually designed features, the quality of which directly affects model performance. While these methods have achieved good results, manually designing features is a very time-consuming and labor-intensive task, requiring a certain level of expert knowledge.

[0006] Deep learning can automatically learn high-level abstract features suitable for specific tasks in an end-to-end manner and has achieved good performance in many natural language processing tasks. In attribute category extraction based on deep learning, recurrent neural networks such as LSTM (Long Short Term Memory) and GRU (Gated Recurrent Unit) are widely used, mainly because these recurrent neural networks can capture sentence order information and long-distance dependency information well.

[0007] In related technologies, pre-trained language models have made significant progress and achieved great success in many natural language tasks, including attribute category extraction. Large-scale pre-trained language models, trained on large datasets, can acquire a wealth of knowledge and possess powerful semantic representation capabilities.

[0008] While deep learning-based methods have made some progress, these attribute category extraction methods still have some shortcomings. For example, existing attribute category extraction methods have weak zero-shot learning capabilities; they ignore document domain information and have a weak ability to model global attribute information; they have not effectively explored supervised contrastive learning; and they are insufficient in their ability to model part-of-speech tags related to attribute categories.

[0009] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0010] This disclosure provides a text classification method, apparatus, electronic device, and computer-readable storage medium, which at least to some extent overcomes the problems in related technologies that ignore the domain information of documents and have a weak ability to model global attribute information.

[0011] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.

[0012] According to one aspect of this disclosure, a text classification method is provided, the method comprising: acquiring text to be classified, wherein the text includes at least one statement, each statement including at least two words; inputting the text to be classified into a trained text category prediction model, and outputting the text category of the text, wherein the text category includes an attribute category and / or a domain category, the text category prediction model including a format conversion layer, a vector processing layer, a target information processing layer, and a prediction layer, wherein the format conversion layer is used to convert the text to be classified into text of a predetermined format, the vector processing layer is used to perform vectorization processing on the text of the predetermined format to obtain a word vector of each word in the text of the predetermined format, the target information processing layer is used to process the word vector of each word in the text of the predetermined format to obtain a global vector of the text, and the prediction layer is used to predict the attribute category and domain category of the text based on the global vector and the word vector of each word in the text.

[0013] In some exemplary embodiments of this disclosure, based on the foregoing scheme, the format conversion layer converts the statements in the text according to a preset template to obtain statements in a predetermined format.

[0014] In some exemplary embodiments of this disclosure, based on the foregoing scheme, the vector processing layer uses a word segmentation model to segment sentences of a predetermined format and vectorizes the words in each sentence to obtain word vectors for each word in the text of the predetermined format, wherein the word vectors include attribute vectors and domain vectors.

[0015] In some exemplary embodiments of this disclosure, based on the foregoing scheme, the target information processing layer includes a dynamic masking attention module and a part-of-speech (POS) awareness attention module; wherein, the dynamic masking attention module generates first weight information of the word vector of each word in the text, and the dynamic masking attention module is used to obtain a vector containing attribute information and / or domain information; the POS awareness attention module generates second weight information of the word vector of each word in the text, and the dynamic masking attention module is used to obtain a vector containing attribute information and / or domain information; and a global vector corresponding to the text to be classified is generated based on the word vector of each word in the text, the first weight information, and the second weight information.

[0016] In some exemplary embodiments of this disclosure, based on the foregoing scheme, generating first weight information of the word vector of each word in the text through the dynamic mask attention module includes: determining attribute information and domain information corresponding to the word vector of each word in the text based on the word vector of each word in the text; determining the attention coefficient of the semantic association between the domain category and the attribute category corresponding to the word vector of each word in the text based on the attribute information and the domain information; and generating first weight information of the word vector of each word in the text based on the attribute information and domain information corresponding to the word vector of each word in the text and the attention coefficient of the semantic association between the domain category and the attribute category.

[0017] In some exemplary embodiments of this disclosure, based on the foregoing scheme, the first weight information of the word vector of each word in the text is generated by the following formula;

[0018]

[0019]

[0020] in, h represents the word vector of the i-th word in the c-th sentence of the text to be classified. d Represents the neighborhood vector. This represents the semantic relationship between the i-th word in the text to be classified and the d1 domain category; h a Represents an attribute vector. This represents the semantic relationship between the i-th word in the text to be classified and the c1 attribute category; This represents the semantic relationship between the i-th word in the text to be classified and the d1 domain category and the c1 attribute category; The attention coefficient represents the semantic association between the word vector of the i-th word and the domain category and attribute category; This represents the first weight information of the i-th word generated using the dynamic mask attention model; -inf represents negative infinity.

[0021] In some exemplary embodiments of this disclosure, based on the foregoing scheme, generating second weight information of the word vector of each word in the text through the dynamic mask attention module includes: generating association parameters of the domain category and attribute category of the word vector according to the attribute vector and domain vector corresponding to the word vector of each word in the text; determining the attention coefficient of the semantic association relationship between the domain category and attribute category of the word vector of each word in the text according to the part-of-speech tag of the word vector; and generating second weight information of the word vector of each word in the text according to the association parameters and attention coefficients corresponding to the word vector of each word in the text.

[0022] In some exemplary embodiments of this disclosure, based on the foregoing scheme, the second weight information of the word vector of each word in the text is generated by the following formula:

[0023]

[0024] in, In this context, ; indicates vector concatenation; W represents the association between the domain category and attribute category of the word vector of the i-th word in the text to be classified; W and b are learnable parameters of the part-of-speech awareness attention module. The attention coefficient represents the part-of-speech tag of the i-th word in the text to be classified; pos(w i ) is used to determine the part of speech of the word vector corresponding to the i-th word in the text to be classified, where noun represents a noun and adj represents an adjective; This represents the second weight information of the i-th word generated by the part-of-speech awareness attention module.

[0025] In some exemplary embodiments of this disclosure, based on the foregoing scheme, before inputting the text to be classified into a trained text category prediction model and outputting the text category of the text to be classified, the method further includes: training a pre-built deep learning model to obtain the text category prediction model.

[0026] In some exemplary embodiments of this disclosure, the text category prediction model is trained based on the foregoing scheme, including: generating a target loss function using the following formula:

[0027]

[0028]

[0029]

[0030] L = L acl +λ(L dcl +L m )+L pos ;

[0031] Where N represents the number of training samples in a batch, and n represents the length of the statement. This represents the word probability distribution predicted by the pre-built deep learning model for the i-th word in the j-th training data. L represents the true word distribution probability of the i-th word in the j-th training data. m This represents the mask word prediction loss function; This represents the part-of-speech distribution of the i-th word in the j-th training data predicted by a pre-built deep learning model. L represents the true part-of-speech probability of the i-th word in the j-th training data. pos denoted as the loss function for predicting noun and adjective part-of-speech; k represents the number of samples in the i-th domain; τ and λ represent the hyperparameters of the pre-built deep learning model. This represents the global attribute vector for the i-th statement. This represents the domain vector of the i-th statement; This represents the global attribute vector of the j-th statement, which shares the same domain as the i-th statement. L represents the domain vector of the j-th statement that shares the same domain as the i-th statement; dcl This represents the domain-specific self-supervised contrastive loss function. L represents the attribute category vector in the j-th statement that has the same attribute category as the i-th statement; aclThe target loss function represents the self-supervised contrastive loss function for attribute categories. The parameters of the pre-built deep learning model are adjusted through the target loss function to obtain the trained text category prediction model.

[0032] According to another aspect of this disclosure, a text classification apparatus is also provided, the apparatus comprising: a text acquisition module for acquiring text to be classified, wherein the text includes at least one sentence, each sentence including at least two words; a text classification module for inputting the text to be classified into a trained text category prediction model and outputting the text category of the text, wherein the text category includes an attribute category and / or a domain category, the text category prediction model including a format conversion layer, a vector processing layer, a target information processing layer, and a prediction layer, wherein the format conversion layer is used to convert the text to be classified into text of a predetermined format, the vector processing layer is used to perform vectorization processing on the text of the predetermined format to obtain a word vector of each word in the text of the predetermined format, the target information processing layer is used to process the word vector of each word in the text of the predetermined format to obtain a global vector of the text, and the prediction layer is used to predict the attribute category and domain category of the text based on the global vector and the word vector of each word in the text.

[0033] According to another aspect of this disclosure, an electronic device is also provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform any of the above-described text classification methods by executing the executable instructions.

[0034] According to another aspect of this disclosure, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements any of the above-described text classification methods.

[0035] The text classification method, apparatus, electronic device, and computer-readable storage medium provided in the embodiments of this disclosure first acquire text to be classified, including multiple words. Then, the text to be classified is input into a trained text category prediction model, which outputs the text category, including attribute category and / or domain category. Further, the text category prediction model in the embodiments of this disclosure can vectorize the words in the text to be classified to obtain word vectors for each word. Through the vector processing layer in the text category prediction model, the processed word vectors are used to generate a global vector for the text. The prediction layer in the text category prediction model can predict the attribute category and domain category of the text based on the word vectors of each word in the text and the generated global vector, thereby overcoming the problem in related technologies that ignore the domain information of the document and have a weak ability to model global attribute information. Furthermore, the global vector in the embodiments of this disclosure helps the text category prediction model capture attribute category related information from a global perspective.

[0036] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0037] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0038] Figure 1 This diagram illustrates a text classification method according to an embodiment of the present disclosure.

[0039] Figure 2 This diagram illustrates a method for generating weight information using a dynamic mask attention module in an embodiment of this disclosure.

[0040] Figure 3 This diagram illustrates a method for generating weight information using a part-of-speech awareness attention module in an embodiment of this disclosure.

[0041] Figure 4 A framework diagram of a text category prediction model according to an embodiment of this disclosure is shown;

[0042] Figure 5 This diagram illustrates the processing of a text category prediction model according to an embodiment of the present disclosure.

[0043] Figure 6 This diagram illustrates a text classification device according to an embodiment of the present disclosure.

[0044] Figure 7A diagram of an electronic device according to an embodiment of this disclosure is shown. Detailed Implementation

[0045] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0046] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0047] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0048] Figure 1 This diagram illustrates a text classification method according to an embodiment of the present disclosure, such as... Figure 1 As shown in the embodiments of this disclosure, the text classification method includes the following steps:

[0049] S102, Obtain the text to be classified, wherein the text includes: at least one statement, and each statement includes at least two words.

[0050] In one optional embodiment, the text to be classified in this disclosure may be text consisting of one or more statements, each statement including more than one word. That is, the text in this disclosure may be text consisting of a statement consisting of two words, or text consisting of multiple statements consisting of multiple words.

[0051] S 104, input the text to be classified into the trained text category prediction model, and output the text category, where the text category includes attribute category and / or domain category. The text category prediction model includes a format conversion layer, a vector processing layer, a target information processing layer, and a prediction layer. The format conversion layer is used to convert the text to be classified into text of a predetermined format. The vector processing layer is used to vectorize the text of the predetermined format to obtain the word vector of each word in the text of the predetermined format. The target information processing layer is used to process the word vector of each word in the text of the predetermined format to obtain the global vector of the text. The prediction layer is used to predict the attribute category and domain category of the text based on the global vector and the word vector of each word in the text.

[0052] In one optional embodiment, the text category prediction model in this disclosure specifically includes a format conversion layer, a vector processing layer, a target information processing layer, and a prediction layer. When the text to be classified is input into the text category prediction model, the text is converted into text in a predetermined format at the format conversion layer, the purpose of which is to obtain sentences that conform to the text category prediction model can process. Then, the text in the predetermined format is vectorized by the vector processing layer to obtain the word vector of each word in the text in the predetermined format. Then, the word vector of each word in the text in the predetermined format is processed by the target information processing layer to obtain the global vector of the text. Finally, the global vector is passed through the prediction layer to predict the attribute category and domain category of the text.

[0053] The text classification method in this embodiment first obtains text to be classified, which includes multiple words. Then, the text to be classified is input into a trained text category prediction model, which outputs the text category, including attribute category and / or domain category. Specifically, the text category prediction model in this embodiment can vectorize the words in the text to be classified to obtain word vectors for each word. Through the vector processing layer in the text category prediction model, the processed word vectors are used to generate a global vector for the text. The prediction layer in the text category prediction model can predict the attribute category and domain category of the text based on the word vectors of each word in the text and the generated global vector, thereby overcoming the problem in related technologies that ignore the domain information of the document and have a weak ability to model global attribute information. Furthermore, the global vector in this embodiment helps the text category prediction model capture attribute category related information from a global perspective.

[0054] In one optional embodiment, the format conversion layer converts the statements in the text according to a preset template to obtain statements in a predetermined format.

[0055] Specifically, in this embodiment of the disclosure, a pre-constructed prompt template is used to transform all statements in the obtained text to be classified according to the pre-constructed prompt template to obtain statements that meet the requirements of the prompt template.

[0056] Furthermore, the specific format of the pre-constructed prompt template is as follows: <text>The domain category it belongs to, and the aspect category it belongs to. Among them, <text>This refers to the statements in the obtained text to be classified. `{domain}` and `{aspect category}` are placeholders, representing the domain category and the specific attribute category contained in each statement, respectively. The context of the statements in the obtained text to be classified is S = {w1, w2, ..., w...}. n } and the corresponding domain category c d and attribute category c a The sentences constructed based on the prompt template are w1, w2, ..., w n The field it belongs to is C. d The attribute category is c a .

[0057] In one optional embodiment, this disclosure constructs a text category prediction model based on a pre-constructed prompt template, thereby realizing a zero-shot attribute category extraction method. This not only overcomes the problem of poor zero-shot learning ability of attribute category extraction methods in related technologies, but also effectively unifies attribute category extraction and domain classification in one model through the prompt template. It has domain transfer capability and can better apply the attribute categories and domain-related knowledge learned from the training set to the attribute category extraction of the target domain or the domain with missing labels.

[0058] In one optional embodiment, the vector processing layer uses a word segmentation model to segment the sentences in a predetermined format and vectorizes the words in each sentence to obtain the word vector of each word in the text in the predetermined format, wherein the word vector includes attribute vector and domain vector.

[0059] Furthermore, in the vector processing layer of this embodiment, a word segmentation model is used to first segment the sentences in the text of a predetermined format, and then each word in the text of the predetermined format is vectorized to obtain the word vector of each word in the text of the predetermined format. Specifically, the sentences constructed according to the prompt template are input into the word segmentation model to obtain the word vector of each word in the sentences, wherein... <text>The corresponding word vector for each word in the text to be classified is The word vector for each word corresponding to {domain} is The word vector for each word corresponding to {aspect category} is in, Let i represent the word vector of the i-th word in the sentence. Let $\mathbf{i}$ represent the word vector of the $i$-th word in the domain ${domain}$. This represents the word vector of the i-th word in the attribute category. The word vectors of the domain are then averaged using average pooling to obtain the domain vector h. d The word vectors of the attribute category are average-pooled to obtain the attribute category vector h. a .

[0060] In one optional embodiment, the target information processing layer includes a dynamic masking attention module and a part-of-speech (POS) awareness attention module; wherein, the dynamic masking attention module generates first weight information of the word vector of each word in the text, and the dynamic masking attention module is used to obtain vectors containing attribute information and / or domain information; the POS awareness attention module generates second weight information of the word vector of each word in the text, and the dynamic masking attention module is used to obtain vectors containing attribute information and / or domain information; based on the word vector of each word in the text, the first weight information, and the second weight information, a global vector corresponding to each sentence in the text to be classified is generated.

[0061] Furthermore, the target information processing layer in this embodiment includes a dynamic masking attention module and a part-of-speech awareness attention module. More specifically, the dynamic masking attention module in this embodiment is used to dynamically capture and model domain and attribute category-related word information based on domain and attribute information, while the part-of-speech awareness attention module is used to explicitly capture and model noun and adjective information related to domain and attribute information.

[0062] In one alternative embodiment, such as Figure 2 As shown, the first weight information of the word vector for each word in the text is generated through the dynamic mask attention module, including the following steps:

[0063] S201, Based on the word vector of each word in the text, determine the attribute information and domain information corresponding to the word vector of each word in the text.

[0064] S202, Based on attribute information and domain information, determine the attention coefficient of the semantic association between the domain category and attribute category corresponding to the word vector of each word in the text.

[0065] S203, based on the attention coefficients of the attribute information and domain information corresponding to the word vector of each word in the text, as well as the semantic association between the domain category and the attribute category, generate the first weight information of the word vector of each word in the text.

[0066] In an optional embodiment, the dynamic mask attention module receives the word vector, domain vector and attribute vector of each word in the text in a predetermined format as input, and generates the first weight information of the word vector of each word in the text through the following formulas (1) to (5);

[0067]

[0068]

[0069]

[0070]

[0071]

[0072] in, h represents the word vector of the i-th word in the c-th sentence of the text to be classified. d Represents the neighborhood vector. h represents the semantic relationship between the i-th word in the text to be classified and the d1 domain category; a Represents an attribute vector. This represents the semantic relationship between the i-th word in the text to be classified and the c1 attribute category; This represents the semantic relationship between the i-th word in the text to be classified and the d1 domain category and the c1 attribute category; The attention coefficient represents the semantic association between the word vector of the i-th word and the domain category and attribute category; This represents the first weight information of the i-th word generated using the dynamic mask attention model; -inf represents negative infinity.

[0073] In one alternative embodiment, such as Figure 3 As shown, the second weight information of the word vector of each word in the text is generated through the part-of-speech awareness attention module. The specific process includes the following steps:

[0074] S301, Based on the attribute vector and domain vector corresponding to the word vector of each word in the text, generate the domain category and attribute category association parameters of the word vector.

[0075] S302, based on the part-of-speech of the word corresponding to the word vector, determine the attention coefficient of the semantic association between the domain category and attribute category of the word vector of each word in the text.

[0076] S303, based on the association parameters and attention coefficients corresponding to the word vectors of each word in the text, generate the second weight information of the word vectors of each word in the text.

[0077] Specifically, this embodiment uses a part-of-speech awareness attention module to generate second weight information for the word vector of each word in the text, explicitly modeling attribute category-related words and their part-of-speech information. This overcomes the weakness of attribute category extraction methods in related technologies in modeling attribute category-related words and their part-of-speech information. Many nouns are generally attribute category-related words, and adjectives are generally sentiment words; these two types of words can better reflect the attribute category information being described. Modeling attribute-related words such as nouns and adjectives and their part-of-speech information can further improve the model's ability to understand and model attribute categories.

[0078] In an optional embodiment, the part-of-speech awareness attention module in this disclosure receives the word vector, domain vector and attribute vector of each word in the text in a predetermined format as input, and generates the second weight information of the word vector of each word in the text through the following formulas (6) to (8);

[0079]

[0080]

[0081]

[0082] in, The symbol ";" in the text indicates vector concatenation. W represents the association between the domain category and attribute category of the word vector of the i-th word in the text to be classified; W and b are the learnable parameters of the part-of-speech awareness attention module. The attention coefficient represents the part-of-speech tag of the i-th word in the text to be classified; pos(w i ) is used to determine the part of speech of the word vector corresponding to the i-th word in the text to be classified, where noun represents a noun and adj represents an adjective; This represents the second weight information of the i-th word generated by the part-of-speech awareness attention module.

[0083] In an optional embodiment, a global vector is obtained by aggregating the first weight information output by the dynamic masking attention module and the second weight information output by the part-of-speech awareness attention module. Specifically, the global vector corresponding to the text to be classified is generated by the following formulas (9) to (11):

[0084]

[0085]

[0086]

[0087] in, This represents the first weight information of the i-th word generated by the dynamic mask attention module. This represents the second weight information of the i-th word generated by the part-of-speech awareness attention module, where n is the number of words in the text to be classified. Let i represent the word vector of the i-th word in the sentence. It is the global vector of each sentence in the text to be classified.

[0088] Furthermore, this embodiment generates a global vector for each sentence in the text through a target information processing layer. This explicitly models the document's domain information and global attribute information, overcoming the shortcomings of related technologies in attribute category extraction methods, which neglect document domain information and have weak capabilities in modeling global attribute information. Document domain information plays a crucial role in attribute category identification; different domains have different attribute categories, and attribute category information can also aid in determining the document's domain. In addition, this embodiment helps text category prediction models capture attribute category-related information from a global perspective.

[0089] In an optional embodiment, the prediction layer in this disclosure mainly consists of a multilayer perceptron, which outputs the domain category and related attribute category of each sentence in the text to be classified, the words corresponding to the mask in each sentence of the text to be classified, and part-of-speech information. Specifically, this disclosure outputs the text category of the text through the following formulas (12) to (15):

[0090]

[0091]

[0092]

[0093]

[0094] in, h is the global vector of each sentence in the text to be classified. d h represents the neighborhood vector. a Represents an attribute vector. Let y represent the word vector of the i-th word in the sentence. d It is the probability distribution of domain categories, y a It is the probability distribution of attribute categories. This is the word probability distribution of the i-th word. Let be the part-of-speech distribution of the i-th word, and W1, W2, W3, W4, b1, b2, b3, b4 are the learnable parameters of the model.

[0095] In an optional embodiment, before inputting the text to be classified into a trained text category prediction model and outputting the text category of the text to be classified, the method further includes: training a pre-built deep learning model to obtain a text category prediction model.

[0096] In one optional embodiment, training a pre-built deep learning model to obtain a text category prediction model includes: generating a target loss function using the following formula;

[0097]

[0098]

[0099]

[0100]

[0101]

[0102]

[0103] L = L acl +λ(L dcl +L m )+L pos (twenty two)

[0104] Where N represents the number of training samples in a batch, and n represents the length of the statement. This represents the word probability distribution predicted by the pre-built deep learning model for the i-th word in the j-th training data. L represents the true word distribution probability of the i-th word in the j-th training data. m This represents the mask word prediction loss function; This represents the part-of-speech distribution of the i-th word in the j-th training data predicted by a pre-built deep learning model. L represents the true part-of-speech probability of the i-th word in the j-th training data. pos denoted as the loss function for predicting noun and adjective part-of-speech; k represents the number of samples in the i-th domain; τ and λ represent the hyperparameters of the pre-built deep learning model. This represents the global attribute vector for the i-th statement. This represents the domain vector of the i-th statement; This represents the global attribute vector of the j-th statement, which shares the same domain as the i-th statement. L represents the domain vector of the j-th statement that shares the same domain as the i-th statement; dcl This represents the domain-specific self-supervised contrastive loss function. L represents the attribute category vector in the j-th statement that has the same attribute category as the i-th statement; acl This represents the attribute category self-supervised contrastive loss function; by adjusting the parameters of the pre-built deep learning model through the target loss function, a trained text category prediction model is obtained.

[0105] Furthermore, this disclosure innovatively applies supervised contrastive learning to zero-shot attribute category extraction, overcoming the problem that related technologies have not effectively explored supervised contrastive learning in attribute category extraction methods. Through supervised contrastive learning, text category prediction models can better capture the implicit attribute category information in data in zero-shot scenarios, thereby improving the expressive power of the text category prediction model.

[0106] In one optional embodiment, during training of a pre-built deep learning model, the probabilities of masked words in the word segmentation model are modified, increasing the masking probabilities of nouns and adjectives to 50% and 30%, respectively, while keeping the masking probabilities of other parts of speech unchanged. Nouns are generally attribute-category related words, and adjectives are also attribute-category and domain-category related words.

[0107] Specifically, in this embodiment, the noun and adjective part-of-speech prediction loss function is used to enhance the model's ability to perceive the parts of speech of nouns and adjectives, and to explicitly obtain the part-of-speech information of nouns and adjectives; the domain category self-supervised contrastive loss function is used to bring data with consistent domain categories closer together, while data with inconsistent attribute categories are further apart; the attribute category self-supervised contrastive loss function brings data with consistent attribute categories closer together, while data with inconsistent attribute categories are further apart.

[0108] Furthermore, the framework diagram of the text category prediction model in this embodiment is as follows: Figure 4 As shown, the text processing procedure to be classified is as follows: Figure 5 As shown, the text category prediction model includes a format conversion layer, a vector processing layer, a target information processing layer, and a prediction layer. The format conversion layer converts all sentences in the obtained text to be classified according to a pre-constructed prompt template to obtain sentences in a predetermined format. The vector processing layer uses a word segmentation model to segment and vectorize the sentences in the predetermined format to obtain word vectors for each word in the text in the predetermined format. The target information processing layer includes a dynamic mask attention module and a part-of-speech awareness attention module. Through the dynamic mask attention module and the part-of-speech awareness attention module, the weight information of the word vectors of each word in the text in the predetermined format is obtained to obtain the global vector corresponding to each sentence in the text to be classified. The prediction layer outputs the domain category and attribute category of each sentence in the text to be classified based on the global vector and the word vectors of each word in the text in the predetermined format.

[0109] Based on the same inventive concept, this disclosure also provides a text classification device, as shown in the following embodiments. Since the principle by which this device solves the problem is similar to that of the method embodiments described above, the implementation of this device embodiment can refer to the implementation of the method embodiments described above, and repeated details will not be elaborated further.

[0110] Figure 6 This diagram illustrates a text classification device according to an embodiment of the present disclosure, such as... Figure 6 As shown, the device includes:

[0111] The text acquisition module 601 is used to acquire text to be classified, wherein the text includes at least one statement, and each statement includes at least two words.

[0112] The text classification module 602 is used to input the text to be classified into a trained text category prediction model and output the text category. The text category includes attribute category and / or domain category. The text category prediction model includes a format conversion layer, a vector processing layer, a target information processing layer, and a prediction layer. The format conversion layer is used to convert the text to be classified into text of a predetermined format. The vector processing layer is used to vectorize the text of the predetermined format to obtain the word vector of each word in the text of the predetermined format. The target information processing layer is used to process the word vector of each word in the text of the predetermined format to obtain the global vector of the text. The prediction layer is used to predict the attribute category and domain category of the text based on the global vector and the word vector of each word in the text.

[0113] In one optional embodiment, the format conversion layer in the device converts the statements in the text according to a preset template to obtain statements in a predetermined format.

[0114] In one optional embodiment, the vector processing layer uses a word segmentation model to segment the sentences in a predetermined format and vectorizes the words in each sentence to obtain the word vector of each word in the text in the predetermined format, wherein the word vector includes attribute vector and domain vector.

[0115] In one optional embodiment, the target information processing layer includes a dynamic masking attention module and a part-of-speech (POS) awareness attention module; wherein, the dynamic masking attention module generates first weight information of the word vector of each word in the text, and the dynamic masking attention module is used to obtain vectors containing attribute information and / or domain information; the POS awareness attention module generates second weight information of the word vector of each word in the text, and the dynamic masking attention module is used to obtain vectors containing attribute information and / or domain information; based on the word vector of each word in the text, the first weight information, and the second weight information, a global vector corresponding to each sentence in the text to be classified is generated.

[0116] In one optional embodiment, a first weight information for the word vector of each word in the text is generated by a dynamic mask attention module. The device is further configured to determine the attribute information and domain information corresponding to the word vector of each word in the text based on the word vector of each word in the text; determine the attention coefficient of the semantic association between the domain category and the attribute category corresponding to the word vector of each word in the text based on the attribute information and the domain information; and generate the first weight information for the word vector of each word in the text based on the attribute information and domain information corresponding to the word vector of each word in the text, as well as the attention coefficient of the semantic association between the domain category and the attribute category.

[0117] In one optional embodiment, the first weight information of the word vector of each word in the text is generated by the above formulas (1) to (5);

[0118] In one optional embodiment, a second weight information for the word vector of each word in the text is generated by a dynamic mask attention module. The device is further configured to generate association parameters of the domain category and attribute category of the word vector based on the attribute vector and domain vector corresponding to the word vector of each word in the text; determine the attention coefficient of the semantic association relationship between the domain category and attribute category of the word vector of each word in the text based on the part-of-speech tag of the word vector; and generate the second weight information for the word vector of each word in the text based on the association parameters and attention coefficients corresponding to the word vector of each word in the text.

[0119] In an optional embodiment, second weight information of the word vector of each word in the text is generated by the above formulas (6) to (8);

[0120] In one optional embodiment, before inputting the text to be classified into the trained text category prediction model and outputting the text category of the text to be classified, the device further includes: a model training module for training a pre-built deep learning model to obtain a text category prediction model.

[0121] In an optional embodiment, the model training module is also used to generate the target loss function using the above formulas (16) to (22).

[0122] Those skilled in the art will understand that various aspects of this disclosure can be implemented as a system, method, or program product. Therefore, various aspects of this disclosure can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."

[0123] The following reference Figure 7 To describe an electronic device 70 according to such an embodiment of the present disclosure. Figure 7 The electronic device 700 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0124] like Figure 7 As shown, the electronic device 700 is manifested in the form of a general-purpose computing device. The components of the electronic device 700 may include, but are not limited to: at least one processing unit 701, at least one storage unit 702, and a bus 703 connecting different system components (including storage unit 702 and processing unit 701).

[0125] The storage unit stores program code, which can be executed by the processing unit 701, causing the processing unit 701 to perform the steps described in the "Exemplary Methods" section above according to various exemplary embodiments of this disclosure.

[0126] In some embodiments, when an electronic device is used to control, for example, the text classification method described above, the processing unit 701 may execute the following steps of the method embodiments described above:

[0127] Obtain the text to be classified, wherein the text includes at least one statement, and each statement includes at least two words;

[0128] The text to be classified is input into a trained text category prediction model, which outputs the text category. The text category includes attribute category and / or domain category. The text category prediction model includes a format conversion layer, a vector processing layer, a target information processing layer, and a prediction layer. The format conversion layer is used to convert the text to be classified into a predefined format. The vector processing layer is used to vectorize the predefined format text to obtain the word vector of each word in the predefined format text. The target information processing layer is used to process the word vector of each word in the predefined format text to obtain the global vector of the text. The prediction layer is used to predict the attribute category and domain category of the text based on the global vector and the word vector of each word in the text.

[0129] Storage unit 702 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 7021 and / or a cache memory unit 7022, and may further include a read-only memory unit (ROM) 7023.

[0130] Storage unit 702 may also include a program / utility 7024 having a set (at least one) program module 7025, such program module 7025 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0131] Bus 703 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0132] Electronic device 700 can also communicate with one or more external devices 704 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 700, and / or with any device that enables electronic device 700 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 705. Furthermore, electronic device 700 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 706. As shown, network adapter 706 communicates with other modules of electronic device 700 via bus 703. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 700, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0133] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0134] In particular, according to embodiments of this disclosure, the process described above with reference to the flowchart can be implemented as a computer program product, which includes a computer program that, when executed by a processor, implements the above-described text classification method.

[0135] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, which may be a readable signal medium or a readable storage medium. A program product capable of implementing the methods described above is stored thereon. In some possible implementations, various aspects of this disclosure may also be implemented as a program product including program code, which, when run on a terminal device, causes the terminal device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure.

[0136] More specific examples of computer-readable storage media in this disclosure may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0137] In this disclosure, a computer-readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of transmitting, propagating, or transmitting a program for use by or in connection with an instruction execution system, apparatus, or device.

[0138] Optionally, the program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0139] In practical implementation, program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0140] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0141] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0142] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0143] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.< / text> < / text> < / text>

Claims

1. A method of text classification, characterized by, The method comprises: acquiring a text to be classified, wherein the text comprises at least one sentence, and each sentence comprises at least two words; inputting the text to be classified into a trained text category prediction model to output a text category of the text, wherein the text category comprises an attribute category and / or a domain category, the text category prediction model comprises a format conversion layer, a vector processing layer, a target information processing layer, and a prediction layer, the format conversion layer is used to convert the text to be classified into a text in a predetermined format, the vector processing layer is used to perform vectorization processing on the text in the predetermined format to obtain a word vector of each word in the text in the predetermined format, the target information processing layer is used to process the word vector of each word in the text in the predetermined format to obtain a global vector of the text, and the prediction layer is used to predict the attribute category and the domain category of the text according to the global vector and the word vector of each word in the text; the target information processing layer comprises a dynamic mask attention module and a part-of-speech perception attention module; wherein the dynamic mask attention module is used to obtain a first weight information of the word vector of each word in the text, and the dynamic mask attention module is used to obtain a vector containing attribute information and / or domain information; the part-of-speech perception attention module is used to obtain a second weight information of the word vector of each word in the text, and the part-of-speech perception attention module is used to obtain a vector containing noun and adjective information related to the attribute information and / or the domain information; and the global vector corresponding to each sentence of the text to be classified is generated according to the word vector of each word in the text, the first weight information, and the second weight information.

2. The text classification method of claim 1, wherein, The format conversion layer converts the sentences in the text according to a preset template to obtain sentences in a predetermined format.

3. The text classification method of claim 1, wherein, The vector processing layer performs word segmentation processing on the sentences in the predetermined format by using a word segmentation model, and performs vectorization processing on the words in each sentence to obtain a word vector of each word in the text in the predetermined format, wherein the word vector comprises an attribute vector and a domain vector.

4. The text classification method of claim 1, wherein, The first weight information of the word vector of each word in the text is generated by the dynamic mask attention module, comprising: determining attribute information and domain information corresponding to the word vector of each word in the text according to the word vector of each word in the text; determining an attention coefficient of a semantic association relationship between the domain category and the attribute category corresponding to the word vector of each word in the text according to the attribute information and the domain information; generating the first weight information of the word vector of each word in the text according to the attribute information and the domain information corresponding to the word vector of each word in the text and the attention coefficient of the semantic association relationship between the domain category and the attribute category.

5. The text classification method of claim 4, wherein, The first weight information of the word vector of each word in the text is generated by the following formula: wherein, represents a word vector of the i-th word in the c-th sentence of the text to be classified, represents a domain vector, represents a semantic relationship of the i-th word in the text to be classified and the d1 domain category; represents an attribute vector, represents a semantic relationship of the i-th word in the text to be classified and the c1 attribute category; represents a semantic relationship of the i-th word in the text to be classified, the d1 domain category, and the c1 attribute category; represents an attention coefficient obtained by the semantic association relationship between the word vector of the i-th word and the domain category and the attribute category; represents the first weight information of the i-th word generated by the dynamic mask attention model; -inf represents negative infinity.

6. The text classification method of claim 1, wherein, The second weight information of the word vector of each word in the text is generated by the part-of-speech perception attention module, comprising: generating an association parameter of the domain category and the attribute category of the word vector according to the attribute vector and the domain vector corresponding to the word vector of each word in the text; and determine, according to the part-of-speech of the word corresponding to the word vector, an attention coefficient of a semantic association relationship between a domain category and an attribute category of the word vector of each word in the text; generate second weight information of the word vector of each word in the text according to the associated parameter and the attention coefficient corresponding to the word vector of each word in the text.

7. The text classification method of claim 6, wherein, The second weight information of the word vector of each word in the text is generated by the following formula: wherein, represents vector splicing; represents vector splicing; represents the association relationship between the domain category and the attribute category of the word vector of the i-th word in the text to be classified; W and b are learnable parameters of the part-of-speech-aware attention module; represents the attention coefficient of the part-of-speech of the i-th word in the text to be classified; is used to judge the part-of-speech of the word corresponding to the word vector of the i-th word in the text to be classified, noun represents noun, and adj represents adjective; represents the second weight information of the i-th word generated by the part-of-speech-aware attention module.

8. The text classification method of claim 1, wherein, Before inputting the text to be classified into the trained text category prediction model and outputting the text category of the text to be classified, the method further comprises: training a pre-constructed deep learning model to obtain the text category prediction model.

9. The text classification method of claim 1, wherein, Training a pre-constructed deep learning model to obtain the text category prediction model comprises: The target loss function is generated by the following formula: wherein N represents the number of batch training samples, n represents the length of the sentence, represents the word probability distribution of the i-th word in the j-th training data predicted by the pre-constructed deep learning model, represents the true word distribution probability of the i-th word in the j-th training data, represents the mask word prediction loss function; represents the part-of-speech distribution of the i-th word in the j-th training data predicted by the pre-constructed deep learning model, represents the true part-of-speech distribution probability of the i-th word in the j-th training data, represents the noun and adjective part-of-speech prediction loss function; k represents the number of i-th same field samples, and represents the hyperparameters of the pre-constructed deep learning model; represents the global attribute vector of the i-th sentence, represents the domain vector of the i-th sentence; represents the global attribute vector of the j-th sentence with the same domain as the i-th sentence, represents the domain vector of the j-th sentence with the same domain as the i-th sentence; represents the domain category self-supervised contrast loss function; represents the attribute category vector in the j-th sentence with the same attribute category as the i-th sentence; represents the attribute category self-supervised contrast loss function; The parameters of the pre-constructed deep learning model are adjusted by the target loss function to obtain the trained text category prediction model.

10. A text classification apparatus characterized by comprising: The device comprises: A text acquisition module for acquiring a text to be classified, wherein the text comprises at least one sentence, and each sentence comprises at least two words; A text classification module for inputting the text to be classified into a trained text category prediction model and outputting the text category of the text, wherein the text category comprises an attribute category and / or a domain category, the text category prediction model comprises a format conversion layer, a vector processing layer, a target information processing layer, and a prediction layer, the format conversion layer is configured to convert the text to be classified into a text in a predetermined format, the vector processing layer is configured to perform vectorization processing on the text in the predetermined format to obtain a word vector of each word in the text in the predetermined format, the target information processing layer is configured to process the word vector of each word in the text in the predetermined format to obtain a global vector of the text, and the prediction layer is configured to predict the attribute category and the domain category of the text according to the global vector and the word vector of each word in the text. The target information processing layer comprises a dynamic mask attention module and a part-of-speech perception attention module, wherein the dynamic mask attention module is configured to obtain a vector containing attribute information and / or domain information to generate first weight information of the word vector of each word in the text, the part-of-speech perception attention module is configured to obtain a vector containing noun and adjective information related to the attribute information and / or the domain information to generate second weight information of the word vector of each word in the text, and the global vector corresponding to each sentence of the text to be classified is generated according to the word vector of each word in the text, the first weight information, and the second weight information.

11. An electronic device, comprising: comprises: a processor; and a memory for storing executable instructions of the processor; The processor is configured to execute the text classification method of any one of claims 1-9 by executing the executable instructions.

12. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the text classification method of any one of claims 1-9.

Images