Text classification method and device, model processing method and device, equipment and medium

Abstract

The invention provides a text classification method and device, a model processing method and device, equipment and a medium, and can be applied to the technical field of artificial intelligence. The text classification method comprises the steps that a to-be-processed text and label set information are received, the label set information comprises multiple labels and relation information, and the relation information comprises the mutual exclusion relation among the multiple labels; utilizing a neural network model to process the features of the to-be-processed text according to the features used for representing the mutual exclusion relationship among the plurality of tags, and determining the matching degree between each of the plurality of tags and the to-be-processed text, the matching degree between the to-be-processed text and the first label and the matching degree between the to-be-processed text and the second label are in negative correlation, and the first label and the second label are two mutually exclusive labels in the multiple labels; according to the matching degrees between the multiple labels and the to-be-processed text, matching labels of the to-be-processed text are generated, and the matching labels comprise the labels, of which the matching degrees with the to-be-processed text are larger than a preset threshold value, in the multiple labels.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, specifically to a text classification method, model processing method, apparatus, device, and medium. Background Technology

[0002] Multi-label text classification is a task in natural language processing that aims to assign one or more relevant category labels to a given text, thereby classifying the text according to these category labels. Unlike traditional single-label classification, which generates a unique label for each text, multi-label classification allows multiple labels to be generated for a single text.

[0003] In related technologies, there are instances where multiple labels generated for a single text data contradict each other, resulting in poor text classification performance. Summary of the Invention

[0004] In view of the above problems, this application provides a text classification method, a model processing method, an apparatus, a device, and a medium.

[0005] According to a first aspect of this application, a text classification method is provided, comprising: receiving a text to be processed and a tag set information, the tag set information including multiple tags and relationship information, the relationship information including mutually exclusive relationships between the multiple tags; using a neural network model, processing features of the text to be processed based on features including those used to characterize the mutually exclusive relationships between the multiple tags, determining the matching degree between each of the multiple tags and the text to be processed, wherein the matching degree between the text to be processed and a first tag and the matching degree between the text to be processed and a second tag are negatively correlated, the first tag and the second tag being two mutually exclusive tags among the multiple tags; generating matching tags for the text to be processed based on the matching degree between each of the multiple tags and the text to be processed, the matching tags including tags among the multiple tags whose matching degree with the text to be processed is greater than a preset threshold.

[0006] According to embodiments of this application, the relationship information also includes co-occurrence relationships among multiple tags, where co-occurrence relationships indicate that the frequency of two tags that are simultaneously text meets a preset requirement. Using a neural network model, based on features including those characterizing mutual exclusion relationships among multiple tags, the feature information of the text to be processed is processed to determine the matching degree between each of the multiple tags and the text to be processed. This includes: constructing a co-occurrence relationship matrix and a mutual exclusion relationship matrix based on co-occurrence and mutual exclusion relationships; performing at least one feature fusion operation on the co-occurrence relationship matrix and the mutual exclusion relationship matrix respectively to obtain fused relationship features among multiple tags; and generating the matching degree between each of the multiple tags and the text to be processed based on the fused relationship features among multiple tags and the features of the text to be processed.

[0007] According to an embodiment of this application, based on the fusion relationship features between multiple tags and the features of the text to be processed, a matching degree between each of the multiple tags and the text to be processed is generated, including: performing a first linear transformation and a second linear transformation on the features of the text to be processed to obtain the key features and value features of the text to be processed; processing the fusion relationship features, key features and value features based on a neural network layer with a multi-head cross-attention mechanism to obtain output features including text feature information related to each of the multiple tags; performing linear classification on the output features and standardizing them using a preset activation function to obtain the matching degree between each of the multiple tags and the text to be processed.

[0008] According to embodiments of this application, at least one feature fusion operation is performed on the co-occurrence relation matrix and the mutual exclusion relation matrix to obtain fused relation features between multiple labels, including: determining the original relation features, reverse relation features, and self-loop relation features of the co-occurrence relation features and the mutual exclusion relation features respectively; performing weighted summation and feature extraction on the co-occurrence relation features and the mutual exclusion relation features according to the weight information corresponding to the original relation features, reverse relation features, and self-loop relation features respectively to obtain intermediate fused relation features; and performing a preset number of weighted summation and feature extraction operations on the intermediate fused relation features according to the weight information corresponding to the original relation features, reverse relation features, and self-loop relation features respectively to obtain fused relation features.

[0009] According to embodiments of this application, the method further includes: using a multi-head self-attention encoder to obtain features of the text to be processed, the features including contextual information.

[0010] According to a second aspect of this application, a model processing method is provided, comprising: obtaining training text for a neural network model, the training text including a text to be processed, a set of label information including multiple labels and relational information, and the matching degree between the text to be processed and each of the multiple labels and the text to be processed, the relational information including mutual exclusion relationships between the multiple labels; using the neural network model, generating a predicted matching degree between each of the multiple labels and the text to be processed based on the mutual exclusion relationships between the text to be processed and the multiple labels in the training text; using a median distributional balance loss function to determine the predicted matching degree and the loss information between the matching degree; and adjusting the parameters of the neural network model based on the loss information.

[0011] A third aspect of this application provides a model processing apparatus, comprising: a receiving module for receiving a text to be processed and a set of tags, the set of tags including multiple tags and relational information, the relational information including mutually exclusive relationships between the multiple tags; a classification module for processing features of the text to be processed using a neural network model, based on features including those characterizing the mutually exclusive relationships between the multiple tags, to determine the matching degree between each of the multiple tags and the text to be processed, wherein the matching degree between the text to be processed and a first tag and the matching degree between the text to be processed and a second tag are negatively correlated, the first tag and the second tag being two mutually exclusive tags among the multiple tags; and a first generation module for generating matching tags for the text to be processed based on the matching degree between each of the multiple tags and the text to be processed, the matching tags including tags among the multiple tags whose matching degree with the text to be processed is greater than a preset threshold.

[0012] A fourth aspect of this application provides a model processing apparatus, comprising: an acquisition module for acquiring training text of a neural network model, the training text including a text to be processed, label set information including multiple labels and relational information of the multiple labels, and matching degrees between the text to be processed and each of the multiple labels and the text to be processed, the relational information including mutual exclusion relationships between the multiple labels; a second generation module for generating predicted matching degrees between each of the multiple labels and the text to be processed using the neural network model, based on the mutual exclusion relationships between the text to be processed and the multiple labels in the training text; a loss calculation module for determining the predicted matching degree and the loss information between the matching degree using a median distribution balance loss function; and an adjustment module for adjusting the parameters of the neural network model based on the loss information.

[0013] A fifth aspect of this application provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0014] A sixth aspect of this application also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0015] A seventh aspect of this application also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method. Attached Figure Description

[0016] The above-mentioned contents, other objects, features and advantages of this application will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0017] Figure 1The illustrations depict application scenarios of text classification methods, model processing methods, apparatuses, devices, media, and products according to embodiments of this application.

[0018] Figure 2 A flowchart illustrating a text classification method according to an embodiment of this application is shown schematically;

[0019] Figure 3 A schematic diagram of a neural network model according to an embodiment of this application is shown.

[0020] Figure 4 A flowchart illustrating a model processing method according to an embodiment of this application is shown schematically.

[0021] Figure 5 This schematic diagram illustrates a structural block diagram of a text classification apparatus according to an embodiment of the present application;

[0022] Figure 6 This schematic diagram illustrates a structural block diagram of a model processing apparatus according to an embodiment of the present application;

[0023] Figure 7 A block diagram schematically illustrates an electronic device suitable for implementing text classification and model processing methods according to embodiments of this application. Detailed Implementation

[0024] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0025] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0026] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0027] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0028] Among the related technologies, the main methods are based on machine learning and deep learning to generate multiple matching tags for text.

[0029] Machine learning methods primarily transform multi-label problems into a series of single-label sub-problems, which are then solved independently using single-label algorithms. Deep learning-based methods utilize deep learning models to extract richer feature representations from text through multi-level nonlinear transformations and end-to-end learning, resulting in greater accuracy and generalization ability. For example, the Head-to-Tail Network (HTTN) multi-label classification model transfers meta-knowledge learned from a corpus-rich head label classifier to a corpus-poor tail label classifier. Meta-knowledge refers to the mapping from parameters of a few-text network to parameters of a many-text network, aiming to improve the generalization of the tail label classifier.

[0030] However, the related technologies do not delve into the relationships between tags, resulting in low correlation or contradictions among the multiple tags generated for the text, leading to low quality of the generated tags.

[0031] Based on this, embodiments of this application provide a text classification method, which sets multiple preset tags and relationship information between the tags in the tag set information used for classifying the text to be processed and the text classification, so that when using the tag set information to classify the text to be processed, the relationship information between the multiple tags is taken into account, and multiple tags matching the text to be processed are generated, avoiding the situation where the multiple tags generated for the text to be processed do not contradict each other.

[0032] It should be noted that the text classification method, apparatus, electronic device, and storage medium specified in this disclosure can be used in the fields of artificial intelligence and fintech, and also in various other fields besides artificial intelligence and fintech. The application areas of the text classification method, apparatus, electronic device, and storage medium provided in the embodiments of this disclosure are not limited.

[0033] In the technical solution disclosed herein, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, and displayed data) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with relevant laws, regulations, and standards, necessary confidentiality measures have been taken, and they do not violate public order and good morals. Corresponding operation entry points are provided for users to choose to authorize or refuse.

[0034] In scenarios involving automated decision-making using personal information, the text classification method, model processing method, apparatus, device, and system provided in this disclosure all offer users corresponding operation entry points, allowing them to choose to agree to or reject the automated decision result; if the user chooses to reject, the process proceeds to the expert decision-making stage. Here, "automated decision-making" refers to the activity of automatically analyzing and evaluating an individual's behavioral habits, interests, or economic, health, and credit status through computer programs, and then making a decision. Here, "expert decision-making" refers to the activity of making decisions by personnel who specialize in a particular field, possess specialized experience, knowledge, and skills, and have reached a certain level of professional expertise.

[0035] Figure 1 The illustrations depict application scenarios of text classification methods, model processing methods, apparatuses, devices, media, and products according to embodiments of this application.

[0036] like Figure 1 As shown, application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0037] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0038] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0039] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0040] It should be noted that the text classification method and model processing method provided in this application embodiment can generally be executed by server 105. Correspondingly, the text classification device and model processing device provided in this application embodiment can generally be located in server 105. The text classification method and model processing method provided in this application embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the text classification device and model processing device provided in this application embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0041] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0042] The following will be based on Figure 1 The described scene, through Figures 2-3 A text classification method according to embodiments of this application will be described in detail.

[0043] Figure 2 A flowchart illustrating a text classification method according to an embodiment of this application is shown schematically.

[0044] like Figure 2 As shown, the text classification method of this embodiment includes operations S210 to S230.

[0045] In operation S210, the text to be processed and tag set information are received. The tag set information includes multiple tags and relationship information, which includes the mutual exclusion relationship between multiple tags.

[0046] The tag set information includes multiple tags that are candidate tags for the text to be processed, and the relationship information of multiple tags can be represented by a form, a relationship graph, or a relationship matrix.

[0047] The mutual exclusion relationship between multiple labels can be determined based on text data that uses multiple labels to complete text classification. For example, if the multiple labels include label A and label B, we can count the first probability that label B exists in the text where label A exists, and the second probability that label B exists in the text where labels A exists. By comparing the first probability and the second probability, if the first probability and the second probability are not equal, it is determined that there is a relationship between label A and label B. If the first probability is less than the second probability, it is determined that there is a mutual exclusion relationship between label A and label B.

[0048] Furthermore, the mutual exclusion relationships between multiple tags can also be determined based on expert experience.

[0049] In operation S220, a neural network model is used to process the features of the text to be processed, including features used to characterize the mutual exclusion relationship between multiple labels, to determine the matching degree between each of the multiple labels and the text to be processed.

[0050] Neural network models can be obtained through pre-training. The input of a neural network model can include the text to be processed and information about the set of labels. The output of a neural network model can be the matching degree between each label and the text to be processed.

[0051] The degree of matching between each tag and the text to be processed can represent the probability that the text to be processed belongs to each tag.

[0052] In the matching degree between each generated tag and the text to be processed, if the first tag and the second tag among multiple tags have a mutual exclusion relationship, then the matching degree between the text to be processed and the first tag and the matching degree between the text to be processed and the second tag are negatively correlated. The higher the matching degree between the first tag and the text to be processed, the lower the matching degree between the second tag and the text to be processed.

[0053] In operation S230, matching tags for the text to be processed are generated based on the matching degree between each of the multiple tags and the text to be processed. The matching tags include tags among the multiple tags whose matching degree with the text to be processed is greater than a preset threshold.

[0054] By setting a negative correlation between the matching degree of mutually exclusive tags and the text to be processed among multiple tags, and selecting tags with a matching degree greater than that of the text to be processed as the matching tags of the text to be processed during the text classification process, the matching tags of the text to be processed will not include two tags with a mutual exclusive relationship at the same time. This avoids the existence of contradictory tags among the multiple tags generated for the text to be processed, and improves the generation effect of multiple tags for the text to be processed.

[0055] Furthermore, the relationship information can include co-occurrence relationships between multiple tags. Co-occurrence relationships can indicate that the frequency of two tags being text tags meets a preset requirement, or that the probability of two tags appearing simultaneously is greater than a preset threshold. For example, if the third and fourth tags among multiple tags have a co-occurrence relationship, then if the third tag exists among the multiple tags of the text, the probability that the fourth tag also exists among the multiple tags of the text is greater than a preset probability threshold.

[0056] It is understandable that if two tags in a set of multiple tags are mutually exclusive, then these two tags must not be co-occurring.

[0057] The matching degree between multiple tags and the text to be processed can be determined based on the mutual exclusion and co-occurrence relationships between them.

[0058] According to embodiments of this application, a neural network model is used to process feature information of the text to be processed based on features including those characterizing mutual exclusion relationships between multiple tags, to determine the matching degree between each tag and the text to be processed. This includes: constructing a co-occurrence relation matrix and a mutual exclusion relation matrix based on co-occurrence and mutual exclusion relationships. Then, at least one feature fusion operation is performed on the co-occurrence relation matrix and the mutual exclusion relation matrix respectively to obtain fused relation features between the multiple tags. Finally, based on the fused relation features between the multiple tags and the features of the text to be processed, the matching degree between each tag and the text to be processed is generated.

[0059] If the tag set information includes n tags, then the co-occurrence relation matrix or the mutual exclusion relation matrix can be set as follows: The matrix M, where, This indicates whether there is a co-occurrence relationship or a mutual exclusion relationship between the i-th label and the j-th label.

[0060] Convolutional neural networks can be used to extract features at least once from the co-occurrence relation matrix and the mutual exclusion relation matrix, respectively. Alternatively, graph neural networks can be used to extract features at least once from the images containing the co-occurrence relation matrix and the images containing the mutual exclusion relation matrix, respectively. This yields co-occurrence relation features and mutual exclusion relation features.

[0061] By overlaying co-occurrence and mutual exclusion features through channels, a fused relationship feature that includes both co-occurrence and mutual exclusion features can be obtained.

[0062] Alternatively, a weighted summation of co-occurrence and mutual exclusion features can be performed to obtain fusion features.

[0063] By mapping the co-occurrence and mutual exclusion relationships between multiple tags into co-occurrence and mutual exclusion matrices, and performing feature extraction on the co-occurrence and mutual exclusion matrices at least once, the extracted co-occurrence and mutual exclusion features are fused to obtain fused relationship features. The fused relationship features are then used to generate the matching degree between each tag and the features of the text to be processed. This achieves the determination of the correlation between multiple tags and the text to be processed based on the mutual exclusion and co-occurrence relationships between multiple tags, making the matching degree of tags with co-occurrence relationships similar to that of the text to be processed, and making the matching degree of tags with mutual exclusion relationships different from that of the text to be processed.

[0064] Co-occurrence and mutual exclusion features can be represented as feature matrices. These features can be divided along the main diagonal, and different weights can be used to fuse the feature information in different regions of the feature matrix.

[0065] It's understandable that co-occurrence and mutual exclusion relationships can be unidirectional. For example, if text contains tag A, there's a high probability that tag B also exists; however, if text contains tag B, there's a lower probability that tag A also exists. This can be understood as the co-occurrence of tag A and tag B differing from the co-occurrence of tag B and tag A.

[0066] According to embodiments of this application, at least one feature fusion operation is performed on the co-occurrence relation matrix and the mutual exclusion relation matrix to obtain fused relation features between multiple labels, including: determining the original relation features, reverse relation features, and self-loop relation features of the co-occurrence relation features and the mutual exclusion relation features respectively; performing weighted summation and feature extraction on the co-occurrence relation features and the mutual exclusion relation features according to the weight information corresponding to the original relation features, reverse relation features, and self-loop relation features respectively to obtain intermediate fused relation features; and performing a preset number of weighted summation and feature extraction operations on the intermediate fused relation features according to the weight information corresponding to the original relation features, reverse relation features, and self-loop relation features respectively to obtain fused relation features.

[0067] In addition, a neural network layer including a multi-head cross-attention mechanism can be set in the neural network model. The multi-head cross-attention mechanism can be used to process the fusion relationship features and the features of the text to be processed to obtain the matching degree between each label and the text to be processed.

[0068] According to an embodiment of this application, based on the fusion relationship features between multiple tags, a matching degree between each tag and the text to be processed is generated, including: performing a first linear transformation and a second linear transformation on the features of the text to be processed to obtain key features and value features of the text to be processed; processing the fusion relationship features, key features, and value features based on a neural network layer with a multi-head cross-attention mechanism to obtain output features including text feature information related to each of the multiple tags; performing linear classification on the output features and standardizing them using a preset activation function to obtain the matching degree between each tag and the text to be processed.

[0069] The fusion relationship features between multiple tags can be used as query features.

[0070] Two different weight matrices can be used to perform two different linear transformations on the features of the text to be processed, resulting in key features and value features. The key features of the text can be used to represent its context index features. The value features can be used to represent the specific content or essential information of the text's features. There is a correspondence between the key features and the value features.

[0071] The neural network layer using the multi-head cross-attention mechanism performs similarity matching based on query features and key features to determine the context index features that match each of the multiple labels represented by the query features. The value features corresponding to the context index features matched by each label are aggregated to obtain text feature information including the value features matched by each label. The text feature information of the features matched by each label is superimposed to obtain the output features of the neural network layer using the multi-head attention mechanism.

[0072] A fully connected layer can be set in the assignment model to map the output features to multiple scalar values, and a preset activation function can be used to standardize the multiple scalar values ​​to obtain the matching degree between each label and the text to be processed.

[0073] By using a neural network layer with a multi-head attention mechanism to extract features for the labels, the intrinsic relationship between the labels and the text to be processed is explored from multiple levels. This enables the accurate determination of feature information in the text to be processed that matches each label with the contextual features in the text to be processed, and improves the accuracy of obtaining the matching degree between multiple labels and the text to be processed.

[0074] Among them, for the features of the text to be processed, a multi-head self-attention encoder can be used to enhance the contextual features in the text to be processed.

[0075] According to embodiments of this application, the method further includes: using a multi-head self-attention encoder to obtain features of the text to be processed, the features including contextual information.

[0076] Figure 3 A schematic diagram of a neural network model according to an embodiment of this application is shown.

[0077] The following is Figure 3 The illustrative diagram of the neural network model illustrates the process of text classification based on the text to be processed and the set of labels.

[0078] exist Figure 3 In this context, the text to be processed is the user's comment text. First, the label input layer 301 receives labels C1, C2...Cu, as well as the co-occurrence matrix between the labels. and mutual exclusion matrix The system receives comment texts W1, W2, ..., Wn through the comment text input layer 302. The label Cj may include a sequence of label phrases. j is a positive integer less than or equal to u, q is a positive integer greater than 2, and q represents the tag phrase length.

[0079] The word vector representation module 303 includes a pre-trained model based on whole word masking, which performs character-level modeling of text and tags. This pre-trained model can include a semantic embedding module, a sequence embedding module, and a boundary embedding module. The semantic embedding module converts text words into vector form; the sequence embedding module captures the positional information of each character in the text sequence, as changes in positional information may cause subtle changes in the semantic interpretation of words; and the boundary embedding module identifies the different sentence segments in the text. By using the pre-trained model based on whole word masking to process tags and comment text, tag embedding information and text embedding information are obtained, overcoming the limitations of word vector representation methods in related technologies when dealing with the problem of polysemy. Whole word masking (wwm) is a training optimization technique for pre-trained models, used to solve the problem of semantic fragmentation of words that may occur during masking training after word segmentation. In WWWM, if a word is segmented into multiple sub-words by a segmentation standard, when any of the sub-words is selected as the masking target, all the sub-words of that word will be masked together, so as to ensure that the model can better learn and understand the meaning of the complete word during the pre-training stage.

[0080] By transforming comment text and tags into the same vector space using the same pre-trained language model, words with similar semantics will be expressed in a similar space, thus ensuring that the implicit correlation between the meaning of the text and the meaning of the tags is effectively preserved.

[0081] After processing the comment texts W1, W2...Wn, the word vector representation module 303 obtains a set of character-level word vectors. ,in, , The word vectors are obtained by processing the comment text Wk using a pre-trained model, where k is a positive integer less than or equal to n.

[0082] The pre-trained model of the word vector representation module 303 includes a multi-dimensional final hidden layer, which contains weight information. The pre-trained model uses the weights of the final hidden layer as the text representation g of the labeled phrase sequence, where... The coded representation p of the tag is obtained by averaging the character vectors of the tag. p can be determined using the following formula:

[0083]

[0084] in, k is the dimension of the last hidden layer of the pre-trained model.

[0085] By aggregating the encoded representations p of each tag, we obtain the tag embedding information E. lable , .

[0086] For word vector E text The positional encoding layer 305 can be used to utilize the sequential information of the text sequence to input the word vector matrix E. text Add location encoding to inject location information, as shown in the following formula:

[0087]

[0088] in, M is the position embedding matrix, and M is the mask matrix corresponding to the text vector matrix. This represents the XOR operation. The calculation method for the elements in the i-th row, 2j-th column, and 2j+1-th column of matrix P is shown in the following formula.

[0089]

[0090] Where i is a positive integer less than or equal to n; j is a positive integer less than or equal to n. is a positive integer; d is a preset even number used to represent the embedding dimension of the transformer model.

[0091] Textual contextual semantic information is extracted using a multi-head self-attention mechanism layer 306. In the h-head attention mechanism, Q, K, and V are represented as shown in the following formula:

[0092]

[0093] in, Corresponding to the query for each attention, key, value, This represents the dimension of the feature space for single-head attention. , , The calculation method is shown in the following formula:

[0094]

[0095] in, , representing the corresponding model training weight matrix, The attention result is calculated using the following formula:

[0096]

[0097] Considering that the text length varies in each batch, a masking operation is performed on the result of multiplying the Q and K matrices, with padding values ​​set to... The final representation is obtained by connecting the attention results of each head, as shown in the following formula:

[0098]

[0099] in, .

[0100] Then, the residual and normalization layer 307, the bit-by-bit feedforward network 308, and the residual and normalization layer 309 can be used to... The processing is as shown in the following formula:

[0101]

[0102] The feedforward neural network consists of two fully connected layers. The following formula is used to determine:

[0103]

[0104] in, b1 and b2 are both parameters to be trained. .

[0105] Text feature vectors are obtained by stacking e-layer encoder layers. Where e is the number of layers in the pre-built encoder, .

[0106] Label feature vector of each of the 304 layers of the graph convolutional neural network The label feature vectors from the previous layer of the graph convolutional neural network can be used to... The update yielded the following: , Let be the hidden state corresponding to the label feature vector 'a' of the l-th layer of the graph convolutional neural network, where 'a' is a positive integer less than or equal to 'u'. The label features are initialized as follows: The extended formula aggregates the co-occurrence and mutual exclusion relationships between tags, and the tag feature update method is shown in the following formula:

[0107]

[0108] Among them, co-occurrence related space Mutually exclusive associative space Includes primitive relations, inverse relations, and spin relations; functions Used to aggregate tag hidden layer states and relationship hidden layer states. , This represents the corresponding training weight matrix. and These represent the embedding of coexistence relationships and mutual exclusion relationships at the l-th level, respectively. Based on the intuitive understanding of label relationships, the settings are as follows: , and express, , The following formula can be used to determine this:

[0109]

[0110] in Represents the lower trigonometric functions of a matrix. Represents trigonometric functions on a matrix. This represents the function of the main diagonal of the matrix.

[0111] and These represent coexistence embedding and mutual exclusion embedding, respectively, based on the intuitive understanding of label relationships. The relation embedding update method is shown in the following formula:

[0112]

[0113] in, To enable the sharing of weight matrices for features, so that all relation features are shared in the same linear space.

[0114] A combined multi-relation graph convolutional network can be trained using a combination of cross-entropy loss and relation-based label embedding distance. Let the final label node embedding of label i be determined by... The relationship loss function is shown in the following formula.

[0115]

[0116] Among them, co-occurrence related space Mutually exclusive associative space It includes original relations, inverse relations, and spin relations.

[0117] A multi-head cross-attention mechanism layer 310 can be used to extract unique textual semantic features for tags. This mechanism can explore the intrinsic relationship between tags and comment text in detail from multiple perspectives and levels.

[0118] Multi-head attention to obtain the final labeled text features The encoder output is the query Q. As the key K and value V, the formula for calculating the text feature A of a specific label is as follows:

[0119]

[0120] in, Indicates will Mapped to the The parameters of each subspace, Indicates the parameters for multi-head attention training. Used to represent The dimension of a vector.

[0121] The input to the multi-class output layer 311 is a text feature matrix that fuses label associations. It consists of two parts: a fully connected layer and a sigmoid activation function. The fully connected layer processes text features... Convert it to a one-dimensional vector, and the calculation method is shown in the following formula.

[0122]

[0123] in, For trainable weight vectors, This is a bias term.

[0124] The multi-class output layer 311 uses the sigmoid function to... Perform the transformation and constrain it to Within the interval, the probability of a comment text belonging to each tag is expressed. The calculation method is shown in the following formula.

[0125]

[0126] This application also provides a model processing method for training an untrained neural network model to obtain the neural network model in the above-mentioned text classification method.

[0127] Figure 4 A flowchart illustrating a model processing method according to an embodiment of this application is shown schematically.

[0128] like Figure 4 As shown, the text classification method of this embodiment includes operations S410 to S440.

[0129] The S410 is used to obtain the training text of the neural network model.

[0130] The training text includes the text to be processed, a set of labels including multiple labels and relational information, and the matching degree between the text to be processed and each of the multiple labels and the text to be processed. The relational information includes the mutual exclusion relationship between the multiple labels.

[0131] In operation S420, a neural network model is used to generate the predicted matching degree between each label and the text to be processed, based on the mutual exclusion relationship between the text to be processed and multiple labels in the training text.

[0132] In operation S430, the medium distribution balanced loss function is used to determine the predicted matching degree and the loss information between the matching degree.

[0133] The distribution-balanced loss function (DB) can be used to mitigate the impact of class imbalance on neural network models. The distribution-balanced loss function is mainly based on the basic binary cross-entropy loss function (BCE) and incorporates a weight rebalancing strategy.

[0134] When operating S440, the parameters of the neural network model are adjusted based on the loss information.

[0135] By designing and using the DB loss function, a scaling factor λ and parameters specific to class bias are introduced. , scaling factor λ and Generated based on pre-tuned model parameters, it helps to mitigate the excessive penalty effect on negative text, thereby reducing redundant co-occurrence information between labels during multi-label classification. It can also reduce the text weight allocation of high-frequency head categories, alleviating the excessive suppression of negative text while also reducing the weight allocation of head category text.

[0136] Based on the above text classification method, this application also provides a text classification device. The following will combine... Figure 5 The device is described in detail.

[0137] Figure 5 A schematic block diagram of a text classification device according to an embodiment of this application is shown.

[0138] like Figure 5 As shown, the text classification device 500 of this embodiment includes a receiving module 510, a classification module 520, and a first generation module 530.

[0139] The receiving module 510 is used to receive the text to be processed and tag set information. The tag set information includes multiple tags and relationship information, which includes the mutual exclusion relationship between the multiple tags.

[0140] The classification module 520 is used to process the features of the text to be processed using a neural network model, based on features including those used to characterize the mutual exclusion relationship between multiple labels, to determine the matching degree between each of the multiple labels and the text to be processed. The matching degree between the text to be processed and the first label and the matching degree between the text to be processed and the second label are negatively correlated. The first label and the second label are two mutually exclusive labels among the multiple labels.

[0141] The first generation module 530 is used to generate matching tags for the text to be processed based on the matching degree between multiple tags and the text to be processed. The matching tags include tags among the multiple tags whose matching degree with the text to be processed is greater than a preset threshold.

[0142] According to embodiments of this application, any plurality of modules among the receiving module 510, the classification module 520, and the first generation module 530 can be merged into one module, or any one of these modules can be split into multiple modules. Alternatively, at least a portion of the functionality of one or more of these modules can be combined with at least a portion of the functionality of other modules and implemented in one module. According to embodiments of this application, at least one of the receiving module 510, the classification module 520, and the first generation module 530 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any appropriate combination of any of these three implementation methods. Alternatively, at least one of the receiving module 510, the classification module 520, and the first generation module 530 can be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.

[0143] According to an embodiment of this application, the relationship information also includes co-occurrence relationships between multiple tags. The co-occurrence relationship indicates that the frequency of two tags that are both text tags meets a preset requirement. The classification module uses a neural network model to process the feature information of the text to be processed based on features including those used to characterize the mutual exclusion relationships between multiple tags, and determines the matching degree between each of the multiple tags and the text to be processed: constructing a co-occurrence relationship matrix and a mutual exclusion relationship matrix based on the co-occurrence relationship matrix and the mutual exclusion relationship matrix; performing at least one feature fusion operation on the co-occurrence relationship matrix and the mutual exclusion relationship matrix respectively to obtain the fusion relationship features between multiple tags; and generating the matching degree between each of the multiple tags and the text to be processed based on the fusion relationship features between multiple tags and the features of the text to be processed.

[0144] According to an embodiment of this application, the classification module generates the matching degree between each of the multiple labels and the text to be processed in the following manner based on the fusion relationship features between multiple labels and the features of the text to be processed: performing a first linear transformation and a second linear transformation on the features of the text to be processed to obtain the key features and value features of the text to be processed; processing the fusion relationship features, key features and value features based on a neural network layer with a multi-head cross-attention mechanism to obtain output features including text feature information related to each of the multiple labels; performing linear classification on the output features and standardizing them using a preset activation function to obtain the matching degree between each of the multiple labels and the text to be processed.

[0145] According to an embodiment of this application, the classification module performs at least one feature fusion operation on the co-occurrence relation matrix and the mutual exclusion relation matrix respectively to obtain fused relation features between multiple labels: The original relation features, reverse relation features, and self-loop relation features of the co-occurrence relation features and the mutual exclusion relation features are determined respectively; according to the weight information corresponding to the original relation features, reverse relation features, and self-loop relation features, the co-occurrence relation features and the mutual exclusion relation features are weighted and summed and feature extracted respectively to obtain intermediate fused relation features; according to the weight information corresponding to the original relation features, reverse relation features, and self-loop relation features, the intermediate fused relation features are weighted and summed a preset number of times and feature extracted to obtain fused relation features.

[0146] According to an embodiment of this application, the apparatus further includes an encoding module, which is used to: acquire features of the text to be processed using a multi-head self-attention encoder, the features including contextual information.

[0147] Based on the above model processing method, this application also provides a model processing apparatus. The following will be combined with... Figure 6 The device is described in detail.

[0148] Figure 6A schematic block diagram of a model processing apparatus according to an embodiment of this application is shown.

[0149] like Figure 6 As shown, the model processing device 600 of this embodiment includes an acquisition module 610, a second generation module 620, a loss calculation module 630, and an adjustment module 640.

[0150] The acquisition module 610 is used to acquire the training text of the neural network model. The training text includes the text to be processed, label set information including multiple labels and relational information of the multiple labels, and the matching degree between the text to be processed and each of the multiple labels and the text to be processed. The relational information includes the mutual exclusion relationship between the multiple labels.

[0151] The second generation module 620 is used to generate the predicted matching degree between each of the multiple labels and the text to be processed based on the mutual exclusion relationship between the text to be processed and the multiple labels in the training text using a neural network model.

[0152] The loss calculation module 630 is used to determine the loss information between the predicted matching degree and the matching degree using the medium distribution balance loss function.

[0153] The adjustment module 640 is used to adjust the parameters of the neural network model based on the loss information.

[0154] Figure 7 A block diagram schematically illustrates an electronic device suitable for implementing text classification and model processing methods according to embodiments of this application.

[0155] like Figure 7 As shown, an electronic device 700 according to an embodiment of this application includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage portion 708 into a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this application.

[0156] RAM 703 stores various programs and data required for the operation of electronic device 700. Processor 701, ROM 702, and RAM 703 are interconnected via bus 704. Processor 701 executes various operations of the method flow according to embodiments of this application by executing programs in ROM 702 and / or RAM 703. It should be noted that the programs may also be stored in one or more memories other than ROM 702 and RAM 703. Processor 701 may also execute various operations of the method flow according to embodiments of this application by executing programs stored in said one or more memories.

[0157] According to embodiments of this application, the electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to a bus 704. The electronic device 700 may also include one or more of the following components connected to the input / output (I / O) interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the input / output (I / O) interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 710 as needed so that computer programs read from it can be installed into the storage section 708 as needed.

[0158] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.

[0159] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this application, the computer-readable storage medium may include ROM 702 and / or RAM 703 and / or one or more memories other than ROM 702 and RAM 703 described above.

[0160] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code enables the computer system to implement the text classification method and model processing method provided in the embodiments of this application.

[0161] When the computer program is executed by the processor 701, it performs the functions defined in the system / apparatus of this application embodiment. According to the embodiments of this application, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0162] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 709, and / or installed from a removable medium 711. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0163] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 709, and / or installed from the removable medium 711. When the computer program is executed by the processor 701, it performs the functions defined in the system of this application embodiment. According to the embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0164] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0165] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0166] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.

Claims

1. A text classification method, characterized in that, The method includes: Receive text to be processed and tag set information, wherein the tag set information includes multiple tags and relationship information, wherein the relationship information includes the mutual exclusion relationship between the multiple tags; Using a neural network model, the features of the text to be processed are processed based on features including those used to characterize the mutual exclusion relationship between the multiple tags, and the matching degree between each of the multiple tags and the text to be processed is determined. The matching degree between the text to be processed and the first tag and the matching degree between the text to be processed and the second tag are negatively correlated. The first tag and the second tag are two mutually exclusive tags among the multiple tags. Based on the matching degree between each of the plurality of tags and the text to be processed, matching tags for the text to be processed are generated. The matching tags include tags among the plurality of tags whose matching degree with the text to be processed is greater than a preset threshold.

2. The method according to claim 1, characterized in that, The relationship information also includes co-occurrence relationships between multiple tags, where the co-occurrence relationship indicates that the frequency of two tags that are both text meets a preset requirement; The method of using a neural network model to process the feature information of the text to be processed based on features including those characterizing the mutual exclusion relationships between the multiple tags, and determining the matching degree between each of the multiple tags and the text to be processed, includes: Based on the co-occurrence relationship and the mutual exclusion relationship, construct the co-occurrence relationship matrix and the mutual exclusion relationship matrix; Perform at least one feature fusion operation on the co-occurrence relation matrix and the mutual exclusion relation matrix respectively to obtain the fused relation features among the multiple labels; Based on the fusion relationship features between the multiple tags and the features of the text to be processed, a matching degree between each of the multiple tags and the text to be processed is generated.

3. The method according to claim 2, characterized in that, The step of generating a matching degree between each of the multiple tags and the text to be processed based on the fusion relationship features between the multiple tags and the features of the text to be processed includes: The features of the text to be processed are subjected to a first linear transformation and a second linear transformation, respectively, to obtain the key features and value features of the text to be processed. A neural network layer based on a multi-head cross-attention mechanism processes the fusion relationship features, the key features, and the value features to obtain output features that include text feature information related to each of the multiple tags. The output features are linearly classified and standardized using a preset activation function to obtain the matching degree between each of the multiple tags and the text to be processed.

4. The method according to claim 2, characterized in that, The step of performing at least one feature fusion operation on the co-occurrence relation matrix and the mutual exclusion relation matrix respectively to obtain the fused relation features among the multiple labels includes: Determine the original relationship features, reverse relationship features, and self-loop relationship features of the co-occurrence relationship features and the mutual exclusion relationship features, respectively; According to the weight information corresponding to the original relation features, reverse relation features and self-loop relation features, the co-occurrence relation features and the mutually exclusive relation features are weighted and summed and feature extracted to obtain intermediate fusion relation features. Based on the weight information corresponding to the original relation features, reverse relation features, and self-loop relation features, the intermediate fusion relation features are subjected to a preset number of weighted summations and feature extraction operations to obtain the fusion relation features.

5. The method according to any one of claims 2 to 4, characterized in that, The method further includes: A multi-head self-attention encoder is used to obtain features of the text to be processed, including contextual information.

6. A model processing method, characterized in that, The method includes: The training text of the neural network model is obtained. The training text includes the text to be processed, a set of labels including multiple labels and relational information of the multiple labels, and the matching degree between the text to be processed and each of the multiple labels and the text to be processed. The relational information includes the mutual exclusion relationship between the multiple labels. Using the neural network model, based on the mutual exclusion relationship between the text to be processed in the training text and the multiple tags, a predicted matching degree between each of the multiple tags and the text to be processed is generated; The loss information between the predicted matching degree and the matching degree is determined using the medium distribution balance loss function; Based on the loss information, the parameters of the neural network model are adjusted.

7. A text classification device, characterized in that, The device includes: A receiving module is used to receive text to be processed and tag set information, wherein the tag set information includes multiple tags and relationship information, wherein the relationship information includes mutual exclusion relationships between the multiple tags; A classification module is used to process the features of the text to be processed using a neural network model, based on features including those representing the mutual exclusion relationship between the multiple labels, to determine the matching degree between each of the multiple labels and the text to be processed. The matching degree between the text to be processed and the first label and the matching degree between the text to be processed and the second label are negatively correlated. The first label and the second label are two mutually exclusive labels among the multiple labels. The first generation module is used to generate matching tags for the text to be processed based on the matching degree between each of the plurality of tags and the text to be processed. The matching tags include tags among the plurality of tags whose matching degree with the text to be processed is greater than a preset threshold.

8. A model processing device, characterized in that, The device includes: The acquisition module is used to acquire the training text of the neural network model. The training text includes the text to be processed, a tag set information including multiple tags and relationship information of the multiple tags, and the matching degree between the text to be processed and each of the multiple tags and the text to be processed. The relationship information includes the mutual exclusion relationship between the multiple tags. The second generation module is used to generate a predicted matching degree between each of the multiple labels and the text to be processed, based on the mutual exclusion relationship between the text to be processed and the multiple labels in the training text, using the neural network model. The loss calculation module is used to determine the loss information between the predicted matching degree and the matching degree using the medium distribution balance loss function; The adjustment module is used to adjust the parameters of the neural network model based on the loss information.

9. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1-6.

10. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1-6.

Images