A method and system for hierarchical multi-label text classification
Through the combination of BERT pre-trained model and dual-stream model, global and local features are extracted and parameter β trade-offs are performed, which solves the problem of low and rough classification accuracy in hierarchical multi-label text classification, achieving higher text classification accuracy .
Patent Information
- Application Number
- CN202110660691.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-06-15
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2041-06-15
AI Technical Summary
The prior art has problems of low classification accuracy and rough classification in hierarchical multi-label text classification, which has failed to effectively solve the problems of learning differential characteristics of subclass labels and filtering of parent-class features.
The word vector initialization is performed by using BERT pre-trained model, and the global and local feature extraction is performed in combination with the dual-flow model. The global and local category information is traded down through the dynamic adjustment model of parameter β to predict text classification results, ensuring that each level can capture directly from the original features. information.
The accuracy of text classification is improved, the problems of low and rough classification accuracy are solved, and better classification results are obtained through the overall learning of global flows and local flows and hierarchical local learning.
Smart Images

Figure CN113590815B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computers, and in particular to a method and system for hierarchical multi-label text classification. Background Art
[0002] In the text classification task, according to the number of labels corresponding to each sample, it can be divided into single-label classification and multi-label classification. In the multi-label classification task, according to the hierarchical relationship of the labels: it can be divided into hierarchical multi-label classification and parallel multi-label classification. Generally speaking, multi-label classification is parallel by default. For example, a movie may be a "comedy" and a "romance movie". Such category labels are parallel and have no hierarchical structure; and the news we watch includes life, technology, and entertainment. Among them, the entertainment category is divided into movie channels and TV series channels, etc. Such category labels have a hierarchical structure, so this type of task is called hierarchical multi-label classification.
[0003] In related technologies, the HFT-CNN model is used for pre-training. During the training of the HFT-CNN model, features are transferred and fine-tuned layer by layer. The CNN model is suitable for capturing local features. However, when classifying sub-class labels, they are very dependent on differential features that are different from the parent class, and these differential features are unnecessary for the parent class classification. If the features of the parent class are used for initialization, the parent class features are theoretically preserved, but the differential features that affect the sub-class labels are not easy to learn and may be filtered or weakened during the learning process of the parent class labels, reducing the classification accuracy.
[0004] Currently, in the related technologies, when classifying hierarchical multi-label texts, there are problems of low classification accuracy and rough classification, and no effective solution has been proposed. Summary of the invention
[0005] The embodiments of the present application provide a method and system for classifying hierarchical multi-label texts, so as to at least solve the problems of low classification accuracy and rough classification when classifying hierarchical multi-label texts in the related art.
[0006] In a first aspect, an embodiment of the present application provides a method for hierarchical multi-label text classification, the method comprising:
[0007] Obtain a text sequence, and initialize the word vector of the text sequence through the BERT pre-trained model to obtain a text matrix;
[0008] The text matrix is processed by a two-stream model to obtain global category information and local category information respectively, wherein the two-stream model includes a global stream and a local stream, the global stream extracts global features from the text matrix layer by layer through a feedforward neural network layer to obtain global features of a subclass layer, and performs global prediction on the text sequence through the global features of the subclass layer to obtain global category information, and the local stream extracts features from the global features of the current subclass layer through each layer of an external feedforward neural network to obtain local features of the current subclass layer, and performs local prediction on the text sequence through the local features of the current subclass layer to obtain local category information of each layer;
[0009] The model is dynamically adjusted by parameter β to weigh the global category information and the local category information of each layer, and the text classification result P is predicted. F :
[0010]
[0011] Among them, ⊙ represents vector concatenation, β is the trade-off value, P G Represents global category information, represents the local category information of each layer, and |H| represents the number of levels.
[0012] In some embodiments, extracting global features from the text matrix layer by layer through a feedforward neural network layer to obtain global features of a subclass layer, and performing global prediction on the text sequence through the global features of the subclass layer to obtain global category information includes:
[0013] The first layer of feedforward neural network is used to extract global features from the text matrix to obtain the first subclass global feature vector
[0014]
[0015] Concatenate the first subclass global feature vector with the text matrix to obtain a first concatenation vector, and extract global features from the first concatenation vector through a second layer of feedforward neural network to obtain a second subclass global feature vector;
[0016] By analogy, the global feature vector of each layer of subclasses is concatenated with the text matrix to obtain a concatenated vector, and the global feature is extracted from the concatenated vector through the current layer feedforward neural network to obtain the global feature vector of the current layer subclass
[0017]
[0018] The text sequence is globally predicted by the subclass global feature vector to obtain the global category information PG :
[0019]
[0020] Among them, x represents the D-dimensional text matrix, G represents global, h represents the level, W represents the weight matrix parameter, and b represents the bias parameter. and σ both represent nonlinear activation functions, ⊙ represents vector concatenation, and |H| represents the number of levels.
[0021] In some embodiments, the feature extraction of the global features of the current subclass layer by the external feedforward neural network of each layer to obtain the local features of the current subclass layer, and the local prediction of the text sequence by the local features of the current subclass layer to obtain the local category information of each layer includes:
[0022] The global features of the subclass layer are linearly mapped to the input vector of the external feedforward neural network of the current subclass layer, and feature extraction is performed to obtain the local feature vector of the current subclass layer
[0023]
[0024] Among them, h represents the level, L represents the local part, W represents the mapping weight parameter, and b represents the mapping bias parameter. represents the nonlinear activation function;
[0025] The text sequence is locally predicted by the local features of the current sub-class layer to obtain the local category information of each layer.
[0026]
[0027] Among them, h represents the level, L represents local, W represents the weight matrix parameter, b represents the bias parameter, and σ represents the nonlinear activation function.
[0028] In some embodiments, initializing the word vector of the text sequence by using the BERT pre-trained model includes:
[0029] The text matrix is obtained by performing feature extraction on the text sequence through the Transformer module integrated with the attention mechanism in the BERT pre-training model.
[0030] In some embodiments, before performing feature extraction and classification prediction on the text matrix through the two-stream model, the method includes:
[0031] The cross entropy loss of the true label of each category in the text sequence is calculated by the two-stream model, and the parameters are optimized through neural network back propagation to train the optimal two-stream model.
[0032] In some embodiments, the text classification result P is predicted. F Thereafter, the method comprises:
[0033] The text classification result P F The value of each dimension is compared with a preset threshold, and when the value is greater than the threshold, the category label of the dimension is set on the text sequence.
[0034] In some embodiments, before initializing the word vector of the text sequence through the BERT pre-training model, the method further includes:
[0035] The BERT model is pre-trained by using open source Chinese text of characters, and parameters of the BERT model are fine-tuned in combination with relevant corpus information to obtain the BERT pre-trained model.
[0036] In a second aspect, an embodiment of the present application provides a system for hierarchical multi-label text classification, the system comprising:
[0037] An acquisition module is used to acquire a text sequence and initialize the word vector of the text sequence through a BERT pre-trained model to obtain a text matrix;
[0038] A dual-stream module, used for processing the text matrix through a dual-stream model to obtain global category information and local category information respectively, wherein the dual-stream model includes a global stream and a local stream, the global stream extracts global features from the text matrix layer by layer through a feedforward neural network layer to obtain global features of a subclass layer, and performs global prediction on the text sequence through the global features of the subclass layer to obtain global category information, and the local stream extracts features from the global features of the current subclass layer through each layer of an external feedforward neural network to obtain local features of the current subclass layer, and performs local prediction on the text sequence through the local features of the current subclass layer to obtain local category information of each layer;
[0039] The prediction module is used to dynamically adjust the model through the parameter β to weigh the global category information and the local category information of each layer, and predict the text classification result P F :
[0040]
[0041] Among them, ⊙ represents vector concatenation, β is the trade-off value, PG represents global category information, represents the local category information of each layer, and |H| represents the number of levels.
[0042] In a third aspect, an embodiment of the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for hierarchical multi-label text classification as described in the first aspect above is implemented.
[0043] In a fourth aspect, an embodiment of the present application provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for hierarchical multi-label text classification as described in the first aspect above.
[0044] Compared with the related art, the hierarchical multi-label text classification method provided in the embodiment of the present application obtains a text sequence, and initializes the word vector of the text sequence through the BERT pre-trained model to obtain a text matrix; then, the text matrix is processed through a two-stream model to obtain global category information and local category information respectively, wherein the two-stream model includes a global stream and a local stream, the global stream extracts global features of the text matrix layer by layer through a feedforward neural network layer to obtain the global features of the sub-class layer, and globally predicts the text sequence through the global features of the sub-class layer to obtain global category information, the local stream extracts features from the global features of the current sub-class layer through each layer of an external feedforward neural network to obtain the local features of the current sub-class layer, and locally predicts the text sequence through the local features of the current sub-class layer to obtain the local category information of each layer; finally, the model is dynamically adjusted through the parameter β to weigh the global category information and the local category information of each layer, and the text classification result is predicted.
[0045] This application uses a dual-stream semantic model combined with a BERT pre-trained model. In the process of feature transmission layer by layer, each layer splices the original text features to the current layer for local flow learning of sub-class level features, so that each layer can directly capture the direct information required for current layer classification from the original features. This not only enriches the classification features of the deep network, but also makes full use of the label relationship information in multi-level classification tasks. By performing overall learning and hierarchical local learning through the global flow and local flow respectively, better classification results can be effectively obtained, the accuracy of text classification is improved, and the problems of low classification accuracy and rough classification when classifying hierarchical multi-distant label texts are solved. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0047] Figure 1 is a flowchart of a method for hierarchical multi-label text classification according to an embodiment of the present application;
[0048] Figure 2 is a network structure flow chart of a dual-stream model according to an embodiment of the present application;
[0049] Figure 3 is a structural block diagram of a system for hierarchical multi-label text classification according to an embodiment of the present application;
[0050] Figure 4 It is a schematic diagram of the internal structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION
[0051] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application. In addition, it can also be understood that although the efforts made in this development process may be complex and lengthy, for ordinary technicians in the field related to the contents disclosed in the present application, some changes such as design, manufacturing or production based on the technical contents disclosed in the present application are only conventional technical means, and should not be understood as insufficient contents disclosed in the present application.
[0052] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.
[0053] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantitative limitation, and may represent the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to greater than or equal to two. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships, for example, "A and / or B" can represent: A exists alone, A and B exist at the same time, and B exists alone. The terms "first", "second", "third" and the like involved in the present application are merely used to distinguish similar objects and do not represent a specific ordering of the objects.
[0054] This application provides a method for hierarchical multi-label text classification. Figure 1 is a flowchart of a method for hierarchical multi-label text classification according to an embodiment of the present application. Figure 1 As shown, the process includes the following steps:
[0055] Step S101, obtain a text sequence, and initialize the word vector of the text sequence through the BERT pre-trained model to obtain a text matrix; preferably, in this embodiment, the Transformer module that integrates the attention mechanism in the BERT pre-trained model is used to extract features of the text sequence, which can extract richer semantic features in the original text and obtain a better text matrix vector, which is conducive to accelerating the convergence of the model and improving the model effect.
[0056] In some embodiments, before initializing the word vector of the text sequence through the BERT pre-trained model, the BERT model is pre-trained through the open source Chinese text of the word, and the parameters of the BERT model are fine-tuned in combination with the relevant corpus information to obtain the BERT pre-trained model. The BERT pre-trained model is used as a feature extractor to obtain the text matrix input into the two-stream model.
[0057] Step S102, processing the text matrix through the dual-stream model to obtain global category information and local category information respectively, wherein the dual-stream model includes a global stream and a local stream, the global stream extracts global features from the text matrix layer by layer through a feedforward neural network layer to obtain global features of the subclass layer, and performs global prediction on the text sequence through the global features of the subclass layer to obtain global category information, the local stream extracts features from the global features of the current subclass layer through each layer of an external feedforward neural network to obtain local features of the current subclass layer, and performs local prediction on the text sequence through the local features of the current subclass layer to obtain local category information of each layer;
[0058] Preferably, in this embodiment, the text matrix is processed by a dual-stream model to obtain global category information and local category information respectively, wherein the dual-stream model includes a global stream and a local stream; Figure 2 is a network structure flow chart of a dual-stream model according to an embodiment of the present application, such as Figure 2 As shown, the specific process of implementing the global flow network structure is as follows:
[0059] S1, input the text matrix into the feedforward neural network layer of the global flow, extract the global features of the text matrix through the first layer of feedforward neural network, and obtain the global feature vector of the first subclass The formula is shown in equation 1 below:
[0060]
[0061] Among them, x represents the D-dimensional text matrix, |D| is the number of text features, G represents global, W represents the weight matrix parameter, and b represents the bias parameter. represents a nonlinear activation function;
[0062] S2, concatenate the first subclass global feature vector and the original text matrix to obtain a first concatenation vector, which is the input of the second layer of feedforward neural network, and extract the second subclass global feature vector through the second layer of feedforward neural network. In this embodiment, as the network level deepens, the input features will become more abstract relative to the shallow general features, and the original features may be gradually lost. Therefore, by concatenating the original text matrix, the calculation of the deep network can still retain rich semantic features to ensure the accuracy of classification;
[0063] S3, and so on, concatenate the global feature vector of each layer of subclasses with the original text matrix to obtain a concatenated vector, and extract global features from the concatenated vector through the current layer feedforward neural network to obtain the global feature vector of the current layer subclass The formula is shown in equation 2 below:
[0064]
[0065] Among them, x represents the D-dimensional text matrix, G represents global, h represents the level, W represents the weight matrix parameter, and b represents the bias parameter. represents a nonlinear activation function, ⊙ represents vector concatenation;
[0066] S4, perform global prediction on the text sequence through the sub-class global feature vector obtained above to obtain the final global category information P G , P G is a |C|-dimensional vector, where |C| is the total number of categories and the global category information P G Each dimension in represents the probability that the input text sequence belongs to each corresponding category, and the calculation formula is shown in the following formula 3:
[0067]
[0068] Among them, x represents the D-dimensional text matrix, G represents global, h represents the level, W represents the weight matrix parameter, b represents the bias parameter, σ represents the nonlinear activation function, ⊙ represents vector concatenation, and |H| represents the number of levels.
[0069] Furthermore, the specific process of implementing the local flow network structure is as follows:
[0070] S1, linearly map the global features of the subclass layer to the input vector of the external feedforward neural network of the current subclass layer, and extract features through the external feedforward neural network to obtain the local feature vector of the current subclass layer The formula is shown in equation 4 below:
[0071]
[0072] Among them, h represents the level, L represents the local part, W represents the mapping weight parameter, and b represents the mapping bias parameter. represents the nonlinear activation function;
[0073] S2, the local features of the current subclass layer obtained by step S1 Perform local prediction on the text sequence to obtain local category information of each layer Yes|C h |-dimensional vector, where |C h | is the number of categories contained in the hth level, local category information Each dimension in represents the probability that the input text sequence belongs to each category at the hth level, and the calculation formula is shown in the following formula 5:
[0074]
[0075] Among them, h represents the level, L represents local, W represents the weight matrix parameter, b represents the bias parameter, and σ represents the nonlinear activation function.
[0076] It should be noted that the above mentioned Both σ and σ represent nonlinear activation functions, preferably, Take the ReLU activation function and σ take the Sigmoid activation function.
[0077] Specifically, Figure 2 As shown, in this embodiment, the text matrix obtained by initializing the word vector of the BERT pre-trained model is input into the two-stream model. First, the text matrix enters the first layer of the global stream neural network for global feature extraction to obtain the first subclass global feature vector Then After concatenating with the text matrix to obtain the first concatenated vector, it is input into the second layer neural network of the global stream for global feature extraction to obtain the second subclass global feature vector at the same time, After linear mapping as the input vector of the local flow, it enters the first layer of the local flow neural network for feature extraction to obtain the local feature vector Afterwards, through Perform local prediction to obtain the local category information of the first layer Similarly to the above, the second subclass global feature vector of the global flow can be obtained in turn pass Get the local feature vector of the local flow and the local category information of the second layer Then, the third subclass global feature vector of the global flow is obtained in the same way pass Get the local feature vector of the local flow and the local category information of the third layer Finally, through The feature vector obtained by concatenating with the original text matrix performs global prediction on the text sequence to obtain the final global category information P G ; It should be noted that the hollow circles in the figure represent vector splicing.
[0078] In this embodiment, the concatenation of the original text matrix in the global stream not only directly enriches the semantic features of the deep network, but also enables each level in the local stream to directly capture the direct information required for the current layer classification from the original features, and obtain the detailed category feature information of each level, making the classification result more detailed and clear. In addition, this embodiment uses two information streams to capture feature information of different levels from the global and local perspectives, which can improve the accuracy of text classification;
[0079] Step S103, dynamically adjust the model through parameter β to weigh the global category information and the local category information of each layer, and predict the text classification result P F , P F is a |C|-dimensional vector, which indicates the probability that the input text sequence belongs to each category. The calculation formula is shown in Equation 6 below:
[0080]
[0081] Among them, ⊙ represents vector concatenation, β is the trade-off value, P G Represents global category information, represents the local category information of each layer, and |H| represents the number of levels.
[0082] Preferably, in this implementation, β is used to weigh the importance of global flow and local flow, wherein β can be set to 0.5 or can be set based on prior knowledge.
[0083] Specifically, Figure 2 As shown, the three-layer local category information of the local flow obtained in step S102 is Splice and get the global category information P G The concatenated category information vector is combined with the global category information vector P G Perform enhanced summation and predict the text classification result P F Among them, the parameter β is used to weigh the importance of global flow and local flow, and the prediction is dynamically adjusted.
[0084] In some of the embodiments, before performing feature extraction and classification prediction on the text matrix through the two-stream model, the cross entropy loss of the true label of each category in the text sequence is calculated through the two-stream model, and parameter optimization is performed through neural network back propagation to train the optimal two-stream model.
[0085] In some embodiments, the text classification result P is predicted. F Afterwards, the text classification result P F The value of each dimension is compared with the preset threshold. When the value is greater than the threshold, the category label corresponding to the dimension is set on the text sequence to achieve multi-label hierarchical classification of the text.
[0086] Through the above steps S101 to S103, in this embodiment, a dual-stream semantic model combined with a BERT pre-trained model, in the process of feature transmission layer by layer, each layer splices the original text features to the current layer, which can not only extract global stream features with rich semantics and obtain global category information, but also use the local stream to learn sub-class level features, so that each level can directly capture the direct information required for the current layer classification from the original features, and obtain the fine category feature information of each level. Through the overall learning of the global and local streams and the hierarchical local learning, the problems of low classification accuracy and rough classification when classifying hierarchical multi-distant label texts are solved, and the accuracy of text classification is effectively improved.
[0087] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0088] This embodiment also provides a system for hierarchical multi-label text classification, which is used to implement the above-mentioned embodiments and preferred implementation modes, and will not be repeated here. As used below, the terms "module", "unit", "subunit", etc. can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.
[0089] Figure 3 is a structural block diagram of a system for hierarchical multi-label text classification according to an embodiment of the present application, such as Figure 3 As shown, the system includes an acquisition module 31, a dual-stream module 32 and a prediction module 33:
[0090] An acquisition module 31 is used to acquire a text sequence, and initialize the word vector of the text sequence through the BERT pre-trained model to obtain a text matrix; a dual-stream module 32 is used to process the text matrix through the dual-stream model to obtain global category information and local category information respectively, wherein the dual-stream model includes a global stream and a local stream, the global stream extracts global features from the text matrix layer by layer through a feedforward neural network layer to obtain the global features of the sub-class layer, and performs global prediction on the text sequence through the global features of the sub-class layer to obtain global category information, the local stream extracts features from the global features of the current sub-class layer through each layer of an external feedforward neural network to obtain the local features of the current sub-class layer, and performs local prediction on the text sequence through the local features of the current sub-class layer to obtain the local category information of each layer; a prediction module 33 is used to dynamically adjust the model through a parameter β to perform a trade-off calculation on the global category information and the local category information of each layer, and predict the text classification result.
[0091] Through the above system, a dual-stream semantic model combined with a BERT pre-trained model in this embodiment, in the process of feature transmission layer by layer, each layer splices the original text features to the current layer, which can not only extract global stream features with rich semantics and obtain global category information, but also use the local stream to learn sub-class level features, so that each level can directly capture the direct information required for the current layer classification from the original features, and obtain the detailed category feature information of each level. Through the overall learning of the global and local streams and the hierarchical local learning, the problems of low classification accuracy and rough classification when classifying hierarchical multi-distant label texts are solved, and the accuracy of text classification is effectively improved.
[0092] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.
[0093] In addition, it should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.
[0094] This embodiment further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.
[0095] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.
[0096] In addition, in combination with the method for hierarchical multi-label text classification in the above embodiment, the present application embodiment can provide a storage medium for implementation. The storage medium stores a computer program; when the computer program is executed by a processor, any of the methods for hierarchical multi-label text classification in the above embodiment is implemented.
[0097] In one embodiment, a computer device is provided, which may be a terminal. The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a method for hierarchical multi-label text classification is implemented. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covering the display screen, or a key, trackball, or touchpad provided on the housing of the computer device, or an external keyboard, touchpad, or mouse, etc.
[0098] In one embodiment, Figure 4 is a schematic diagram of the internal structure of an electronic device according to an embodiment of the present application, such as Figure 4 As shown, an electronic device is provided, which may be a server, and its internal structure diagram may be as shown in Figure 4 As shown. The electronic device includes a processor, a network interface, an internal memory and a non-volatile memory connected through an internal bus, wherein the non-volatile memory stores an operating system, a computer program and a database. The processor is used to provide computing and control capabilities, the network interface is used to communicate with an external terminal through a network connection, the internal memory is used to provide an environment for the operation of the operating system and the computer program, the computer program is executed by the processor to implement a hierarchical multi-label text classification method, and the database is used to store data.
[0099] Those skilled in the art will understand that Figure 4 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.
[0100] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
[0101] Those skilled in the art should understand that the technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0102] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.
Claims
1. A method for hierarchical multi-label text classification, characterized in that: The method comprises: Obtain a text sequence, and initialize the word vector of the text sequence through the BERT pre-trained model to obtain a text matrix; The text matrix is processed by a two-stream model to obtain global category information and local category information respectively, wherein the two-stream model includes a global stream and a local stream, the global stream extracts global features from the text matrix layer by layer through a feedforward neural network layer to obtain global features of a subclass layer, and globally predicts the text sequence through the global features of the subclass layer to obtain global category information, the local stream extracts features from the global features of the current subclass layer through each layer of an external feedforward neural network to obtain local features of the current subclass layer, and locally predicts the text sequence through the local features of the current subclass layer to obtain local category information of each layer, The extracting of global features from the text matrix layer by layer through the feedforward neural network layer to obtain global features of the subclass layer, and performing global prediction on the text sequence through the global features of the subclass layer to obtain global category information includes: The first layer of feedforward neural network is used to extract global features from the text matrix to obtain the first subclass global feature vector Concatenate the first subclass global feature vector with the text matrix to obtain a first concatenation vector, and extract global features from the first concatenation vector through a second layer of feedforward neural network to obtain a second subclass global feature vector; By analogy, the global feature vector of each layer of subclasses is concatenated with the text matrix to obtain a concatenated vector, and the global feature is extracted from the concatenated vector through the current layer feedforward neural network to obtain the global feature vector of the current layer subclass The text sequence is globally predicted by the subclass global feature vector to obtain the global category information P G : Among them, x represents the D-dimensional text matrix, G represents global, h represents the level, W represents the weight matrix parameter, b represents the bias parameter, Ф and σ both represent nonlinear activation functions, ⊙ represents vector concatenation, and |H| represents the number of levels; The model is dynamically adjusted by parameter β to weigh the global category information and the local category information of each layer, and the text classification result P is predicted. F : Among them, ⊙ represents vector concatenation, β is the trade-off value, P G Represents global category information, represents the local category information of each layer, and |H| represents the number of levels.
2. The method according to claim 1, characterized in that The feature extraction of the global features of the current subclass layer by the external feedforward neural network of each layer to obtain the local features of the current subclass layer, and the local prediction of the text sequence by the local features of the current subclass layer to obtain the local category information of each layer includes: The global features of the subclass layer are linearly mapped to the input vector of the external feedforward neural network of the current subclass layer, and feature extraction is performed to obtain the local feature vector of the current subclass layer Among them, h represents the level, L represents the local part, and W T represents the mapping weight parameter, b T Represents the mapping bias parameter; The text sequence is locally predicted by the local features of the current sub-class layer to obtain the local category information of each layer. Among them, h represents the level, L represents local, W represents the weight matrix parameter, b represents the bias parameter, and σ represents the nonlinear activation function.
3. The method according to claim 1, characterized in that: Initializing the word vector of the text sequence by using the BERT pre-training model includes: The text matrix is obtained by performing feature extraction on the text sequence through the Transformer module integrated with the attention mechanism in the BERT pre-training model.
4. The method according to claim 1, characterized in that: Before performing feature extraction and classification prediction on the text matrix through the two-stream model, the method includes: The cross entropy loss of the true label of each category in the text sequence is calculated by the two-stream model, and the parameters are optimized through neural network back propagation to train the optimal two-stream model.
5. The method according to claim 1, characterized in that In the prediction, the text classification result P is obtained F Thereafter, the method comprises: The text classification result P F The value of each dimension is compared with a preset threshold, and when the value is greater than the threshold, the category label of the dimension is set on the text sequence.
6. The method according to claim 1, characterized in that Before initializing the word vector of the text sequence through the BERT pre-training model, the method further includes: The BERT model is pre-trained by using open source Chinese text of characters, and parameters of the BERT model are fine-tuned in combination with relevant corpus information to obtain the BERT pre-trained model.
7. A hierarchical multi-label text classification system, characterized in that: The system comprises: An acquisition module is used to acquire a text sequence and initialize the word vector of the text sequence through a BERT pre-trained model to obtain a text matrix; A dual-stream module is used to process the text matrix through a dual-stream model to obtain global category information and local category information respectively, wherein the dual-stream model includes a global stream and a local stream, the global stream extracts global features from the text matrix layer by layer through a feedforward neural network layer to obtain global features of a subclass layer, and performs global prediction on the text sequence through the global features of the subclass layer to obtain global category information, the local stream extracts features from the global features of the current subclass layer through each layer of an external feedforward neural network to obtain local features of the current subclass layer, and performs local prediction on the text sequence through the local features of the current subclass layer to obtain local category information of each layer, The extracting of global features from the text matrix layer by layer through a feedforward neural network layer to obtain global features of a subclass layer, and performing global prediction on the text sequence through the global features of the subclass layer to obtain global category information includes: The first layer of feedforward neural network is used to extract global features from the text matrix to obtain the first subclass global feature vector Concatenate the first subclass global feature vector with the text matrix to obtain a first concatenation vector, and extract global features from the first concatenation vector through a second layer of feedforward neural network to obtain a second subclass global feature vector; By analogy, the global feature vector of each layer of subclasses is concatenated with the text matrix to obtain a concatenated vector, and the global feature is extracted from the concatenated vector through the current layer feedforward neural network to obtain the global feature vector of the current layer subclass The text sequence is globally predicted by the subclass global feature vector to obtain the global category information P G : Among them, x represents the D-dimensional text matrix, G represents global, h represents the level, W represents the weight matrix parameter, b represents the bias parameter, Ф and σ both represent nonlinear activation functions, ⊙ represents vector concatenation, and |H| represents the number of levels; The prediction module is used to dynamically adjust the model through the parameter β to weigh the global category information and the local category information of each layer, and predict the text classification result P F : Among them, ⊙ represents vector concatenation, β is the trade-off value, P G Represents global category information, represents the local category information of each layer, and |H| represents the number of levels.
8. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to execute the method for hierarchical multi-label text classification according to any one of claims 1 to 6.
9. A storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to execute the method for hierarchical multi-label text classification according to any one of claims 1 to 6 when running.
