Text classification method and device based on hybrid neural network and electronic equipment

Abstract

This application provides a text classification method, apparatus, and electronic device based on a hybrid neural network, relating to the field of natural language processing technology. The hybrid neural network includes a bidirectional recurrent neural network unit, a global modeling unit, and a classification unit. Specifically, the method includes: extracting local features from a dense vector matrix of unstructured text data using the bidirectional recurrent neural network unit to obtain a local temporal feature matrix of the unstructured text data. Then, explicit positional encoding is added to each local temporal feature representation in the local temporal feature matrix, and the local temporal feature matrix is ​​input to the global modeling unit. Based on a first preset attention mechanism built into the global modeling unit, global correlation modeling is performed on the local temporal feature matrix to obtain a global feature matrix. Finally, based on the global feature matrix, the text classification result of the unstructured text data is output by the classification unit. The method of this application can improve the accuracy of text classification.

Description

Technical Field

[0001] This application relates to the field of natural language processing technology, and in particular to a text classification method, apparatus and electronic device based on hybrid neural networks. Background Technology

[0002] In today's data-driven business environment, enterprises, institutions, and research organizations face the need for efficient processing of massive amounts of unstructured text data (such as reports, logs, emails, and social media content). This unstructured text data is usually in natural language form and lacks a unified format, thus urgently requiring classification through automated methods.

[0003] In the existing technology, text classification methods based on recurrent neural networks are widely used in the processing of unstructured text data. Its typical process includes: (1) text preprocessing (such as word embedding and denoising); (2) constructing a recurrent neural network structure, capturing the sequence dependencies of unstructured text data step by step through time steps, and obtaining the global features of unstructured text data; (3) the output layer uses the Softmax function or the Sigmoid function for text classification.

[0004] However, when dealing with extremely long unstructured text data, recurrent neural networks have limited ability to capture long-distance dependencies in such texts, which may lead to key features being ignored and affect the accuracy of text classification. Summary of the Invention

[0005] This application provides a text classification method, apparatus, and electronic device based on a hybrid neural network to solve the technical problem of low text classification accuracy in the prior art.

[0006] In a first aspect, this application provides a text classification method based on a hybrid neural network, wherein the hybrid neural network includes a bidirectional recurrent neural network unit, a global modeling unit, and a classification unit, and the method includes:

[0007] Obtain a dense vector matrix of unstructured text data; extract local features from the dense vector matrix using the bidirectional recurrent neural network unit to obtain a local temporal feature matrix of the unstructured text data; wherein, each row of the dense vector matrix represents the dense vector representation of each sub-word unit in the unstructured text data, and the dense vector representation represents a high-dimensional continuous number used to describe the sub-word unit;

[0008] Explicit positional encoding is added to each local temporal feature representation in the local temporal feature matrix, and the local temporal feature matrix with explicit positional encoding is input to the global modeling unit; based on the first preset attention mechanism built into the global modeling unit, global correlation modeling is performed on the local temporal feature matrix to obtain the global feature matrix;

[0009] Based on the global feature matrix, the classification unit outputs the text classification result of the unstructured text data.

[0010] Optionally, the bidirectional recurrent neural network unit includes a forward recurrent neural network and a backward recurrent neural network;

[0011] The dense vector matrix is ​​processed by the bidirectional recurrent neural network unit to extract local features, resulting in a local temporal feature matrix of the unstructured text data, including:

[0012] Each word unit corresponding to the dense vector matrix is ​​set as the time step of the bidirectional recurrent neural network unit;

[0013] Based on the aforementioned forward recurrent neural network, the forward hidden state of each time step is calculated, and based on the aforementioned backward recurrent neural network, the backward hidden state of each time step is calculated; wherein, the forward hidden state represents the aggregated representation of the preceding information of the time step, and the backward hidden state represents the aggregated representation of the following information of the time step.

[0014] Based on the forward and backward hidden states at each time step, a local temporal feature matrix of the unstructured text data is generated.

[0015] Optionally, based on the feedforward recurrent neural network, the feedforward hidden state at each time step is calculated, and based on the feedback recurrent neural network, the feedback hidden state at each time step is calculated, including:

[0016] Length padding and masking are performed on the sub-word units corresponding to each time step to obtain the real sub-words and padded sub-words of the sub-word units;

[0017] Based on the aforementioned forward recurrent neural network, the forward hidden state of the real words in each time step is calculated, and based on the aforementioned backward recurrent neural network, the backward hidden state of the real words in each time step is calculated.

[0018] Optionally, based on the forward and backward hidden states at each time step, a local temporal feature matrix of the unstructured text data is generated, including:

[0019] Integrate the forward and backward hidden states of each time step to generate local temporal feature representations for each time step;

[0020] Based on the second preset attention mechanism built into the bidirectional recurrent neural network unit, weights are set for each local temporal feature representation; based on each local temporal feature representation after setting the weights, a local temporal feature matrix of the unstructured text data is generated.

[0021] Optionally, explicit positional encoding is added to each local temporal feature representation in the local temporal feature matrix, including:

[0022] The dimension index and number of dimensions of each local temporal feature representation in the local temporal feature matrix are determined, as well as the position index of the sub-word unit corresponding to each local temporal feature representation; wherein, the dimension index represents the temporal position of the local temporal feature representation in the local temporal feature matrix, and the position index represents the position of the sub-word unit in the unstructured text data;

[0023] Explicit position codes are calculated based on the dimension index, the number of dimensions, and the position index, and the explicit position codes are added to the corresponding local temporal feature representations.

[0024] Optionally, the hybrid neural network further includes a preprocessing unit;

[0025] Obtain the dense vector matrix of unstructured text data, including:

[0026] Acquire unstructured text data and input the unstructured text data into the preprocessing unit;

[0027] Based on the word segmenter built into the preprocessing unit, the unstructured text data is segmented to obtain a sequence of sub-word units of the unstructured text data;

[0028] Word embedding processing is performed on each sub-word unit included in the sub-word unit sequence to obtain a dense vector representation of each sub-word unit; a dense vector matrix of the unstructured text data is generated based on the dense vector representation.

[0029] Optionally, word embedding processing is performed on each word unit included in the word unit sequence to obtain a dense vector representation of each word unit, including:

[0030] Determine the target identifier corresponding to each sub-word unit in the sub-word unit sequence; match the target identifier with the sub-word identifiers included in the preset word embedding matrix; wherein, the preset word embedding matrix includes multiple different sub-word identifiers and a preset dense vector representation corresponding to each sub-word identifier;

[0031] The sub-word identifiers that successfully match the target identifier in the preset word embedding matrix, and their corresponding preset dense vector representations, are determined as the dense vector representations of the sub-word units.

[0032] Secondly, this application provides a text classification device based on a hybrid neural network, wherein the hybrid neural network includes a bidirectional recurrent neural network unit, a global modeling unit, and a classification unit, and the device includes:

[0033] The acquisition module is used to acquire dense vector matrices of unstructured text data;

[0034] An extraction module is used to extract local features from the dense vector matrix through the bidirectional recurrent neural network unit to obtain a local temporal feature matrix of the unstructured text data; wherein, each row of the dense vector matrix represents the dense vector representation of each sub-word unit in the unstructured text data, and the dense vector representation represents a high-dimensional continuous number used to describe the sub-word unit;

[0035] The modeling module is used to add explicit positional encoding to each local temporal feature representation in the local temporal feature matrix, and input the local temporal feature matrix with added explicit positional encoding to the global modeling unit; based on the first preset attention mechanism built into the global modeling unit, global correlation modeling is performed on the local temporal feature matrix to obtain the global feature matrix;

[0036] The classification module is used to output the text classification result of the unstructured text data through the classification unit based on the global feature matrix.

[0037] Optionally, the bidirectional recurrent neural network unit includes a forward recurrent neural network and a backward recurrent neural network;

[0038] The extraction module includes:

[0039] A submodule is configured to set each word unit corresponding to the dense vector matrix as the time step of the bidirectional recurrent neural network unit.

[0040] The first calculation submodule is used to calculate the forward hidden state of each time step based on the forward recurrent neural network, and to calculate the backward hidden state of each time step based on the backward recurrent neural network; wherein the forward hidden state represents the aggregated representation of the preceding information of the time step, and the backward hidden state represents the aggregated representation of the following information of the time step.

[0041] The first generation submodule is used to generate the local temporal feature matrix of the unstructured text data based on the forward and backward hidden states at each time step.

[0042] Optionally, the first computing submodule includes:

[0043] The processing component is used to perform length padding and masking on the sub-word units corresponding to each time step to obtain the real sub-words and padded sub-words of the sub-word units;

[0044] A computing component is used to calculate the forward hidden state of the real words in each time step based on the forward recurrent neural network, and to calculate the backward hidden state of the real words in each time step based on the backward recurrent neural network.

[0045] Optionally, the first generation submodule includes:

[0046] An integration component is used to integrate the forward and backward hidden states of each time step to generate local temporal feature representations for each time step;

[0047] The component is configured to set weights for each local temporal feature representation based on the second preset attention mechanism built into the bidirectional recurrent neural network unit.

[0048] A generation component is used to generate a local temporal feature matrix of the unstructured text data based on the weighted representations of each local temporal feature.

[0049] Optionally, the modeling module includes:

[0050] The determination submodule is used to determine the dimension index and dimension number of each local temporal feature representation in the local temporal feature matrix, as well as the position index of the sub-word unit corresponding to each local temporal feature representation; wherein, the dimension index represents the temporal position of the local temporal feature representation in the local temporal feature matrix, and the position index represents the position of the sub-word unit in the unstructured text data;

[0051] The second calculation submodule is used to calculate the explicit position code based on the dimension index, the number of dimensions, and the position index;

[0052] Add a submodule to add the explicit positional encoding to the corresponding local temporal feature representation.

[0053] Optionally, the hybrid neural network further includes a preprocessing unit;

[0054] The acquisition module includes:

[0055] The `get` submodule is used to retrieve unstructured text data;

[0056] An input submodule is used to input the unstructured text data into the preprocessing unit;

[0057] The processing submodule is used for:

[0058] Based on the word segmenter built into the preprocessing unit, the unstructured text data is segmented to obtain a sequence of sub-word units of the unstructured text data;

[0059] Word embedding processing is performed on each sub-word unit included in the sub-word unit sequence to obtain a dense vector representation of each sub-word unit;

[0060] The second generation submodule is used to generate a dense vector matrix of the unstructured text data based on the dense vector representation.

[0061] Optionally, the processing submodule includes:

[0062] A determining component is used to determine the target identifier corresponding to each sub-word unit included in the sub-word unit sequence;

[0063] A matching component is used to match the target identifier with the sub-word identifiers included in a preset word embedding matrix; wherein, the preset word embedding matrix includes multiple different sub-word identifiers and a preset dense vector representation corresponding to each sub-word identifier;

[0064] The determining component is further configured to determine the sub-word identifier that successfully matches the target identifier in the preset word embedding matrix and its corresponding preset dense vector representation as the dense vector representation of the sub-word unit.

[0065] Thirdly, this application provides an electronic device, comprising:

[0066] Memory and processor;

[0067] The memory stores computer-executed instructions;

[0068] The processor executes computer execution instructions stored in the memory to implement the text classification method based on a hybrid neural network as described in the first aspect above and in various possible implementations.

[0069] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the text classification method based on a hybrid neural network as described in the first aspect and various possible implementations above.

[0070] Fifthly, this application provides a computer program product comprising a computer program that, when executed by a processor, implements the text classification method based on a hybrid neural network as described in the first aspect and various possible implementations above.

[0071] This application provides a text classification method, apparatus, and electronic device based on a hybrid neural network. It obtains a dense vector matrix of unstructured text data, and extracts local features from this dense vector matrix using a bidirectional recurrent neural network unit within the hybrid neural network to obtain a local temporal feature matrix of the unstructured text data. Each row of the dense vector matrix represents the dense vector representation of each sub-word unit in the unstructured text data, and this dense vector representation represents a high-dimensional continuous number used to describe the sub-word unit. Explicit positional encoding is then added to each local temporal feature representation in the local temporal feature matrix, and the local temporal feature matrix with explicit positional encoding is input to the global modeling unit in the hybrid neural network. Based on the first preset attention mechanism built into the global modeling unit, global correlation modeling is performed on the local temporal feature matrix to obtain a global feature matrix. Finally, based on the global feature matrix, the classification unit in the hybrid neural network outputs the text classification result of the unstructured text data. By using a hierarchical hybrid neural network architecture, combining the local feature extraction capability of the bidirectional recurrent neural network unit with the global attention mechanism of the global modeling unit, it simultaneously focuses on local and global dependencies, enhancing the ability to extract key features. Meanwhile, by adding explicit positional encoding to local features, the lack of sequential information in the local temporal feature matrix output by the bidirectional recurrent neural network unit is compensated for, ensuring that the global modeling unit can accurately determine the location of local features. Therefore, the method in this application can improve the accuracy of text classification. Attached Figure Description

[0072] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0073] Figure 1 A schematic diagram of the structure of the hybrid neural network for the text classification method based on the hybrid neural network provided in the embodiments of this application;

[0074] Figure 2 A flowchart illustrating the text classification method based on a hybrid neural network provided in this application embodiment;

[0075] Figure 3 A schematic diagram of the structure of a text classification device based on a hybrid neural network provided in an embodiment of this application;

[0076] Figure 4 A hardware structure diagram of the electronic device provided in this application.

[0077] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0078] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0079] The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented, for example, in orders other than those illustrated or described herein.

[0080] In this application, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0081] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0082] In today's data-driven business environment, enterprises, institutions, and research organizations face the need for efficient processing of massive amounts of unstructured text data. For example, in the financial industry, it is necessary to distinguish between customer profiles, transaction records, and market data; in the medical field, it is necessary to identify patient privacy information and clinical research data; and in government scenarios, it is necessary to distinguish between confidential documents and public information. Therefore, there is an urgent need for a text classification method that can adapt to diverse data characteristics to support core business scenarios such as data governance, privacy protection, and intelligent decision-making.

[0083] In the existing technology, text classification methods based on recurrent neural networks are widely used in the processing of unstructured text data. Its typical process includes: (1) text preprocessing (such as word embedding and denoising); (2) constructing a recurrent neural network structure, capturing the sequence dependencies of unstructured text data step by step through time steps, and obtaining the global features of unstructured text data; (3) the output layer uses the Softmax function or the Sigmoid function for text classification.

[0084] However, unstructured text data, such as government documents or medical records, often contains thousands of words. Recurrent neural networks (RNNs) are prone to introducing noise or losing important information when processing extremely long unstructured text data, leading to the neglect of key features. Therefore, RNNs have limited ability to capture long-range dependencies in extremely long unstructured text data, thus affecting the accuracy of text classification.

[0085] Starting with the problems in existing technologies, the inventors first analyzed the advantages and disadvantages of recurrent neural networks (RNNs), finding that RNNs are better at capturing short-range dependencies. They then proposed using RNNs for local feature extraction. To obtain global features based on local features, the inventors conceived of using an attention mechanism to capture global dependencies based on local features. Furthermore, to ensure compatibility between RNNs and the attention mechanism, the inventors added explicit positional encoding to local features, enabling the attention mechanism to understand and utilize the positional information of these local features. Thus, the combined use of RNNs and the attention mechanism improves the accuracy of text classification.

[0086] The technical solution of this application and how it solves the above-mentioned technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0087] This application provides a text classification method based on a hybrid neural network. Figure 1 A schematic diagram of the hybrid neural network structure of the text classification method based on hybrid neural networks provided in this application embodiment is shown below. Figure 1 As shown, the hybrid neural network in this embodiment includes a preprocessing unit, a bidirectional recurrent neural network unit, a global modeling unit, and a classification unit.

[0088] based on Figure 1 The hybrid neural network shown, Figure 2 A flowchart illustrating the text classification method based on a hybrid neural network provided in this application is shown below. Figure 2 As shown, this text classification method based on hybrid neural networks includes:

[0089] S201. Obtain the dense vector matrix of the unstructured text data; extract local features from the dense vector matrix using a bidirectional recurrent neural network unit to obtain the local temporal feature matrix of the unstructured text data.

[0090] The interpretable, dense vector matrix consists of rows that correspond to a dense vector representation of a sub-word unit in unstructured text data. The dense vector representation is a high-dimensional, continuous number used to describe the sub-word unit, which is obtained through word segmentation.

[0091] Specifically, unstructured text data to be classified is obtained, and the unstructured text data is input into the preprocessing unit of the hybrid neural network. Inside the preprocessing unit, word segmentation and word embedding are performed on the unstructured text data.

[0092] It should be noted that the preprocessing unit uses a built-in word segmenter to segment unstructured text data, resulting in a sequence of sub-word units. For example, if the obtained unstructured text data is "I love learning", the resulting sub-word unit sequence after segmentation is {"I", "love", "learning"}. It should be understood that "I", "love", and "learning" are the three sub-word units included in the sub-word unit sequence.

[0093] Furthermore, word embedding is performed on each word unit in the word unit sequence to obtain a dense vector representation of each word unit. Explain, the essence of word embedding is mapping discrete symbols (i.e., word units) to geometric relationships in a continuous vector space.

[0094] Specifically, the word embedding process for each sub-word unit is as follows: First, the target identifier corresponding to each sub-word unit is determined, where each sub-word unit has a unique target identifier. Then, the target identifier is matched with the sub-word identifiers included in the preset word embedding matrix. The preset word embedding matrix can be considered a special type of database, containing multiple different sub-word identifiers and their corresponding preset dense vector representations. Each row of the preset word embedding matrix corresponds to a preset dense vector representation of a sub-word identifier. Therefore, if a sub-word identifier that matches (i.e., is the same as) the target identifier is found in the preset word embedding matrix, the preset dense vector representation corresponding to that sub-word identifier is the dense vector representation of the sub-word unit.

[0095] After obtaining the dense vector representations of each sub-word unit, a dense vector matrix of unstructured text data is generated based on these representations. For example, assuming that the dense vector representation of the sub-word unit "I" is [h1_1, h1_2, h1_3, h1_4, h1_5, h1_6], the dense vector representation of the sub-word unit "love" is [h2_1, h2_2, h2_3, h2_4, h2_5, h2_6], and the dense vector representation of the sub-word unit "learn" is [h3_1, h3_2, h3_3, h3_4, h3_5, h3_6], then the generated dense vector matrix is:

[0096]

[0097] Clearly, each row of the dense vector matrix corresponds to a dense vector representation of a sub-word unit, arranged from top to bottom according to the position of each sub-word unit in the unstructured text data. Understandably, the number of rows in the dense vector matrix is ​​the number of sub-word units, and the number of columns is the embedding dimension of the preset word embedding matrix. In the example above, the embedding dimension of the preset word embedding matrix is ​​6, so the number of columns in the dense vector matrix is ​​also 6.

[0098] Next, the dense vector matrix is ​​input into a bidirectional recurrent neural network unit. Bidirectional recurrent neural network units excel at capturing local contextual dependencies in the input sequence; the output features of each element in the input sequence contain its local contextual information (the words preceding and following it). Therefore, this embodiment uses a bidirectional recurrent neural network unit to extract local features from the dense vector matrix.

[0099] It should be noted that the bidirectional recurrent neural network (BRNN) unit processes the dense vector matrix according to the positional order of the elements in the dense vector matrix; this positional order is the time step of the BRNN unit. Since the dense vector matrix includes the dense vector representations of each word unit in the unstructured text data, this embodiment sets each word unit as the time step of the BRNN unit. It should be understood that setting each word unit as a time step already specifies the positional order of each word unit in the unstructured text data.

[0100] It is worth noting that the bidirectional recurrent neural network unit includes a forward recurrent neural network and a backward recurrent neural network. The forward recurrent neural network processes the dense vector matrix in a forward order, while the backward recurrent neural network processes the dense vector matrix in a reverse order.

[0101] Interpretationally, for the unstructured text data "I love learning", the three sub-word units "I", "love", and "learn" are all time steps of the bidirectional recurrent neural network unit. The feedforward recurrent neural network processes the dense vector representations included in the dense vector matrix in the order of "I" → "love" → "learn", while the feedback recurrent neural network processes the dense vector representations included in the dense vector matrix in the order of "learn" → "love" → "I".

[0102] As can be seen, the forward recurrent neural network calculates the forward hidden state at each time step, while the backward recurrent neural network calculates the backward hidden state at each time step. The forward hidden state refers to the aggregated representation of the preceding context information at each time step, and the backward hidden state refers to the aggregated representation of the following context information at each time step. For example, in the three time steps "I," "love," and "learning," the forward hidden state of the "love" time step is the preceding context information obtained by feature extraction from the dense vector representation of "I," and its backward hidden state is the following context information obtained by feature extraction from the dense vector representation of "learning."

[0103] It should be noted that, since the lengths of each word unit may be inconsistent, it is necessary to perform length padding and masking on each word unit before or after the time step in which they are set as bidirectional recurrent neural network units. Length padding ensures that all word units have the same length, while masking distinguishes between real words and padded words within the padded word units. Masking generates a mask matrix based on the padded word units, which identifies the boundaries between real and padded words.

[0104] It is worth noting that the forward recurrent neural network and the backward recurrent neural network only calculate the forward hidden state and backward hidden state of the real words in the word unit corresponding to each time step.

[0105] Furthermore, the forward and backward hidden states at each time step are integrated to generate local temporal feature representations for each time step. Then, a pre-defined attention mechanism built into the bidirectional recurrent neural network unit is used to assign weights to each local temporal feature representation to enhance attention to key time steps (i.e., sub-word units). Based on the weighted local temporal feature representations, a local temporal feature matrix of the unstructured text data can be generated.

[0106] In this model, the number of rows in the local temporal feature matrix represents the number of sub-word units, and the number of columns represents the hidden state dimension of the bidirectional recurrent neural network unit. Assuming the hidden state dimension of the bidirectional recurrent neural network unit is 4, for the dense vector matrix of the unstructured text data "I love learning," after passing through a forward recurrent neural network and a backward recurrent neural network, the local temporal feature representations at each time step are obtained: the local temporal feature representation of "I" is [e1_1, e1_2, e1_3, e1_4], the local temporal feature representation of the sub-word unit "love" is [e2_1, e2_2, e2_3, e2_4], and the local temporal feature representation of the sub-word unit "learn" is [e3_1, e3_2, e3_3, e3_4]. The local temporal feature matrix is ​​as follows:

[0107]

[0108] It should be understood that the local temporal feature representation at each time step includes the contextual information of the corresponding sub-word unit.

[0109] S202. Add explicit positional encoding to each local temporal feature representation in the local temporal feature matrix, and input the local temporal feature matrix with added explicit positional encoding into the global modeling unit; based on the first preset attention mechanism built into the global modeling unit, perform global correlation modeling on the local temporal feature matrix to obtain the global feature matrix.

[0110] The interpretable local temporal feature matrix output by the bidirectional recurrent neural network unit already contains an implicit encoding of the processing order of each local temporal feature representation, i.e., the positional information of each word unit in the unstructured text data. However, this positional information cannot be understood and used by the global modeling unit. Therefore, it is necessary to add explicit positional encoding to each local temporal feature representation in the local temporal feature matrix to ensure that the global modeling unit can know the absolute position of each local temporal feature representation.

[0111] Specifically, the dimension index and number of dimensions of each local temporal feature representation in the local temporal feature matrix are determined, as well as the position index of the sub-word unit corresponding to each local temporal feature representation. Then, explicit positional encoding is calculated based on the dimension index, number of dimensions, and position index, and explicit positional encoding is added to the corresponding local temporal feature representation.

[0112] Among them, the dimension index refers to the temporal position of the local temporal feature representation in the local temporal feature matrix, and the position index refers to the position of the sub-word unit in the unstructured text data.

[0113] In one possible implementation, for local temporal feature representations where the temporal position in the local temporal feature matrix is ​​even, the formula for calculating the explicit positional encoding (PE) is as follows:

[0114]

[0115] In the formula, pos represents the position index of the sub-word unit. The dimension index representing the local temporal feature representation. This represents the number of dimensions in the local temporal feature representation.

[0116] For local temporal feature representations where the temporal position in the local temporal feature matrix is ​​odd, the calculation formula for its explicit positional encoding is as follows:

[0117]

[0118] After adding explicit positional encoding to each local temporal feature representation in the local temporal feature matrix, the local temporal feature matrix with added explicit positional encoding is input into the global modeling unit. The global modeling unit then uses its built-in preset attention mechanism to perform correlation modeling on the local temporal feature matrix.

[0119] It should be noted that this embodiment uses a transformer model as the global modeling unit to model global context dependencies for the local temporal feature matrix, thereby enhancing the capture of key local temporal feature representations. Furthermore, the global modeling unit in this embodiment uses a multi-head attention mechanism as its built-in preset attention mechanism.

[0120] Understandably, global correlation modeling enables the transformer model to capture long-distance dependencies, avoiding classification bias caused by insufficient extraction of local temporal feature representations. The parallel computing characteristics of multi-head attention significantly reduce the computational complexity of long text processing, thereby improving classification efficiency while maintaining classification accuracy.

[0121] In the process of associating and modeling local temporal feature matrices using multi-head attention, the parameter configuration of the global modeling unit can be changed in real time according to the characteristics of the local temporal feature matrices. For example, the number of multi-head attention heads and the number of layers in the transformer model can be dynamically adjusted according to the sequence length of the local temporal feature matrix (i.e., the number of local temporal feature representations).

[0122] For example, suppose the pre-set sequence length threshold is 512. If the sequence length of the local temporal feature matrix is ​​less than 512, then 4 attention heads and 2 transformer layers are enabled; if the sequence length of the local temporal feature matrix is ​​greater than or equal to 512, then 8 attention heads and 4 transformer layers are enabled.

[0123] S203. Based on the global feature matrix, output the text classification results of the unstructured text data through the classification unit.

[0124] Specifically, the global feature matrix output by the global modeling unit is input into the classification unit. In the classification unit, global pooling is used to reduce the dimensionality of the global feature matrix, generating a fixed-size global feature vector. Further, a fully connected layer combined with a softmax / sigmoid function is used to implement text classification, ultimately obtaining the text classification results for the unstructured text data.

[0125] The fully connected layer is used to perform a linear transformation on the global feature vector and generate the corresponding original score. The softmax / sigmoid function is used to convert the original score output by the fully connected layer into a probability value.

[0126] The text classification method based on hybrid neural networks provided in this application obtains a dense vector matrix of unstructured text data and sets each sub-word unit corresponding to the dense vector matrix as a time step of a bidirectional recurrent neural network unit in the hybrid neural network. Then, based on the feedforward recurrent neural network included in the bidirectional recurrent neural network unit, the forward hidden state of each time step is calculated, and based on the feedback recurrent neural network included in the bidirectional recurrent neural network unit, the backward hidden state of each time step is calculated. Based on the forward and backward hidden states of each time step, a local temporal feature matrix of the unstructured text data is generated. Next, the dimension index and dimension number of each local temporal feature representation in the local temporal feature matrix, as well as the position index of the sub-word unit corresponding to each local temporal feature representation, are determined. Explicit positional encoding is calculated based on the dimension index, dimension number, and position index, and explicit positional encoding is added to the corresponding local temporal feature representation. Further, the local temporal feature matrix with added explicit positional encoding is input into the global modeling unit in the hybrid neural network. Based on the first preset attention mechanism built into the global modeling unit, global correlation modeling is performed on the local temporal feature matrix to obtain the global feature matrix. Finally, based on the global feature matrix, the classification unit in the hybrid neural network outputs the text classification results for the unstructured text data. By employing a hierarchical hybrid neural network architecture, combining the local feature extraction capabilities of the bidirectional recurrent neural network unit with the global attention mechanism of the global modeling unit, both local and global dependencies are addressed, enhancing the extraction capability of key features. Simultaneously, by adding explicit positional encoding to local features, the missing sequence information in the local temporal feature matrix output by the bidirectional recurrent neural network unit is compensated for, ensuring that the global modeling unit can accurately determine the location of local features. Therefore, the method in this application can improve the accuracy of text classification.

[0127] Figure 3 A schematic diagram of the structure of the text classification device based on a hybrid neural network provided in the embodiments of this application is shown below. Figure 3As shown, the text classification device 300 based on a hybrid neural network includes: an acquisition module 301, an extraction module 302, a modeling module 303, and a classification module 304.

[0128] Among them, the acquisition module 301 is used to acquire the dense vector matrix of unstructured text data;

[0129] Extraction module 302 is used to extract local features from the dense vector matrix through a bidirectional recurrent neural network unit to obtain a local temporal feature matrix of unstructured text data; wherein, each row of the dense vector matrix represents the dense vector representation of each sub-word unit in the unstructured text data, and the dense vector representation represents the high-dimensional continuous numbers used to describe the sub-word unit.

[0130] Modeling module 303 is used to add explicit positional encoding to each local temporal feature representation in the local temporal feature matrix, and input the local temporal feature matrix with added explicit positional encoding to the global modeling unit; based on the first preset attention mechanism built into the global modeling unit, global correlation modeling is performed on the local temporal feature matrix to obtain the global feature matrix;

[0131] The classification module 304 is used to output the text classification results of unstructured text data through the classification unit based on the global feature matrix.

[0132] Optionally, the bidirectional recurrent neural network unit includes a forward recurrent neural network and a backward recurrent neural network;

[0133] Extraction module 302 includes:

[0134] Submodule 3021 is configured to set each word unit corresponding to the dense vector matrix as the time step of the bidirectional recurrent neural network unit;

[0135] The first calculation submodule 3022 is used to calculate the forward hidden state of each time step based on the forward recurrent neural network, and to calculate the backward hidden state of each time step based on the backward recurrent neural network; wherein, the forward hidden state represents the aggregated representation of the preceding information of the time step, and the backward hidden state represents the aggregated representation of the following information of the time step.

[0136] The first generation submodule 3023 is used to generate a local temporal feature matrix of unstructured text data based on the forward and backward hidden states at each time step.

[0137] Optionally, the first computational submodule 3022 includes:

[0138] The processing component is used to perform length padding and masking on the sub-word units corresponding to each time step to obtain the real sub-words and padded sub-words of the sub-word units;

[0139] The computational component is used to compute the forward hidden state of the real words in each time step based on a forward recurrent neural network, and to compute the backward hidden state of the real words in each time step based on a backward recurrent neural network.

[0140] Optionally, the first generation submodule 3023 includes:

[0141] An integration component is used to integrate the forward and backward hidden states of each time step to generate local temporal feature representations for each time step;

[0142] The configuration component is used to set weights for each local temporal feature representation based on the second preset attention mechanism built into the bidirectional recurrent neural network unit.

[0143] The generation component is used to generate a local temporal feature matrix of unstructured text data based on the weighted representations of each local temporal feature.

[0144] Optional, modeling module 303 includes:

[0145] The determination submodule 3031 is used to determine the dimension index and dimension number of each local temporal feature representation in the local temporal feature matrix, as well as the position index of the sub-word unit corresponding to each local temporal feature representation; wherein, the dimension index represents the temporal position of the local temporal feature representation in the local temporal feature matrix, and the position index represents the position of the sub-word unit in the unstructured text data;

[0146] The second calculation submodule 3032 is used to calculate the explicit position code based on the dimension index, the number of dimensions, and the position index;

[0147] Add submodule 3033 to add explicit positional encoding to the corresponding local temporal feature representation.

[0148] Optionally, the hybrid neural network also includes a preprocessing unit;

[0149] Module 301 includes:

[0150] The acquisition submodule 3011 is used to acquire unstructured text data;

[0151] Input submodule 3012 is used to input unstructured text data into the preprocessing unit;

[0152] Processing submodule 3013 is used for:

[0153] Based on the word segmenter built into the preprocessing unit, unstructured text data is segmented to obtain a sequence of sub-word units of unstructured text data;

[0154] Word embedding is performed on each sub-word unit in the sub-word unit sequence to obtain a dense vector representation of each sub-word unit;

[0155] The second generation submodule 3014 is used to generate a dense vector matrix of unstructured text data based on dense vector representation.

[0156] Optionally, the processing submodule 3013 includes:

[0157] The component is used to determine the target identifier corresponding to each sub-word unit included in the sub-word unit sequence;

[0158] A matching component is used to match a target identifier with sub-word identifiers included in a preset word embedding matrix; wherein, the preset word embedding matrix includes multiple different sub-word identifiers and a preset dense vector representation corresponding to each sub-word identifier;

[0159] The component is also used to determine the sub-word identifiers that successfully match the target identifier in the preset word embedding matrix, and their corresponding preset dense vector representations, as dense vector representations of the sub-word units.

[0160] The text classification device based on a hybrid neural network provided in this application can be used to execute the text classification method based on a hybrid neural network in any of the above embodiments. Its implementation principle and technical effect are similar, and will not be described again here.

[0161] It should be noted that the division of the various modules in the above device is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software via processing element calls; they can be fully implemented in hardware; or some modules can be implemented in software via processing element calls, while others are implemented in hardware. Additionally, these modules can be fully or partially integrated together, or implemented independently. The processing element here can be an integrated circuit with signal processing capabilities. During implementation, each step of the above method or each of the above modules can be completed through the integrated logic circuits in the hardware of the processor element or through software instructions.

[0162] Figure 4 A hardware structure diagram of the electronic device provided in this application. For example... Figure 4 As shown, the electronic device 400 includes:

[0163] Processor 401 and memory 402;

[0164] The memory stores the instructions that the computer executes;

[0165] The processor executes computer execution instructions stored in memory 402, causing the electronic device to perform the text classification method based on a hybrid neural network as described above.

[0166] It should be understood that the processor 401 described above can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor. The memory 402 may include high-speed random access memory (RAM), and may also include non-volatile memory (NVM), such as at least one disk storage device, and may also be a USB flash drive, external hard drive, read-only memory, disk, or optical disc, etc.

[0167] The electronic device provided in this application embodiment can be used to execute the text classification method based on hybrid neural networks provided in any of the above method embodiments. Its implementation principle and technical effect are similar, and will not be described again here.

[0168] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the text classification method based on a hybrid neural network as described above.

[0169] This application also provides a computer program product, which includes a computer program that, when executed by a processor, is used to implement the text classification method based on a hybrid neural network as described above.

[0170] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0171] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0172] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.

[0173] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0174] When integrated units / modules are implemented in hardware, the hardware can be digital circuits, analog circuits, etc. The physical implementation of the hardware structure includes, but is not limited to, transistors, memristors, etc. Unless otherwise specified, the processor can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC, etc. Unless otherwise specified, the storage unit can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc.

[0175] If the integrated unit / module is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0176] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as these combinations of technical features do not contradict each other, they should be considered within the scope of this specification.

[0177] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0178] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A text classification method based on a hybrid neural network, characterized in that, The hybrid neural network includes a bidirectional recurrent neural network unit, a global modeling unit, and a classification unit; the method includes: Obtain a dense vector matrix of unstructured text data; extract local features from the dense vector matrix using the bidirectional recurrent neural network unit to obtain a local temporal feature matrix of the unstructured text data; wherein, each row of the dense vector matrix represents the dense vector representation of each sub-word unit in the unstructured text data, and the dense vector representation represents a high-dimensional continuous number used to describe the sub-word unit; Explicit positional encoding is added to each local temporal feature representation in the local temporal feature matrix, and the local temporal feature matrix with explicit positional encoding is input to the global modeling unit; based on the first preset attention mechanism built into the global modeling unit, global correlation modeling is performed on the local temporal feature matrix to obtain the global feature matrix; Based on the global feature matrix, the classification unit outputs the text classification result of the unstructured text data.

2. The method according to claim 1, characterized in that, The bidirectional recurrent neural network unit includes a forward recurrent neural network and a backward recurrent neural network; The dense vector matrix is ​​processed by the bidirectional recurrent neural network unit to extract local features, resulting in a local temporal feature matrix of the unstructured text data, including: Each word unit corresponding to the dense vector matrix is ​​set as the time step of the bidirectional recurrent neural network unit; Based on the aforementioned forward recurrent neural network, the forward hidden state of each time step is calculated, and based on the aforementioned backward recurrent neural network, the backward hidden state of each time step is calculated; wherein, the forward hidden state represents the aggregated representation of the preceding information of the time step, and the backward hidden state represents the aggregated representation of the following information of the time step. Based on the forward and backward hidden states at each time step, a local temporal feature matrix of the unstructured text data is generated.

3. The method according to claim 2, characterized in that, Based on the feedforward recurrent neural network, the feedforward hidden state at each time step is calculated, and based on the feedback recurrent neural network, the feedback hidden state at each time step is calculated, including: Length padding and masking are performed on the sub-word units corresponding to each time step to obtain the real sub-words and padded sub-words of the sub-word units; Based on the aforementioned forward recurrent neural network, the forward hidden state of the real words in each time step is calculated, and based on the aforementioned backward recurrent neural network, the backward hidden state of the real words in each time step is calculated.

4. The method according to claim 2, characterized in that, Based on the forward and backward hidden states at each time step, a local temporal feature matrix of the unstructured text data is generated, including: Integrate the forward and backward hidden states of each time step to generate local temporal feature representations for each time step; Based on the second preset attention mechanism built into the bidirectional recurrent neural network unit, weights are set for each local temporal feature representation; based on each local temporal feature representation after setting the weights, a local temporal feature matrix of the unstructured text data is generated.

5. The method according to claim 1, characterized in that, Add explicit positional encoding to each local temporal feature representation in the local temporal feature matrix, including: The dimension index and number of dimensions of each local temporal feature representation in the local temporal feature matrix are determined, as well as the position index of the sub-word unit corresponding to each local temporal feature representation; wherein, the dimension index represents the temporal position of the local temporal feature representation in the local temporal feature matrix, and the position index represents the position of the sub-word unit in the unstructured text data; Explicit position codes are calculated based on the dimension index, the number of dimensions, and the position index, and the explicit position codes are added to the corresponding local temporal feature representations.

6. The method according to any one of claims 1 to 5, characterized in that, The hybrid neural network also includes a preprocessing unit; Obtain the dense vector matrix of unstructured text data, including: Acquire unstructured text data and input the unstructured text data into the preprocessing unit; Based on the word segmenter built into the preprocessing unit, the unstructured text data is segmented to obtain a sequence of sub-word units of the unstructured text data; Word embedding processing is performed on each sub-word unit included in the sub-word unit sequence to obtain a dense vector representation of each sub-word unit; a dense vector matrix of the unstructured text data is generated based on the dense vector representation.

7. The method according to claim 6, characterized in that, Word embedding processing is performed on each sub-word unit included in the sub-word unit sequence to obtain a dense vector representation of each sub-word unit, including: Determine the target identifier corresponding to each sub-word unit in the sub-word unit sequence; match the target identifier with the sub-word identifiers included in the preset word embedding matrix; wherein, the preset word embedding matrix includes multiple different sub-word identifiers and a preset dense vector representation corresponding to each sub-word identifier; The sub-word identifiers that successfully match the target identifier in the preset word embedding matrix, and their corresponding preset dense vector representations, are determined as the dense vector representations of the sub-word units.

8. A text classification device based on a hybrid neural network, characterized in that, The hybrid neural network includes a bidirectional recurrent neural network unit, a global modeling unit, and a classification unit; the device includes: The acquisition module is used to acquire dense vector matrices of unstructured text data; An extraction module is used to extract local features from the dense vector matrix through the bidirectional recurrent neural network unit to obtain a local temporal feature matrix of the unstructured text data; wherein, each row of the dense vector matrix represents the dense vector representation of each sub-word unit in the unstructured text data, and the dense vector representation represents a high-dimensional continuous number used to describe the sub-word unit; The modeling module is used to add explicit positional encoding to each local temporal feature representation in the local temporal feature matrix, and input the local temporal feature matrix after adding explicit positional encoding to the global modeling unit; based on the first preset attention mechanism built into the global modeling unit, global correlation modeling is performed on the local temporal feature matrix to obtain the global feature matrix; The classification module is used to output the text classification result of the unstructured text data through the classification unit based on the global feature matrix.

9. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the text classification method based on a hybrid neural network as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the text classification method based on a hybrid neural network as described in any one of claims 1 to 7.

Images