A Method for Constructing a Hierarchical Classification Model for Government Procurement Items

Abstract

This invention provides a method for constructing a hierarchical classification model for government procurement items. The method first constructs and encodes a hierarchical structure for the procurement items. Then, it performs text cleaning and word segmentation on the procurement project names, using a continuous bag-of-words model to train the corpus to obtain word vectors, or directly obtaining word vectors using a BERT Chinese pre-trained model, thus obtaining an initial text representation. Finally, the text representation and label representation are fed into the model for training. The three-layer network contained in the HA-BiGRU model can effectively solve the above problems. The text encoding layer with a BiGRU network as the encoder can further extract semantic information from the context; the hierarchical attention recurrent network layer can model the dependencies between layers and enhance the association between text and labels through a text-label attention module; the hybrid prediction layer integrates the local and global losses of hierarchical labels, optimizing the model by reducing the total loss.

Description

Technical Field

[0001] This invention relates to the field of text classification methods with a hierarchical label structure, and more specifically, to a method for constructing a hierarchical classification model for government procurement items. Background Technology

[0002] With the advent of the information age, the booming bidding industry has ushered in new opportunities and challenges. Government procurement includes goods, works, and services, and the main procuring entities are government agencies, public institutions, and social organizations at all levels. In government procurement, procurement items are a very effective research variable, capable of characterizing the procurement situation, project budget, and funding of various categories from different dimensions and levels, providing valuable information for procurement decisions. However, procurement items are numerous and difficult to distinguish. Associating appropriate categories through procurement project information facilitates hierarchical retrieval. Hierarchical management of procurement items can save procurement personnel significant time and effort, optimizing office processes. Therefore, using mainstream natural language processing technology to predict corresponding procurement items from textual information in procurement projects is of great significance. It not only fills in item information, providing complete data for subsequent statistical analysis, but also simplifies the office processes of government procurement personnel and improves office efficiency.

[0003] In multi-label classification problems, transforming the problem into multiple single-label binary classification problems is a common approach. However, this can lead to decreased model performance when the number of label types is too large. Therefore, for multi-label classification, there are two main ways to represent the relationships between labels. One is to flatten the labels, i.e., globalize the classification, where all labels belong to the same level. The other is to establish hierarchical labels, i.e., localize the classification, where labels have a hierarchical relationship from top to bottom. In most cases, flattening the labels is the more common method, such as decision trees and Naive Bayes, which directly predict the labels at the last layer. However, with large datasets, flattening the hierarchical labels can drastically increase the number of categories, increasing sparsity and leading to poor prediction results. Therefore, fully utilizing the semantic relationships of the text, the hierarchical structure between labels, and the relevance between labels and text is a technical problem that those skilled in the art need to solve.

[0004] In 2001, Sun and Lim proposed a top-down Support Vector Machine (SVM) classification method. Venset et al. proposed the Clus-HMCL model, which predicts hierarchical labels based on decision trees. In 2017, Liu et al. proposed a method to address extreme imbalance in label categories, using dynamic max pooling, binary cross-entropy loss, and hidden layers to improve overall performance. Wehrman's hierarchical label classification network effectively integrates label localization and label globalization. Generally, upper-level labels have fewer categories and higher discriminative power. If information learned from upper-level labels can be applied to the classification of lower-level labels, prediction efficiency will be improved. Therefore, in subsequent research, researchers have focused on maximizing the relationships between hierarchical labels, and have mostly introduced attention mechanisms to capture the semantic relationships in the text, the hierarchical structure between labels, and the relevance between labels and text. Zichao Yang proposed using hierarchical attention to capture the relationships between words and sentences. Irsan proposed using CNN (Convolutional Neural Networks) to build a top-level classifier, combining vocabulary and semantics by calculating the average value of word vectors and statistical word frequencies to improve model performance. KShimura proposed the HFT-CNN model, which uses a convolutional neural network (CNN) as the base classifier to build a hierarchical label system. It passes the parameters of upper-layer labels to lower-layer labels for fine-tuning, effectively utilizing the correlation between upper and lower-layer labels. S Lai proposed the Recurrent Convolutional Neural Network (RCNN) model, using a recurrent structure to learn contextual information as much as possible. Previous research has rarely considered the relationship between text and labels. Wehrmann et al. constructed a multi-output neural network model to optimize the global and local structure of hierarchical label classification. Each layer of this model has a local output, and a local loss function applies to the gradient backpropagation of the corresponding layer. The global output represents the cumulative result of the entire network and performs gradient backpropagation across all layers. Building on previous research, Wei Huang proposed the HARNN model, which models the dependencies between different layers in a top-down manner, and then uses an attention mechanism to capture the relationship between text and the hierarchical structure. Mu Yixiang invented a method and apparatus based on obtaining a hierarchical classifier and text classification. The method includes establishing a hierarchical category structure of the hierarchical classifier; labeling training samples according to the hierarchical category structure; extracting feature information of training samples; and at each trunk node, using the labeled data of the corresponding trunk node and the feature information of the training samples, training and obtaining the classifier model of the corresponding trunk node through a classifier trainer, and forming a hierarchical classifier with a hierarchical category structure with the corresponding classifier model of each trunk node.Yang Han et al. invented a method for constructing a news text classification model based on a hierarchical multi-label system. This method can adaptively determine the weight of each word, increase the confidence of different hierarchical categories, and reduce the adverse effects caused by inaccurate manual labeling. Summary of the Invention

[0005] This invention provides a method for constructing a hierarchical classification model for government procurement items. The method has a reasonable computational efficiency, simplifies the office process for government procurement personnel, and improves office efficiency.

[0006] To achieve the above-mentioned technical effects, the technical solution of the present invention is as follows:

[0007] A method for constructing a hierarchical classification model for government procurement items includes the following steps:

[0008] S1: Construct a tree structure for hierarchical labels of government procurement items and encode the labels;

[0009] S2: Perform text cleaning and word segmentation on the name of the government procurement project, obtain word vectors after word segmentation, and concatenate the word vectors to obtain a word vector matrix;

[0010] S3: Design a general attention model and a hierarchical attention model. Input the encoded representation of the text and the encoded representation of the label into the general attention model and the hierarchical attention model respectively for classification to obtain local classification results and global classification results.

[0011] Further, in step S2, the government procurement project name is subjected to text cleaning and word segmentation to obtain the word vectors of the segmented text, including the following steps:

[0012] 1) Divide the text into a logical sequence of words;

[0013] 2) The word segmentation results are represented by word embedding and text encoding, mapping the sequence into a fixed-length vector; the vector can well represent the features of the sequence;

[0014] 3) The BERT model is used to embed words into the segmented results. BERT is based on a bidirectional Transformer encoder and uses random masking word syntax to achieve pre-trained deep bidirectional representation. Furthermore, character-level word vectors and model parameters are obtained by training on corpora from encyclopedias.

[0015] Furthermore, in step S3, the hierarchical attention model has a three-layer structure: a document encoding layer, a hierarchical attention recurrent network layer, and a hybrid prediction layer; the document encoding layer encodes the text and hierarchical category structure using an embedding layer, and then uses a Bi-LSTM model or a Bi-GRU model to further enhance the semantic relationships of the text; the hierarchical attention recurrent network layer models the dependencies between different levels step by step using the hierarchical structure in a top-down manner; at each category level, the hierarchical attention memory unit is repeated to capture the association between the text and the category.

[0016] Furthermore, in step S3, the text and hierarchical category structure are encoded using an embedding layer, and then the semantic relationships of the text are further enhanced using a Bi-LSTM model or a Bi-GRU model, including the following steps:

[0017] The Bi-LSTM is a combination of forward LSTM and backward LSTM; LSTM learns and utilizes the sequence dependencies of data, and selects to remember and forget some information by controlling the output of the input gate, forget gate and output gate; Bi-LSTM learns longer sequence information and can also integrate information from the preceding and following context.

[0018] The principle of Bi-GRU is the same as that of Bi-LSTM. It uses a GRU network with opposite directions to extract text features and then concatenates them. GRU selects to remember and forget certain information by controlling the output of the reset gate and update gate.

[0019] The semantic enhancement matrix of the text is obtained through the Bi-LSTM or the Bi-CRU. The labels of each layer are embedded, an initial embedding matrix is ​​randomly generated, and then the obtained semantic enhancement matrix and the initial embedding matrix are input into the hierarchical attention recurrent network layer.

[0020] Furthermore, the hierarchical attention memory unit comprises three modules: a text-label attention module, a classification prediction module, and a hierarchical dependency module.

[0021] The text-label attention module is used to capture the correlation between text and hierarchical categories, and to obtain the relationship representation between hierarchical category labels and text, and the attention matrix between hierarchical category labels and text;

[0022] The classification prediction module is used to integrate the semantic representation of the original text and the text category representation that incorporates information from the previous level, to generate a unified representation for each level and perform category prediction.

[0023] The hierarchical dependency module is used to model the dependency relationships between different levels by preserving hierarchical information for each level.

[0024] Furthermore, for the category labels in layer h, different categories contribute differently to the prediction results, thus allowing for the correction of the attention matrix:

[0025] For the h-th level of the hierarchical category structure, the input to the text-label attention module is the full text representation V and the corresponding level representation S. h and the weight parameter w passed from the previous level. h-1 :

[0026]

[0027]

[0028]

[0029]

[0030] The average embedding of N words is obtained using average pooling. This represents the relationship between category labels and text at the h level. It is the attention matrix between the category labels and text at the h layer. This is the class prediction output vector of the h-th layer. Since the root node, i.e., the 0th layer, does not contain any information, the weight parameters w of the 0th layer are initialized with the element 1. 0 .

[0031] Furthermore, the goal of the text-label attention module is to capture the correlation between text and hierarchical categories, and to output a representation of the relationship between hierarchical category labels and text. Attention matrix between hierarchical category labels and text Weight parameters of the previous level Multiplying it by the text representation V, we get:

[0032] V h =w h-1 .V

[0033] V h This text representation incorporates information from the previous level. Because different words in a text often express different meanings, we need Num for the h-th level label. h The semantic relationships of the entire text are represented by vectors of dimension S, using the embedding matrix S of the h-level tags. h To calculate the attention for the corresponding category:

[0034]

[0035]

[0036] It is a randomly generated weight matrix, O h This represents V after activation using the tanh function. h . W i h Let represent the normalized text attention score for the i-th category in layer h, where each element in the vector represents the contribution of the word at the corresponding position to the i-th category; then the attention matrix... With text representation V h Multiply, we get The row average of Mh is obtained

[0037]

[0038] The classification prediction module aims to integrate the semantic representation of the original text with the text category representation that incorporates information from the previous level, generating a unified representation for each level and performing category prediction. As a representation of the h-th layer It is a randomly initialized weight matrix. That is the corresponding error vector. Represents a nonlinear activation function; It is a weight matrix. σ is the corresponding error vector, and σ is the sigmoid function.

[0039] Furthermore, the hierarchical dependency module models the dependencies between different levels by preserving hierarchical information for each level. For the category label of the h-th level, different categories contribute differently to the prediction results, thus allowing for the correction of the attention matrix:

[0040]

[0041]

[0042] This represents the corrected attention matrix, and the `broadcast` function represents tiling the vector along a certain dimension. By performing row averaging, we obtained the weighted attention vector. right By tiling, we obtain the association weight matrix w between the text and the h-th layer labels. h :

[0043]

[0044] This represents the association weight between the i-th word and the hierarchical information.h This will also be used as input to the attention memory unit of the next level; after passing through the classification prediction module of each layer, the corresponding hierarchical representation vector can be obtained. and prediction probability vector For hierarchical label classification, predicting both local and global labels is important; a hybrid prediction layer is constructed, which uses both local and global information for the final output, and the hierarchical representation vector of each layer is used. Aggregate, obtain Then for A L Calculate the row average Let P G The output represents the probability of global prediction:

[0045]

[0046] P G =σ(W M ·A G +b M )

[0047] b G and b M Each of these is an error vector in its corresponding dimension. P G It is a continuous vector, and the element P at the i-th position is... G (i)=p(c i |x), indicating that it belongs to the label set. The conditional probability of the i-th label is calculated; the conditional probabilities of local and global predictions are integrated to obtain the final prediction output P. F :

[0048]

[0049] α is a parameter that determines the proportion of global and local information, and can be adjusted according to the actual prediction results. After passing through the document encoding layer, hierarchical attention recurrent network layer, and hybrid prediction layer, the final model structure is obtained:

[0050] The loss function is defined as the sum of the local loss and the global loss. During the model training phase, for each document, the local classification loss is calculated. and global classification loss The calculation method is as follows:

[0051] Y G It is a binary vector containing all the labels within the hierarchy. It is a binary vector containing only all labels of the h layer, Y G and Representing the true value of the label category, ε(.,.)=yi log(p(y i ))+(1–y i )log(1-p(y i () represents the binary cross-entropy function, and the optimization objective of the model is to minimize the sum of the global loss and the local loss.

[0052]

[0053] Θ represents all parameters of the model, λ Θ The regularization parameter is obtained using the Adam optimization algorithm. Minimization operation.

[0054] Furthermore, the construction process of the general attention model is as follows:

[0055] D = {w1, w2, ..., w N} represents the text after word embedding:

[0056]

[0057]

[0058]

[0059] Considering that words have varying degrees of influence on sentences, a word attention mechanism was added to score the attention of each word, resulting in an attention-weighted sentence vector:

[0060] u i =tanh(W o h i +b o )

[0061]

[0062]

[0063] The final prediction vector is:

[0064] p = softmax(W c s+b c )

[0065] Use the negative log-likelihood of the correct label corresponding to the sample as the training loss:

[0066]

[0067] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0068] The method of this invention first constructs and encodes a hierarchical structure for the procurement items, then performs text cleaning and word segmentation on the procurement item names, and uses a continuous bag-of-words model to train the corpus to obtain word vectors or directly obtains word vectors using a BERT Chinese pre-trained model to obtain the initial text representation. Finally, the text representation and label representation are fed into the model for training. The three-layer network contained in the HA-BiGRU model can effectively solve the above problems. The text encoding layer with BiGRU network as encoder can further extract semantic information of the context, the hierarchical attention recurrent network layer can model the dependencies between layers, and enhance the association between text and labels through the text-label attention module, and the hybrid prediction layer integrates the local loss and global loss of hierarchical labels, optimizing the model by reducing the total loss. Experiments show that this invention achieves good performance on various evaluation metrics. Attached Figure Description

[0069] Figure 1 This is a schematic diagram of the hierarchical structure encoding of the present invention;

[0070] Figure 2 This is a diagram of the Bert word vector learning network of the present invention;

[0071] Figure 3 This is a diagram showing the overall structure of the hierarchical attention model HA-BiRNN of this invention;

[0072] Figure 4 This is a schematic diagram of the document representation layer network of the hierarchical attention model of the present invention;

[0073] Figure 5 This is a schematic diagram of the hierarchical attention memory unit in the hierarchical attention recurrent network of the hierarchical attention model of the present invention;

[0074] Figure 6 This is a detailed diagram of the hierarchical attention memory unit in the hierarchical attention recurrent network of the hierarchical attention model of the present invention;

[0075] Figure 7 This is a diagram showing the overall structure of the A-BiRNN, a common attention contrast model of this invention.

[0076] Figure 8 The following is a flowchart of an experimental procedure for a preferred embodiment of the present invention. Detailed Implementation

[0077] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this patent.

[0078] To better illustrate this embodiment, some parts in the accompanying drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions;

[0079] It will be understood by those skilled in the art that certain well-known structures and their descriptions may be omitted in the accompanying drawings.

[0080] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments:

[0081] This invention proposes a method for constructing a text classification model based on hierarchical attention, using the names of government procurement projects to classify procurement items. Ordinary attention models directly use Bi-LSTM or Bi-GRU models to encode and extract features from the text matrix, adding an attention mechanism between words, and outputting multi-label classification results through model decoding. The hierarchical attention model, however, uses Bi-LSTM or Bi-GRU models to further extract semantics from the text matrix, adding a text-hierarchical attention module, and improving model performance by optimizing local and global classification losses. Its main steps include:

[0082] S101. Perform text cleaning and word segmentation on the names of government procurement projects, use the CBOW model or the Bert Chinese pre-trained model to obtain word vectors, and concatenate the word vectors to obtain the word vector matrix of each text.

[0083] In a hierarchical labeling structure, the type of each item is determined by the type of its parent item. The structure between hierarchical label categories can use a tree structure or a directed acyclic graph (DAG) structure; this invention uses a tree structure.

[0084] Multi-label classification refers to a sample belonging to more than one label category; for example, natural sciences can include categories such as mathematics, physics, and chemistry. Hierarchical label classification is multi-label classification with a hierarchical structure. See the appendix for visualization of hierarchical structure coding. Figure 1 The height of the hierarchical structure is defined as H, meaning there are H levels of tags. The tag structure is as follows: C 1 The first-level label, C H The label represents the last layer. Let Num be the set of labels for the i-th layer. i This represents the total number of categories for the labels at level i. Define Num. all The total number of categories for all labels. A hierarchical category structure is defined based on structure C, containing a partially ordered set (C, <).

[0085] Documents and their corresponding tag hierarchy are represented by D (Document) and T (Tree), respectively, and the input set is x = (D1, T1), (D2, T2), ..., (D M ,T M Each document consists of a sequence of words of a certain length, represented as D. i={w1,w2,…,w N Each hierarchical structure is represented as T. i ={l1,l2,…,l H}, where l i ∈C i That is, label l i It belongs to the i-th layer structure.

[0086] The label structure of the i-th layer is C i The total number of categories is Num i The label vector of this layer is constructed using one-hot encoding, and its length is equal to the total number of categories. The overall label vector is encoded in the same way. Hierarchical label vector V label ={v l 1 ,v l 2 ,…,v l H ,v l all This vector is ultimately used to calculate the classification loss and evaluate the classification performance.

[0087] The Jieba word segmenter is used to divide the text into reasonable word sequences. Before segmentation, regular expression matching is used to filter out numbers and punctuation marks from the text. The length of each document is fixed at N, w i Representing the i-th word in the text sequence, document D i ={w1,w2,…,w N If the length of the text sequence is less than N, pad it to N. If the length of the text sequence is greater than N, truncate it at the Nth word.

[0088] This invention uses two methods for word embedding. One method involves training the corpus using a CBOW model, resulting in word vectors that contain contextual information about the purchase name data. The second method directly uses the BERT Chinese pre-trained model to obtain word vectors as the initial word embeddings.

[0089] The bag-of-words model is a simple neural network model. The vocabulary size is set to S, the hidden layer size to U, and the connectivity to be fully connected. The model's input consists of words encoded using one-hot methods from the vocabulary, {x1, x2, ..., x...}. S Only the value at the position corresponding to the word is 1, and the rest are 0.

[0090] The CBOW model maps words to a geometric space, using geometric relationships to reflect the semantic connections between words. Unlike one-hot encoding, the CBOW model learns information from the corpus, making the word embeddings more suitable for our target task. (Document D) i ={w1,w2,…,w N After word vector transformation, the text is represented as a matrix. Assume the dimension of the word vectors is u. The CBOW model is trained on local corpora, resulting in high optimization efficiency.

[0091] See the appendix for the Bert word vector learning network. Figure 2 The BERT model is based on the Transformer and is trained using a large-scale unlabeled corpus. Through pre-training, it obtains semantic representations of text containing rich semantic information.

[0092] The Transformer is generally an encoder-decoder architecture. The encoder consists of six identical layers, each containing two sub-layers: a multi-head self-attention layer and a fully connected layer. Each sub-layer is residually connected, and the output dimension of each layer is 512. The decoder also consists of six identical layers, but adds a multi-head attention layer to process the encoder's output.

[0093] Self-attention mechanisms are used to determine the influence of context on input words. First, three vectors are generated based on the encoder's input vector: a query vector, a key vector, and a value vector. These three vectors are then mapped to obtain the output. The dimensions of the input queries and all keys are d. k The dimension of values ​​is d v Take the dot product of the query and all keys, then divide by . The weights of all values ​​are obtained by applying a softmax function. The Transformer is better able to capture bidirectional relationships in statements.

[0094]

[0095] BERT is based on a bidirectional Transformer encoder and uses random masking lexical syntax to achieve pre-trained deep bidirectional representations. It also obtains character-level word vectors and model parameters by training on corpora from encyclopedias. BERT's input includes character encoding, sentence encoding, and positional encoding.

[0096] S102. Design a general attention model A-BiRNN and a hierarchical attention model HA-BiRNN. Input the encoded representations of the text and labels into the general attention model and the hierarchical attention model respectively for classification, obtaining local classification results and global classification results. Use relevant evaluation metrics to assess the model's performance.

[0097] The overall framework of the model of this invention is shown in the appendix. Figure 3 It mainly consists of three parts: a document encoding layer, a hierarchical attention recurrent network layer, and a hybrid prediction layer. This invention uses a lightweight Bi-GRU network to replace the Bi-LSTM network in the network structure, reducing network parameters and mitigating overfitting.

[0098] The hierarchical attention model document representation layer network of this invention is shown in the appendix. Figure 4 First, a document embedding layer is applied to encode the text and hierarchical category structure separately. The segmented text is then embedded to obtain a vector D = {w1, w2, ..., w...}. N Then, a Bi-LSTM model or a Bi-GRU model is used to further enhance the semantic relationships of the text.

[0099] LSTM can learn and utilize the sequence dependencies of data, selectively remembering and forgetting certain information by controlling the outputs of the input, forget, and output gates. The output of each node depends on the most recent context in the input sequence, rather than the initial input entering the network. Bi-LSTM can better learn bidirectional semantic dependencies.

[0100] Let input represent the input gate, forget represent the forget gate, and output represent the output gate; C represents the cell state, and h is the output of the current cell state. This represents the updated cell state. σ represents the sigmoid function. The cell state is continuously updated as the network moves forward, and the model parameters are continuously updated through backpropagation of errors.

[0101] forget t =σ(W forget ·[h t-1 ,x t ]+b forget )

[0102] input t =σ(W input ·[h t-1 ,x t ]+b input )

[0103]

[0104]

[0105] output t =σ(W output ·[h t-1 ,x t ]+b output )

[0106] h t =output t *tanh(C t )

[0107] Bi-LSTM combines forward and backward LSTM, enabling it to learn longer sequence information and integrate contextual information. Taking a document as an example, the original input of the Bi-LSTM network is D = {w1, w2, ..., w...} N The hidden layer vectors of the network are calculated as follows:

[0108]

[0109]

[0110]

[0111] and These are the hidden layer outputs of the nth word in the forward LSTM and backward LSTM, respectively. It is a combination of the two, where u is the number of hidden nodes in the unidirectional LSTM. The final representation of the document is as follows:

[0112] GRU is a simplified version of LSTM, also known as a gated recurrent unit structure, containing a reset gate and an update gate. The update gate determines how much information from the previous hidden layer is selected for the current hidden layer. xt and ht-1 represent the model input and the output of the previous hidden layer, respectively. Wu is the parameter of the update gate, and Wr is the parameter of the reset gate. C represents the cell state, and h is the output of the current cell state. This represents the updated cell state. σ represents the sigmoid function. GRU uses the sigmoid function to update and filter information. The sigmoid function discards invalid information and retains important information. The reset gate determines the degree of forgetting of historical information, i.e., the importance of the output of the previous hidden layer.

[0113] G u =σ(W u ·[h t-1 ,x t ]+b u )

[0114] G r =σ(W r ·[h t-1 ,x t ]+b t )

[0115]

[0116]

[0117] h r =C t

[0118] The GRU model has fewer components and significantly fewer training parameters than the LSTM model, which reduces overfitting and improves model efficiency. The principle of Bi-GRU is the same as that of Bi-LSTM: it uses GRU networks in opposite directions to extract text features and then concatenates them.

[0119]

[0120]

[0121]

[0122] and These are the hidden layer outputs of the nth word in the forward GRU and backward GRU, respectively. It is a combination of the two, where u is the number of hidden nodes in a one-way GRU. The final representation of the document is as follows:

[0123] The semantic enhancement matrix V = {h1, h2, ..., h} of the text is obtained through Bi-LSTM or Bi-CRU. N For each layer of labels, embedding is performed, and an initial embedding matrix S = {S} is randomly generated. 1 ,S 2 ,…,S H},in da represents the dimension of the embedding matrix of the labels in the i-th layer. V and S are input to a hierarchical attention-based recurrent network layer, which outputs the final prediction result.

[0124] The hierarchical attention model of this invention, the hierarchical attention recurrent network, and the hierarchical attention memory unit are shown in the appendix. Figure 5 For a detailed explanation of the hierarchical attention model, hierarchical attention recurrent network, and hierarchical attention memory unit of this invention, please refer to the appendix. Figure 6It models the dependencies between different levels in a top-down, hierarchical manner. At each category level, hierarchical attention memory units are repeated to capture the association between text and categories. Simultaneously, the network can pass relevant hierarchical semantic information to the next layer, conforming to people's reading habits from shallow to deep.

[0125] The hierarchical attention memory unit comprises three modules: a text-label attention module, a classification prediction module, and a hierarchical dependency module. The text-label attention module calculates the association value between the text and the hierarchical label; the classification prediction module generates a unified representation and prediction of the corresponding hierarchical category; and the hierarchical dependency module models the dependencies between different hierarchical categories. The text-label attention module is represented by TCA (Text-Category Attention), the classification prediction module by CPM (ClassPrediction Module), and the hierarchical dependency module by CDM (Class Dependency Module).

[0126] For the h-th level of the hierarchical category structure, the input to the text-label attention module is the full text representation V and the corresponding level representation S. h and the weight parameter w passed from the previous level. h-1 .

[0127]

[0128]

[0129]

[0130]

[0131] The average embedding of N words is obtained using average pooling. This represents the relationship between category labels and text at the h level. It is the attention matrix between the category labels and text at the h layer. This is the class prediction output vector of the h-th layer. Since the root node, i.e., the 0th layer, does not contain any information, the weight parameters w of the 0th layer are initialized with the element 1. 0 .

[0132] The goal of the text-label attention module is to capture the correlation between text and hierarchical categories, and output a representation of the relationship between hierarchical category labels and text. Attention matrix between hierarchical category labels and text Weight parameters of the previous level Multiplying it by the text representation V, we get:

[0133] V h =w h-1 .V

[0134] V h This is a text representation that incorporates information from the previous level. Because different words in text often express different meanings, we need Num for the h-th level label. h The semantic relationships of the entire text are represented by vectors of dimension S. The embedding matrix S of the h-level tags is used. h To calculate the attention for the corresponding category.

[0135]

[0136]

[0137] It is a randomly generated weight matrix, O h This represents V after activation using the tanh function. h . W i h This represents the normalized text attention score for the i-th category in the h-th layer, where each element in the vector represents the contribution of the word at the corresponding position to the i-th category.

[0138] Attention matrix With text representation V h Multiply, we get The row average of Mh is obtained

[0139]

[0140]

[0141] The classification prediction module aims to integrate the semantic representation of the original text with the text category representation that incorporates information from the previous level, generating a unified representation for each level and performing category prediction. Using... As a representation of the h-th layer

[0142] It is a randomly initialized weight matrix. That is the corresponding error vector. This represents a non-linear activation function.

[0143] It is a weight matrix. σ is the corresponding error vector, and σ is the sigmoid function.

[0144] The hierarchical dependency module models the dependencies between different levels by preserving hierarchical information for each level. For the category label of the h-th level, different categories contribute differently to the prediction results, thus allowing for the adjustment of the attention matrix.

[0145]

[0146]

[0147] This represents the corrected attention matrix, and the `broadcast` function represents tiling the vector along a certain dimension. By performing row averaging, we obtained the weighted attention vector.

[0148] right By tiling, we obtain the association weight matrix w between the text and the h-th layer labels. h

[0149]

[0150] This represents the association weight between the i-th word and the hierarchical information. h It will also be used as input to the next level of attention memory unit.

[0151] After passing through the classification prediction module at each layer, the corresponding hierarchical representation vector can be obtained. and prediction probability vector For hierarchical label classification, predicting both local and global labels is important. Therefore, we constructed a hybrid prediction layer, using both local and global information for the final output. This involves creating a hierarchical representation vector for each layer. Aggregate, obtain Then for A L Calculate the row average Let P G This represents the probability output of the global prediction.

[0152]

[0153] P G =σ(W M ·A G +b M )

[0154] b G and b M Each of these is an error vector in its corresponding dimension. P G It is a continuous vector, and the element P at the i-th position is... G(i)=p(c i |x), indicating that it belongs to the label set. The conditional probability of the i-th label.

[0155] By integrating the conditional probabilities of local and global predictions, the final prediction output P is obtained. F

[0156]

[0157] α is a parameter that determines the proportion of global and local information, and can be adjusted according to the actual prediction situation.

[0158] After passing through the document encoding layer, the hierarchical attention recurrent network layer, and the hybrid prediction layer, the final model structure is obtained.

[0159] The loss function of this invention is defined as the sum of the local loss and the global loss. During the training phase of the model, for each document, the local classification loss is... and global classification loss The calculation method is as follows:

[0160]

[0161]

[0162] Y G It is a binary vector containing all the labels within the hierarchy. It is a binary vector containing only all the labels of the h layer. Y G and This represents the actual value of the label category. ε(.,.)=y i log(p(y i ))+(1–y i )log(1-p(y i )) represents the binary cross-entropy function. The optimization objective of the model is to minimize the sum of the global loss and the local loss.

[0163]

[0164] Θ represents all parameters of the model, λ Θ These are the regularization parameters. The Adam optimization algorithm is used. Minimization operation.

[0165] The overall structure of the ordinary attention contrast model A-BiRNN of this invention is shown in the appendix. Figure 7 For convenience, this invention will refer to both Bi-LSTM and Bi-GRU as BiRNN. This invention does not consider the hierarchical structure of multi-label systems and constructs a BiRNN model based on word-level attention, namely A-BiRNN.

[0166] D = {w1, w2, ..., w N} represents the text after the word is embedded.

[0167]

[0168]

[0169]

[0170] Considering that words have varying degrees of influence on sentences, this invention incorporates a word attention mechanism. Each word is scored for attention, resulting in an attention-weighted sentence vector.

[0171] u i =tanh(W o h i +b o )

[0172]

[0173]

[0174] The final prediction vector is:

[0175] p = softmax(W c s+b c )

[0176] Use the negative log-likelihood of the correct label for the sample as the training loss:

[0177]

[0178] The experimental procedure for preferred examples of this invention is shown in the appendix. Figure 8 This example uses government procurement data, sourced from all government procurement announcements on the China Government Procurement Network (http: / / www.ccgp.gov.cn / ) from 2018 to 2019. Procurement projects with a single source will specify the project name, procurement source, procurement amount, winning supplier, price details, review date, review comments, and contact information. Tendered procurement, limited competitive procurement, and competitive negotiation will provide a brief description of the supplier qualification requirements.

[0179] After text cleaning techniques such as regular expression matching, the data features retained by this invention are the announcement title, procurement project name, item, procuring unit, administrative region, and announcement time.

[0180] This experiment sets the text length to a fixed value of 20, i.e., N = 20, and the document representation is D = {w1, w2, ..., w NTwo methods were used to convert words into word vectors. The first method used the CBOW model, trained on the corpus of all project procurement names. A word frequency cutoff value of 2 was set, excluding words appearing less than twice from the training scope. The context window size was 5, and the output word vector dimension was 300. The second method used the BERT Chinese pre-trained model (chinese). L -12 H -768 A -12, select the second-to-last layer vector of the output layer as the word vector, and the dimension of the output word vector is 768.

[0181] This experiment uses As an evaluation metric, in the confusion matrix of the PR (Precision Recall) curve, TP (True Positive) indicates that positive examples are correctly classified as positive examples, FN (False Negative) indicates that positive examples are incorrectly classified as negative examples, TN (True Negative) indicates that negative examples are incorrectly classified as positive examples, and FP (False Positive) indicates that negative examples are correctly classified as negative examples.

[0182]

[0183]

[0184] Since the problem to be solved by this invention is a multi-label classification problem, therefore, for all labels The i-th type of label in the TP is defined. i FP i FN i TN i To calculate precision and recall

[0185]

[0186]

[0187] The Micro-Average F1 score is also a method for evaluating multi-label classification models, and it effectively handles the label imbalance problem. Assuming there are M samples, y i j =1 indicates that the j-th sample truly belongs to the i-th label; otherwise, y = 1. i j =-1. Similarly, This indicates that the predicted j-th sample belongs to the i-th label class; otherwise...

[0188]

[0189] Therefore, Micro-F1 can be represented by two metrics: Micro-Precision and Micro-Recall.

[0190]

[0191] The higher the values ​​of the Micro-Precision, Micro-Recall, and Micro-F1 metrics, the better the model's predictive performance. The values ​​of the first three evaluation metrics adjust continuously with the threshold, so the area under the average PR curve is also a good indicator, often simply referred to as... The higher the value of this indicator, the better the model performs.

[0192] This experiment compares the model's performance in three aspects:

[0193] First, regarding the methods of obtaining word vectors, this experiment employed two approaches: the first was using the CBOW model to train the corpus, and the second was directly using the BERT Chinese pre-trained model. We are interested in the impact of these two word embedding methods (i.e., the accuracy of word vectors in semantic representation) on the model. We examine whether the word vectors obtained using the BERT Chinese pre-trained model have similar performance to those obtained through corpus training.

[0194] Second, regarding the Bi-LSTM and Bi-GRU encoding methods, both are derivative models of recurrent neural networks (RNNs). Each has its advantages: Bi-LSTM has a complex structure but can handle long texts well, while Bi-GRU has a simple structure and high computational efficiency. This experiment compared the semantic representation performance and computational complexity of the two text encoding networks, selecting the network model more suitable for short texts with large amounts of data.

[0195] Third, regarding hierarchical attention and ordinary attention. Ordinary attention mechanisms are limited to interactions between words or sentences in text. This invention proposes a hierarchical label-text attention structure in the model. Therefore, a control experiment was designed, with one group using the A-BiRNN model with simple inter-word attention and the other group using the HA-BiRNN model with hierarchical text-label attention. This invention demonstrates through this experiment that hierarchical attention significantly improves the performance of text classification tasks with hierarchical label structures. The eight models used in this experiment are shown in the table below:

[0196] CBOW Bert A-BiLSTM-CBOW A-BiLSTM-Bert A-BiGRU-CBOW A-BiGRU-Bert HA-BiLSTM-CBOW HA-BiLSTM-Bert HA-BiGRU-CBOW HA-BiGRU-Bert

[0197] In this experiment, the parameter settings for the A-BiRNN model were consistent with those for Bi-LSTM and Bi-GRU. For vectors obtained using CBOW, the word vector dimension was set to 300; for vectors obtained using CBOW, the word vector dimension was set to 768. The BiRNN hidden layer dimension was 256, the fully connected layer dimension was 512, and the attention layer dimension was 40. Each batch of training data consisted of 256 data points.

[0198] In this experiment, the parameter settings for the HA-BiRNN model were consistent with those for Bi-LSTM and Bi-GRU. For vectors obtained using CBOW, the word vector dimension was set to 300; for vectors obtained using CBOW, the word vector dimension was set to 768. The dimension d of the label vector representation was also set. a =200, the dimension of the BiRNN hidden layer is 256, and the dimension of the fully connected layer is 512. The global information retention rate α = 0.5. The amount of data trained in each batch is 256. The experimental results of the 8 models in this experiment are shown in the table below:

[0199]

[0200]

[0201] By comparing the experimental results in the table, the following conclusions can be drawn:

[0202] The word embedding results of CBOW and BERT pre-trained models are similar. The final prediction results of BERT and CBOW word embedding methods are close, verifying that the word vectors obtained by the BERT Chinese pre-trained model can also meet the requirements of this experiment. Both HA-BiGRU-Bert and HA-BiGRU-CBOW achieve good results in terms of performance metrics. It achieved an optimal value of 0.708, while also achieving an F1 score of 0.623. HA-BiLSTM-CBOW both achieved excellent results on the F1 score scale. The value reached 0.707, and the F1 score was 0.625. In the ABiRNN model, the A-BiGRU-Bert model achieved a score of 0.625 on the F1 score. The value reached 0.697, and the F1 score reached 0.590. The A-BiGRU-CBOW model achieved this on the F1 score. The A-BiGRU model achieved a score of 0.697 and an F1 score of 0.599. Furthermore, the A-BiLSTM model outperformed the A-BiGRU model in both aspects.

[0203] Hierarchical attention outperforms ordinary attention. Since BERT word embeddings and CBOW word embeddings have similar performance, this experiment uses the results of CBOW word embeddings for demonstration and analysis. Among the A-BiRNN models, A-BiGRU-CBOW performs best, with the following metrics... The score is 0.697, and the F1 score is 0.582. Therefore, it can be concluded that... In terms of both the F1 score and the F1 score, the hierarchical attention model HA-BiRNN of this invention significantly outperforms the ordinary attention model A-BiRNN.

[0204] The Bi-GRU structure outperforms the Bi-LSTM structure. The time complexity of each model shows that models based on GRU networks have lower time complexity than those based on LSTM networks. HA-BiLSTM completes one epoch in 3020 seconds, HA-BiGRU in 1755 seconds, and A-BiLSTM in 1416 seconds, with a total epoch time of 631 seconds. While the HA-BiGRU model slightly outperforms HA-BiLSTM in classification prediction, it reduces time complexity by half. Therefore, choosing the BiGRU network as the text encoding layer in this invention is highly meaningful.

[0205] The same or similar labels correspond to the same or similar parts;

[0206] The positional relationships depicted in the accompanying drawings are for illustrative purposes only and should not be construed as limiting this patent.

[0207] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A method for constructing a hierarchical classification model for government procurement items, characterized in that, Includes the following steps: S1: Construct a tree structure for hierarchical labels of government procurement items and encode the labels; S2: Perform text cleaning and word segmentation on the name of the government procurement item, obtain the word vectors of the segmented text, and concatenate the word vectors to obtain a word vector matrix; S3: Design a general attention model and a hierarchical attention model. Input the encoded representation of the text and the encoded representation of the label into the general attention model and the hierarchical attention model respectively for classification to obtain local classification results and global classification results. The hierarchical attention model has a three-layer structure: a document encoding layer, a hierarchical attention recurrent network layer, and a hybrid prediction layer. The document encoding layer encodes the text and hierarchical category structure using embedding layers, and then uses a Bi-LSTM model or a Bi-GRU model to further enhance the semantic relationships of the text. The hierarchical attention recurrent network layer models the dependencies between different levels step by step using the hierarchical structure in a top-down manner. At each category level, the hierarchical attention memory unit is repeated to capture the association between the text and the category. The hierarchical attention memory unit comprises three modules: a text-label attention module, a classification prediction module, and a hierarchical dependency module. The text-label attention module is used to capture the correlation between text and hierarchical categories, and to obtain the relationship representation between hierarchical category labels and text, and the attention matrix between hierarchical category labels and text; The classification prediction module is used to integrate the semantic representation of the original text and the text category representation that incorporates information from the previous level, to generate a unified representation for each level and perform category prediction. The hierarchical dependency module is used to model the dependency relationships between different levels by preserving hierarchical information for each level. For the category labels in layer h, different categories contribute differently to the prediction results, thus allowing for the correction of the attention matrix: For the h-th level of the hierarchical category structure, the input to the text-label attention module is: the full text representation V, and the corresponding level representation S. h and the weight parameter w passed from the previous level. h-1 : The average embedding of N words is obtained using average pooling. This represents the relationship between category labels and text at the h level. It is the attention matrix between the category labels and text at the h layer. This is the class prediction output vector of the h-th layer. Since the root node, i.e., the 0th layer, does not contain any information, the weight parameters w of the 0th layer are initialized with the element 1. 0 .

2. The method for constructing a hierarchical classification model for government procurement items according to claim 1, characterized in that, In step S2, the government procurement item names are cleaned and segmented into words to obtain word vectors of the segmented text, including the following steps: 1) Divide the text into a logical sequence of words; 2) The word segmentation results are represented by word embedding and text encoding, mapping the sequence into a fixed-length vector; the vector can well represent the features of the sequence; 3) The Bert model is used to embed words into the segmented results. Bert is based on a bidirectional Transformer encoder and uses random masking word syntax to achieve pre-trained deep bidirectional representation. It also obtains character-level word vectors and model parameters by training on corpora from encyclopedias.

3. The method for constructing a hierarchical classification model for government procurement items according to claim 1, characterized in that, In step S3, the text and hierarchical category structure are encoded using an embedding layer, and then the semantic relationships of the text are further enhanced using a Bi-LSTM model or a Bi-GRU model, including the following steps: The Bi-LSTM is a combination of forward LSTM and backward LSTM; LSTM learns and utilizes the sequence dependencies of data, and selects to remember and forget some information by controlling the output of the input gate, forget gate and output gate; Bi-LSTM learns longer sequence information and can also integrate information from the preceding and following context. The principle of Bi-GRU is the same as that of Bi-LSTM. It uses a GRU network with opposite directions to extract text features and then concatenates them. GRU selects to remember and forget certain information by controlling the output of the reset gate and update gate. The semantic enhancement matrix of the text is obtained through Bi-LSTM or Bi-GRU, the labels of each layer are embedded, an initial embedding matrix is ​​randomly generated, and then the obtained semantic enhancement matrix and the initial embedding matrix are input into the hierarchical attention recurrent network layer.

4. The method for constructing a hierarchical classification model for government procurement items according to claim 1, characterized in that, The goal of the text-label attention module is to capture the correlation between text and hierarchical categories, and output a representation of the relationship between hierarchical category labels and text. Attention matrix between hierarchical category labels and text Weight parameters of the previous level Multiplying it by the text representation V, we get: V h =w h-1 ·V V h This text representation incorporates information from the previous level. Because different words in a text often express different meanings, we need Num for the h-th level label. h The semantic relationships of the entire text are represented by vectors of dimension S, using the embedding matrix S of the h-level tags. h To calculate the attention for the corresponding category: It is a randomly generated weight matrix, O h This represents V after activation using the tanh function. h , Let represent the normalized text attention score for the i-th category in layer h, where each element in the vector represents the contribution of the word at the corresponding position to the i-th category; then the attention matrix... With text representation V h Multiply, we get The row average of Mh is obtained 5. The method for constructing a hierarchical classification model for government procurement items according to claim 4, characterized in that, The classification prediction module aims to integrate the semantic representation of the original text with the text category representation that incorporates information from the previous level, generating a unified representation for each level and performing category prediction. As a representation of the h-th layer It is a randomly initialized weight matrix. That is the corresponding error vector. Represents a nonlinear activation function; It is a weight matrix. σ is the corresponding error vector, and σ is the sigmoid function. The hierarchical dependency module models the dependencies between different levels by preserving hierarchical information for each level. For the category label of the h-th level, different categories contribute differently to the prediction results, thus enabling the correction of the attention matrix. This represents the corrected attention matrix. The broadcast function represents tiling the vector along a certain dimension. By performing row averaging, we obtained the weighted attention vector. right By tiling, we obtain the association weight matrix w between the text and the h-th layer labels. h : w represents the association weight between the i-th word and the hierarchical information. h It will also be used as input to the attention memory unit at the next level; After passing through the classification prediction module at each layer, the corresponding hierarchical representation vector can be obtained. and prediction probability vector For hierarchical label classification, predicting both local and global labels is important; a hybrid prediction layer is constructed, which uses both local and global information for the final output, and the hierarchical representation vector of each layer is used. Aggregate, obtain Then for A L Calculate the row average Let P G The output represents the probability of global prediction: P G =σ(W M ·A G +b M ) b G and b M Each is an error vector in the corresponding dimension, P G It is a continuous vector, and the element P at the i-th position is... G (i)=p(c i |x), indicating that it belongs to the label set. The conditional probability of the i-th label is calculated; the conditional probabilities of local and global predictions are integrated to obtain the final prediction output P. F : α is a parameter that determines the proportion of global and local information, and can be adjusted according to the actual prediction results. After passing through the document encoding layer, hierarchical attention recurrent network layer, and hybrid prediction layer, the final model structure is obtained: The loss function is defined as the sum of the local loss and the global loss. During the model training phase, for each document, the local classification loss is calculated. and global classification loss The calculation method is as follows: Y G It is a binary vector containing all the labels within the hierarchy. It is a binary vector containing only all labels of the h layer, Y G and Representing the true value of the label category, ε(.,.)=y i log(p(y i ))+(1-y i )log(1-p(y i () represents the binary cross-entropy function, and the optimization objective of the model is to minimize the sum of the global loss and the local loss. Θ represents all parameters of the model, λ Θ The regularization parameter is obtained using the Adam optimization algorithm. Minimization operation.

6. The method for constructing a hierarchical classification model for government procurement items according to claim 5, characterized in that, The construction process of the ordinary attention model is as follows: D = {w1, w2, ..., w N } represents the text after word embedding: Considering that words have varying degrees of influence on sentences, a word attention mechanism was added to score the attention of each word, resulting in an attention-weighted sentence vector: u i =tanh(W o h i +b o ) The final prediction vector is: p=softmax(W c s+b c ) Use the negative log-likelihood of the correct label corresponding to the sample as the training loss:

Images