Bidding text classification method and computer equipment

Abstract

The invention discloses a bidding and tendering text classification method, which comprises the following steps of: obtaining each core word of a to-be-processed bidding and tendering text, constructing a graph structure by taking each core word as a node, and obtaining a feature matrix and a co-occurrence matrix; a first Transform network is adopted to process the feature matrix, and word-level coding features are obtained; determining each sentence vector, and processing each sentence vector, the word-level coding feature and the co-occurrence matrix by adopting a second Transform network to obtain a sentence-level coding feature; processing the sentence-level coding feature and the co-occurrence matrix by adopting a Transform variant to obtain a graph-level coding feature; fusing the word-level coding features, the sentence-level coding features and the graph-level coding features to obtain fused features; and processing the fusion features based on a multi-label text classification model, and outputting a multi-label classification result of the to-be-processed bidding and tendering text. According to the method, semantic information of different hierarchical features (words, sentences and graphs) of the bidding text can be captured, and the efficiency and accuracy of bidding text classification are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and particularly relates to a bidding text classification method and a computer device. BACKGROUND

[0002] Bidding is a widely used commercial activity in public procurement, construction projects, goods supply and service industry, and its purpose is to select the most suitable supplier or contractor through an open, fair and just way. The bidding text is an important link in the bidding process, which contains detailed information of the bidding project, so that potential bidders can understand the project and decide whether to participate in bidding. According to the intelligent classification of the bidding text according to the industry to which the project belongs, the bidders can accurately locate the bidding project, obtain effective information in time, and improve the efficiency of resource allocation, so as to improve the bidding competitiveness and ensure the maximization of benefits.

[0003] The bidding text has the characteristics of complex hierarchical structure, long text content, various formats, and scattered effective information, which reduces the accuracy of the bidding text classification processing, and makes it difficult for users to quickly obtain effective bidding information. SUMMARY

[0004] The purpose of the present application is to provide a bidding text classification method to solve the problem of low accuracy of bidding text classification processing and difficulty for users to quickly obtain effective bidding information.

[0005] In a first aspect, the present application provides a bidding text classification method, which comprises:

[0006] Obtaining each core word of the to-be-processed bidding text, constructing a graph structure with each core word as a node, obtaining a feature matrix and a co-occurrence matrix of the graph structure, wherein the feature matrix represents the word vector of each core word in the graph structure, and the co-occurrence matrix represents the co-occurrence relationship between each core word in the graph structure;

[0007] Processing the feature matrix by using a first Transformer network to obtain corresponding word-level encoding features;

[0008] Determining each sentence vector based on the to-be-processed bidding text and each core word, processing each sentence vector, word-level encoding feature and co-occurrence matrix by using a second Transformer network to obtain corresponding sentence-level encoding features;

[0009] Processing the sentence-level encoding features and the co-occurrence matrix by using a Transformer variant to obtain corresponding graph-level encoding features, wherein the Transformer variant comprises a single-head attention mechanism layer and a normalization layer;

[0010] The word-level coding feature, the sentence-level coding feature and the graph-level coding feature are fused to obtain a fused feature;

[0011] The trained multi-label text classification model is used to process the fused feature, and a multi-label classification result of the to-be-processed bidding text is output.

[0012] Optionally, the second Transformer network includes six second Transformer layers, each second Transformer layer includes a single-head attention mechanism layer, a normalization layer and a feedforward network layer, each sentence vector is determined based on the to-be-processed bidding text and each core word, and the second Transformer network is used to process each sentence vector, the word-level coding feature and the co-occurrence matrix to obtain corresponding sentence-level coding features, including:

[0013] For each sentence in the to-be-processed bidding text, each core word appearing in the sentence is determined, and an average pooling operation is performed on the word vectors of the appearing core words to obtain a sentence vector of the sentence.

[0014] The word-level coding feature, the co-occurrence matrix and the weight coefficient of the co-occurrence matrix are multiplied to obtain an intermediate vector, and the intermediate vector and each sentence vector are added to obtain an input sentence vector.

[0015] The single-head attention mechanism layer is used to process the input sentence vector and a mask matrix to obtain a second attention vector output by the single-head attention mechanism layer, wherein the mask matrix is determined based on the adjacency relationship between nodes in the graph structure.

[0016] The feedforward network layer is used to perform nonlinear processing on the second attention vector to obtain the sentence-level coding feature. Optionally, the trained multi-label text classification model is used to process the fused feature, including:

[0017] A bidding text training set is constructed, wherein the bidding text training set includes a plurality of bidding text samples and a plurality of true labels corresponding to each bidding text sample;

[0018] The pre-trained multi-label text classification model is used to train the bidding text training set to obtain a plurality of predicted labels for each bidding text sample.

[0019] A cross-entropy loss function is determined based on the true labels and the predicted labels corresponding to each bidding text sample, the label similarity values between each true label and the similarity weight coefficients.

[0020] The pre-trained multi-label text classification model is parameter-adjusted based on the cross-entropy loss function, and continues to be trained until a training end condition is reached, and a trained multi-label text classification model is obtained.

[0021] Optionally, a cross-entropy loss function is determined based on the real label and the predicted label corresponding to each bidding text sample, the similarity weight value between each real label, and the similarity weight coefficient, and the cross-entropy loss function comprises:

[0022] A hierarchical label tree is constructed based on the industry classification data, wherein the hierarchical label tree is formed by a plurality of nodes, each node represents a semantic label corresponding to a classification category in the industry classification data, and the connection relationship between the nodes represents the relationship between the corresponding semantic labels.

[0023] Random walk is performed in the hierarchical label tree to obtain a plurality of label sequences corresponding thereto;

[0024] The plurality of label sequences are input into a Skip-Gram model to obtain label embedding vectors of each semantic label;

[0025] Based on the label embedding vectors, a label similarity value between each label is determined.

[0026] Optionally, each core word of the to-be-processed bidding text is obtained, comprising:

[0027] The to-be-processed bidding text is subjected to word segmentation processing to obtain each word segment contained in the to-be-processed bidding text;

[0028] A BERT model is used to generate a word vector corresponding to each word segment;

[0029] An average word vector is calculated based on each word vector;

[0030] A score of each word vector is determined according to the distance between each word vector and the average word vector;

[0031] The N word segments with the highest scores are determined as each core word of the to-be-processed bidding text;

[0032] Each sentence in which each core word is located in the to-be-processed bidding text is found, and each sentence is constructed into an optimized bidding text.

[0033] Optionally, a graph structure is constructed with each core word as a node, and a feature matrix and a co-occurrence matrix of the graph structure are obtained, comprising:

[0034] A graph structure is constructed based on the optimized bidding text with each core word as a node;

[0035] Based on the graph structure, a feature matrix and a co-occurrence matrix of the graph structure are determined, wherein

[0036] The feature matrix is determined based on the word vector of the core word of each node;

[0037] Based on the normalized co-occurrence frequency between any two core words in the graph structure, a co-occurrence matrix is ​​constructed. The formula for calculating the normalized co-occurrence frequency is as follows:

[0038]

[0039] Among them, E ij Indicates the core word v i and core word v j The standardized co-occurrence frequency between them, co_count(v i ,v j () indicates the core word v within a preset sliding window. i and core word v j The frequency of co-occurrence, freq(v i ) indicates the core word v i The frequency of occurrence in the optimized bidding text, freq(v) j ) indicates the core word v j Frequency of appearance in the optimized bidding documents.

[0040] Optionally, the first Transformer network includes four first Transformer layers. Each first Transformer layer includes a multi-head attention mechanism layer, a first normalization layer, a feedforward network layer, and a second normalization layer. The first Transformer network is used to process the feature matrix to obtain the corresponding word-level encoded features, including:

[0041] The feature matrix is ​​processed by a multi-head attention mechanism layer to obtain the first attention vector output by the multi-head attention mechanism layer;

[0042] Based on the feedforward network layer and the second normalization layer, the first attention vector is processed to obtain word-level encoded features.

[0043] Optionally, a Transformer variant is used to process the sentence-level encoded features and the co-occurrence matrix to obtain the corresponding graph-level encoded features, including:

[0044] By fusing sentence-level encoded features with graph structures and using the co-occurrence matrix as the attention bias input to a Transformer variant, graph-level encoding is achieved, and graph-level encoded features are obtained.

[0045] Optionally, the Transformer variant includes two Transformer layers, each consisting of a single-head attention mechanism layer and a normalization layer. The Transformer variant is used to process sentence-level encoded features and the co-occurrence matrix to obtain the corresponding graph-level encoded features, including:

[0046] The sentence-level coding features are input into a Transformer layer, linear transformation is performed on the sentence-level coding features to generate a query vector, a key vector and a value vector;

[0047] The query vector is normalized to obtain a normalized query vector;

[0048] Self-attention mechanism calculation is performed based on the co-occurrence matrix, the normalized query vector, the key vector and the value vector to obtain corresponding graph-level coding features, and a formula of the self-attention mechanism calculation is as follows:

[0049]

[0050] wherein E represents the co-occurrence matrix, LayerNorm(HW q ) represents the normalized query vector matrix, HW q represents the query vector, HW k represents the key vector, and HW v represents the value vector, represents the dimension of the key vector, represents a scaling factor, and σ represents an activation function.

[0051] In a second aspect, the present application provides a computer device, comprising a memory and a processor, the memory stores a computer program, and the processor implements the steps of the bidding text classification method as described above when executing the computer program.

[0052] The present application constructs a graph structure taking each core word of the to-be-processed bidding text as a node, encodes and processes the graph structure through a three-level Transformer network, processes the core term and the long document sparse attention at different semantic representation levels (word, sentence and graph) respectively by using corresponding Transformer networks, fully captures the hierarchical features of the bidding text, improves the text feature capturing ability of the model, performs multi-label prediction on the text through a multi-label text classification model, can automatically classify the bidding text into different industry categories, has good creativity and practicality in the industry multi-label text classification task of large-scale bidding text, improves the efficiency and accuracy of the bidding text classification, and helps the bidding party and the bidding party to quickly locate the related information of the bidding project. BRIEF DESCRIPTION OF DRAWINGS

[0053] Figure 1 A first flowchart of the bidding text classification method provided by the embodiment of the present application;

[0054] Figure 2 A model structure schematic diagram of the bidding text classification method provided by the embodiment of the present application;

[0055] Figure 3A second flowchart of the bidding text classification method provided by the embodiment of the present application is provided;

[0056] Figure 4 A third flowchart of the bidding text classification method provided by the embodiment of the present application is provided;

[0057] Figure 5 A fourth flowchart of the bidding text classification method provided by the embodiment of the present application is provided;

[0058] Figure 6 A fifth flowchart of the bidding text classification method provided by the embodiment of the present application is provided;

[0059] Figure 7 A sixth flowchart of the bidding text classification method provided by the embodiment of the present application is provided;

[0060] Figure 8 A seventh flowchart of the bidding text classification method provided by the embodiment of the present application is provided;

[0061] Figure 9 A system block diagram of the computer device provided by the embodiment of the present application is provided. DETAILED DESCRIPTION

[0062] The present application will be described in detail below with reference to the specific embodiments shown in the drawings, but these embodiments do not limit the present application, and the structural, method, or functional changes made by those of ordinary skill in the art based on these embodiments are all within the protection scope of the present application.

[0063] Please refer to Figure 1 The present application provides a bidding text classification method, which comprises steps S101-S106.

[0064] S101, obtain each core word of the bidding text to be processed, and construct a graph structure with each core word as a node, obtain a feature matrix and a co-occurrence matrix of the graph structure, wherein the feature matrix represents the word vector of each core word in the graph structure, and the co-occurrence matrix represents the co-occurrence relationship between each core word in the graph structure.

[0065] Bidding is a common procurement method, which selects appropriate suppliers or contractors through an open, fair, and just procedure. Usually, the detailed information of the bidding project is expressed in the form of a bidding text. Taking the bidding document as the data basis, the corresponding graph structure is constructed.

[0066] Obtain each core word of the to-be-processed bidding text. The core word can be understood as a word with high importance in the bidding text, or a word with high frequency of occurrence in the document. For example, the core words are "bill of quantities", "qualification requirements", etc. Illustratively, a pre-trained language model can be used to perform information extraction on the to-be-processed bidding text to obtain each core word of the to-be-processed bidding text.

[0067] A graph structure is a data format used to represent a network with certain relationships. The graph structure includes nodes and edges between nodes. The edges between nodes are used to represent the association relationship between the nodes. Taking each core word as a node and the association relationship between each core word as an edge between nodes, the to-be-processed bidding text is converted into a corresponding graph structure, denoted as g=(V, E, X), where V represents the set of nodes in the graph structure, i.e., the core word set of the to-be-processed bidding text. If the to-be-processed bidding text includes N core words, there are N nodes in the graph structure. E is the set of edges between nodes, i.e., the co-occurrence matrix, which is used to represent the relationship between each node in the graph structure, i.e., the co-occurrence relationship between each core word. X is a feature matrix used to represent the word vector of each core word in the graph structure.

[0068] By converting the to-be-processed bidding text into corresponding graph structure information, the association relationship features between the core words in the text can be obtained based on the text information, thereby improving the utilization rate of the information in the document and enabling the document to be more accurately represented.

[0069] In S102, the first Transformer network is used to process the feature matrix to obtain corresponding word-level encoding features.

[0070] As shown in the model network architecture diagram. Figure 2 The feature matrix is input into the first Transformer network, and the first Transformer network performs feature encoding processing on the feature matrix to obtain corresponding word-level encoding features. By processing the feature matrix through the first Transformer network, the context semantic features between each core word in the to-be-processed bidding text and the relevance between each core word can be fully learned, thereby enabling the word-level feature representation of each core word to be accurately determined. For example, the semantic association between the core words "EPC general contractor" and "design and construction integration".

[0071] In S103, each sentence vector is determined based on the to-be-processed bidding text and each core word, and the second Transformer network is used to process each sentence vector, the word-level encoding features, and the co-occurrence matrix to obtain corresponding sentence-level encoding features.

[0072] The bidding text usually has long text content and scattered effective information. If the model directly processes the sentences in the bidding text, the complexity of the model processing increases. Therefore, the sentence vectors of each sentence are determined based on the to-be-processed bidding text and each core word. Each core word can reflect important information in the bidding text. Therefore, the sentence vectors of each sentence determined based on each core word can accurately express the semantic information of the bidding text.

[0073] The second Transformer network inputs the sentence vectors, the word-level coding features, and the co-occurrence matrix, and performs feature coding processing on the sentence vectors, the word-level coding features, and the co-occurrence matrix to obtain corresponding sentence-level coding features. Through the second Transformer network, the sentence vectors are processed, and the co-occurrence matrix and the word-level coding features of the graph structure are introduced at the same time. The co-occurrence matrix can represent the relationship between the core words and can learn the context semantic features between the sentence levels in the to-be-processed bidding text, so as to accurately determine the sentence feature representation of the to-be-processed bidding text.

[0074] In S104, the sentence-level coding features and the co-occurrence matrix are processed by using a Transformer variant to obtain corresponding graph-level coding features. The Transformer variant includes a single-head attention mechanism layer and a normalization layer.

[0075] The Transformer variant is an improvement of the architecture of the Transformer network and includes the single-head attention mechanism layer and the normalization layer. The sentence-level coding features and the co-occurrence matrix are input into the Transformer variant, and the sentence-level coding features and the co-occurrence matrix are coded to obtain corresponding graph-level coding features. By combining the information of the graph structure and the co-occurrence matrix, the semantic feature information of the text level in the to-be-processed bidding text can be learned, so that the semantic information of the to-be-processed bidding text can be accurately obtained.

[0076] In S105, the word-level coding features, the sentence-level coding features, and the graph-level coding features are fused to obtain fused features.

[0077] Exemplarily, the word-level coding features and the corresponding word weight coefficients output by the first Transformer network, the sentence-level coding features and the corresponding sentence weight coefficients output by the second Transformer network, and the graph-level coding features and the corresponding graph-level weight coefficients output by the Transformer variant are input into the normalization layer for processing to obtain the fused features.

[0078] Exemplarily, the fused features H final are represented as:

[0079]

[0080] wherein, is represented as a word-level encoding feature, W w is represented as a word weight coefficient, is represented as a sentence-level encoding feature, W s is represented as a sentence weight coefficient, is represented as a graph-level encoding feature, W g is represented as a graph-level weight coefficient, LayerNorm is represented as layer normalization.

[0081] S106, based on the trained multi-label text classification model, processing the fusion features to output a multi-label classification result of the to-be-processed bidding text.

[0082] Multi-label text classification is a task in natural language processing, which aims to assign multiple relevant class labels to a text, and these class labels usually have relevance or coexistence. The label of the text can be a word that can express the content, semantics or features of the text.

[0083] Through the multi-label text classification model, the fusion features are processed, and the multi-label text classification model can identify the relevant class labels of the to-be-processed bidding text, for example, the labels are "engineering construction", "municipal engineering", "greening engineering", etc.

[0084] In the embodiment of the present application, the graph structure is constructed by taking each core word of the to-be-processed bidding text as a node, which can better capture the complex semantic relationship of the bidding text; the graph structure is encoded and processed by a three-level Transformer network to obtain corresponding word-level encoding features, sentence-level encoding features and graph-level encoding features, respectively. In different semantic representation levels (word, sentence, graph), the corresponding Transformer network is used for processing, the core term is combined with the long document sparse attention, the hierarchical features (such as term→clause→document) of the bidding text are jointly modeled, the features of the bidding text are fully captured and the importance of each part is evaluated, and the capturing ability of the model for the text features is improved. Through the multi-label text classification model, the multi-label prediction of the text can be performed, the bidding text can be automatically divided into different industry categories, and the method has good creativity and practicality in the industry multi-label text classification task of large-scale bidding text, which improves the efficiency and accuracy of the bidding text classification, and helps the bidding party and the bidding party to quickly locate the related information of the bidding project.

[0085] One embodiment of the present application, as shown in Figure 3 the core words of the to-be-processed bidding text are obtained, including steps S301-S306.

[0086] S301, perform word segmentation on the to-be-processed bidding text to obtain each word segmentation included in the to-be-processed bidding text.

[0087] All sentences in the to-be-processed bidding text are subjected to word segmentation processing, and stop words are removed to obtain each word segmentation included in the to-be-processed bidding text. Stop words refer to common words, empty words or functional words that are considered to have less contribution to the text theme or semantic analysis in text processing. They frequently appear in the text, but usually lack distinguishing degree, so they are often filtered or ignored in text mining, natural language processing (NLP) and other tasks to improve model efficiency and effect. For example, de, le, zhe, shi, according to, etc.

[0088] S302, generate a word vector corresponding to each word segmentation using a BERT model.

[0089] BERT model is an open source language model applied to natural language processing tasks. The BERT model has a multi-layer Ttransformer structure. The Transformer structure is a network structure based on attention mechanism (Attention). This network structure can calculate the mutual relationship between each word in a text and all words in the text, and calculate the importance (weight) of each word based on the mutual relationship between the words, thereby obtaining a new expression (such as a new vector) for each word. The above new expression not only involves the features of the word itself, but also involves the relationship between other words and this word, so it is more global than traditional word vectors. The BERT model can learn the context relationship between words in the text.

[0090] Each word segmentation w is input into the BERT model, and a word vector h corresponding to each word segmentation is generated using the BERT model. w , d is the embedding dimension of the BERT model, usually 512, 768 or 1024.

[0091] S303, calculate an average word vector based on each word vector.

[0092] An average word vector h of the to-be-processed bidding text is calculated based on each word vector, and the average word vector h avg The calculation formula is:

[0093] h avg = avg(h w ).

[0094] S304, determine the score of each word vector according to the distance between each word vector and the average word vector.

[0095] For each word vector, the distance between the word vector and the average word vector is calculated, and the score of the word vector is calculated based on the distance.

[0096] Exemplarily, the calculation formula of the score of the word vector is:

[0097]

[0098] wherein Score(w) represents the score of the word vector, h w represents the word vector, h avg represents the average word vector.

[0099] S305, determining N core words with the highest scores as the core words of the to-be-processed bidding text.

[0100] Based on the scores of the word vectors, N word vectors with the highest scores are taken, and the corresponding N words are determined as the N core words of the to-be-processed bidding text.

[0101] S306, searching for the sentences in which the core words are located in the to-be-processed bidding text, and constructing the sentences as the optimized bidding text.

[0102] According to the N core words, the sentences in which each core word is located in the to-be-processed bidding text are constructed as the optimized bidding text. The optimized bidding text is converted into a corresponding graph structure.

[0103] Exemplarily, the optimized bidding text D is D={s1, s2, …, s M M represents the number of sentences related to the core words, N i words.

[0104] In the embodiment of the application, each word vector is obtained by processing each word in the to-be-processed bidding text through the BERT model, and the core words of the text are determined based on the scores of the word vectors. The context relationship of the words in the text can be learned more accurately, and the core words of the text can be obtained through clustering, so that the semantic information of the text can be accurately expressed.

[0105] In an embodiment of the application, a graph structure is constructed by taking each core word as a node, and a feature matrix and a co-occurrence matrix of the graph structure are obtained, including: taking each core word as a node, constructing a graph structure based on the optimized bidding text, and determining a feature matrix and a co-occurrence matrix of the graph structure based on the graph structure. The feature matrix is determined based on the word vector of the core word of each node. The co-occurrence matrix is constructed according to the standardized co-occurrence frequency between any two core words of the graph structure.

[0106] Exemplarily, the feature matrix Each row corresponds to a word vector of a core word, |V| = N, and d is the embedding dimension of the BERT model.

[0107] For example, co-occurrence matrix This is the edge set of a graph structure, used to represent the relationships between nodes in the graph structure, i.e., the co-occurrence relationships between core terms, such as the co-occurrence relationship between "qualification requirements" and "bidder qualifications". A co-occurrence matrix is ​​constructed based on the standardized co-occurrence frequencies between two core terms.

[0108] For example, the formula for calculating the normalized co-occurrence frequency is as follows:

[0109]

[0110] Among them, E ij Indicates the core word v i and core word v j The standardized co-occurrence frequency between them, co_count(v i ,v j () indicates the core word v within a preset sliding window. i and core word v j The frequency of co-occurrence, freq(v i ) indicates the core word v i The frequency of occurrence in the optimized bidding text, freq(v) j ) indicates the core word v j The frequency of occurrence of words in the optimized bidding text. For example, the sliding window can be set to 5-10 words. The normalized co-occurrence frequency between any two core words is used to represent the number of times the two core words co-occur in the text information. The higher the co-occurrence frequency, the more important the relationship between the two core words; the lower the co-occurrence frequency, the less important the relationship between the two core words.

[0111] One embodiment of this application, such as Figure 4 As shown, the first Transformer network is used to process the feature matrix to obtain the corresponding word-level encoded features, including steps S401-S402.

[0112] S401, a multi-head attention mechanism layer is used to process the feature matrix to obtain the first attention vector output by the multi-head attention mechanism layer;

[0113] S402, based on the feedforward network layer and the second normalization layer, processes the first attention vector to obtain word-level encoded features.

[0114] The first Transformer network includes 4 first Transformer layers, each of which includes a multi-head attention mechanism layer, a first normalization layer, a feedforward network layer, and a second normalization layer. The feature matrix is input into the first Transformer network, and the feature matrix is linearly transformed to generate corresponding query vectors, key vectors, and value vectors. The query vector matrix, the key vector matrix, and the value vector matrix are calculated based on the multi-head attention mechanism to obtain first attention vectors. The multi-head self-attention layer obtains different granularity and diversity of representation by introducing multiple independent attention heads and performing attention calculation on each attention head. The first attention vectors are processed through the feedforward network layer and the second normalization layer to output word-level encoding features. The feedforward network layer receives the output of the multi-head self-attention layer and performs nonlinear transformation and mapping through a fully connected layer. The feedforward network layer can be composed of two linear layers and an activation function, wherein the linear layer is responsible for linear transformation, and the activation function introduces nonlinear features. The normalization layer normalizes the output of the feedforward network layer to have similar scales in different dimensions, which helps to accelerate training, improve the generalization ability of the model, and alleviate the gradient vanishing problem.

[0115] Exemplarily, the input of the first first Transformer layer is represented as:

[0116]

[0117] The input of the lth first Transformer layer is represented as:

[0118]

[0119] wherein, L w is 4, representing 4 first Transformer layers.

[0120] The structure of the first Transformer network is represented as follows:

[0121]

[0122] The calculation of multi-head attention is represented as follows:

[0123]

[0124] wherein, represents the query vector, represents the key vector, represents the value vector, and h represents the number of attention heads.

[0125] The calculation of the feedforward network layer is represented as follows:

[0126] FFN(x) = ReLU(xW1 + b1)W2 + b2;

[0127] wherein W1 represents a weight matrix of the first linear layer, W2 represents a weight matrix of the second linear layer, b1 represents a bias term of the first linear layer, and b2 represents a bias term of the second linear layer.

[0128] In the embodiments of the present application, the first Transformer network is used to encode and process the feature matrix, learn the context semantic features between the core words in the to-be-processed bidding text, and the correlation between the core words, so as to accurately determine the word-level feature representation of each core word in the text.

[0129] As shown in one of the embodiments of the present application, Figure 5 the second Transformer network is used to process the sentence vectors, the word-level encoding features and the co-occurrence matrix to obtain corresponding sentence-level encoding features, including steps S501-S504.

[0130] S501, for each sentence in the to-be-processed bidding text, determining each core word appearing in the sentence, performing average pooling operation on the word vectors of each core word appearing, and obtaining a sentence vector of the sentence;

[0131] S502, multiplying the word-level encoding features, the co-occurrence matrix and the weight coefficient of the co-occurrence matrix to obtain an intermediate vector, and adding the intermediate vector and each sentence vector to obtain an input sentence vector;

[0132] S503, using a single-head attention mechanism layer to process the input sentence vector and a mask matrix to obtain a second attention vector output by the single-head attention mechanism layer, wherein the mask matrix is determined based on the adjacency relationship between the nodes in the graph structure.

[0133] S504, performing non-linear processing on the second attention vector based on a feedforward network layer to obtain a sentence-level encoding feature.

[0134] The second Transformer network includes 6 second Transformer layers, and each second Transformer layer includes a single-head attention mechanism layer, a normalization layer and a feedforward network layer. For the second Transformer network processing the sentence vector, 6 second Transformer layers are used, which can more accurately learn the overall information at the sentence level, and the number of processing at the sentence level is much less than the number of processing at the word level, so that using a network with more levels will not cause the processor to be down, which can more reasonably utilize the processing resources and improve the learning effect.

[0135] Exemplarily, based on the above-obtained optimized to-be-processed bidding text, for each sentence in the text, each core word appearing in the sentence is determined, and an average pooling operation is performed on the word vectors of each core word appearing to obtain a sentence vector representing the overall information of the sentence. A single-head attention mechanism layer is used to process the input sentence vector and the mask matrix to obtain a second attention vector output by the single-head attention mechanism layer, and a feedforward network layer is used to perform nonlinear processing on the second attention vector to obtain a sentence-level encoding feature.

[0136] For example, the average pooling operation is represented as:

[0137]

[0138] The word-level encoding feature, the co-occurrence matrix and the weight coefficient of the co-occurrence matrix are multiplied to obtain an intermediate vector, the intermediate vector and each sentence vector are added to obtain an input sentence vector, and the input sentence vector is input into a second Transformer network for encoding processing.

[0139] Exemplarily, the second Transformer network is represented as:

[0140]

[0141] wherein, represents an input sentence vector input into the lth second Transformer network, represents a vector output by the (l-1)th second Transformer network, and E represents a co-occurrence matrix. represents a word-level encoding feature output by the first Transformer network, and η represents a weight coefficient of the co-occurrence matrix.

[0142] Exemplarily, the structure of the second Transformer network is represented as follows:

[0143] LongformerBlock(x)=FFN L (LayerNorm(x+Attention(x))).

[0144] Exemplarily, the calculation method of attention is represented as follows:

[0145]

[0146] wherein, M represents a mask matrix used to mask illegal attention calculation, so that attention calculation is controlled within the structural neighbor range of nodes, and M is represented as:

[0147]

[0148] The calculation of the feedforward network layer is as follows:

[0149] FFN L (x) = GeLU(xW1 + b1)W2 + b2;

[0150] wherein W1 represents a weight matrix of the first linear layer, W2 represents a weight matrix of the second linear layer, b1 represents a bias term of the first linear layer, and b2 represents a bias term of the second linear layer.

[0151] In the embodiment of the present application, when processing the sentence-level encoding of the text, the traditional Transformer network structure is improved, the network structure is simplified to reduce the complexity of network calculation, and the co-occurrence matrix is injected into the Transformer network to learn the context semantic features between the sentence levels in the text, so as to accurately determine the sentence feature representation of the to-be-processed bidding text.

[0152] In one embodiment of the present application, the sentence-level encoding features and the co-occurrence matrix are processed by using a Transformer variant to obtain corresponding graph-level encoding features, including: fusing the sentence-level encoding features into a graph structure, inputting the co-occurrence matrix as an attention bias into the Transformer variant, realizing graph-level encoding, and obtaining graph-level encoding features.

[0153] As shown in FIG. 6, one specific embodiment of the present application fuses the sentence-level encoding features into a graph structure, inputs the co-occurrence matrix as an attention bias into the Transformer variant, realizes graph-level encoding, and obtains graph-level encoding features, including steps S601-S603. Figure 6

[0154] S601, inputting the sentence-level encoding features into a Transformer layer, performing linear transformation on the sentence-level encoding features to generate a query vector, a key vector and a value vector;

[0155] S602, performing normalization processing on the query vector to obtain a normalized query vector;

[0156] S603, performing self-attention mechanism calculation based on the co-occurrence matrix, the normalized query vector, the key vector and the value vector to obtain corresponding graph-level encoding features.

[0157] ​The Transformer variant includes 2 Transformer layers, each of which includes a single-head attention mechanism layer and a normalization layer. The sentence-level encoding features are input into the Transformer layer, and the sentence-level encoding features are linearly transformed to generate query vectors, key vectors, and value vectors. The co-occurrence matrix, the normalized query vectors, the key vectors, and the value vectors are used to perform self-attention mechanism calculation to obtain corresponding graph-level encoding features.

[0158] Exemplarily, the input of the Transformer variant is represented as:

[0159]

[0160] wherein L g represents the number of Transformer variants, and is 2, and E represents the co-occurrence matrix.

[0161] Exemplarily, the formula of the self-attention mechanism calculation is represented as:

[0162]

[0163] wherein E represents the co-occurrence matrix, LayerNorm(HW q ) represents the normalized query vectors, HW q represents the query vectors, HW k represents the key vectors, and HW v represents the value vectors. represents the dimension of the key vectors, represents the scaling factor, and σ represents the activation function.

[0164] Considering the sparsity of the co-occurrence matrix, the co-occurrence matrix is sparsified, that is:

[0165] E = Sparsify(E, k = 10);

[0166] The above sparsification indicates that the co-occurrence matrix retains the 10 elements with the largest absolute values in each row, and the other elements are set to 0, so as to reduce the complexity of model calculation and improve the interpretability of the co-occurrence matrix.

[0167] Since the complexity of graph-level processing is higher, too many network levels will cause high occupation of computing resources and easily cause lag failure, but the learning ability of single-layer network is limited and cannot learn the features of graph structure well, therefore, the embodiment of the application improves the traditional Transformer network, the Transformer variant adopts 2 Transformer layers, each Transformer layer only has a single-head attention mechanism layer and a normalization layer, so as to reduce the complexity of model calculation, fuse the sentence-level encoding features with the graph structure information, take the co-occurrence matrix as the attention bias input of the Transformer variant, realize the graph-level encoding, and learn the semantic feature information of the text level of the to-be-processed bidding text, so as to accurately obtain the semantic information of the to-be-processed bidding text.

[0168] One embodiment of the application is shown as Figure 7 The cross-entropy loss function is determined based on the real label and the predicted label corresponding to each bidding text sample, the similarity weight value between each real label, and the similarity weight coefficient, including steps S701-S704.

[0169] S701, constructing a hierarchical label tree based on industry classification data, wherein the hierarchical label tree is formed by a plurality of nodes, each node representing a semantic label corresponding to a classification category in the industry classification data, and the connection relationship between the nodes represents the relationship between the corresponding semantic labels.

[0170] According to the industry classification data (for example, GB / T 4754-2017), a hierarchical label tree is constructed, the hierarchical label tree is formed by a plurality of nodes, each node representing a semantic label corresponding to a classification category in the industry classification data, and the connection relationship between the nodes represents the relationship between the corresponding semantic labels. For example:

[0171] · Engineering construction

[0172] · Municipal engineering

[0173] · Greening engineering

[0174] · Road traffic engineering

[0175] · House building engineering

[0176] In the above examples, the nodes are "engineering construction", "municipal engineering", "greening engineering", etc.

[0177] S702, random walk in the hierarchical label tree to obtain a plurality of label sequences corresponding thereto.

[0178] Random walk, also known as random walk, refers to a network structure diagram, starting with a certain node, then randomly moving to its neighbor position with a certain probability, and then randomly moving from the neighbor position until t steps (t is a pre-set parameter, t is a positive integer greater than or equal to 1), so as to obtain a "sentence" (sequence) composed of t "words" (nodes).

[0179] Based on the multiple random walk mode, the sequence obtained by randomly walking from each node in the hierarchical label tree as the starting point can be understood that each node random walk sequence can have multiple, based on the random walk sequence of different starting points, multiple label sequences can be obtained. For example, the label sequence is: "engineering construction→municipal engineering→greening engineering", "engineering construction→municipal engineering→road traffic engineering".

[0180] S703, input multiple label sequences into the Skip-Gram model to obtain label embedding vectors of each semantic label.

[0181] The Skip-Gram model is a model that learns the context information according to the middle word to learn the relationship between the words in the sentence. The Skip-Gram model includes an input layer, a hidden layer and an output layer. By inputting multiple label sequences into the Skip-Gram model, the relationship between each semantic label can be learned, and the label embedding vector of each semantic label can be obtained.

[0182] S704, based on each label embedding vector, determine the label similarity value between each label.

[0183] According to each label embedding vector, the distance between each label embedding vector is calculated, and based on the calculated distance, the label similarity value between each label is determined.

[0184] Exemplarily, the Euclidean distance between any two label vectors is calculated by using the Euclidean distance, and the calculation method of the Euclidean distance is:

[0185]

[0186] Wherein, Dis(i,j) represents the Euclidean distance between the label embedding vector and the label embedding vector .

[0187] The calculation formula of the label similarity value is represented as:

[0188]

[0189] In the embodiments of the present application, a hierarchical label tree is constructed based on industry classification data, an embedding vector of a semantic label of an industry category is determined by using random walk and a Skip-Gram model, and then the similarity of each label can be determined, so that the similarity and correlation of each industry category can be determined.

[0190] As shown in one of the embodiments of the present application, Figure 8 the fusion features are processed based on the trained multi-label text classification model, including steps S801-S804.

[0191] S801, a bidding text training set is constructed, wherein the bidding text training set includes a plurality of bidding text samples and a plurality of true labels corresponding to each bidding text sample;

[0192] S802, the pre-trained multi-label text classification model is used to train the bidding text training set, and a plurality of predicted labels of each bidding text sample are obtained;

[0193] S803, a cross-entropy loss function is determined based on the true labels and the predicted labels corresponding to each bidding text sample, the label similarity values between each true label, and the similarity weight coefficients;

[0194] S804, the pre-trained multi-label text classification model is adjusted in parameters based on the cross-entropy loss function, and continues to be trained until a training end condition is reached, and a trained multi-label text classification model is obtained.

[0195] By constructing the bidding text training set and training, the trained multi-label text classification model is obtained. In the process of model training, a weighted loss function is introduced, and the loss weight is adjusted according to the semantic distance between the labels. Therefore, the cross-entropy loss function is determined based on the true labels and the predicted labels corresponding to each bidding text sample, the label similarity values between each true label, and the similarity weight coefficients.

[0196] Exemplarily, the cross-entropy loss function is represented as:

[0197]

[0198] α k = maxSim(l, k) e [0, 1];

[0199] wherein, represents the predicted label, represents the true label, N represents the number of samples, λ represents the similarity weight coefficient, and ɑ k represents the label similarity value.

[0200] In the training process of the multi-label text classification model, the similarity between labels is considered in the cross-entropy loss function of the model, the loss weight is adjusted according to the semantic distance between labels, the classification performance of the model is optimized, and especially when dealing with infrequent labels and fine-grained labels. Similar labels can be given higher weights to solve the long-tail distribution problem (rare industry label sample shortage) and improve the accuracy of small sample industry classification.

[0201] The embodiment of the present application also provides a computer readable storage medium, and the computer readable storage medium stores a computer program. The computer program is executed by a processor to implement the steps of any one of the above bidding text classification methods.

[0202] Figure 9 The computer device provided by the embodiment of the present application is a hardware structure schematic diagram of a computer device. Figure 9 The computer device shown includes a processor 901, a communication interface 902, a memory 903 and a communication bus 904. The processor 901, the communication interface 902 and the memory 903 communicate with each other through the communication bus 904. Among them, Figure 9 The connection mode between the processor 901, the communication interface 902 and the memory 903 shown is only exemplary. In the implementation process, the processor 901, the communication interface 902 and the memory 903 can also be connected with each other in communication through other connection modes other than the communication bus 904.

[0203] The memory 903 can be used to store a computer program, which can include instructions and data, to implement the steps of any one of the above bidding text classification methods. In the embodiment of the present application, the memory 903 can be various types of storage media, such as random access memory (RAM), read only memory (ROM), non-volatile RAM (NVRAM), programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, optical storage and register, etc. The memory 903 can include a hard disk and / or memory.

[0204] The processor 901 can be a general processor, which can be a processor that performs specific steps and / or operations by reading and executing computer programs (e.g., computer programs) stored in a memory (e.g., the memory 903), and can use data stored in the memory (e.g., the memory 903) in the process of executing the steps and / or operations. The general processor can be, for example but not limited to, a central processing unit (CPU). In addition, the processor 901 can also be a special-purpose processor, which can be a processor specially designed to perform specific steps and / or operations. The special-purpose processor can be, for example but not limited to, ASIC, FPGA, etc. In addition, the processor 901 can also be a combination of multiple processors, such as a multi-core processor.

[0205] The communication interface 902 can include an input / output (I / O) interface, a physical interface, and a logical interface for realizing the interconnection of devices inside the network device, and an interface for realizing the interconnection of the network device with other devices (e.g., network devices). The communication network can be an Ethernet, a radio access network (RAN), a wireless local area network (WLAN), etc. The communication interface 902 can be a module, a circuit, a transceiver, or any device capable of realizing communication.

[0206] In the implementation process, each step of the above method can be completed by the integrated logic circuit of hardware in the processor 901 or the instructions in the form of software. The method disclosed in combination with the embodiments of the present application can be directly embodied as a hardware processor for execution, or a combination of hardware and software modules in the processor for execution. The software module can be located in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, register, etc. The storage medium is located in the memory 903, and the processor 901 reads the information in the memory 903 and combines the hardware to complete the steps of the above method. To avoid repetition, it will not be described in detail here.

[0207] Although the preferred embodiments of the present application have been disclosed for illustrative purposes, those skilled in the art will realize that various modifications, additions and substitutions are possible without departing from the scope and spirit of the present application disclosed by the appended claims.

Claims

1. A method of classifying a tendering text, characterized by, The method comprises: acquiring each core word of a to-be-processed bidding text, constructing a graph structure with each core word as a node, acquiring a feature matrix and a co-occurrence matrix of the graph structure, wherein the feature matrix represents the word vector of each core word in the graph structure, and the co-occurrence matrix represents the co-occurrence relationship between each core word in the graph structure; processing the feature matrix by using a first Transformer network to obtain corresponding word-level encoding features; determining each sentence vector based on the to-be-processed bidding text and each core word, processing the each sentence vector, the word-level encoding features and the co-occurrence matrix by using a second Transformer network to obtain corresponding sentence-level encoding features; processing the sentence-level encoding features and the co-occurrence matrix by using a Transformer variant to obtain corresponding graph-level encoding features, wherein the Transformer variant comprises a single-head attention mechanism layer and a normalization layer; fusing the word-level encoding features, the sentence-level encoding features and the graph-level encoding features to obtain fused features; processing the fused features based on a trained multi-label text classification model to output a multi-label classification result of the to-be-processed bidding text.

2. The tender text classification method of claim 1, wherein, The second Transformer network comprises six second Transformer layers, each second Transformer layer comprises a single-head attention mechanism layer, a normalization layer and a feedforward network layer, each sentence vector is determined based on the to-be-processed bidding text and each core word, the each sentence vector, the word-level encoding features and the co-occurrence matrix are processed by using the second Transformer network to obtain corresponding sentence-level encoding features, which comprises: for each sentence in the to-be-processed bidding text, determining each core word appearing in the sentence, performing average pooling operation on the word vectors of the appearing each core word to obtain a sentence vector of the sentence; multiplying the word-level encoding features, the co-occurrence matrix and the weight coefficient of the co-occurrence matrix to obtain an intermediate vector, and adding the intermediate vector and each sentence vector to obtain an input sentence vector; processing the input sentence vector and a mask matrix by using the single-head attention mechanism layer to obtain a second attention vector output by the single-head attention mechanism layer, wherein the mask matrix is determined based on the adjacency relationship between each node in the graph structure; performing nonlinear processing on the second attention vector based on the feedforward network layer to obtain the sentence-level encoding features.

3. The tender text classification method of claim 1, wherein, Processing the fused features based on a trained multi-label text classification model comprises: constructing a bidding text training set, wherein the bidding text training set comprises a plurality of bidding text samples and a plurality of true labels corresponding to each bidding text sample; training the bidding text training set by using a pre-trained multi-label text classification model to obtain a plurality of predicted labels for each bidding text sample; determining a cross-entropy loss function based on the true labels and the predicted labels corresponding to each bidding text sample, the label similarity values between each true label and the similarity weight coefficients; and The pre-trained multi-label text classification model is parameter-adjusted by using the cross-entropy loss function and continues to be trained until a training end condition is reached, so as to obtain a trained multi-label text classification model.

4. The tender text classification method of claim 3, wherein, A cross-entropy loss function is determined based on the real labels and the predicted labels corresponding to each bidding text sample, the similarity weight values between the real labels, and the similarity weight coefficients, and the cross-entropy loss function includes: A hierarchical label tree is constructed based on the industry classification data, wherein the hierarchical label tree is formed by a plurality of nodes, each node represents a semantic label corresponding to a classification category in the industry classification data, and the connection relationship between the nodes represents the relationship between the corresponding semantic labels; Random walk is performed in the hierarchical label tree to obtain a plurality of label sequences; The plurality of label sequences are input into a Skip-Gram model to obtain label embedding vectors of each semantic label; Based on the label embedding vectors, a label similarity value between each label is determined.

5. The method of claim 1, wherein, Each core word of a to-be-processed bidding text is obtained, including: The to-be-processed bidding text is segmented to obtain each segmented word included in the to-be-processed bidding text; A BERT model is used to generate a word vector corresponding to each segmented word; An average word vector is calculated based on each word vector; A score of each word vector is determined according to the distance between each word vector and the average word vector; The N segmented words with the highest scores are determined as the core words of the to-be-processed bidding text; Each sentence in which the core words are located is searched in the to-be-processed bidding text, and each sentence is constructed into an optimized bidding text.

6. The tender text classification method of claim 5, wherein, A graph structure is constructed with each core word as a node, and a feature matrix and a co-occurrence matrix of the graph structure are obtained, including: A graph structure is constructed based on the optimized bidding text with each core word as a node; Based on the graph structure, a feature matrix and a co-occurrence matrix of the graph structure are determined, wherein The feature matrix is determined based on the word vector of the core word of each node; The co-occurrence matrix is constructed according to the standardized co-occurrence frequency between any two core words of the graph structure, wherein the calculation formula of the standardized co-occurrence frequency is as follows: wherein E ij represents the normalized co-occurrence frequency between core words v i and core words v j , co_count(v i ,v j ) represents the co-occurrence frequency of core words v i and core words v j in a preset sliding window, freq(v i ) represents the frequency of core words v i in the optimized bidding text, and freq(v j ) represents the frequency of core words v j in the optimized bidding text.

7. The tender text classification method of claim 1, wherein, The first Transformer network includes four first Transformer layers, each first Transformer layer includes a multi-head attention mechanism layer, a first normalization layer, a feedforward network layer, and a second normalization layer, the feature matrix is processed by using the first Transformer network to obtain corresponding word-level encoding features, including: The multi-head attention mechanism layer is used to process the feature matrix to obtain a first attention vector output by the multi-head attention mechanism layer; The first attention vector is processed based on the feedforward network layer and the second normalization layer to obtain the word-level encoding features.

8. The method of claim 1, wherein, The sentence-level encoding features and the co-occurrence matrix are processed by using a Transformer variant to obtain corresponding graph-level encoding features, including: The sentence-level coding feature fusion graph structure is input into a Transformer variant with a co-occurrence matrix as an attention bias to realize graph-level coding and obtain graph-level coding features.

9. The tender text classification method of claim 8, wherein, The Transformer variant includes two Transformer layers, each of which includes a single-head attention mechanism layer and a normalization layer, and the sentence-level coding features and the co-occurrence matrix are processed by the Transformer variant to obtain corresponding graph-level coding features, including: The sentence-level coding features are input into the Transformer layer, and linear transformation is performed on the sentence-level coding features to generate query vectors, key vectors and value vectors; The query vectors are normalized to obtain normalized query vectors; Self-attention mechanism calculation is performed based on the co-occurrence matrix, the normalized query vectors, the key vectors and the value vectors to obtain corresponding graph-level coding features, and a formula of the self-attention mechanism calculation is represented as: where e denotes the co-occurrence matrix, LayerNorm(HW q ) denotes the normalized query vector matrix, HW q denotes the query vector, HW k denotes the key vector, HW v denotes the value vector, denotes the dimension of the key vector, denotes the scaling factor, and σ denotes the activation function. 10.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-9. The processor executes the computer program to realize the steps of the bidding text classification method in any one of claims 1 to 9.

Images