Business data element encoding generation method based on transformer-encoder

Abstract

The application provides a business data element coding generation method based on a Transformer-Encoder, relates to the technical field of coding design, and builds a business data element coding generation method model based on the Transformer-Encoder, which comprises a whole feature extractor and a local feature extractor. The whole feature extractor extracts the features of the whole label tree according to an input vector. The local feature extractor takes the hierarchical features generated in the feature extraction process of the whole feature extractor as input, and generates feature vector representations for each layer of the label tree. The model not only considers the whole features of the label tree, but also fully extracts the hierarchical features of the label tree, effectively improving the accuracy of the business data element coding. In addition, in order to process sequence data and capture the position information between words, the model introduces a position variable, so that the business data element coding is better.

Description

Technical Field

[0001] This invention relates to the field of coding design technology, and in particular to a method for generating business data element codes based on Transformer-Encoder. Background Technology

[0002] Coding design is a crucial aspect of information system design. A good business data element encoder can significantly facilitate information statistics and retrieval. In researching element coding technology for grassroots social grid governance, business data encompasses public service data, grid governance entity data, social and legal attributes, etc., and the element coding of this business data is designed in a hierarchical coding format. Generally, a hierarchical multi-label classifier can be used to implement element coding for business data.

[0003] Hierarchical multi-label classifiers have a wide range of applications, such as text classification, product recommendation, movie and TV show recommendation, and search engine optimization. In these scenarios, by modeling the relationships between labels, the label categories of text data can be predicted more accurately, improving classification performance and accuracy.

[0004] Traditional hierarchical multi-label classifiers primarily focus on the overall or local features of the labels, neglecting the sequence information in the text. While this approach performs well in certain scenarios, its effectiveness is less than ideal when considering the dependencies between labels and the significant impact of text sequence information on the classification results.

[0005] In existing technologies, patent CN114896388A describes a hierarchical multi-label text classification method based on hybrid attention: this patent uses pre-trained word vectors as word embeddings and extracts information between labels through a relational network. Patent CN114912433A describes a hierarchical multi-label text classification method, apparatus, electronic device, and storage medium: this patent combines the text vector to be identified with L hierarchical label vectors to determine the M hierarchical labels to which the text belongs.

[0006] The above method does not consider the sequence relationship of the sample data, but only considers the global classifier or local classifier of the hierarchical label. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention provides a method for generating business data element encoding based on Transformer-Encoder.

[0008] A method for generating business data element encoding based on Transformer-Encoder includes:

[0009] Step 1: Define the scenarios of grassroots social grid governance in a symbolic way, specifically including defining data representation, label representation, and hierarchical representation;

[0010] The data is represented as follows: in the scenario of grassroots social grid governance, a sample X corresponds to a label set Y, where the sample... d represents the number of features for each character in sample X, and n represents the number of characters contained in each sample.

[0011] The labels are represented as follows: the label set Y is a tree structure, using... This represents the structure of the tag tree;

[0012] The hierarchy is represented as T h T represents the set of tags in the h-th level tag tree. 1 The set contains only one node, which is called the root node. Each sample's label is marked by the label represented by the root node.

[0013] Step 2: Preprocess the data text: including label hierarchy and dataset partitioning;

[0014] Step 2.1: In the context of grassroots social grid governance, assign different levels to the tags based on the relationships between them, and generate a tag tree;

[0015] Step 2.2: Divide the dataset D into training set, validation set and test set in a ratio of 6:2:2.

[0016] Step 3: Using the Transformer-Encoder-based business data element encoding generation method model, perform character embedding and positional encoding on the vector representation of characters, and input it into the business data element encoding generation method model;

[0017] The business data element encoding generation method model utilizes Transformer-Encoder and attention mechanism to extract local and global features of the label tree, and is a hybrid feature extractor;

[0018] Step 3.1: Using characters as input, convert the characters in the text into vector representations using character embeddings;

[0019] Step 3.2: Provide the location information, i.e., the location code, of each character to the business data element encoding generation method model;

[0020] The location code is represented by the following formula:

[0021]

[0022] In the formula, pos represents the position of a character in the sentence, and i represents the dimension. 2k represents an even-numbered dimension, and 2k+1 represents an odd-numbered dimension.

[0023] Step 3.3: Add the word embedding result and the positional encoding result to generate the input vector, represented as... x here t The vector representation of the t-th character, Embedding(x t ) represents x t Perform word embedding operation. Indicates positional encoding;

[0024] Step 4: Perform self-attention operation on the input vector generated in Step 3 to capture the correlation between each position in the sequence;

[0025] Step 4.1 Set x t This is represented as a vector representation of the t-th character. It is the t-th input vector, obtained by adding the word embedding and positional encoding in step 3, that is... Will Multiply by the three weight matrices respectively: W Q W K W V ,at this time Three linear transformations were performed, resulting in three new vectors: q t ,k t ,v t ;

[0026] Step 4.2: For all word vectors x of sample X t Perform word embedding and positional encoding to obtain the input vector Here is composed of multiple The assembled matrix will be composed of each The three vectors obtained are concatenated to form three matrices: Q, K, and V. The calculation formula is as follows:

[0027]

[0028]

[0029]

[0030] Step 4.3: Based on the three matrices Q, K, V obtained in Step 4.2, calculate the input vector using formula (5). Self-attention:

[0031]

[0032] In the formula, softmax() represents the multi-class activation function;

[0033] Step 5: Perform residual connections and data normalization on the self-attention results;

[0034] Step 5.1: The business data element encoding generation method model performs a residual join operation, which adds the output of the previous layer to the input of the current layer, as shown below:

[0035]

[0036] In the formula, Self-Attention() represents the execution of a self-attention mechanism on Q, K, and V, and Z 2 It is for Z 1 Perform the residual join operation.

[0037] Step 5.2: After the residual join of the business data element coding generation method model, a standardization layer is added to standardize the output data of the previous layer. The standardization operation is represented as follows:

[0038]

[0039] In the formula, μ and σ represent Z respectively. 2 The mean and standard deviation of Z, a and b are learnable scale factors and deviation terms, therefore Z 3 Z represents 2 The result after standardization.

[0040] Step 6: The business data element coding generation method model further processes the data by applying nonlinear transformations;

[0041] Step 6.1: Implement the nonlinear transformation by adding several forward propagation nonlinear activation functions, expressed as:

[0042] Z 4 =max(0,Z) 3 ·W1+b1)·W2+b2 (8)

[0043] In the formula, W1 and W2 are learnable weight matrices, and b1 and b2 are learnable bias vectors, therefore Z 4 Z represents 3 The result after performing the nonlinear transformation.

[0044] Step 6.2: Place Z 4 Performing residual joins and standardization is described as follows:

[0045]

[0046] Here, `Add&Normalize()` performs residual joins and normalization, thus obtaining the result of the input vector passing through a Transformer-Encoder layer, represented as...

[0047] Step 7: Encapsulate Steps 4-6 into an internal structure of a Transformer-Encoder, and concatenate a Transformer-Encoder for each layer of the label tree to obtain the overall feature extractor, thereby obtaining the overall features of the label tree;

[0048] Step 7.1: The result obtained in Step 6 For the features of the first layer of the label tree, and Add them together, and use the result as the input vector for the second layer of the label tree.

[0049] Step 7.2: Repeat steps 4-6 to obtain the features of the second layer. Concatenate a Transformer-Encoder for each level of the label tree;

[0050] Step 7.3: Continue repeating steps 4-6 to obtain... For the output of the last layer The output Z obtained after concatenating the Transformer-Encoder is G As a feature of the tag tree as a whole.

[0051] Step 8: What was obtained in Step 7 For each layer of the label tree, the features are treated as input vectors and fed into the Transformer-Encoder to obtain the features of each layer of the label tree, that is, to obtain the local features of each layer of the label tree.

[0052] Step 8.1: Represent the features of each layer in Step 7. These are used as inputs to the Transformer-Encoder to obtain the local feature representations of each layer.

[0053] Step 8.2: Perform a concatenation operation on the local feature representations of each layer to obtain the local feature extractor, which can be expressed as:

[0054]

[0055] Z here L This represents the local features of the tag tree.

[0056] Step 9: Calculate the overall and local features of the label tree using weighted calculations to obtain the final business data element encoding results;

[0057] The weighted summation of overall and local features is expressed as follows:

[0058] Z = α·Z G +(1-α)·Z L (11)

[0059] α is used to control the weight ratio between global and local features, so Z represents the final encoding result.

[0060] The beneficial effects of adopting the above technical solution are as follows:

[0061] This invention provides a business data element encoding generation method based on Transformer-Encoder. By utilizing Transformer-Encoder technology, a global feature extractor and a local feature extractor are built. Based on this, this invention carefully designs a novel encoder architecture to effectively improve model performance. Specifically, it has the following beneficial effects:

[0062] First, using Transformer-Encoder technology, a global feature extractor and a local feature extractor were built. This not only considers the overall features of the label tree, but also fully extracts the hierarchical features of the label tree, effectively improving the accuracy of business data element encoding. Transformer-Encoder can make full use of the self-attention mechanism to automatically learn the dependencies between different unknowns, effectively solving the problem of performance degradation of traditional models when inputting long texts.

[0063] Second: Consider the sequence relationship between texts and embed the position into the model input to better encode business data elements. Attached Figure Description

[0064] Figure 1 This is an overall flowchart of an embodiment of the present invention;

[0065] Figure 2 This is a schematic diagram illustrating the calculation of the Q, K, V matrices in an embodiment of the present invention;

[0066] Figure 3 This is an architecture diagram of the business data element encoding generation method model based on Transformer-Encoder in an embodiment of the present invention. Detailed Implementation

[0067] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0068] A method for generating business data element encoding based on Transformer-Encoder, such as Figure 1 As shown, the method includes the following steps:

[0069] Step 1: Define the scenarios of grassroots social grid governance in a symbolic way, specifically including defining data representation, label representation, and hierarchical representation;

[0070] The data is represented as follows: in the scenario of grassroots social grid governance, a sample X corresponds to a label set Y, where the sample... d represents the number of features for each character in sample X, and n represents the number of characters contained in each sample.

[0071] The labels are represented as follows: the label set Y has a tree structure, and the labels at higher levels have more general characteristics, such as basketball and football both belonging to ball games. This represents the structure of the tag tree;

[0072] The hierarchy is represented as T h T represents the set of tags in the h-th level tag tree. 1 The set contains only one node, which is called the root node. Each sample's label is marked by the label represented by the root node.

[0073] Step 2: Preprocess the data text: including label hierarchy and dataset partitioning;

[0074] Step 2.1: In the context of grassroots social grid governance, assign different levels to the tags based on the relationships between them, and generate a tag tree;

[0075] Step 2.2: Divide the dataset D into training set, validation set and test set in a ratio of 6:2:2.

[0076] Step 3: Using the Transformer-Encoder-based business data element encoding generation method model, perform character embedding and positional encoding on the vector representation of characters, and input it into the business data element encoding generation method model;

[0077] The business data element encoding generation method model utilizes Transformer-Encoder and attention mechanisms to extract local and global features of the label tree, and is a hybrid feature extractor; the model architecture is as follows: Figure 3As shown, the model first processes character features, calculating their word embeddings and positional encodings to generate an input vector. This input vector is then fed into a global feature extractor consisting of h layers of Transformer-Encoders to generate the global features of the label tree, where h represents the number of layers. The output of each layer of the global feature extractor is used as the input to a local feature extractor to obtain the hierarchical features of each layer of the label tree. A concatenation operation is performed on these hierarchical features to generate the final local features. Finally, the weights of the global and local features are calculated to achieve the element encoding of the business data.

[0078] Step 4: Perform self-attention operation on the input vector generated in Step 3 to capture the correlation between each position in the sequence;

[0079] Step 4.1 Set x t This is represented as a vector representation of the t-th character. It is the t-th input vector, obtained by adding the word embedding and positional encoding in step 3, that is... Will Multiply by the three weight matrices respectively: W Q W K W V ,at this time Three linear transformations were performed, resulting in three new vectors: q t ,k t ,v t ;

[0080] Step 4.2: For all word vectors x of sample X t Perform word embedding and positional encoding to obtain the input vector Here is composed of multiple The concatenated matrix is ​​essentially still multiple input vectors. Furthermore, the model input is batch input, capable of processing multiple input vectors at once. Therefore, for convenience, [the matrix is...]. Also known as the input vector. It will be composed of each The three vectors obtained are concatenated to form three matrices: Q, K, and V. The calculation formula is as follows:

[0081]

[0082]

[0083]

[0084] The calculation process is as follows Figure 2 As shown.

[0085] Step 4.3: Based on the three matrices Q, K, V obtained in Step 4.2, calculate the input vector using formula (5). Self-attention:

[0086]

[0087] In the formula, softmax() represents the multi-class activation function;

[0088] Step 5: Perform residual connections and data normalization on the self-attention results;

[0089] Step 5.1: To address the vanishing gradient problem and improve model accuracy, the business data element encoding generation method performs a residual join operation, which adds the output of the previous layer to the input of the current layer, as shown below:

[0090]

[0091] In the formula, Self-Attention() represents the execution of a self-attention mechanism on Q, K, and V, and is a simplified representation of formula (5), which is expressed as Z. 1 Therefore Z 2 It is for Z 1 Perform the residual join operation.

[0092] Step 5.2: To improve the model's computation speed, a standardization layer is added after the residual connection of the business data element encoding generation method model to standardize the output data of the previous layer. The standardization operation is represented as follows:

[0093]

[0094] In the formula, μ and σ represent Z respectively. 2 The mean and standard deviation of Z, a and b are learnable scale factors and deviation terms, therefore Z 3 Z represents 2 The result after standardization.

[0095] Step 6: The business data element encoding generation method model further processes the data by applying nonlinear transformations to better fit the data.

[0096] Step 6.1: Implement the nonlinear transformation by adding several forward propagation nonlinear activation functions, expressed as:

[0097] Z 4 =max(0,Z) 3 ·W1+b1)·W2+b2 (8)

[0098] In the formula, W1 and W2 are learnable weight matrices, and b1 and b2 are learnable bias vectors, therefore Z 4 Z represents 3 The result after performing the nonlinear transformation.

[0099] Step 6.2: Place Z 4 Performing residual joins and standardization is described as follows:

[0100]

[0101] Here, `Add&Normalize()` performs residual joins and normalization, thus obtaining the result of the input vector passing through a Transformer-Encoder layer, represented as...

[0102] Step 7: Encapsulate Steps 4-6 into an internal structure of a Transformer-Encoder, and concatenate a Transformer-Encoder for each layer of the label tree to obtain the overall feature extractor, thereby obtaining the overall features of the label tree;

[0103] Step 7.1: The result obtained in Step 6 For the features of the first layer of the label tree, and Add them together, and use the result as the input vector for the second layer of the label tree.

[0104] Step 7.2: Repeat steps 4-6 to obtain the features of the second layer. Concatenate a Transformer-Encoder for each level of the label tree;

[0105] Step 7.3: Continue repeating steps 4-6 to obtain... For the output of the last layer The output Z obtained after concatenating the Transformer-Encoder is G As a feature of the tag tree as a whole.

[0106] Step 8: What was obtained in Step 7 For each layer of the label tree, its features are treated as input vectors and fed into the Transformer-Encoder to obtain the features of each layer, i.e., to obtain the local features of each layer of the label tree. Figure 3 As shown.

[0107] Step 8.1: Represent the features of each layer in Step 7. These are used as inputs to the Transformer-Encoder to obtain the local feature representations of each layer.

[0108] Step 8.2: Perform a concatenation operation on the local feature representations of each layer to obtain the local feature extractor, which can be expressed as:

[0109]

[0110] Z here L This represents the local features of the tag tree.

[0111] Step 9: Calculate the overall and local features of the label tree using weighted calculations to obtain the final business data element encoding results;

[0112] To integrate local and global features and adjust their weights, a weighted sum is applied to the global and local features, expressed as follows:

[0113] Z = α·Z G +(1-α)·Z L (11)

[0114] α controls the weight ratio between global and local features, therefore Z represents the final encoding result. During training, the cross-entropy loss function is used, and the Adam optimizer is employed. The cross-entropy loss function is as follows:

[0115]

[0116] in, Y represents the actual probability of the i-th sample with respect to the j-th label. ij Let N be the expected probability of the i-th sample with the j-th label. N represents the number of samples, and L represents the number of labels.

[0117] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A method for generating business data element encoding based on Transformer-Encoder, characterized in that, Includes the following steps: Step 1: Define the scenarios of grassroots social grid governance in a symbolic way, specifically including defining data representation, label representation, and hierarchical representation; Step 2: Preprocess the data text: including label hierarchy and dataset partitioning; Step 2.1: In the context of grassroots social grid governance, assign different levels to the tags based on the relationships between them, and generate a tag tree; Step 2.2: Divide the dataset D into training set, validation set, and test set in a 6:2:2 ratio; Step 3: Using the Transformer-Encoder-based business data element encoding generation method model, the vector representation of characters is embedded and positionally encoded to generate an input vector, which is then input into the business data element encoding generation method model. The business data element encoding generation method model utilizes Transformer-Encoder and attention mechanism to extract local and global features of the label tree, and is a hybrid feature extractor; Step 4: Perform self-attention operation on the input vector generated in Step 3 to capture the correlation between each position in the sequence; Step 5: Perform residual connections and data normalization on the self-attention results; Step 6: The business data element coding generation method model further processes the data by applying nonlinear transformations; Step 7: Encapsulate Steps 4-6 into an internal structure of a Transformer-Encoder, and concatenate a Transformer-Encoder for each layer of the label tree to obtain the overall feature extractor, thereby obtaining the overall features of the label tree; Step 8: The features of each layer of the label tree obtained in Step 7 are treated as input vectors and input into the Transformer-Encoder to obtain the local features of each layer of the label tree; Step 9: Calculate the overall and local features of the label tree using weighted calculations to obtain the final business data element encoding results.

2. The business data element encoding generation method based on Transformer-Encoder according to claim 1, characterized in that, The data mentioned in step 1 is represented as follows: In the scenario of grassroots social grid governance, a sample X corresponds to a label set Y, where the sample... d represents the number of features for each character in sample X, and n represents the number of characters contained in each sample; The labels are represented as follows: the label set Y is a tree structure, using... This represents the structure of the tag tree; The hierarchy is represented as, using This represents the set of labels in the h-th level of the label tree. The set contains only one node, which is called the root node. Each sample's label is marked by the label represented by the root node.

3. The method for generating business data element encoding based on Transformer-Encoder according to claim 2, characterized in that, Step 3 specifically includes the following steps: Step 3.1: Using characters as input, convert the characters in the text into vector representations using character embeddings; Step 3.2: Provide the location information, i.e., the location code, of each character to the business data element encoding generation method model; The location code is represented by the following formula: (1)； In the formula, pos represents the position of a character in the sentence, and i represents the dimension. 2k represents an even-numbered dimension, and 2k+1 represents an odd-numbered dimension; Step 3.3: Add the word embedding result and the positional encoding result to generate the input vector, represented as... , here The vector representation of the t-th character. Indicates to Perform word embedding operation. This indicates the position code.

4. The method for generating business data element encoding based on Transformer-Encoder according to claim 3, characterized in that, Step 4 specifically includes the following steps: Step 4.1 will This is represented as a vector representation of the t-th character. It is the t-th input vector, obtained by adding the word embedding and positional encoding in step 3, i.e. ;Will Multiply by the three weight matrices respectively: ,at this time Three linear transformations were performed, resulting in three new vectors: ; Step 4.2: For all word vectors of sample X Perform word embedding and positional encoding to obtain the input vector , here is composed of multiple The assembled matrix will be composed of each The three vectors obtained are concatenated to form three matrices: The calculation formula is expressed as: (2)； (3)； (4)； Step 4.3: Based on the three matrices obtained in Step 4.2 The input vector is calculated using formula (5). Self-attention: (5)； In the formula, softmax() represents the multi-class activation function.

5. The method for generating business data element encoding based on Transformer-Encoder according to claim 4, characterized in that, Step 5 specifically includes the following steps: Step 5.1: The business data element encoding generation method model performs a residual join operation, which adds the output of the previous layer to the input of the current layer, as shown below: (6)； In the formula, Self-Attention() represents the execution of a self-attention mechanism on Q, K, and V, and Z 2 It is for Z 1 Perform residual join operations; Step 5.2: After the residual join of the business data element coding generation method model, a standardization layer is added to standardize the output data of the previous layer. The standardization operation is represented as follows: (7)； In the formula, μ and σ represent respectively The mean and standard deviation are given, and a and b are learnable scale factors and deviation terms, therefore... express The result after standardization.

6. The method for generating business data element encoding based on Transformer-Encoder according to claim 5, characterized in that, Step 6 specifically includes the following steps: Step 6.1: Implement the nonlinear transformation by adding several forward propagation nonlinear activation functions, expressed as: (8)； In the formula and It is a learnable weight matrix. and It is a learnable bias vector, therefore express The result after performing the nonlinear transformation; Step 6.2: [The sentence is incomplete and requires more context to be translated accurately.] Performing residual joins and standardization is described as follows: (9)； Here This indicates that residual connections and normalization operations are performed, thereby obtaining the result of the input vector passing through a single Transformer-Encoder layer, represented as... .

7. The method for generating business data element encoding based on Transformer-Encoder according to claim 6, characterized in that, Step 7 specifically includes the following steps: Step 7.1: The result obtained in Step 6 For the features of the first layer of the label tree, and Add them together, and use the result of the addition as the input vector for the second layer of the label tree; Step 7.2: Repeat steps 4-6 to obtain the features of the second layer. For each layer of the label tree, a Transformer-Encoder is concatenated. Step 7.3: Continue repeating steps 4-6 to obtain... In the output of the last layer The output obtained after chaining Transformer-Encoder As a feature of the tag tree as a whole.

8. The method for generating business data element encoding based on Transformer-Encoder according to claim 7, characterized in that, Step 8 specifically includes the following steps: Step 8.1: Represent the features of each layer in Step 7. These are used as inputs to the Transformer-Encoder to obtain the local feature representations of each layer. ; Step 8.2: Perform a concatenation operation on the local feature representations of each layer to obtain the local feature extractor, which can be expressed as: (10)； Here This represents the local features of the tag tree.

9. A method for generating business data element encoding based on Transformer-Encoder according to claim 8, characterized in that, The weighted calculation of the overall and local features of the label tree in step 9 specifically includes: (11) Z is used to control the weight ratio between global and local features, and thus represents the final encoding result.

Images