Keyword clustering extraction method based on segmented recursion and multi-query attention mechanism

Abstract

The invention discloses a keyword clustering extraction method based on segmented recursion and a multi-query attention mechanism, solves the problems of long text input limitation and context semantic disorder caused by factors such as context, cultural differences and language structures, and introduces two important mechanisms of segmented recursion and relative position coding. The method aims to construct the relation between the fragments, and the latter enables the position information of the vocabularies to be effectively integrated during segmentation processing, and the layer output of the previous fragment is cached and is reused in the processing of the next fragment, so that a large amount of redundant calculation is avoided, and the input limitation of a long text is solved. Relative position coding can play a better role in processing a long-distance dependency relationship and generalizing to sequences with different lengths. According to the multi-query cross attention mechanism, by sharing Key and Value weights, parameter operation occupation is reduced, and the reasoning speed of the model is increased.

Description

Technical Field

[0001] The present invention relates to the field of computer data analysis, and in particular to a keyword clustering extraction method based on segmented recursion and multi-query attention mechanism. Background Art

[0002] As the Internet is entering the intelligent era, big data, cloud computing, and artificial intelligence will usher in revolutionary developments around the world. The amount of business information data is growing explosively. Since most business text information contains irrelevant information, mining sparse but valuable information from massive data has become a new breakthrough point. Through research and analysis, we can accurately mine information and provide exclusive personalized services. We can accurately locate and extract knowledge resources in the business field, analyze text data, and mine the most valuable content, among which keyword extraction is an important part. Keyword extraction refers to the process of extracting high-quality phrases from a given corpus, and is a core basic problem in the field of natural language processing. It can quickly mine the most valuable information in business literature and shorten the cycle of understanding the industry.

[0003] The natural language algorithms commonly used today are limited in their ability to complete keyword extraction tasks. For example, the traditional BERT model only supports fixed-length sequence inputs (e.g. Transformer supports 512 words), and requires padding for input sequences that are insufficient in length, while trimming for sequences that exceed the length limit. The input restrictions on long texts and the differences in fine-grained semantic understanding make it difficult to avoid semantic information loss when processing long commercial information texts, resulting in poor keyword extraction performance. Summary of the invention

[0004] In order to overcome the above-mentioned deficiencies of the prior art, the present invention provides a keyword clustering extraction method based on segmented recursion and multi-query attention mechanism.

[0005] The technical solution adopted by the present invention is: a keyword clustering extraction method based on segmented recursion and multi-query attention mechanism, comprising the following steps;

[0006] S1. Collect information to build a data set, obtain information from multiple commercial long texts, and use the obtained information as a training data set;

[0007] S2, data preprocessing, converting the input long text into a standard format;

[0008] S3, data model training, the data model training includes an early stage of data model training, a mid stage of data model training and a late stage of data model training;

[0009] S301, in the early stage of data model training, the relative position of the sentence is encoded in a relative position manner, and the full sentence position of the vocabulary is no longer considered;

[0010] S302, in the middle of data model training, the vector is encoded and decoded. The encoding adopts a segmented recursive mechanism to encode the vector, and the decoding adopts a multi-query attention mechanism to allow all heads to share the same parameters;

[0011] S303, at the end of data model training, the output is also the number of words in the current sentence plus the size of the preceding and following tags, with a representation vector of dimension 768, the representation vectors of the candidate words are extracted as a vector data set, and the trained BERT model is used to screen each text block to obtain several candidate words, and the representation vectors of the candidate words are extracted as a vector data set;

[0012] S4, randomly input a long commercial text into the trained BERT model, and output the representation vector of the candidate word as the candidate word vector;

[0013] S5. Use the K-Means clustering algorithm to iteratively calculate the distance between the candidate word vector and the cluster center, sort and obtain the candidate words corresponding to the first N shortest distances as keyword samples;

[0014] Further, the S1 includes:

[0015] S101, read all the text information of the data set, read each text and regard it as ending with a line break character " / n", and split each text information into short sentence blocks as a long text, and separate each short sentence block into several candidate words [X 0 , X 1 , X 2 …X T ], where X 0 , X 1 , X 2 …X T Indicates different candidate words;

[0016] S102, for each unstructured long text block X t The data is preprocessed by word segmentation, part-of-speech screening, deduplication, and stop word removal to obtain the input format of CLS sentence SEP, where CLS represents the head tag and SEP represents the tail tag;

[0017] Furthermore, in the step S301, at the token stage, the short sentence is calculated by convolution with a candidate word list size of [30522, 768], where 30522 represents the number of candidate words, 768 represents the vector dimension of the candidate word, and the output is the number of candidate words in the current short sentence plus the size of the preceding and following tokens, and the embedding vector representation of the dimension is 768;

[0018] Furthermore, the relative position in step S301 is, when the jth word is queried, the position encoding vector corresponding to the i-th word is r j-i , and when the query word becomes the kth word, the position code corresponding to the i-th word becomes r k-i ,The position information of a word is based only on the current query word;

[0019] Furthermore, the step S302 includes:

[0020] S3021, get the entire sentence and candidate word vectors and input them into the encoding layer stage. Each encoding block introduces a segmented recursive division strategy, using an additional storage space with a fixed length to save the hidden state of the previous segment. When processing the current segment, these stored hidden states will interact and integrate information with the current segment, thereby establishing a connection between the two segments. The output of the nth layer of the second segment is integrated with the calculation method of context semantics.

[0021]

[0022] Among them, the subscripts τ and τ+1 are used to represent two adjacent segments. For n-1 layers, the hidden states of two adjacent segments are expressed as and Indicates that it is used normally during forward propagation Value, two adjacent fragments are spliced ​​according to the sequence length to obtain It means concatenating two sequences of length T into a sequence of length 2T, which does not participate in gradient regression and is only used as pure data;

[0023] S3022, the hidden state of the nth layer and The joint sequence is combined with the learnable linear transformation matrix with different weights to obtain the set of query q, key k and value v. The specific calculation process is as follows:

[0024]

[0025] The output tensor is split from the semantic level, that is, each head first obtains a set of query q, key k and value v for attention mechanism calculation as part of the representation vector of the entire sentence and candidate word;

[0026] S3023, the attention weight is obtained by the current long sequence method, and then the features are integrated to establish the association relationship between the two fragments. The specific calculation process is as follows:

[0027]

[0028] in, is the encoding vector corresponding to the current fragment, Respectively represent the parameter values ​​obtained by the current layer;

[0029] S3024, cache the layer output of the previous fragment and reuse it in the processing of the next fragment, and so on for all fragments. The calculation method is as follows:

[0030]

[0031] Where N is the number of layers in the model, S is the number of all segments, and depend represents a general dependency relationship. The encoding vector corresponding to the entire sentence and the candidate word and the hidden state corresponding to the current segment are extracted, which means outputting the vector representation corresponding to the entire sentence and the candidate word.

[0032] S3025, construct a mask matrix Mask matrix at the first layer of each decoding block, and convert the QK calculated at the encoding layer into T Multiply the masked attention score matrix bit by bit with the Mask matrix to get the masked attention score matrix. Multiply the masked attention score matrix with the V matrix to get the output vector of the entire sentence and candidate words. All heads share the same key and value matrix, and assign different weights to each position in the input sequence, so as to effectively capture context information and obtain the encoding vectors of the entire sentence and candidate words output by the encoding layer;

[0033] S3026, each row of the encoding vector, i.e., the multidimensional matrix, represents the decoding vector of each candidate word, and passes through the feedforward network, i.e., the fully connected layer, so as to keep the vector dimension represented by each word the same, and the residual structure outputs the vector matrix, which is calculated as follows:

[0034]

[0035] Furthermore, the step S5 includes:

[0036] S501, using the word vector set as a sample, clustering initialization, selecting K initial cluster centers, setting a threshold, calculating the distance between each candidate word vector and each cluster center, and assigning it to the cluster center closest to it, recalculating the mean of all word vector sets in its cluster center, and obtaining a new cluster center. The cluster center point update formula is calculated as follows:

[0037]

[0038] in, is the coordinate vector of the original cluster center, The sample position vector is used to find the new centroid by weighted averaging. Repeat the allocation and update steps until the cluster center no longer changes significantly, that is, the distance between the new cluster center and the old cluster center is less than a preset threshold.

[0039] S502: According to the clustering results, select the words with high frequency or representativeness from each cluster as keywords. Use the Euclidean distance calculation method to calculate the distance between each word and the cluster center; sort in ascending order according to the obtained distance, and select the top N words as text keywords.

[0040] Beneficial effects of the invention:

[0041] The present invention solves the problem of long text input limitations and contextual semantic barriers caused by factors such as context, cultural differences, and language structure, and introduces two important mechanisms: segmented recursion and relative position encoding. The former aims to build the relationship between segments, while the latter enables the effective integration of vocabulary position information during segment processing. By caching the layer output of the previous segment and reusing it in the processing of the next segment, this method avoids a large amount of redundant calculations, thereby solving the input limitations of long texts. Relative position encoding can perform better in processing long-distance dependencies and generalizing to sequences of different lengths. The multi-query cross-attention mechanism reduces the memory occupied by parameter operations and improves the reasoning speed of the model by sharing Key and Value weights.

[0042] 1. Two important mechanisms, segment recursion and relative position encoding, are introduced into the word embedding vector. The former aims to build the relationship between segments, while the latter enables the effective integration of vocabulary position information during segment processing. By caching the layer output of the previous segment and reusing it in the processing of the next segment, this method avoids a lot of redundant calculations, thereby solving the input limitation of long texts.

[0043] 2. The introduction of the relative position encoding mechanism enables the effective integration of vocabulary position information during segment processing, which can provide better performance in processing long-distance dependencies and generalizing to sequences of different lengths;

[0044] 3. At the same time, the decoding layer in the BERT network architecture was improved, and a multi-query cross-attention mechanism was added. By sharing Key and Value weights, the number of parameters was reduced, the reasoning speed of the model was significantly improved, and long texts could be processed faster.

[0045] 4. The improved BERT model is combined with KMearns clustering to continuously optimize the clustering results in an iterative manner, so that the objects within each cluster are as close as possible, while the objects between different clusters are as separated as possible. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A keyword extraction flow chart of the keyword clustering extraction method based on segmented recursion and multi-query attention mechanism provided by the present invention.

[0047] Figure 2 The present invention provides a flowchart of the encoding layer-decoding layer architecture of the BERT model for the keyword clustering extraction method based on segmented recursion and multi-query attention mechanism.

[0048] Figure 3 The keyword clustering extraction method based on segmented recursion and multi-query attention mechanism provided by the present invention is a schematic diagram of the semantic merging of previous and next word fragments based on the segmented recursive mechanism. DETAILED DESCRIPTION

[0049] In order to more clearly understand the technical solution of the present invention, the present invention is further described below in conjunction with the accompanying drawings;

[0050] like Figure 1-Figure 3 As shown, the keyword clustering extraction method based on segmented recursion and multi-query attention mechanism includes the following steps:

[0051] S1. Collect information to build a data set, obtain information from multiple commercial long texts, and use the obtained information as a training data set;

[0052] S2, data preprocessing, converting the input long text into a standard format;

[0053] S3, data model training, the data model training includes an early stage of data model training, a mid stage of data model training and a late stage of data model training;

[0054] S301, in the early stage of data model training, the relative position of the sentence is encoded in a relative position manner, and the full sentence position of the vocabulary is no longer considered;

[0055] S302, in the middle of data model training, the data is encoded and decoded. The encoding adopts a segmented recursive mechanism to encode the vector, and the decoding adopts a multi-query attention mechanism to allow all heads to share the same parameters; the Head here refers to the number of independent parallel sub-layers of the attention mechanism.

[0056] S303, at the end of data model training, the output is also the total number of words in the current sentence plus the size of the preceding and following tags, with a representation vector of dimension 768, the representation vectors of the candidate words are extracted as the vector data set, and the trained BERT model is used to screen each text block to obtain several candidate words, and the representation vectors of the candidate words are extracted as the vector data set.

[0057] During the overall training phase, the model can automatically determine the importance of different elements in the input sequence to generate a specific output when processing text at the encoding and decoding layers. Several text blocks with higher importance are selected until the BERT model converges;

[0058] S4, randomly input a long commercial text into the trained BERT model, and output the representation vector of the candidate word as the candidate word vector;

[0059] S5. Use the K-Means clustering algorithm to iteratively calculate the distance between the candidate word vector and the cluster center, sort and obtain the candidate words corresponding to the first N shortest distances as keyword samples;

[0060] Specifically, the S1 includes:

[0061] S101, read all the text information of the data set, read each text and regard it as ending with a line break character " / n", and split each text information into short sentence blocks as a long text, and separate each short sentence block into several candidate words [X 0 , X 1 , X 2 …X T ], where X 0 , X 1 , X 2 …X T Indicates different candidate words;

[0062] S102, for each unstructured long text block X t The data is preprocessed by word segmentation, part-of-speech screening, deduplication, and stop word removal to obtain the input format of CLS sentence SEP, where CLS represents the head tag and SEP represents the tail tag;

[0063] Specifically, in the step S301, at the token stage, the short sentence is calculated by convolution with a candidate word list size of [30522, 768], where 30522 represents the number of candidate words, 768 represents the vector dimension of the candidate word, and the output is the number of candidate words in the current short sentence plus the size of the preceding and following tokens, and the embedding vector representation of the dimension is 768;

[0064] Specifically, the relative position in step S301 is, when the jth word is queried, the position encoding vector corresponding to the i-th word is rj-i , and when the query word becomes the kth word, the position code corresponding to the i-th word becomes r k-i ,The position information of a word is based only on the current query word;

[0065] Specifically, the step S302 includes:

[0066] S3021, obtain the entire sentence and candidate word vectors and input them into the encoding layer stage. Each encoding block introduces a segmented recursive strategy, which uses an additional fixed-length storage space to save the hidden state of the previous segment. When processing the current segment, these stored hidden states will interact and integrate information with the current segment, thereby establishing a connection between the two segments. The output of the nth layer of the second segment is integrated with the calculation method of context semantics.

[0067]

[0068] Among them, the subscripts τ and τ+1 are used to represent two adjacent segments. For n-1 layers, the hidden states of two adjacent segments are expressed as and Indicates that it is used normally during forward propagation Value, two adjacent fragments are spliced ​​according to the sequence length to obtain It means concatenating two sequences of length T into a sequence of length 2T. It does not participate in gradient regression and is only used as pure data. The hidden state refers to the internal state calculated by the model when processing the input of each time series.

[0069] S3022, the hidden state of the nth layer and The joint sequence is combined with the learnable linear transformation matrix with different weights to obtain the set of query Qurey, key Key and value Value. The specific calculation process is as follows:

[0070]

[0071] The output tensor is split from the semantic level, that is, each head first obtains a set of query q, key k and value v for attention mechanism calculation as part of the representation vector of the entire sentence and candidate word;

[0072] S3023, the attention weight is obtained by the current long sequence method, and then the features are integrated to establish the association relationship between the two fragments. The specific calculation process is as follows:

[0073]

[0074] in, is the encoding vector corresponding to the current fragment, They respectively represent the parameter values ​​obtained by the current fragment;

[0075] S3024, cache the layer output of the previous fragment and reuse it in the processing of the next fragment, and so on for all fragments. The calculation method is as follows:

[0076]

[0077] Among them, N is the number of layers of the model, S is the number of all fragments, and depend represents a general dependency relationship. This method avoids a lot of redundant calculations. The final output of the last fragment depends on the first-layer output of the first fragment, which greatly reduces the calculation efficiency and solves the long text input limitation of Bert and the training model. The encoding vector corresponding to the entire sentence and the candidate word and the hidden state corresponding to the current fragment are extracted, which means outputting the vector representation corresponding to the entire sentence and the candidate word;

[0078] S3025. Construct a mask matrix Mask matrix at the first layer of each decoding block. The mask matrix will recalculate the value of each pixel in the image. T Multiply the masked attention score matrix bit by bit with the Mask matrix to get the masked attention score matrix. Multiply the masked attention score matrix with the V matrix to get the output vector of the entire sentence and candidate words. All heads share the same key and value matrix, and assign different weights to each position in the input sequence, so as to effectively capture the context information and obtain the encoding vectors of the entire sentence and candidate words output by the encoding layer, as shown in the following multidimensional matrix Z.

[0079]

[0080] S3026, each row of the encoding vector, i.e., the multidimensional matrix, represents the decoding vector of each candidate word, and passes through the feedforward network, i.e., the fully connected layer, so as to keep the vector dimension represented by each word the same, and the residual structure outputs the vector matrix, which is calculated as follows:

[0081]

[0082] Specifically, the step S5 includes:

[0083] S501, using the word vector set as a sample, clustering initialization, selecting K initial cluster centers, setting a threshold, calculating the distance between each candidate word vector and each cluster center, and assigning it to the cluster center closest to it, recalculating the mean of all word vector sets in its cluster center, and obtaining a new cluster center. The cluster center point update formula is calculated as follows:

[0084]

[0085] in, is the coordinate vector of the original cluster center, The sample position vector is used to find the new centroid by weighted averaging. Repeat the allocation and update steps until the cluster center no longer changes significantly, that is, the distance between the new cluster center and the old cluster center is less than a preset threshold.

[0086] S502: According to the clustering results, select the words with high frequency or representativeness from each cluster as keywords. Use the Euclidean distance calculation method to calculate the distance between each word and the cluster center; sort in ascending order according to the obtained distance, and select the top N words as text keywords.

[0087] The present invention solves the problem of long text input limitations and contextual semantic barriers caused by factors such as context, cultural differences, and language structure, and introduces two important mechanisms: segmented recursion and relative position encoding: the former aims to build the relationship between segments, while the latter enables the effective integration of vocabulary position information during segment processing. By caching the layer output of the previous segment and reusing it in the processing of the next segment, this method avoids a large amount of redundant calculations, thereby solving the input limitations of long texts. Relative position encoding can perform better in processing long-distance dependencies and generalizing to sequences of different lengths. The multi-query cross-attention mechanism reduces the number of parameters and improves the reasoning speed of the model by sharing Key and Value weights.

[0088] The above description is only a preferred embodiment of the present invention, so all equivalent changes or modifications made according to the structure, characteristics and principles described in the scope of the patent application of the present invention are included in the scope of the patent application of the present invention.

Claims

1. A keyword clustering extraction method based on segmented recursion and multi-query attention mechanism, characterized in that: The steps include: S1. Collect information to build a data set, obtain information from multiple commercial long texts, and use the obtained information as a training data set; S2, data preprocessing, converting the input long text into a standard format; S3, data model training, the data model training includes an early stage of data model training, a mid stage of data model training and a late stage of data model training; S301, in the early stage of data model training, the relative position of the sentence is encoded in a relative position manner, and the full sentence position of the vocabulary is no longer considered; S302, in the middle of data model training, the vector is encoded and decoded. The encoding adopts a segmented recursive mechanism to encode the vector, and the decoding adopts a multi-query attention mechanism to allow all heads to share the same parameters; S303, at the end of data model training, the output is also the number of words in the current sentence plus the size of the preceding and following tags, with a representation vector of dimension 768, the representation vectors of the candidate words are extracted as a vector data set, and the trained BERT model is used to screen each text block to obtain several candidate words, and the representation vectors of the candidate words are extracted as a vector data set; S4, randomly input a long commercial text into the trained BERT model, and output the representation vector of the candidate word as the candidate word vector; S5. Use the K-Means clustering algorithm to iteratively calculate the distance between the candidate word vector and the cluster center, and sort and obtain the candidate words corresponding to the first N shortest distances as keyword samples.

2. The keyword clustering extraction method based on segmented recursion and multi-query attention mechanism according to claim 1 is characterized in that: The S1 includes: S101, read all the text information of the data set, read each text and regard it as ending with a line break " / n", and split each text information into short sentence blocks as a long text, and divide each short sentence block into several candidate words [X0, X1, X2...X T ], where X0, X1, X2…X T Indicates different candidate words; S102, for each unstructured long text block X t The data is preprocessed by word segmentation, part-of-speech screening, deduplication and stop word removal to obtain the input format of CLS sentence SEP, where CLS represents the head tag and SEP represents the tail tag.

3. The keyword clustering extraction method based on segmented recursion and multi-query attention mechanism according to claim 1 is characterized in that: In the step S301, at the token stage, the short sentence is calculated by convolution with a candidate word list size of [30522, 768], where 30522 represents the number of candidate words, 768 represents the vector dimension of the candidate word, and the output is the number of candidate words in the current short sentence plus the size of the preceding and following tags, and the embedding vector representation of the dimension is 768.

4. The keyword clustering extraction method based on segmented recursion and multi-query attention mechanism according to claim 1 is characterized in that: The relative position in step S301 is, when the jth word is queried, the position encoding vector corresponding to the ith word is r j-i , and when the query word becomes the kth word, the position code corresponding to the i-th word becomes r k-i , the position information of a word is based only on the current query word.

5. The keyword clustering extraction method based on segmented recursion and multi-query attention mechanism according to claim 1 is characterized in that: The step S302 includes: S3021, get the entire sentence and candidate word vectors and input them into the encoding layer stage. Each encoding block introduces a segmented recursive division strategy, using an additional storage space with a fixed length to save the hidden state of the previous segment. When processing the current segment, these stored hidden states will interact and integrate information with the current segment, thereby establishing a connection between the two segments. The output of the nth layer of the second segment is integrated with the calculation method of context semantics. Among them, the subscripts τ and τ+1 are used to represent two adjacent segments. For n-1 layers, the hidden states of two adjacent segments are expressed as and Indicates that it is used normally during forward propagation Value, two adjacent fragments are spliced ​​according to the sequence length to obtain It means concatenating two sequences of length T into a sequence of length 2T, which does not participate in gradient regression and is only used as pure data; S3022, the hidden state of the nth layer and The joint sequence is combined with the learnable linear transformation matrix with different weights to obtain the set of query q, key k and value v. The specific calculation process is as follows: The output tensor is split from the semantic level, that is, each head first obtains a set of query q, key k and value v for attention mechanism calculation as part of the representation vector of the entire sentence and candidate word; S3023. The attention weight is obtained by the current long sequence method, and then the features are integrated to establish the association relationship between the two fragments. The specific calculation process is as follows: in, is the encoding vector corresponding to the current fragment, Respectively represent the parameter values ​​obtained by the current layer; S3024, cache the layer output of the previous fragment and reuse it in the processing of the next fragment, and so on for all fragments. The calculation method is as follows: Where N is the number of layers in the model, S is the number of all segments, and depend represents a general dependency relationship. The encoding vector corresponding to the entire sentence and the candidate word and the hidden state corresponding to the current segment are extracted, which means outputting the vector representation corresponding to the entire sentence and the candidate word. S3025, construct a mask matrix Mask matrix at the first layer of each decoding block, and convert the QK calculated at the encoding layer into T Multiply the masked attention score matrix bit by bit with the Mask matrix, and multiply the masked attention score matrix with the V matrix to get the output vector of the entire sentence and candidate words. All heads share the same key and value matrix, and assign different weights to each position in the input sequence, so as to effectively capture context information and obtain the encoding vectors of the entire sentence and candidate words output by the encoding layer; S3026, each row of the encoding vector, i.e., the multidimensional matrix, represents the decoding vector of each candidate word, and passes through the feedforward network, i.e., the fully connected layer, so as to keep the vector dimension represented by each word the same, and the residual structure outputs the vector matrix, which is calculated as follows:

6. The keyword clustering extraction method based on segmented recursion and multi-query attention mechanism according to claim 1 is characterized in that: The S5 step includes: S501, using the word vector set as a sample, clustering initialization, selecting K initial cluster centers, setting a threshold, calculating the distance between each candidate word vector and each cluster center, and assigning it to the cluster center closest to it, recalculating the mean of all word vector sets in its cluster center, and obtaining a new cluster center. The cluster center point update formula is calculated as follows: in, is the coordinate vector of the original cluster center, The sample position vector is used to find the new centroid by weighted averaging. Repeat the allocation and update steps until the cluster center no longer changes significantly, that is, the distance between the new cluster center and the old cluster center is less than a preset threshold. S502: According to the clustering results, select the words with high frequency or representativeness from each cluster as keywords. Use the Euclidean distance calculation method to calculate the distance between each word and the cluster center; sort in ascending order according to the obtained distance, and select the top N words as text keywords.

Images