A keyword / sequence extraction method through feature set dimensionality reduction

Abstract

The present invention discloses a keyword / sequence extraction method through feature set dimensionality reduction, including processing a text set to obtain a feature set; obtaining a vector representation of the feature set; reducing the dimension of the feature set to obtain a feature set after dimensionality reduction and a mapping relationship between the feature set and the feature set after dimensionality reduction; keyword / sequence extraction. The present invention can not only improve the quality of keyword / sequence extraction, and reduce the computing power and memory overhead required for vector calculation in the keyword / sequence extraction model training process of massive data, but the dimensionality reduction algorithm module therein can also be well applied to the dimensionality reduction of high-dimensional feature sets (similarity cannot be calculated at one time).

Description

Technical Field

[0001] The present invention relates to the technical field of keyword extraction, and in particular to a keyword / sequence extraction method through dimensionality reduction of feature sets. Background Art

[0002] Keyword / sequence extraction is to extract the words or sequences that best express the text, which is a common task in natural language understanding. Commonly used keyword / sequence extraction models include tf-idf, textRank, calculating word / sequence frequency, and selecting top-n words / sequences based on semantic similarity. Regardless of which model is used, it is modeled based on a set of text features. The features dimension (the number of features obtained through word segmentation and n-gram) and the diversity of features (elements in features) (representations in different languages, different tenses, and different styles) will affect the performance of the model (accuracy, calculation speed, etc.) and the model's requirements for equipment (computing power and memory). For example:

[0003] 1) The diversity of feature representations will affect the model extraction results and feature dimensions:

[0004] 2) As the dimension of features increases, the requirements for equipment for model training will also increase:

[0005] exp: For massive data, the dimensions of features will be very large (for example, millions, tens of millions, or billions), and the iterative calculation process within the model will not be able to run due to insufficient memory due to the large vector dimensions. Summary of the invention

[0006] To solve the above technical problems, the purpose of the present invention is to provide a keyword / sequence extraction method through feature set dimensionality reduction, which method first obtains a feature set features (after duplicate removal); secondly, converts features into a vector representation features_embedding; then, according to the similarity and threshold between features_embedding elements (which can be manually set according to different tasks and different requirements), features are reduced in dimension to obtain a feature set features_reduction after dimensionality reduction and a mapping relationship map between features and features_reduction; finally, the feature set features_reduction after dimensionality reduction is used to perform keyword / sequence extraction.

[0007] The purpose of the present invention is achieved through the following technical solutions:

[0008] A keyword / sequence extraction method by reducing the dimension of a feature set, comprising:

[0009] A. Process the text set to obtain the feature set;

[0010] B. Obtain the vector representation of the feature set;

[0011] C. Reduce the dimension of the feature set to obtain the feature set after dimension reduction and the mapping relationship map between the feature set after dimension reduction;

[0012] D. Keyword / sequence extraction.

[0013] Compared with the prior art, one or more embodiments of the present invention may have the following advantages:

[0014] It can not only improve the quality of keyword / sequence extraction and reduce the computing power and memory overhead required for vector calculation during the training of keyword / sequence extraction models for massive data, but the dimensionality reduction algorithm module can also be well applied to the dimensionality reduction of high-dimensional feature sets (similarity cannot be calculated at one time). BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a flow chart of the keyword / sequence extraction method through dimensionality reduction of feature sets. DETAILED DESCRIPTION

[0016] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below in conjunction with embodiments and drawings.

[0017] like Figure 1 As shown in FIG. 1 , a keyword / sequence extraction method based on feature set dimensionality reduction is shown, including:

[0018] Get feature set features: This module mainly processes the text set corpus, obtains the feature set through word segmentation and n-gram, and arranges them in descending order according to the feature frequency to obtain features = [feature_0, feature_1,........, feature_n-1], n>1, features is n-dimensional, and each dimension represents a feature.

[0019] Get the vector representation of features features_embedding: This module is mainly used to obtain the semantic representation of features. It uses PreTrained Language Mode (for example: multi-lingual-sBert, other PTLMs can also be used) to convert each feature into a high-dimensional vector, features_embedding = [f_emb_0, f_emb_1, ..., f_emb_n-1], n>1, where f_emb_n-1 is m-dimensional (m is determined by the PTLMs model parameters, and different dimensions can be selected as needed), so features_embedding is a matrix of n rows and m columns, and f_emb_n-1 represents the vector representation of feature_n-1.

[0020] Features dimensionality reduction: This module mainly reduces the dimensionality of features according to the similarity and threshold between features_embedding elements, that is, reduces the value of n, and finally obtains the reduced-dimensional feature set features_reduction and the mapping relationship between features and features_reduction map(features,features_reduction). The dimensionality reduction process is as follows:

[0021] (1) Divide features into blocks of a certain length L: L <n

[0022] features = [fch_0,fch_1,......,fch_i,......,fch_p-1], p = (n / L+1) rounded down;

[0023] in:

[0024] fch_i=[feature_0,feature_1,.....,features_L-1],L <n;

[0025] fch_p-1=[feature_0,feature_1,......,features_j-1],j<=L.

[0026] (2) Divide features_embedding into blocks by row according to a certain length L:

[0027] features_embedding = [fembch_0, fembch_1,......fembch_i,......, fembch_p - 1], where p = floor(n / L + 1);

[0028] Where:

[0029] fembch_i = [f_emb_0, f_emb_1,......, f_emb_L - 1], with L < n;

[0030] fembch_p - 1 = [f_emb_0, f_emb_1,......, f_emb_j - 1], where j <= L.

[0031] (3) Calculate the similarity S between the i-th and j-th blocks in features i,j , and merge and update features and features_embedding:

[0032] Traversal order: Traverse the diagonal and upper triangle row by row, from left to right, from top to bottom, i.e., i ≤ j < p,

[0033]

[0034] Where:

[0035] S i,j = cosine(fembch_i, fembch_j), where fembch_i corresponds to the vector representation of fch_i, and the same for j

[0036] Expand further:

[0037]

[0038] a) Take i and j in sequence. If i ≤ j < p, then calculate S i,j ;

[0039] b) Based on the set similarity threshold T (0 < T < 1.0) and S i,j perform feature merging. Merging process:

[0040] For If x ≥ y, then continue to traverse the next element;

[0041] If then the y-th feature in the fch_j block merges with the x-th feature in the fch_i block;

[0042] Loop through b1 and b2 until S is fully traversedi,j All elements in .

[0043] c) Update features and features_embedding:

[0044] According to the record S i,j For the merged features, delete the corresponding features in the j-th block fch_j in features, and the dimension of fch_j is reduced;

[0045] According to the record S i,j For the merged features, delete the corresponding feature vector representation in the j-th block fembch_j in features_embedding, and the dimension of fembch_j is reduced.

[0046] d) Until all blocks in S are traversed, due to the reduction of the dimension of features_embedding, S i,j The calculation speed is faster and will get faster with the number of iterations.

[0047] (4) When the dimension of features no longer decreases, the training process ends and the reduced feature set features_reduction and the mapping relationship from features to features_reduction map(features,features_reduction) are obtained.

[0048] Keyword / sequence extraction: This module converts the model input features into corresponding features_reduction according to map(features, features_reduction), and then trains the extraction model to obtain the extraction results.

[0049] The specific embodiments are as follows:

[0050] Task type: After topic clustering, extract keywords / sequences for each topic category;

[0051] Task data: There are 8.32 million documents in total, which are aggregated into 6,000 topic categories

[0052] The processing flow is as follows:

[0053] Get the feature set features: extract the keywords of 8.32 million documents as the feature set. There are 7.23 million features in total, so features = [feature_0, feature_1,........, feature_n-1], n = 7230000.

[0054] Get the vector representation of features features_embedding: This module is mainly for the semantic representation of features. It uses PreTrained Language Mode (for example: multi-lingual-sBert, other PTLMs can also be used) to convert each feature into a high 384-dimensional vector, features_embedding = [f_emb_0, f_emb_1, ..., f_emb_n-1], features_embedding is a matrix of n rows and m columns, n = 7230000, m = 384.

[0055] Features dimension reduction: The parameter L is 100000 (can be adjusted according to device performance), the similarity threshold T is 0.8 (can be adjusted according to task requirements), and the dimension of features_reduction after dimension reduction is 1430000, which is 1 / 5 of the dimension of features.

[0056] Keyword / sequence extraction: This module converts the model input features into corresponding features_reduction according to map(features, features_reduction), and then performs model training. The extracted words / sequences can better express the topic.

[0057] Although the embodiments disclosed in the present invention are as above, the above contents are only embodiments adopted for facilitating the understanding of the present invention and are not intended to limit the present invention. Any technician in the technical field to which the present invention belongs can make any modifications and changes in the form and details of the implementation without departing from the spirit and scope disclosed in the present invention, but the patent protection scope of the present invention shall still be subject to the scope defined in the attached claims.

Claims

1. A keyword / sequence extraction method by reducing the dimension of feature set, characterized in that: include A. Process the text set to obtain the feature set; B. Obtain the vector representation of the feature set; C. Reduce the dimension of the feature set to obtain the feature set after dimension reduction and the mapping relationship between the feature set and the feature set after dimension reduction; D. Keyword / sequence extraction; The C specifically includes: reducing the dimension of features according to features and features_embedding to obtain a feature set features_reduction after dimension reduction and a mapping relationship map between features and features_reduction; the dimension reduction process specifically includes: (1) Divide features into blocks of a certain length L; (2) Divide features_embedding into blocks by row according to a certain length L; (3) Calculate the similarity S between the i-th block and the j-th block in features in sequence i,j , and merge and update features and features_embedding; (4) When the features dimension no longer decreases, the training process ends and the reduced feature set features_reduction and the mapping relationship map from features to features_reduction are obtained; In (1), features = [fch_0, fch_1, ..., fch_i, ..., fch_p-1], p = (n / L+1) rounded down; in: fch_i=[feature_0,feature_1,.....,features_L-1],L <n; fch_p-1=[feature_0,feature_1,......,features_j-1],j<=L; In (2), features_embedding=[fembch_0,fembch_1,...fembch_i,...,fembch_p-1]], p=(n / L+1) round down; in: fembch_i=[f_emb_0,f_emb_1,...,f_emb_L-1],L <n; fembch_p-1=[f_emb_0,f_emb_1,...,f_emb_j-1],j<=L; In (3): Traversal order: traverse the diagonal and upper triangle row by row, from left to right, from top to bottom, that is, i≤j<p, in: S i,j = cosine(fembch_i,fembch_j), fembch_i corresponds to the vector representation of fch_i, j is also Further expansion: In (3): (a) Take i and j in order. If i≤j<p, calculate S i,j ; (b) According to the set similarity thresholds T and S i,j , where 0 < T < 1.0, perform feature merging; (c) Update features and features_embedding; (d) Until all blocks in S are traversed, S i,j The calculation speed is faster, and it will get faster with the number of iterations; In (b), the feature merging process includes: b1) For If x≥y, continue to traverse the next element; b2) If Then the yth feature in the fch_j block is merged with the xth feature in the fch_i block; b3) Loop through b1 and b2 until all S are traversed i,j All elements in In (c), updating features and features_embedding includes: c1) According to the recorded S i,j For the merged features, delete the corresponding features in the j-th block fch_j in features, and the dimension of fch_j is reduced; c2) According to the recorded S i,j For the merged features, delete the corresponding feature vector representation in the j-th block fembch_j in features_embedding, and the dimension of fembch_j is reduced.

2. The keyword / sequence extraction method by feature set dimensionality reduction as claimed in claim 1, characterized in that: In A, a feature set is obtained by word segmentation and n-gram, and is arranged in descending order according to feature frequency to obtain features = [feature_0, feature_1, .........., feature_n-1], n>1, features is n-dimensional, and each dimension represents a feature.

3. The keyword / sequence extraction method by feature set dimensionality reduction as claimed in claim 1, characterized in that: In B, each feature is converted into a high-dimensional vector, features_embedding = [f_emb_0, f_emb_1, ..., f_emb_n-1], n>1, where f_emb_n-1 is m-dimensional. Therefore, features_embedding is a matrix of n rows and m columns, and f_emb_n-1 represents the vector representation of feature_n-1.

Images