A feature extraction method based on a multi-scale grouping interaction neural network

Abstract

This invention discloses a feature extraction method based on a multi-scale grouped interactive neural network. The method includes: labeling several original input data from an artificial intelligence system to obtain a label sequence; establishing a multi-scale grouped interactive neural network containing a multi-scale transformation module, which includes several single-scale transformation layers with different grouping operations arranged in hierarchical order; inputting the label sequences of each original input data into the multi-scale grouped interactive neural network, and outputting a data table with extracted features after processing. This invention adopts an end-to-end feature extraction approach, eliminating the need for manual feature design; it achieves layer-by-layer propagation of global contextual information through a multi-scale grouped interactive mechanism; and its computational complexity is significantly lower than that of structures based on global self-attention. It does not rely on information compression or sparse approximation, maintains the integrity of feature representation, and can be widely applied to various scenarios such as tabular data, sequence data, image data, and biological data.

Description

Technical Field

[0001] This invention relates to a feature extraction method, which falls under the fields of artificial intelligence and deep learning technology, and specifically to a feature extraction method based on a multi-scale grouped interactive neural network. Background Technology

[0002] Feature extraction is a crucial step in machine learning and artificial intelligence systems, aiming to transform raw data into an information representation that effectively expresses the semantics of the data. Traditional fully connected neural networks suffer from large parameter sizes, low computational efficiency, and are prone to overfitting when processing high-dimensional data.

[0003] Convolutional neural networks (CNNs) have achieved good results in image and other data processing through local connections and parameter sharing mechanisms, but their modeling capabilities are mainly limited to local receptive fields, making it difficult to effectively capture long-range dependencies. Recurrent neural networks (RNNs) can model temporal dependencies to some extent in sequence modeling, but they still suffer from problems such as vanishing gradients and low computational efficiency in long sequence scenarios. In recent years, the Transformer structure based on self-attention mechanisms can model global contextual information, but its computational complexity increases quadratically with the input size, resulting in excessive computational and storage overhead in long sequence or large-scale data scenarios, limiting practical applications.

[0004] To address the aforementioned issues, existing technologies have proposed various methods to accelerate or approximate attention mechanisms. However, these methods typically require information compression or the introduction of structural constraints, which can easily lead to a decrease in feature representation capabilities. Therefore, there is an urgent need for a feature extraction method that can reduce computational complexity while maintaining global context modeling capabilities and is applicable to various data types. Summary of the Invention

[0005] To address the problems existing in the background technology, this invention provides a feature extraction method based on a multi-scale grouped interactive neural network. This method effectively reduces computational complexity while ensuring the integrity of feature representation, achieving efficient modeling of long-distance dependencies in high-dimensional data, and is suitable for representation learning and feature modeling of high-dimensional data.

[0006] The technical solution adopted in this invention is: The feature extraction method based on a multi-scale grouped interactive neural network of the present invention includes: Step S1: Perform labeling processing on several raw input data of the artificial intelligence system to obtain a label sequence.

[0007] Step S2: Establish a multi-scale grouping interactive neural network containing a multi-scale transformation module (MST). The multi-scale transformation module (MST) contains several single-scale transformation layers with different grouping operations arranged in hierarchical order.

[0008] Step S3: Input the labeled sequences of each original input data into the multi-scale grouped interactive neural network, and output the data table after feature extraction.

[0009] In step S1, each original input data is tokenized and converted into a token sequence consisting of N tokens, where each token is a d-dimensional feature vector.

[0010] In step S2, the multi-scale grouped interactive neural network includes a multi-scale transformation module (MST) or several sequentially connected MSTs for feature transformation. Each MST includes... K Single-scale conversion layer, K equal to log n N Rounding up, n The preset grouping size is used; the single-scale transformation layer includes a marker-level feature transformation unit, a grouping interaction unit, and a feature normalization unit connected in sequence; the marker-level feature transformation unit specifically performs a marker vector generation operation on each marker token; the grouping interaction unit specifically performs a grouping operation on each marker vector of each marker sequence and then performs a feature fusion operation on the marker vectors within the same group, with the number of marker vectors remaining unchanged after the feature fusion operation on each group; the feature normalization unit specifically performs a normalization operation on each marker vector after the feature fusion operation; each marker vector interacts directly or indirectly with all other marker vectors in a multi-scale transformation module (MST) to model the global context of all marker tokens.

[0011] The grouping operation performed on each tag vector of each tag sequence in the grouping interaction unit is as follows: The hierarchical order settings in the multi-scale conversion module MST K Each single-scale transformation layer has its own index. l , in the marker sequence N The labeled vectors are divided into Q Groups, each group being the largest. n , Q equal N Divide by n The result of rounding up; each marker vector has its own index. i ,That n number system , , … They are respectively the 1st, 2nd, ... K +1 base value; in the... l In the single-scale transformation layer of the layer, the first i The _th marker vector is divided into the _th ... q Groups, Group Number q Specifically as follows: No. i The tag vector in the th ... q The index within each group is: .

[0012] The aforementioned group interaction unit performs feature fusion operations only within the same group, targeting the first... i Tag vectors x i First, determine the number i Tag vectors x i and the first in the same group j Tag vectors x j The correlation value between them is used to determine the interaction weight. Finally, the label vector after feature fusion is obtained. x i for .

[0013] In step S3, the data table after feature extraction includes a token sequence consisting of N interacting tokens, each of which is a d-dimensional feature vector.

[0014] The data table continues to perform a labeling and generalization operation, and then, after being processed by the predictor, outputs the prediction result of the original input data to identify the meaning of the original input data.

[0015] The beneficial effects of this invention are: 1. This invention adopts an end-to-end feature extraction method, eliminating the need for manual feature design.

[0016] 2. This invention achieves the layer-by-layer propagation of global context information through a multi-scale grouping interaction mechanism.

[0017] 3. The computational complexity of this invention is... O ( N log N The performance is significantly lower than that of structures based on global self-attention.

[0018] 4. This invention does not rely on information compression or sparse approximation, and maintains the integrity of feature expression.

[0019] 5. This invention can be widely applied to various scenarios such as tabular data, sequence data, image data, and biological data. Attached Figure Description

[0020] Figure 1 This is an architecture diagram of the feature extraction method based on multi-scale group interaction of the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] This invention provides a feature extraction method based on a multi-scale grouped interactive artificial neural network, comprising the following steps: Step S1: Data Tokenization Processing. The raw input data of the artificial intelligence system is tokenized, converting each input data point into a sequence of N tokens. Each token is... d The 3D feature vector is as follows: When the original input data is numerical data such as tabular data, the numerical data is vector-extended to form a labeled sequence. For example, an instance of the original data is... N When dealing with a dimensional vector, repeat the original vector, repeating it for each dimension. d Sub-composition d Dimension, forming a d × N The matrix.

[0023] When the original input data is sequence-type data such as a string of symbols or words (e.g., genome sequences or text documents), embedding mapping methods are used to represent each symbol or word as a sequence of symbols or words. d dimensional vector, thus forming a d × N The matrix. For example, for a DNA sequence ACCCGT of length N=6, each base symbol is first encoded using one-hot encoding to form a base label sequence, where each label is 4-dimensional, and then processed through the 4-dimensional matrix. d A linear transformation mapping of dimension transforms all labels into d dimension.

[0024] If the original input data already exists N indivual D Dimensional elements ( D≠d Linear projection maps each element to... d Dimension, thus forming a d × N The matrix.

[0025] The aforementioned embedding or linear mapping parameters can be learned during model training.

[0026] Step S2: Multi-scale transformation processing. Establish a multi-scale grouping interactive neural network containing one or more MST modules connected sequentially. Each MST module transforms the labeled features at different scales. Each MST module contains several single-scale transformation layers with different grouping operations arranged hierarchically. The input and output representations of each data instance in each single-scale transformation layer are both single-scale and single-scale. d × N The matrix.

[0027] In practical implementation, the multi-scale grouped interactive neural network includes M Feature transformation is performed by a series of cascaded multiscale transformation modules (MSTs). Each MST includes... K Single-scale conversion layer, K equal to log n N Rounding up, n The preset group size, n <N The single-scale transformation layer includes a token-level feature transformation unit, a grouping interaction unit, and a feature normalization unit connected in sequence. The token-level feature transformation unit performs a token-by-token transformation on each token to obtain a token vector. The token-level feature transformation is a function that operates on a token-by-token basis, specifically performing the same equal-dimensional linear mapping on each token. The grouping interaction unit performs a grouping operation on each token vector of each token sequence and then performs a feature fusion operation on the token vectors within the same group. The number of token vectors after the feature fusion operation remains unchanged for each group. The feature normalization unit performs a normalization operation on each token vector after the feature fusion operation, such as LayerNorm. Each token vector interacts directly or indirectly with all other token vectors in a multi-scale transformation module (MST) to model the global context of all tokens.

[0028] The components of each single-scale transformation layer are as follows: 1) Label-level feature transformation unit: for all label vectors x i Perform linear transformation x i ← Ux i, where ← is the computer assignment symbol. U It is a linear transformation matrix.

[0029] 2) Group interaction unit, which performs the following two functions: 2.1) Grouping operations: To achieve global contextual information modeling while ensuring computational efficiency, different grouping operations are used for the labels in different single-scale transformation layers. Specifically, let the labeling process result in... N Each group contains 10 tags, and each tag is assigned a unique global index number, which indicates the tag's position in the tag sequence. Each group is pre-defined to contain 10 tags. n 1 marker, of which n Less than N In the first single-scale transformation layer, adjacent values ​​are transformed according to the index order of the labels. n Each label is divided into the same group, thus forming multiple groups. In the second single-scale transformation layer, while keeping the number of labels in each group unchanged, the labels within each group are regrouped, so that some labels that did not belong to the same group in the first single-scale transformation layer are assigned to the same group in the second single-scale transformation layer. In subsequent single-scale transformation layers, the labels are grouped again according to the preset grouping rearrangement operation, so that the same label interacts with different label sets in different single-scale transformation layers.

[0030] Specifically, the grouping operation performed on each tag vector of each tag sequence in the grouping interaction unit is as follows: The grouping rule is that the grouping of each label is indexed by a single-scale transformation layer. l and the index of tags n The number system is determined by the base, the first... One digit determines the index within the group, while the other digits determine the group number.

[0031] The hierarchical order settings in the multi-scale conversion module MST K Each single-scale transformation layer has its own index. l , in the marker sequence N The labeled vectors are divided into Q Groups, each group being the largest. n , Q equal N Divide by n The result of rounding up; each marker vector has its own index. i ,That n number system , , … They are respectively the 1st, 2nd, ... K+1 base value; in the... l In the single-scale transformation layer of the layer, the first i The _th marker vector is divided into the _th ... q Groups, Group Number q Specifically as follows: No. i The tag vector in the th ... q The index within each group is: .

[0032] For example, the label sequence length is N=8000, and each subgroup size is n=10. Each label sequence will pass through a total of 4 single-scale transformation layers, where 4 is... The floor function is N divided by n. In each single-scale transformation layer, all labels in the label sequence are divided into Q=800 groups, which is the floor function of N divided by n.

[0033] For example, one of the markers has an index of 1234 in the entire sequence: in the first level single-scale transformation, the marker is in group 234 with an index of 1 within the group; in the second level single-scale transformation, the marker is in group 134 with an index of 2 within the group; in the third level single-scale transformation, the marker is in group 124 with an index of 3 within the group; and in the fourth level single-scale transformation, the marker is in group 123 with an index of 4 within the group.

[0034] With the above grouping method, in any single-scale transformation layer, the label only interacts with the labels in its own group in terms of features; and as multiple single-scale transformation layers are processed layer by layer, the information between the labels can be gradually propagated through the cross-over between different groups, thereby realizing the context modeling of all labels.

[0035] For example, when the number of tags N The group size is 16. n When the value is 4: In the first single-scale transformation layer, the tags are divided into 4 groups according to the index order, and each group contains tags with 4 consecutive indices; in the second single-scale transformation layer, the groups are recombined so that tags from different original groups are assigned to the same group; in the third single-scale transformation layer, the group combination method is further adjusted so that the tags can establish group interaction relationships with more different tags.

[0036] By using the above multi-level grouping and rearrangement method, global information propagation between tags is achieved without introducing global pairwise interactions.

[0037] In one or more implementations, the grouping and rearrangement operation is determined based on the positional features of the tag index at different scales, so that the grouping differs in different single-scale transformation layers according to the index position information used. It should be understood that the above grouping methods are merely illustrative. Any grouping operation that enables the same tag to participate in grouping of different tag sets at different levels in different single-scale transformation layers, and achieves gradual propagation of information between tags through multi-level intra-group interactions, falls within the protection scope of this invention.

[0038] 2.2) Group interaction: In one optional implementation, feature interaction between tags within a group can be achieved in the group interaction unit as follows: Assume a group contains multiple tags, each tag corresponding to a feature vector. For any two tags within the group, a correlation value representing the degree of correlation between them is obtained by linearly mapping their feature vectors; the correlation value is then normalized to obtain the corresponding interaction coefficient, i.e., the weight. After obtaining the interaction coefficient, the feature vectors of each tag within the group are linearly fused to update the feature representation of the current tag.

[0039] Specifically, feature fusion operations, such as scaled dot-product attention, are performed only within the same group in the group interaction unit, without performing label interactions across groups. i Tag vectors x i First, determine the number i Tag vectors x i and the first in the same group j Tag vectors x j The correlation value between them is used to determine the interaction weight. Finally, the label vector after feature fusion is obtained. x i for .

[0040] In one or more alternative implementations, the correlation value can be obtained by vector inner product, and the interaction coefficient can be processed by a normalization function; however, it should be understood that this calculation method is only an illustrative example, and the present invention is not limited to the above calculation method. Any implementation method that can achieve linear fusion of intra-group labeled features should be considered to fall within the protection scope of the present invention.

[0041] 3) Feature normalization unit: used to normalize the labeled features after interaction.

[0042] The above units process the input labeled features in sequence.

[0043] Step S3: Feature Output or Aggregation Processing. After the multi-scale transformation module completes its processing, the labeled features can be further processed according to the actual application requirements. Specifically, the labeled sequences of each original input data are input into a multi-scale grouped interactive neural network, and the processed data is output as a table with extracted features.

[0044] The data table after feature extraction includes a token sequence consisting of N interactive tokens, each interactive token being a d-dimensional feature vector.

[0045] In one implementation, all labeled features are aggregated to obtain a holistic feature vector representing the entire data object. In another implementation, labeled features are directly output for tasks such as sequence labeling and location prediction. The resulting feature representation can be used for downstream tasks such as classification, regression, clustering, anomaly detection, or decision analysis.

[0046] Specifically, the data table continues to perform labeling and generalization operations, such as averaging each dimension, and then outputs the prediction results of the original input data after being processed by the predictor, so as to identify the meaning of the original input data.

[0047] For example, a DNA (deoxyribonucleic acid) sequence of length 100,000, after multi-scale transformation, is represented as a label sequence of length 100,000, where each label is, for example, a d=100-dimensional vector. Depending on the downstream application task, the label sequence can be output directly or the labels can be combined. For example, in a DNA-based population identification task, the average of each dimension of the label features can be calculated to obtain a d=100-dimensional vector output feature. These output vectors can then be further input into a classifier (e.g., a linear predictor) to identify the population.

[0048] Specific embodiments of the present invention are as follows: Example 1: This invention applies the method to long sequence numerical modeling tasks (N=1,0000,000, d=2). Specifically, the long sequence data, consisting of multiple numerical pairs, is labeled, with each sequence element corresponding to a label. The labeled sequence is then processed by a multi-scale transformation module, enabling labels that are geographically distant to gradually establish information associations through multi-level grouping interactions.

[0049] By adopting the above method, effective modeling of key marker information in long sequences is achieved without introducing global pairwise interactions, verifying the applicability of the method of the present invention in long sequence data processing scenarios.

[0050] Example 2: High-dimensional biological data feature extraction: The feature extraction method described in this invention is applied to feature modeling of high-dimensional biological data. Specifically, the gene expression frequencies of single cells are labeled (N=20000, d=100) and input into a multi-scale transformation module. Through changes in grouping operations in different single-scale transformation layers, the gene features can interact with each other across multiple scales.

[0051] The feature representation obtained through the above processing can be used for subsequent sample classification, clustering or state recognition tasks, verifying the applicability of the method of the present invention in high-dimensional feature data scenarios.

[0052] The embodiments described above are only used to illustrate the technical solutions of the present invention, and are not intended to limit the scope of protection of the present invention. Any equivalent substitutions or modifications made to the above technical features based on the concept of the present invention should fall within the scope of protection of the present invention.

Claims

1. A feature extraction method based on a multi-scale grouping interaction neural network, characterized in that, include: Step S1: Perform labeling processing on several raw input data of the artificial intelligence system to obtain a label sequence; Step S2: Establish a multi-scale grouping interactive neural network containing a multi-scale transformation module MST. The multi-scale transformation module MST contains several single-scale transformation layers with different grouping operations set in hierarchical order. Step S3: Input the labeled sequences of each original input data into the multi-scale grouped interactive neural network, and output the data table after feature extraction. 2.The method of claim 1, wherein: In step S1, each original input data is tokenized and converted into a token sequence consisting of N tokens, where each token is a d-dimensional feature vector.

3. The feature extraction method based on a multi-scale grouped interactive neural network according to claim 2, characterized in that: In step S2, the multi-scale grouped interactive neural network includes a multi-scale transformation module (MST) or several sequentially connected MSTs for feature transformation. Each MST includes... K Single-scale conversion layer, K equal to log n N Rounding up, n The preset group size; The single-scale transformation layer includes a marker-level feature transformation unit, a grouping interaction unit, and a feature normalization unit connected in sequence. The marker-level feature transformation unit performs a marker-by-mark operation on each marker token to obtain a marker vector. The grouping interaction unit performs a grouping operation on each marker vector of each marker sequence and then performs a feature fusion operation on the marker vectors within the same group. The number of marker vectors after the feature fusion operation on each group remains unchanged. The feature normalization unit performs a normalization operation on each marker vector after the feature fusion operation. Each marker vector interacts directly or indirectly with all other marker vectors in a multi-scale transformation module (MST) to model the global context of all marker tokens.

4. The feature extraction method based on a multi-scale grouped interactive neural network according to claim 3, characterized in that: The grouping operation performed on each tag vector of each tag sequence in the grouping interaction unit is as follows: The hierarchical order settings in the multi-scale conversion module MST K Each single-scale transformation layer has its own index. l , in the marker sequence N The labeled vectors are divided into Q Groups, each group being of size . n , Q equal N Divide by n The result of rounding up; each marker vector has its own index. i ,That n number system , , … They are respectively the 1st, 2nd, ... K +1 base value; in the... l In the single-scale transformation layer of the layer, the first i The _th marker vector is divided into the _th ... q Groups, Group Number q Specifically as follows: The index of the first i marker vector within the group in the first q group is .

5. The feature extraction method based on a multi-scale grouped interactive neural network according to claim 3, characterized in that: The aforementioned group interaction unit performs feature fusion operations only within the same group, targeting the first... i Tag vectors x i First, determine the number i Tag vectors x i and the first in the same group j Tag vectors x j The correlation value between them is used to determine the interaction weight. Finally, the label vector after feature fusion is obtained. x i for .

6. The feature extraction method based on a multi-scale grouped interactive neural network according to claim 2, characterized in that: In step S3, the data table after feature extraction includes a token sequence consisting of N interacting tokens, each of which is a d-dimensional feature vector.

7. The feature extraction method based on a multi-scale grouped interactive neural network according to claim 2, characterized in that: The data table continues to perform a labeling and generalization operation, and then, after being processed by the predictor, outputs the prediction result of the original input data to identify the meaning of the original input data.

Images