General feature extraction system based on improved transformer structure, extraction method and application
By improving the transformer structure, removing position embedding and masks, and using pre-built vocabulary for feature quantization and normalization, the problem of poor performance of existing feature extractors in general tasks is solved, achieving wider application and more Strong generalization ability.
Patent Information
- Application Number
- CN202410944275.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-15
- Publication Date
- 2024-11-15
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing feature extractors have poor results in general tasks, especially because transformer model design is closely related to natural language processing, which leads to poor application in non-natural language fields, and traditional methods are prone to overfitting and generalization capabilities. weak.
The improved transformer structure is adopted to remove position embedding and self-attention masks, feature quantization and normalization is used for pre-constructed vocabulary, and the linear layer structure is adjusted to meet different task requirements, realizing feature extraction.
It improves the generalization ability of feature extraction, reduces the difficulty of modeling, is suitable for more feature extraction tasks in the non-natural language processing field, and reduces the phenomenon of overfitting.
Smart Images

Figure BDA0004945152330000041 
Figure BDA0004945152330000091 
Figure FDA0004945152320000021
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of feature extraction, and relates to a universal feature extraction system, an extraction method and an application based on an improved transformer structure. Background Art
[0002] With the emergence of OpenAI's ChatGPT, large language models based on the transformer structure have been widely used in the field of NLP. As the industry applies the transformer structure to the field of machine vision (CV), the models in different fields have been unified into the transformer architecture.
[0003] Transformer is a deep learning model for processing sequence data, proposed by Vaswani et al. in 2017. [1] . Unlike traditional recurrent neural networks (RNN) and long short-term memory networks (LSTM), the transformer model relies entirely on the self-attention mechanism to capture long-distance dependencies in the input data. Its core components include two parts: the encoder and the decoder, each of which is composed of multiple identical layers stacked together. This architectural design gives the transformer significant advantages in parallel processing and long-distance dependency modeling.
[0004] In the transformer model, the encoder and decoder are two main components, each with unique structure and functions:
[0005] Encoder: The main task of the encoder is to receive the input sequence and convert it into a set of continuous representations. Each encoder layer contains two main sublayers: Multi-Head Self-Attention Mechanism and Feed-Forward Neural Network. The encoder layer promotes training stability and information flow through layer normalization and residual connection.
[0006] Decoder: The task of the decoder is to generate the output sequence. Each decoder layer also contains two main sublayers: a multi-head self-attention mechanism and a feedforward neural network, and an additional multi-head attention sublayer for receiving the output representation of the encoder. The decoder uses an auto-regressive mechanism to gradually generate outputs, and uses a masking mechanism to ensure that each position can only access the output before the current moment.
[0007] Since the transformer model does not use a loop structure and cannot implicitly capture the position information of the sequence, it is necessary to explicitly add position embedding to represent the position of the elements in the input data. Position embedding adds position information to each element in the input sequence, allowing the model to use position information for more effective sequence modeling. Common implementation methods include sine and cosine functions and learnable position embedding parameters.
[0008] In addition to position encoding, masks play an important role in the transformer model, mainly used in the following two aspects:
[0009] 1. Self-attention mask: During the training process, in order to prevent the model from accessing information at future moments when generating output at a certain position, a self-attention mask is used to shield the input at future moments. This mask is usually called the "Future Mask" to ensure that the model can only access the position information at the current moment and before.
[0010] 2. Padding Mask: In order to process input sequences of different lengths, shorter sequences need to be padded during batch processing. Padding Mask is used to shield the data of the padded part so that these parts will not affect the training and prediction of the model.
[0011] In summary, the transformer model effectively solves the problems of long-distance dependency and parallel computing in sequence data processing through its unique architecture design, position embedding, and mask mechanism, becoming a powerful feature extractor.
[0012] These technical features provide a solid foundation for the application of various natural language processing (NLP) tasks. However, since the transformer was originally designed to meet the processing needs of the natural language processing field, many structures in the model structure are closely related to the characteristics of natural language. These structures limit the application of transformers in other fields.
[0013] The problems existing in current feature extractors when dealing with general tasks include the following:
[0014] First, transformers pay more attention to the features of neighboring positions, while in general feature extraction, the relationship between features is often not strongly correlated with their positions;
[0015] Second, traditional machine learning methods often cause overfitting and poor generalization due to parameter limitations;
[0016] 3. The convolutional neural network (CNN) model is naturally suitable for problems where features such as images have strong correlations on the 2D level due to the influence of the convolution and sliding window mechanism. However, for many other application areas, this correlation does not exist.
[0017] Therefore, existing feature extractors are relatively ineffective in solving the related problems of general feature extraction. Summary of the invention
[0018] In order to address the deficiencies in the prior art, the purpose of the present invention is to provide a general feature extraction system, extraction method and application based on an improved transformer structure, so that the improved transformer model can be applicable to feature extraction work in more non-natural language fields.
[0019] The present invention provides a universal feature extraction system based on an improved transformer structure, the universal feature extraction system comprising: an input and preprocessing module, a feature normalization module, a feature encoding module, an improved transformer module, a task adaptation module, and a result output module;
[0020] The input and preprocessing module receives the input raw feature data and preprocesses the raw feature data; the preprocessing includes processing methods such as data cleaning, data filling, and data conversion, so that the obtained data can better adapt to the training and use of subsequent models;
[0021] The feature normalization module normalizes the preprocessed feature data to obtain feature floating point numbers;
[0022] The feature encoding module quantizes the normalized feature floating point number and the pre-built vocabulary, and encodes them into a unique token id;
[0023] The improved transformer module performs feature extraction on the encoded data;
[0024] The improved transformer module includes an attention module that removes position embedding and self-attention mask, and is used to extract features from the encoded data;
[0025] The task adaptation module adjusts the structure of the linear layer in the improved transformer module according to task requirements;
[0026] The result output module outputs the final feature extraction result.
[0027] The general feature extraction system includes a pre-built vocabulary that meets the predetermined scenario, which replaces the original vocabulary in the transformer; and / or,
[0028] The pre-built vocabulary that meets the predetermined scenario stores a mapping from vocabulary floating point numbers to token ids, and after quantizing the feature floating point numbers corresponding to the feature data into vocabulary floating point numbers, the feature floating point numbers are made to correspond one to one with the token ids; and / or,
[0029] The floating point numbers of the vocabulary are divided evenly and / or according to a normal distribution and / or a dynamic tree.
[0030] The improved transformer module does not include position embedding and self-attention mask;
[0031] The improved transformer module directly uses the input feature vector without adding any position information, and the feature vector processed subsequently does not contain any position code; and / or,
[0032] The improved Transformer module does not use any form of mask in the multi-head self-attention mechanism, allowing the feature vector at each position to interact freely with the feature vectors at all other positions, making full use of the global information between the input features.
[0033] The linear layer in the improved transformer module is a task adaptation layer, which is adapted to different tasks by changing the dimension of the linear layer; and / or,
[0034] The adaptation of the task includes regression or classification tasks;
[0035] When adapting to the regression task, the dimension of the linear layer is converted from [feature_size, hidden_dim]x[hidden_dim, vocab_size] to [feature_size, hidden_dim]x[hidden_dim, 1], where feature_size represents the feature size (or feature dimension), hidden_dim represents the hidden layer dimension, and vocab_size represents the vocabulary size;
[0036] When adapting to classification tasks, the dimension of the linear layer is converted from [feature_size, hidden_dim]x[hidden_dim, vocab_size] to [feature_size, hidden_dim]x[hidden_dim, label_nums], where feature_size represents the feature size (or feature dimension), hidden_dim represents the hidden layer dimension, label_nums represents the number of categories to be classified, and vocab_size represents the vocabulary size.
[0037] The present invention also provides a general feature extraction method implemented by using the general feature extraction system, the feature extraction method comprising:
[0038] Step 1: Normalize the input feature data according to each feature dimension to obtain feature floating point numbers;
[0039] Step 2: quantify the normalized feature floating point numbers into floating point numbers stored in the pre-built vocabulary, and correspond them one-to-one with the token id;
[0040] Step 3: Input the quantized data in step 2 into the improved transformer module;
[0041] Step 4: After feature extraction, the final feature extraction result is obtained.
[0042] The step 1 also includes preprocessing the input original feature data, including data cleaning, data filling and data conversion;
[0043] The data cleaning refers to removing noise data and outliers in the original feature data; the data filling refers to processing and filling missing data in the original feature data; the data conversion refers to converting the format and unifying the units of the input original feature data as needed.
[0044] In step 1, the feature data is normalized to a feature floating point number of [0, 1]; the normalization formula is as follows:
[0045]
[0046] Wherein, feature[i,j] represents the jth sample value of the i-th feature dimension, min(feature[i,:]) represents the minimum sample value of the i-th feature dimension, and max(feature[i,:]) represents the maximum sample value of the i-th feature dimension; and / or,
[0047] In step 2, the pre-constructed vocabulary contains a character string corresponding to a floating point number of [0, 1], which can then be mapped to a token id. The normalized feature floating point number is mapped into normally distributed data according to the proximity principle and / or first, and then quantized into a vocabulary floating point number in the pre-constructed vocabulary to obtain the token id.
[0048] In step 3, the improved transformer module includes one or more blocks consisting of an attention module and a feed-forward network;
[0049] The position embedding module and the self-attention mask module are removed from the attention module;
[0050] In the improved transformer module, the input data is converted into a high-dimensional vector representation through an embedding layer; the attention weight between each position in the input data and all other positions is calculated; the features output by the self-attention mechanism are nonlinearly mapped through two linear transformations and an activation function, and after layer normalization and residual connection, a high-dimensional feature representation after multi-layer processing is obtained.
[0051] In step 3, the dimension of the last linear layer in the transformer module is modified to adapt to tasks including regression and classification.
[0052] When adapting to the regression task, the dimension of the linear layer is converted from [feature_size, hidden_dim]x[hidden_dim, vocab_size] to [feature_size, hidden_dim]x[hidden_dim, 1], where feature_size represents the feature size (or feature dimension), hidden_dim represents the hidden layer dimension, and vocab_size represents the vocabulary size;
[0053] When adapting to classification tasks, the dimension of the linear layer is converted from [feature_size, hidden_dim]x[hidden_dim, vocab_size] to [feature_size, hidden_dim]x[hidden_dim, label_nums], where feature_size represents the feature size (or feature dimension), hidden_dim represents the hidden layer dimension, label_nums represents the number of categories to be classified, and vocab_size represents the vocabulary size.
[0054] The present invention also provides a hardware system for implementing the above-mentioned general feature extraction method, and the hardware system includes: a memory and a processor; a computer program is stored in the memory, and when the computer program is executed by the processor, the above-mentioned general feature extraction method is implemented.
[0055] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned general feature extraction method is implemented.
[0056] The present invention also provides the application of the above-mentioned general feature extraction system, the above-mentioned general feature extraction method, the above-mentioned hardware system or the above-mentioned computer-readable storage medium in feature extraction in the field of non-natural language processing.
[0057] The beneficial effects of the present invention include:
[0058] The method of the present invention uses a transformer-like structure to provide a general large model solution for various fields that require feature extraction (such as finance, real estate, medicine, etc.), which greatly changes the shortcomings of using traditional modeling methods such as easy overfitting and poor generalization ability, and reduces the difficulty of modeling.
[0059] The present invention transforms the transformer model structure, removes position embedding and masking, and makes the model applicable to more non-NLP tasks in principle.
[0060] According to different usage scenarios, the present invention pre-builds a larger vocabulary suitable for the corresponding scenario and replaces the original vocabulary in the transformer, so that the normalized feature data can be converted into a unique token, ensuring the one-to-one correspondence between the token and the feature;
[0061] The present invention can also be applied to different feature extraction tasks by changing the operator dimension of the last layer of the transformer module, and has a wide range of application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS
[0062] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.
[0063] Figure 1 It is a schematic diagram of the transformer network structure in the prior art.
[0064] Figure 2It is a schematic diagram of the network structure in each transformer block in the prior art.
[0065] Figure 3 This is a schematic diagram of the attention network structure in the transformer in the prior art.
[0066] Figure 4 This is a schematic diagram of the FFN network structure in the transformer in the prior art.
[0067] Figure 5 It is a schematic diagram of the attention network structure in the improved transformer in the present invention.
[0068] Figure 6 A schematic diagram showing a dynamic tree representation scheme. DETAILED DESCRIPTION
[0069] The present invention is further described in detail with reference to the following specific examples and drawings. The process, conditions, experimental methods, etc. for implementing the present invention, except for the contents specifically mentioned below, are all common knowledge and common common sense in the art and are not particularly limited by the present invention.
[0070] Feature extraction and modeling have always been important research topics in all walks of life. In the past, modeling was generally carried out through mathematical formulas and machine learning, but these methods are often subject to unfavorable factors such as the difficulty in easily obtaining mathematical models and the easy overfitting of machine learning. In the past two years, models of transformer structures represented by GPT have been widely used in the field of natural language processing (NLP), but due to their structural characteristics, such as the dependence of sequence data, the limitations of position encoding, and the field specificity of feature extraction, it is difficult to generalize them to more fields besides natural language processing (NLP) and machine vision (CV). The present invention innovatively transforms the traditional transformer structure to make it competent for fields other than natural language processing (NLP). At the same time, the transformer's tokenizer is also processed so that the encoding of features will be easier to be understood by the model. For different feature extraction tasks, the present invention also provides a modification scheme for the model structure.
[0071] In the past two years, the decoder-only transformer model has gradually become the mainstream in practical applications. Therefore, this solution mainly considers the decoder-only model architecture. For the encoder-only and encoder-decoder model structures, this solution can be easily promoted due to the high consistency of the internal operators and the overall model architecture.
[0072] The traditional decoder-only structure of a large language model consists of two modules: attention module and feed-forward network (FFN). Figure 2 The complete transformer network consists of a certain number of such blocks connected end to end, so this solution mainly discusses the changes in the model structure in a block. The model structure diagram of the transformer in the prior art is as follows Figure 1 As shown, the structure of the attention module is as follows Figure 3 As shown, the structure of the feedforward network (FFN) is as follows Figure 4 shown.
[0073] The present invention provides a general feature extraction system based on an improved transformer structure, the general feature extraction system comprising: an input and preprocessing module, a feature normalization module, a feature encoding module, an improved transformer module, a task adaptation module, and a result output module; the structure of the attention module in the improved transformer module of the present invention is as follows Figure 5 As shown;
[0074] The input and preprocessing module receives the input raw feature data and preprocesses the raw feature data; the preprocessing includes processing methods such as data cleaning, data filling, and data conversion, so that the obtained data can better adapt to the training and use of subsequent models;
[0075] The feature normalization module normalizes the preprocessed feature data to obtain feature floating point numbers;
[0076] The feature encoding module quantizes the normalized feature floating point number and the pre-built vocabulary, and encodes them into a unique token id;
[0077] The improved transformer module performs feature extraction on the encoded data;
[0078] The improved transformer module includes an attention module that removes position embedding and self-attention mask;
[0079] The task adaptation module adjusts the structure of the linear layer in the improved transformer module according to task requirements;
[0080] The result output module outputs the final feature extraction result.
[0081] The present invention also provides a general feature extraction method implemented by using the general feature extraction system, the feature extraction method comprising:
[0082] Step 1: Normalize the input feature data according to each feature dimension to obtain feature floating point numbers;
[0083] Step 2: quantify the normalized feature floating point numbers into floating point numbers stored in the pre-built vocabulary, and correspond them one-to-one with the token id;
[0084] Step 3: Input the quantized data in step 2 into the improved transformer module;
[0085] Step 4: After feature extraction, the final feature extraction result is obtained.
[0086] The feature data in the present invention is firstly obtained by a normalization algorithm to obtain a normalized feature floating point number. The normalized feature floating point number may be any floating point number between the data range of [0, 1]. However, since the number of vocabulary is limited, any continuous feature floating point number must be quantized into a discrete floating point number existing in the vocabulary by quantization. The quantization methods include uniform quantization, dynamic tree quantization, normal distribution quantization and other schemes. Since the floating point number in the vocabulary has a one-to-one correspondence with the token id, as long as the quantized vocabulary floating point number is obtained, the corresponding token id will be obtained.
[0087] Specifically,
[0088] 1. Position Embedding Processing
[0089] In the prior art, position embedding is added to the attention module of the transformer to describe the mathematical relationship between features at different positions. In natural language, since words often have very important relationships with neighboring words, adding position embedding allows the model to perceive this type of relationship, and also helps the attention model to establish the relationship between features and features at farther positions. However, since the features in general application extraction tasks often do not have (or cannot be known in advance) positional relationships, when using transformers to process general application extraction tasks where there is a high probability that there will be no or unpredictable positional relationships between features, position embedding needs to be removed.
[0090] 2. Self-Attention Mask Processing
[0091] In order to prevent the model from accessing information at future times when generating output at a certain position, the transformer in the prior art usually uses a self-attention mask to shield the input at future times. However, since the features in general application extraction tasks often do not have (or cannot be known in advance) temporal connections, when using the transformer to process general application extraction tasks where there is a high probability that there is no or unpredictable temporal connection between features, the self-attention mask needs to be removed.
[0092] Tokenizer Processing
[0093] Tokenizer is an important component of transformer. It converts text into a format that can be processed by machine learning models in natural language processing. Its main functions include word segmentation, vocabulary construction, text encoding, processing special tags, generating attention masks, and processing batch input. Word segmentation splits text into independent words or subwords. The vocabulary is built by analyzing a large amount of text data, and each word or subword has a unique ID. The text after word segmentation is encoded as an ID sequence, allowing the model to directly process text in digital form.
[0094] In general feature extraction tasks, the numerical ranges of features often vary greatly. In order to avoid a large impact of individual features on other features, the feature values need to be normalized according to the feature dimension (the normalization method is shown in Formula 1) to a floating point number of [0,1]. Simply using the transformer's tokenizer will cause a floating point number to be encoded into multiple tokens, destroying the one-to-one correspondence between tokens and features. Therefore, the approach is to expand the vocabulary to ensure that each floating point number will only be encoded into one token in the end. Assuming that the numbers between 0 and 1 are uniformly encoded into 1000 numbers (that is, the minimum unit is 0.001), then first add 1000 characters to the tokenizer vocabulary, whose values are strings representing floating point numbers of [0,1], such as "0.001": 32000, "0.002": 32001, the number before the colon represents the floating point number to be represented, and the number after the colon is the ID value converted from the word segmentation encoding. Next, the normalized feature floating-point numbers are converted into 1,000 characters added to the vocabulary according to the proximity principle. This ensures that each token is converted into a unique and unique ID, thus avoiding the situation where one feature is converted into multiple IDs.
[0095] The normalization of the input feature data in each feature dimension is calculated by the following formula:
[0096]
[0097] Among them, feature[i,j] represents the j-th sample value of the i-th feature dimension, min(feature[i,:]) represents the minimum sample value of the i-th feature dimension, and max(feature[i,:]) represents the maximum sample value of the i-th feature dimension.
[0098] 4. Different Task Processing
[0099] In the method of the present invention, since different tasks can generally be summarized as regression tasks and classification tasks, multiple tasks such as regression and classification are achieved by changing the last layer of the transformer model.
[0100] When the task type is a regression problem, the dimension of the last linear layer is converted from [feature_size, hidden_dim]x[hidden_dim,vocab_size] to [feature_size,hidden_dim]x[hidden_dim,1], where feature_size represents the feature size, vocab_size represents the vocabulary size, and hidden_dim represents the hidden layer dimension, which is a parameter in the model. The mean square error (MSE) is used as the loss function.
[0101] When the task type is a classification problem, the dimension of the last linear layer is converted from [feature_size, hidden_dim]x[hidden_dim, vocab_size] to [feature_size, hidden_dim]x[hidden_dim, label_nums], where feature_size represents the feature size, hidden_dim represents the hidden layer dimension, vocab_size represents the vocabulary size, and label_nums represents the number of categories to be classified. Cross entropy is used as the loss function.
[0102] The general feature extraction system of the present invention mainly helps general feature tasks to complete feature extraction. The specific implementation steps are:
[0103] (1) Normalize the input feature data according to each feature dimension to obtain feature floating point numbers;
[0104] (2) The normalized feature floating point numbers are mapped into normal distribution data according to the proximity principle and / or first, and then quantized into the floating point numbers of the vocabulary stored in the vocabulary, and corresponded one by one to the token id;
[0105] (3) Input the quantized data in the previous step into the improved transformer module;
[0106] (4) After feature extraction, the final feature extraction result is obtained.
[0107] In a specific implementation, the floating point numbers in the vocabulary mentioned in the present invention may not be divided evenly, but may be divided in other ways, such as by normal distribution or dynamic tree to divide the floating point numbers between [0, 1];
[0108] In addition to converting the normalized features into stored floating-point numbers using the nearest principle mentioned in the text, the normalized features can also be first mapped into state distribution data and then converted into floating-point numbers to make the data more consistent with statistical laws.
[0109] In a specific implementation, the following method can be used for the non-uniform partitioning method:
[0110] (1) Normalize the input feature data according to each feature dimension to obtain feature floating point numbers;
[0111] (2) Using a dynamic tree based on the number of vocabularies (e.g. Figure 6 ) or a list of floating point numbers represented in a normal distribution determination vocabulary;
[0112] (3) The normalized feature floating-point numbers are quantized into floating-point numbers stored in the vocabulary according to the principle of proximity, and correspond one-to-one with the token id;
[0113] (4) Input the data transformed in the previous step into the improved transformer module;
[0114] (5) After feature extraction, the final feature extraction result is obtained.
[0115] In another specific implementation, the normal distribution conversion method may be performed first in the following manner:
[0116] (1) Normalize the input feature data according to each feature dimension to obtain feature floating point numbers;
[0117] (2) The normalized data were transformed into a normal distribution with a mean of 0.5 and a variance of 1;
[0118] (3) The characteristic floating-point numbers converted into normal distribution are quantized into floating-point numbers stored in the vocabulary according to the principle of proximity, and correspond one-to-one with the token id;
[0119] (4) Input the quantized data in the previous step into the improved transformer module;
[0120] (5) After feature extraction, the final feature extraction result is obtained.
[0121] In the process of verifying the universal feature extraction system in the present invention, the effects of the method adopted by the present invention and two traditional machine learning methods were compared on a private financial data set. The final results are shown in the following table:
[0122] Model Accuracy Random Forest 82.14% Gradient Boosting Decision Tree 78.59% This patented method 92.86%
[0123] The values listed in the table are the prediction accuracy, and the higher the value, the better the effect. From the results shown in the table, it can be seen that the method used in the present invention obtains results far better than the other two existing machine learning feature extractors.
[0124] References
[0125] [1]Vaswani A,Shazeer N,Parmar N,et al.Attention is all you need[J].Advances in neural information processing systems,2017,30.
[0126] Various aspects of the present invention are described herein with reference to the flow charts and / or block diagrams of the methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each box of the flow chart and / or block diagram and the combination of each box in the flow chart and / or block diagram can be implemented by computer-readable program instructions.
[0127] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.
[0128] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.
[0129] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present invention. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and the module, program segment or instruction part contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.
[0130] The protection content of the present invention is not limited to the above embodiments. Without departing from the spirit and scope of the present invention, changes and advantages that can be thought of by those skilled in the art are included in the present invention and are protected by the attached claims.
Claims
1. A general feature extraction system based on an improved transformer structure, characterized in that: The general feature extraction system includes: an input and preprocessing module, a feature normalization module, a feature encoding module, an improved transformer module, a task adaptation module, and a result output module; The input and preprocessing module receives the input original feature data and preprocesses the original feature data; The feature normalization module normalizes the preprocessed feature data to obtain feature floating point numbers; The feature encoding module quantizes the normalized feature floating point number and the pre-built vocabulary, and encodes them into a unique token id; The improved transformer module includes an attention module that removes position embedding and self-attention mask, and is used to extract features from the encoded data; The task adaptation module adjusts the structure of the linear layer in the improved transformer module according to task requirements; The result output module outputs the final feature extraction result.
2. The general feature extraction system according to claim 1, characterized in that: The general feature extraction system includes a pre-built vocabulary that meets the predetermined scenario, which replaces the original vocabulary in the transformer; and / or, The pre-built vocabulary that meets the predetermined scenario stores a mapping from vocabulary floating point numbers to token ids, and after quantizing the feature floating point numbers corresponding to the feature data into vocabulary floating point numbers, the feature floating point numbers are made to correspond one to one with the token ids; and / or, The floating point numbers of the vocabulary are divided evenly and / or according to a normal distribution and / or a dynamic tree.
3. The general feature extraction system according to claim 1, characterized in that: The improved transformer module does not include position embedding and self-attention mask; and / or, The improved transformer module directly uses the input feature vector without adding any position information, and the feature vector processed subsequently does not contain any position code; and / or, The improved transformer module does not use any form of mask in the multi-head self-attention mechanism, allowing the feature vector of each position to interact freely with the feature vectors of all other positions, making full use of the global information between the input features.
4. The general feature extraction system according to claim 1, characterized in that: The linear layer in the improved transformer module is a task adaptation layer, which is adapted to different regression or classification tasks by changing the dimension of the linear layer; and / or, When adapting to the regression task, the dimension of the linear layer is converted from [feature_size, hidden_dim]x[hidden_dim, vocab_size] to [feature_size, hidden_dim]x[hidden_dim, 1], where feature_size represents the feature size, hidden_dim represents the hidden layer dimension, and vocab_size represents the vocabulary size; When adapting to the classification task, the dimension of the linear layer is converted from [feature_size, hidden_dim]x[hidden_dim, vocab_size] to [feature_size, hidden_dim]x[hidden_dim, label_nums], where feature_size represents the feature size, hidden_dim represents the hidden layer dimension, label_nums represents the number of categories to be classified, and vocab_size represents the vocabulary size.
5. A general feature extraction method, characterized in that: The feature extraction method comprises: Step 1: Normalize the input feature data according to each feature dimension to obtain feature floating point numbers; Step 2: quantify the normalized feature floating point numbers into floating point numbers stored in the pre-built vocabulary, and correspond them one-to-one with tokenid; Step 3: Input the quantized data in step 2 into the improved transformer module; Step 4: After feature extraction, the final feature extraction result is obtained.
6. The feature extraction method according to claim 5, characterized in that: In step 1, the feature data is normalized to a feature floating point number of [0, 1]; the normalization formula is as follows: Wherein, feature[i,j] represents the jth sample value of the i-th feature dimension, min(feature[i,:]) represents the minimum sample value of the i-th feature dimension, and max(feature[i,:]) represents the maximum sample value of the i-th feature dimension; and / or, In step 2, the character string corresponding to the floating point number [0, 1] contained in the pre-constructed vocabulary is mapped to tokenid, and the normalized feature floating point number is mapped into standard normal distribution data according to the proximity principle and / or first, and then quantized into the vocabulary floating point number in the pre-constructed vocabulary, and the token id is obtained.
7. The feature extraction method according to claim 5, characterized in that: In step 3, the improved transformer module includes one or more blocks consisting of an attention module and a feed-forward network; The position embedding module and the self-attention mask module are removed from the attention module; In the improved transformer module, the input data is converted into a high-dimensional vector representation through an embedding layer; the attention weight between each position in the input data and all other positions is calculated; Through two linear transformations and an activation function, the features output by the self-attention mechanism are nonlinearly mapped. After layer normalization and residual connection, a high-dimensional feature representation after multi-layer processing is obtained; and / or, In step 3, the dimensions of the last linear layer in the transformer module are modified to adapt to tasks including regression and classification. When adapting to the regression task, the dimension of the linear layer is converted from [feature_size, hidden_dim]x[hidden_dim, vocab_size] to [feature_size, hidden_dim]x[hidden_dim, 1], where feature_size represents the feature size, hidden_dim represents the hidden layer dimension, and vocab_size represents the vocabulary size; When adapting to the classification task, the dimension of the linear layer is converted from [feature_size, hidden_dim]x[hidden_dim, vocab_size] to [feature_size, hidden_dim]x[hidden_dim, label_nums], where feature_size represents the feature size, hidden_dim represents the hidden layer dimension, label_nums represents the number of categories to be classified, and vocab_size represents the vocabulary size.
8. A hardware system for implementing the method according to any one of claims 5 to 7, characterized in that: The hardware system includes: a memory and a processor; a computer program is stored in the memory, and when the computer program is executed by the processor, the method according to any one of claims 5 to 7 is implemented.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 5 to 7 is implemented.
10. Application of the feature extraction system as described in any one of claims 1-4, the feature extraction method as described in any one of claims 5-7, the hardware system as described in claim 8 or the computer-readable storage medium as described in claim 9 in feature extraction in the field of non-natural language processing.
