A compression model construction method based on adaptive dictionary
By introducing an adaptive dictionary and a second-layer four-head self-attention structure into the neural network compression model, the problem of degradation of compression effect on the Chinese data set is solved, and a higher compression ratio and data throughput is achieved.
Patent Information
- Application Number
- CN202311819395.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-27
- Publication Date
- 2025-04-08
AI Technical Summary
The compression effect of existing neural networks on Chinese data sets has decreased, the training difficulty is increased, and it consumes a lot of time costs, resulting in long compression time and low throughput.
The compression model construction method based on adaptive dictionary is adopted to generate an adaptive dictionary by repetitive iteration to generate byte pairs, and a compression model is constructed with the self-attention structure of the second layer and the four-headed self-attention structure.
The compression ratio of English data is improved and the compression effect and data throughput on Chinese data exceeds most models.
Smart Images
Figure CN119783734A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of software, and in particular to a compression model construction method based on an adaptive dictionary. Background Art
[0002] At present, when using neural networks for data compression, fixed dictionaries are used, that is, 8-bit single bytes are used as the basic unit, and the dictionary length is 256; or single bits are used as the basic unit, and the model is a binary classification model. Since the model uses an English data set, the single-byte dictionary format meets its basic text requirements. However, in the Chinese data set, a single Chinese character occupies 2 or 3 bytes. Using a single byte to forcibly divide an entire character makes it more difficult to train the model, resulting in a decrease in compression effect on the Chinese data set, increased training difficulty, and excessive time costs, resulting in long compression time and low compression throughput.
[0003] In summary, a compression model construction method based on an adaptive dictionary is needed to address the deficiencies in the prior art. Summary of the invention
[0004] In view of the deficiencies in the prior art, the present invention provides a compression model construction method based on an adaptive dictionary, aiming to solve the above problems.
[0005] To achieve the above object, the present invention provides the following technical solution: a method for constructing a compression model based on an adaptive dictionary, comprising the following steps:
[0006] Step S1: input program configuration parameters and perform parameter configuration;
[0007] Step S2: input source data, perform a loop based on the source data, and generate an adaptive dictionary by repeatedly iterating and generating byte pairs;
[0008] Step S3: The compression model uses word embedding to convert the character index into a word vector, extracts the feature information in the character through the embedding layer, forms a feature representation of the word, and inputs it into the model for conditional probability prediction to obtain a prediction result;
[0009] Step S4: input the prediction result of the compression model into the arithmetic encoder and store and output it through the storage module.
[0010] Optionally, the parameter configuration in step S1 specifically includes the following steps:
[0011] Step S11: The user inputs program configuration parameters, and the parameter processing module parses and performs integrity check on the program configuration parameters, and then transmits the program configuration parameters to the lower module;
[0012] Step S12: receiving the program configuration parameters passed in by the parameter processing module, applying for memory space and initializing variables according to the received program configuration parameters;
[0013] Step S13: the initialization module receives the program configuration parameters transmitted by the parameter processing module, applies for memory space and initializes variables according to the received program configuration parameters.
[0014] Optionally, the generation of the adaptive dictionary in step S2 specifically includes the following steps:
[0015] Step S21: Count all the single bytes and their frequencies and add them directly to the dictionary. The text data is based on single bytes, and two by two are grouped into double-byte pairs. These double-byte pairs are counted and stored in the frequency statistics set, and the location of each byte pair in the file is recorded.
[0016] Step S22: cyclically adding the byte pairs that meet the requirements to the adaptive dictionary, first obtaining the byte pair with the highest frequency in the byte pair set and adding it to the adaptive dictionary, and simultaneously using the auxiliary information and position information to update the byte pair statistical set;
[0017] Step S23: After extracting the byte pair with the highest frequency, add it to the adaptive dictionary, and obtain the position information of the byte pair with the highest frequency in the file. Based on the position information, remove the old byte pairs consisting of the byte pairs contained at the head and tail of the byte pair and the byte pairs before and after it in the byte pair statistical set, and add a new byte pair consisting of the byte pair and the byte pairs before and after it;
[0018] Step S24: repeating steps S21-S23 until the dictionary capacity reaches the requirement or there is no byte that meets the requirement in the byte pair statistical set, and the adaptive dictionary generation ends;
[0019] Step S25: The storage module stores the generated adaptive dictionary in the hard disk.
[0020] Optionally, in step S3, the compression model is constructed by using a two-layer four-head self-attention structure, and the self-attention base layer of the two-layer four-head self-attention structure includes the query matrix W Q , matching matrix W K and the feature matrix W V , query matrix W Q Responsible for extracting query information from word vectors and matching matrix W K Responsible for extracting matching information in word vectors, feature matrix W V Responsible for extracting feature information from word vectors.
[0021] Optionally, each layer of the compression model includes four self-attention heads, each with three matrices And use the query matrix W Q , matching matrix W K and the feature matrix W V Perform a linear transformation on the word vector. The steps of linear transformation include:
[0022] Perform matrix multiplication on the word vector and the matrix to obtain the query vector Q, matching vector K and feature vector V of the word vector. Perform inner product on the query vector Q of the current time step and the matching vector K of all other time steps respectively to obtain the correlation between them, which is the score vector between the current time step and all other steps.
[0023] Optionally, the formula between the query vector Q and the matching vector K is:
[0024]
[0025]
[0026]
[0027] in, represents the Q vector of the i-th attention head at the t-th time step, represents the K vector of the i-th attention head at the t-th time step, represents the V vector of the i-th attention head at the t-th time step, represents the Q trainable linear transformation matrix of the i-th attention head, represents the K trainable linear transformation matrix of the i-th attention head, V represents the trainable linear transformation matrix of the i-th attention head, X t is the embedded feature vector of the byte or byte pair at time t.
[0028] Optionally, the calculation of the i-th attention head is specifically formulated as follows:
[0029]
[0030]
[0031]
[0032] Among them, Q i is the Q vector of the i-th attention head in a single layer, K i is the K vector of the i-th attention head in a single layer, V i is the V vector of the i-th attention head in a single layer, represents the Q vector of the i-th attention head at the t-th time step, represents the K vector of the i-th attention head at the t-th time step, Vt i V represents the vector of the i-th attention head at the t-th time step.
[0033] Optionally, the single-layer i-th attention head is further normalized, specifically:
[0034] Use the softmax function to transform the score:
[0035]
[0036]
[0037] hidden_state i =LayerNorm(Attention i ·W H );
[0038] Among them, Qi is the Q vector of the i-th attention head in a single layer, Ki is the K vector of the i-th attention head in a single layer, and d k is the feature vector dimension of the words in the dictionary, W H is a trainable matrix, hidden-state i is the concatenation of the hidden layer states of the attention head, and V is the feature vector.
[0039] Optionally, the byte pairs relative position The calculation formula is:
[0040]
[0041]
[0042]
[0043]
[0044]
[0045] Among them, Distance is the byte pair relative distance matrix, is a K-vector trainable embedding layer, V is a trainable embedding layer for vectors, E K Distance and The relative distance K vector is obtained by matrix product, E V Distance and The relative distance V vector is obtained by matrix product, is the relative position K value of i and j in the relative distance matrix Distance, that is is the relative position V value of position i and j in the relative distance matrix Distance, that is X t is the embedding feature vector of the byte or byte pair at time t, represents the Q trainable linear transformation matrix of the i-th attention head, They represent the K trainable linear transformation matrices of the i-th attention head respectively.
[0046] Beneficial effects of the present invention:
[0047] 1. In the present invention, an adaptive dictionary and a basic self-attention structure model are applied for the first time in a lossless compression algorithm based on a neural network, and a Chinese data set is applied in the compression algorithm model. The generation of the adaptive dictionary is efficient, fast and memory-saving. After combining the adaptive dictionary scheme, the compression ratio of English data is effectively improved, and the compression of Chinese data is also effective, exceeding most models, and the data throughput rate exceeds most recurrent neural network models, which has certain use value and promotion value. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] Figure 1 It is an overall flow chart of the present invention.
[0049] Figure 2 A flow chart for adaptive dictionary generation.
[0050] Figure 3 A flow chart of the parameter processing module of the present invention.
[0051] Figure 4 A flow chart of the initialization module of the present invention.
[0052] Figure 5 A flow chart of the first stage of the dictionary module of the present invention.
[0053] Figure 6 A flow chart of the second stage of the dictionary module of the present invention.
[0054] Figure 7 A flow chart of the storage module of the present invention.
[0055] Figure 8 A schematic diagram of the dictionary output format of the present invention. DETAILED DESCRIPTION
[0056] In order to more clearly illustrate the embodiments of the invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0057] In conjunction with the instruction manual Figure 1-Figure 8 , further explain the technical solution.
[0058] See Figure 1 , a compression model construction method based on an adaptive dictionary, comprising the following steps:
[0059] Step S1: input program configuration parameters and perform parameter configuration;
[0060] Step S2: input source data, perform a loop based on the source data, and generate an adaptive dictionary by repeatedly iterating and generating byte pairs;
[0061] Step S3: The compression model uses word embedding to convert the character index into a word vector, extracts the feature information in the character through the embedding layer, forms a feature representation of the word, and inputs it into the model for conditional probability prediction to obtain a prediction result;
[0062] Step S4: input the prediction result of the compression model into the arithmetic encoder and store and output it through the storage module.
[0063] Parameter configuration is performed in step S1, which specifically includes the following steps:
[0064] Step S11: The user inputs program configuration parameters, and the parameter processing module parses and performs integrity check on the program configuration parameters, and then transmits the program configuration parameters to the lower module;
[0065] Step S12: receiving the program configuration parameters passed in by the parameter processing module, applying for memory space and initializing variables according to the received program configuration parameters;
[0066] Step S13: the initialization module receives the program configuration parameters transmitted by the parameter processing module, applies for memory space and initializes variables according to the received program configuration parameters.
[0067] The generation of the adaptive dictionary in step S2 specifically includes the following steps:
[0068] Step S21: Count all the single bytes and their frequencies and add them directly to the dictionary. The text data is based on single bytes, and two by two are grouped into double-byte pairs. These double-byte pairs are counted and stored in the frequency statistics set, and the location of each byte pair in the file is recorded.
[0069] Step S22: cyclically adding the byte pairs that meet the requirements to the adaptive dictionary, first obtaining the byte pair with the highest frequency in the byte pair set and adding it to the adaptive dictionary, and simultaneously using the auxiliary information and position information to update the byte pair statistical set;
[0070] Step S23: After extracting the byte pair with the highest frequency, add it to the adaptive dictionary, and obtain the position information of the byte pair with the highest frequency in the file. Based on the position information, remove the old byte pairs consisting of the byte pairs contained at the head and tail of the byte pair and the byte pairs before and after it in the byte pair statistical set, and add a new byte pair consisting of the byte pair and the byte pairs before and after it;
[0071] Step S24: repeating steps S21-S23 until the dictionary capacity reaches the requirement or there is no byte that meets the requirement in the byte pair statistical set, and the adaptive dictionary generation ends;
[0072] Step S25: The storage module stores the generated adaptive dictionary in the hard disk.
[0073] In step S3, the compression model is constructed by using a two-layer four-head self-attention structure. The self-attention base layer of the two-layer four-head self-attention structure includes the query matrix W Q , matching matrix W K and the feature matrix W V , query matrix W Q Responsible for extracting query information from word vectors and matching matrix W K Responsible for extracting matching information in word vectors, feature matrix W V Responsible for extracting feature information from word vectors.
[0074] Each layer of the compressed model includes four self-attention heads, each with three matrices And use the query matrix W Q , matching matrix W K and the feature matrix W V Perform a linear transformation on the word vector. The steps of linear transformation include:
[0075] Perform matrix multiplication on the word vector and the matrix to obtain the query vector Q, matching vector K and feature vector V of the word vector. Perform inner product on the query vector Q of the current time step and the matching vector K of all other time steps respectively to obtain the correlation between them, which is the score vector between the current time step and all other steps.
[0076] In order to solve the problems existing in the dictionary, the byte pair position information processing method is used to avoid the high time complexity caused by iterating the original data to calculate the highest frequency. It only needs to loop the source data once, record the position information and frequency information of the current byte pair, and then iterate to obtain the final dictionary. In terms of the model, a simpler and more effective self-attention structure plus relative position encoding is used to build the model model, solving the problem of slow model training and long compression time leading to low throughput, while improving the compression ratio of the software.
[0077] The specific solution of the adaptive dictionary is:
[0078] First, a loop is performed on the source data to obtain a byte pair set with a byte pair length of 2, and the frequency information and position information of the byte pair are recorded. Then, an adaptive dictionary is generated according to the byte pair set.
[0079] In each iteration, the two adjacent byte pairs are combined according to the byte pair position information to form a new byte pair that is added to the set.
[0080] After obtaining the byte pair with the highest frequency through statistics, the byte pair is added to the dictionary. In the file, the byte pair is regarded as a whole and the byte pair information is re-counted.
[0081] However, in the next statistics, the byte pairs whose statistical information has changed are only related to the byte pairs added to the dictionary last time, or more precisely, are related to their positions. The frequency of the byte pair composed of the head of the byte pair and the previous byte pair connected to its head position is reduced by 1; similarly, the frequency of the byte pair composed of the tail of the byte pair and the next byte pair connected to its tail position is reduced by 1. Accordingly, the byte pair forms two new byte pairs with the previous byte pair and the next byte pair, and the two new byte pairs are added to the byte pair set.
[0082] Repeat the iterative process until the dictionary capacity is reached or there are no byte pairs that meet the requirements. Technical features: Only a single loop is needed to obtain the initial byte pair set information for the source data. Subsequently, only data processing is needed on the set information to obtain the adaptive dictionary. The degree of byte pair position information is much smaller than the total length of the file. The entire file only needs to be looped once during initialization. At the same time, an array space is opened to record the mapping relationship between the front and back head and tail positions of the byte pairs, which can greatly improve the execution speed of the program.
[0083] In the solution of the compression model, the recurrent neural network with serial input is no longer used to build the compression model. Instead, the basic multi-layer multi-head self-attention structure and relative position encoding are used to build the compression model. The compression model is built with a 2-layer 4-head self-attention structure. In the model embedding layer, the position encoding is inserted so that the model can obtain the position information of the time series data. Technical features: For the first time in the field of data compression, a basic and practical self-attention structure is used to build a model. After training the model with an adaptive dictionary, a compression model with higher compression ratio and compression throughput is obtained.
[0084] See Figure 2 ,The adaptive dictionary includes a parameter processing module, an initialization module, a dictionary module and a storage module.
[0085] Parameter processing modules, such as Figure 3 :The user inputs the program configuration parameters, the module parses the parameters and performs integrity check, and then passes the parameters to the lower-level module.
[0086] Initialize the module, such as Figure 4 :Apply for memory space and initialize variables according to the input parameters. This includes: applying for system memory for auxiliary mapping space; reading the file into memory in binary form, and inputting the space resources and source data resources into the dictionary module after success.
[0087] Dictionary modules, such as Figure 5 , Figure 6 :The dictionary module is the main module of the software and also the core module. This module encodes binary data into byte pairs based on bytes, extracts the byte pairs with the highest frequency to form a byte pair set, forms a dictionary after the extraction is completed, and re-segments the binary file into string data based on the generated dictionary. After completion, the system is released.
[0088] The dictionary is built in two stages. In the first stage, see Figure 5 , it is necessary to loop through the entire amount of text data.
[0089] In this loop, two operations need to be completed: 1. Count all the single bytes and their frequencies and add them directly to the dictionary. 2. Take the single byte as the basis of the text data, and group them into double-byte pairs. Count these double-byte pairs and store them in the frequency statistics set, and record the file location of each byte pair. At the same time, use the two file byte pairs to count the front and back positions and map the hash table to record the head and tail positions of each byte pair in the file.
[0090] The second stage, such as Figure 6, to add the byte pairs that meet the requirements to the dictionary in a loop. First, obtain the byte pair with the highest frequency in the byte pair set and add it to the dictionary. At the same time, use the auxiliary information and position information to update the byte pair statistical set, remove redundant byte pairs and add newly added byte pairs. After extracting the byte pair with the highest frequency, add it to the dictionary. At the same time, obtain the position information of the byte pair in the file. Through the position information, the old byte pairs consisting of the byte pairs contained in the head and tail of the byte pair and the byte pairs before and after it are removed from the byte pair statistical set, and a new byte pair consisting of the byte pair and the byte pairs before and after it is added. This cycle is repeated until the dictionary capacity reaches the requirement or there are no byte pairs that meet the requirements in the byte pair statistical set. After the end, the complete dictionary corresponding to the file data and the file data information re-divided according to the dictionary can be obtained.
[0091] Storage module, see Figure 7 : The storage module stores the generated binary dictionary file, visual dictionary file, and output file in the hard disk according to certain format requirements for subsequent operation calls and debugging.
[0092] The storage format of the binary dictionary file is manually specified and written to the specified file in the form of a binary stream to ensure that the correct dictionary information can be parsed. At the same time, the visualization dictionary file and the output file are written to the specified file in the form of UTF-8.
[0093] Dictionary output format, see Figure 8 :In order to maximize the data compression rate, the adaptive dictionary needs to be stored in an efficient storage form after it is generated. The total number of dictionaries occupies 16 bits, the maximum number of bytes in a byte pair occupies 8 bits, and the length of bytes in the file occupied by a byte pair with a byte number of x occupies 24 bits. At the same time, in the process of generating the adaptive dictionary, the source data has been segmented and a segmentation position information has been generated. An iterative loop of the source data in the compression model can be omitted.
[0094] The compression model is:
[0095] The word embedding is used to convert the character index into a word vector. The word embedding needs to be learned by the model itself and can be called a part of the model. The feature information in the character is extracted through the embedding layer to form a feature representation of the word, which is then input into the model for conditional probability prediction.
[0096] The main structure of the model is a 2-layer 4-head self-attention structure to build a compression model, which can not only ensure that the model structure is not too complicated, but also effectively use its advantages to obtain feature information. Each self-attention base layer follows the basic network structure and consists of three main trainable matrices: W Q The query matrix, W K Matching matrix and WV The feature matrix W Q The query matrix is responsible for extracting the query information in the word vector, W K The matching matrix is responsible for extracting the matching information in the word vector, W V The feature matrix is responsible for extracting feature information from word vectors. Each layer has 4 self-attention heads, and each head has 3 matrices Using these three matrices, we perform a linear transformation on the word vector, that is, we perform matrix multiplication on the word vector and the matrix to obtain the query vector Q, matching vector K and feature vector V of the word vector. Since the three matrices are trainable, we can obtain the linear changes we want in the subsequent training process.
[0097] In the implementation of the self-attention layer, the vector dot product is used to calculate the correlation between the sequence time steps. The query vector Q of the current time step is inner-producted with the matching vectors K of all other time steps to obtain the correlation between them, which is called the score vector between the current time step and all other steps. The formula is:
[0098] represents the Q, K, V vector of the i-th attention head at the t-th time step. Represents the trainable linear transformation matrix of the i-th attention head. It represents the attention level of the i-th attention head at time step t to time step j, which is a scalar.
[0099] At the same time, in order to make the model converge more easily during training and prevent gradient disappearance or gradient explosion, it needs to be normalized. where d k is the feature vector dimension of the word. The softmax function can be used to convert the score into a score between 0 and 1. The calculation formula for the hidden layer feature vector of the i-th attention head time step t can be written as: After obtaining the hidden layer states of each attention head, the hidden layer states at time step t are:
[0100] Represented as the concatenation of the hidden layer states of all attention heads. Using a trainable matrix hidden_state t A linear change is performed to ensure that the model can fit more functions, and layer normalization is used on the hidden layer to promote model convergence. The above is the structure and calculation formula of the single-layer multi-head self-attention model in this article, in which the parallel operation of vectors can use matrix operations.
[0101] To optimize and represent, the calculation of the i-th attention head in a single layer can be expressed as:
[0102]
[0103]
[0104]
[0105]
[0106]
[0107] hidden_state i =LayerNorm(Attention i ·W H ).
[0108] Relative position encoding. The context length is fixed, so the current time step has T-1 relative positions to other time steps. There are T characters in total, so in the entire compression model, there are 2T-1 relative positions in total, with the minimum value being -T+1 and the maximum value being T-1. The relative distance matrix Distance = Distance + (T-1) can be obtained. Two trainable embedding layers are used By linearly changing the relative distance matrix, the relative position feature information can be learned during the model training process.
[0109]
[0110]
[0111] in is the encoding of the relative position of i and j in the relative distance matrix. Finally, we can get The calculation formula is divided into two parts:
[0112]
[0113] The rest of the calculation formulas remain unchanged, and the model's prediction results are finally input into the arithmetic encoder to achieve compression.
[0114] The adaptive dictionary and the basic self-attention structure model are applied in the lossless compression algorithm based on neural network, and the Chinese data set is applied in the research of compression algorithm model. The generation of adaptive dictionary is efficient, fast and memory-saving. After combining with the adaptive dictionary scheme, the compression ratio of English data is effectively improved, and the compression of Chinese data is also effective, exceeding most models, and the data throughput rate exceeds most recurrent neural network models.
[0115] The compression results are compared in Table 1 below.
[0116]
[0117] The adaptive dictionary generation effect and the resources used are shown in Table 2 below.
[0118]
[0119] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modification, equivalent substitution or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A compression model construction method based on an adaptive dictionary, characterized in that: The following steps are involved: Step S1: input program configuration parameters and perform parameter configuration; Step S2: input source data, perform a loop based on the source data, and generate an adaptive dictionary by repeatedly iterating and generating byte pairs; Step S3: The compression model uses word embedding to convert the character index into a word vector, extracts the feature information in the character through the embedding layer, forms a feature representation of the word, and inputs it into the model for conditional probability prediction to obtain a prediction result; Step S4: input the prediction result of the compression model into the arithmetic encoder and store and output it through the storage module.
2. The method for constructing a compression model based on an adaptive dictionary according to claim 1, characterized in that: The step S1 performs parameter configuration, which specifically includes the following steps: Step S11: The user inputs program configuration parameters, and the parameter processing module parses and performs integrity check on the program configuration parameters, and then transmits the program configuration parameters to the lower module; Step S12: receiving the program configuration parameters passed in by the parameter processing module, applying for memory space and initializing variables according to the received program configuration parameters; Step S13: the initialization module receives the program configuration parameters transmitted by the parameter processing module, applies for memory space and initializes variables according to the received program configuration parameters.
3. The method for constructing a compression model based on an adaptive dictionary according to claim 1, characterized in that: The generation of the adaptive dictionary in step S2 specifically includes the following steps: Step S21: Count all the single bytes and their frequencies and add them directly to the dictionary. The text data is based on single bytes, and two by two are grouped into double-byte pairs. These double-byte pairs are counted and stored in the frequency statistics set, and the location of each byte pair in the file is recorded. Step S22: cyclically adding the byte pairs that meet the requirements to the adaptive dictionary, first obtaining the byte pair with the highest frequency in the byte pair set and adding it to the adaptive dictionary, and simultaneously using the auxiliary information and position information to update the byte pair statistical set; Step S23: After extracting the byte pair with the highest frequency, add it to the adaptive dictionary, and obtain the position information of the byte pair with the highest frequency in the file. Based on the position information, remove the old byte pairs consisting of the byte pairs contained at the head and tail of the byte pair and the byte pairs before and after it in the byte pair statistical set, and add a new byte pair consisting of the byte pair and the byte pairs before and after it; Step S24: repeating steps S21-S23 until the dictionary capacity reaches the requirement or there is no byte that meets the requirement in the byte pair statistical set, and the adaptive dictionary generation ends; Step S25: The storage module stores the generated adaptive dictionary in the hard disk.
4. The method for constructing a compression model based on an adaptive dictionary according to claim 1, characterized in that: In step S3, the compression model is constructed by using a two-layer four-head self-attention structure. The self-attention base layer of the two-layer four-head self-attention structure includes the query matrix W Q , matching matrix W K and the feature matrix W V , query matrix W Q Responsible for extracting query information from word vectors and matching matrix W K Responsible for extracting matching information in word vectors, feature matrix W V Responsible for extracting feature information from word vectors.
5. The method for constructing a compression model based on an adaptive dictionary according to claim 4, characterized in that: The compression model consists of four self-attention heads per layer, each with three matrices And use the query matrix W Q , matching matrix W K and the feature matrix W V Perform a linear transformation on the word vector. The steps of linear transformation include: Perform matrix multiplication on the word vector and the matrix to obtain the query vector Q, matching vector K and feature vector V of the word vector. Perform inner product on the query vector Q of the current time step and the matching vector K of all other time steps respectively to obtain the correlation between them, which is the score vector between the current time step and all other steps.
6. The method for constructing a compression model based on an adaptive dictionary according to claim 5, characterized in that: The formula between the query vector Q and the matching vector K is: in, represents the Q vector of the i-th attention head at the t-th time step, represents the K vector of the i-th attention head at the t-th time step, represents the V vector of the i-th attention head at the t-th time step, represents the Q trainable linear transformation matrix of the i-th attention head, represents the K trainable linear transformation matrix of the i-th attention head, V represents the trainable linear transformation matrix of the i-th attention head, X t is the embedded feature vector of the byte or byte pair at time t.
7. The method for constructing a compression model based on an adaptive dictionary according to claim 6, characterized in that: The specific formula for the vector calculation of the i-th attention head is: Among them, Q i is the Q vector of the i-th attention head in a single layer, K i is the K vector of the i-th attention head in a single layer, V i is the V vector of the i-th attention head in a single layer, represents the Q vector of the i-th attention head at the t-th time step, represents the K vector of the i-th attention head at the t-th time step, V represents the vector of the i-th attention head at the t-th time step.
8. The compression model construction system based on adaptive dictionary according to claim 7, characterized in that: The single-layer i-th attention head is also normalized, specifically: Use the softmax function to transform the score: hidden_state i =LayerNorm(Attention i ·W H ); Among them, Qi is the Q vector of the i-th attention head in a single layer, Ki is the K vector of the i-th attention head in a single layer, and d k is the feature vector dimension of the words in the dictionary, W H is a trainable matrix, hidden_state i is the concatenation of the hidden layer states of the attention head, and V is the feature vector.
9. The compression model construction system based on adaptive dictionary according to claim 8, characterized in that: The relative position of the byte pairs The calculation formula is: Among them, Distance is the byte pair relative distance matrix, is a K-vector trainable embedding layer, V is a trainable embedding layer for vectors, E K Distance and The relative distance K vector is obtained by matrix product, E V Distance and The relative distance V vector is obtained by matrix product, is the relative position K value of i and j in the relative distance matrix Distance, that is is the relative position V value of position i and j in the relative distance matrix Distance, that is X t is the embedding feature vector of the byte or byte pair at time t, represents the Q trainable linear transformation matrix of the i-th attention head, They represent the K trainable linear transformation matrices of the i-th attention head respectively.