Feature extraction method and device, electronic equipment and storage medium

Abstract

One or more embodiments of the specification provide a feature extraction method and device, an electronic device and a storage medium, the method comprising: determining a query vector, a key vector and a value vector according to target data, wherein the proportion between elements of the query vector and the key vector at a first index position is the same as the proportion between elements of the query vector and the key vector at a second index position, the first index position and the second index position are any two index positions in a first dimension with an interval of 0.5 times the first dimension, and the first dimension is the dimension of the query vector and the key vector; performing position encoding on the query vector and the key vector respectively to obtain an encoding result of the query vector and an encoding result of the key vector; and determining a feature vector of the target data according to the encoding result of the query vector, the encoding result of the key vector and the value vector.

Description

Technical Field

[0001] This specification relates to the field of artificial intelligence technology, and more particularly to a feature extraction method and apparatus, electronic device and storage medium. Background Technology

[0002] In recent years, artificial intelligence technology has developed rapidly, triggering technological innovations in many fields and bringing great convenience to people's work and lives. For example, large language models, due to their advantages such as context awareness, task versatility, and language generation capabilities, can complete various natural language processing tasks, such as text generation, machine translation, and question-answering systems. The level of intelligence of a large language model depends on its parameter scale and training level; that is, the larger the parameter scale and the higher the training level, the higher the level of intelligence of the large language model, and vice versa.

[0003] Understandably, the sequence length of training samples processed by large language models during the training phase is finite. However, the sequence length of data processed by large language models in practical applications is unpredictable and may be longer than the sequence length of training samples, even significantly longer. The extrapolation capability of large language models refers to their ability to process data with sequence lengths longer than those encountered during training. However, in related technologies, the extrapolation capability of large language models is poor, meaning they cannot handle sequences longer than those encountered during training. Summary of the Invention

[0004] In view of the above, one or more embodiments of this specification provide a feature extraction method and apparatus, an electronic device and a storage medium.

[0005] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions:

[0006] According to a first aspect of one or more embodiments of this specification, a feature extraction method is proposed, the method comprising:

[0007] The query vector, key vector, and value vector are determined based on the target data. The ratio between the elements of the query vector and the key vector at the first index position is the same as the ratio between the elements of the query vector and the key vector at the second index position. The first index position and the second index position are any two index positions in the first dimension with an interval of 0.5 times the first dimension. The first dimension is the dimension of the query vector and the key vector.

[0008] The query vector and the key vector are respectively positionally encoded to obtain the encoding result of the query vector and the encoding result of the key vector;

[0009] The feature vector of the target data is determined based on the encoding results of the query vector, the encoding results of the key vector, and the value vector.

[0010] In at least one embodiment of this specification, determining the query vector, key vector, and value vector based on the target data includes:

[0011] The query vector is determined based on the target data and the first projection matrix;

[0012] The folding vector is determined based on the target data and the second projection matrix, and the key vector is determined based on the query vector and the folding vector, wherein any two elements with a second distance between them are equal, and the second distance is 0.5 times the dimension of the folding vector;

[0013] The value vector is determined based on the target data and the third projection matrix.

[0014] In at least one embodiment of this specification, determining the folding vector based on the target data and the second projection matrix includes:

[0015] The target data and the second projection matrix are multiplied to obtain a first vector, wherein the first vector and the folded vector have the same dimension;

[0016] The average value of every two elements in the first vector that are separated by the second distance is determined as the value of the corresponding two elements in the folded vector.

[0017] In at least one embodiment of this specification, determining the key vector based on the query vector and the folded vector includes:

[0018] The key vector is determined by the Hadamard product of the query vector and the folded vector.

[0019] In at least one embodiment of this specification, the step of performing position encoding on the query vector and the key vector respectively to obtain the encoding result of the query vector and the encoding result of the key vector includes:

[0020] The query vector and the key vector are rotated and encoded respectively to obtain the encoding results of the query vector and the key vector.

[0021] In at least one embodiment of this specification, determining the feature vector of the target data based on the encoding result of the query vector, the encoding result of the key vector, and the value vector includes:

[0022] The attention weight matrix is ​​determined based on the encoding results of the query vector and the encoding results of the key vector;

[0023] The feature vector of the target data is determined based on the attention weight matrix and the value vector.

[0024] In at least one embodiment of this specification, determining the attention weight matrix based on the encoding result of the query vector and the encoding result of the key vector includes:

[0025] The attention weight matrix is ​​obtained by multiplying the encoded result of the query vector and the encoded result of the key vector using the Einstein summation algorithm.

[0026] In at least one embodiment of this specification, the method further includes:

[0027] The target data is encoded or decoded based on its feature vector.

[0028] In at least one embodiment of this specification, the target data includes training samples during the training process or data to be processed during the application process.

[0029] According to a second aspect of one or more embodiments of this specification, a feature extraction apparatus is provided, the apparatus comprising:

[0030] The vector module is used to determine a query vector, a key vector, and a value vector based on target data. The ratio between the elements of the query vector and the key vector at the first index position is the same as the ratio between the elements of the query vector and the key vector at the second index position. The first index position and the second index position are any two index positions in the first dimension with an interval of 0.5 times the first dimension. The first dimension is the dimension of the query vector and the key vector.

[0031] An encoding module is used to perform position encoding on the query vector and the key vector respectively, to obtain the encoding result of the query vector and the encoding result of the key vector;

[0032] The feature module is used to determine the feature vector of the target data based on the encoding result of the query vector, the encoding result of the key vector, and the value vector.

[0033] In at least one embodiment of this specification, the vector module is used for:

[0034] The query vector is determined based on the target data and the first projection matrix;

[0035] The folding vector is determined based on the target data and the second projection matrix, and the key vector is determined based on the query vector and the folding vector, wherein any two elements with a second distance between them are equal, and the second distance is 0.5 times the dimension of the folding vector;

[0036] The value vector is determined based on the target data and the third projection matrix.

[0037] In at least one embodiment of this specification, when the vector module is used to determine the folding vector based on the target data and the second projection matrix, it is used for:

[0038] The target data and the second projection matrix are multiplied to obtain a first vector, wherein the first vector and the folded vector have the same dimension;

[0039] The average value of every two elements in the first vector that are separated by the second distance is determined as the value of the corresponding two elements in the folded vector.

[0040] In at least one embodiment of this specification, when the vector module is used to determine the key vector based on the query vector and the folded vector, it is used to:

[0041] The key vector is determined by the Hadamard product of the query vector and the folded vector.

[0042] In at least one embodiment of this specification, the encoding module is used for:

[0043] The query vector and the key vector are rotated and encoded respectively to obtain the encoding results of the query vector and the key vector.

[0044] In at least one embodiment of this specification, the feature module is used for:

[0045] The attention weight matrix is ​​determined based on the encoding results of the query vector and the encoding results of the key vector;

[0046] The feature vector of the target data is determined based on the attention weight matrix and the value vector.

[0047] In at least one embodiment of this specification, when the feature module is used to determine the attention weight matrix based on the encoding result of the query vector and the encoding result of the key vector, it is used to:

[0048] The attention weight matrix is ​​obtained by multiplying the encoded result of the query vector and the encoded result of the key vector using the Einstein summation algorithm.

[0049] In at least one embodiment of this specification, the apparatus further includes a processing module for:

[0050] The target data is encoded or decoded based on its feature vector.

[0051] In at least one embodiment of this specification, the target data includes training samples during the training process or data to be processed during the application process.

[0052] According to a third aspect of one or more embodiments of this specification, an electronic device is provided, comprising:

[0053] processor;

[0054] Memory used to store processor-executable instructions;

[0055] The processor implements the method as described in the first aspect by running the executable instructions.

[0056] According to a fourth aspect of one or more embodiments of this specification, a computer-readable storage medium is provided that stores computer instructions thereon, which, when executed by a processor, implement the steps of the method as described in the first aspect.

[0057] The technical solutions provided in the embodiments of this specification may include the following beneficial effects:

[0058] The feature extraction method provided in this specification is used to extract feature vectors from target data. The query vector and key vector determined by this method both have a first dimension. The first index position and the second index position are any two index positions in the first dimension with an interval of 0.5 times the first dimension. Furthermore, the ratio between the elements of the query vector and the key vector at the first index position is the same as the ratio between the elements of the query vector and the key vector at the second index position. That is, the vector formed by the elements at any two index positions of the query vector with an interval of 0.5 times the first dimension in the complex plane is collinear with the vector formed by the elements at the corresponding index positions of the key vector in the complex plane, i.e., the included angle is 0°. Since the attention score of any element of the query vector encoding result (hereinafter referred to as the first element) and any element of the key vector encoding result (hereinafter referred to as the second element) can be visualized as the vector formed by the elements at any two index positions of the query vector with an interval of 0.5 times the first dimension in the complex plane, and the vector formed by the elements at the corresponding index positions of the key vector, the vector is collinear with the vector formed by the elements at the corresponding index positions of the key vector. The inner product of the vectors formed by the elements at corresponding index positions in the complex plane after rotation (i.e., the inner product of the two vectors after rotation), and the rotation angles of the two vectors depend on the index positions of the first and second elements respectively. In this method, the two vectors are collinear before rotation, so the distance between the index positions of the first and second elements conforms to prior knowledge with the attention scores of the first and second elements (i.e., the greater the distance, the lower the attention score; the smaller the distance, the higher the attention score). This makes the attention scores of any two elements determined by this method more accurate than related techniques, because in related techniques, the two vectors have an angle greater than 0 before rotation. Furthermore, the feature vectors extracted by this method can more accurately represent the correlation between different elements of the target data. Moreover, the extrapolation ability of large language data trained using this method can be greatly improved compared to related techniques, and no model fine-tuning is required when training large language data using this method. Attached Figure Description

[0059] Figure 1 This is a flowchart of a feature extraction method provided in an exemplary embodiment.

[0060] Figure 2 This is a schematic diagram of vector rotation on a complex plane provided in an exemplary embodiment.

[0061] Figure 3 This is a schematic diagram of vector rotation on a complex plane provided in an exemplary embodiment.

[0062] Figure 4 This is a schematic diagram of vector rotation on a complex plane provided in an exemplary embodiment.

[0063] Figure 5This is a flowchart illustrating a method for determining a query vector, a key vector, and a value vector, as provided in an exemplary embodiment.

[0064] Figure 6 This is a flowchart of a feature extraction method provided in an exemplary embodiment.

[0065] Figure 7 This is a schematic diagram of the structure of a device provided in an exemplary embodiment.

[0066] Figure 8 This is a block diagram of a feature extraction apparatus provided in an exemplary embodiment. Detailed Implementation

[0067] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0068] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.

[0069] In recent years, artificial intelligence technology has developed rapidly, triggering technological innovations in many fields and bringing great convenience to people's work and lives. For example, large language models, due to their advantages such as context awareness, task versatility, and language generation capabilities, can complete various natural language processing tasks, such as text generation, machine translation, and question-answering systems. The level of intelligence of a large language model depends on its parameter scale and training level; that is, the larger the parameter scale and the higher the training level, the higher the level of intelligence of the large language model, and vice versa.

[0070] Understandably, the sequence length of training samples processed by large language models during the training phase is finite. However, the sequence length of data processed by large language models in practical applications is unpredictable and may be longer than the sequence length of training samples, even significantly longer. The extrapolation capability of large language models refers to their ability to process data with sequence lengths longer than those encountered during training. However, in related technologies, the extrapolation capability of large language models is poor, meaning they cannot handle sequences longer than those encountered during training.

[0071] Based on this, in a first aspect, at least one embodiment of this specification provides a feature extraction method that can be applied to a self-attention model framework to extract feature vectors from a data sequence to characterize the correlation between different elements within the sequence; this method can improve the accuracy and precision of the feature vectors, thereby significantly improving the extrapolation capability of a large language model trained using this method.

[0072] Please refer to the appendix. Figure 1 The example illustrates the flow of the feature extraction method, including steps S101 to S103.

[0073] In step S101, a query vector, a key vector, and a value vector are determined based on the target data. The ratio between the elements of the query vector and the key vector at the first index position is the same as the ratio between the elements of the query vector and the key vector at the second index position. The first index position and the second index position are any two index positions in the first dimension with an interval of 0.5 times the first dimension. The first dimension is the dimension of the query vector and the key vector.

[0074] The target data can be a sequence of text data, image data, or audio data, such as a text sequence for text data, a pixel sequence for image data, and a frequency sequence for audio data. The target data is the data from which features are to be extracted; that is, this method extracts feature vectors from the target data.

[0075] For example, this method can be applied to the training process of a large language model, where the target data is the training samples during the training process. Alternatively, this method can be applied to the application process of a large language model, where the target data is the data to be processed during the application process.

[0076] As another example, this method can be applied to the encoding stage of a large language model, where the feature vectors of the target data are used to encode the target data; in this case, the self-attention model framework running this method can be placed within the encoder of the large language model. Alternatively, this method can be applied to the decoding stage of a large language model, where the feature vectors of the target data are used to decode the target data; in this case, the self-attention model framework running this method can be placed within the decoder of the large language model.

[0077] It is important to note that the ratio between the elements of the query vector and the key vector at the first index position refers to the ratio between the elements of the query vector at the first index position and the elements of the key vector at the first index position; similarly, the ratio between the elements of the query vector and the key vector at the second index position refers to the ratio between the elements of the query vector at the second index position and the elements of the key vector at the second index position. For example, the query vector is a d-dimensional vector Q(q0, q1, q2, ..., q...). j , ..., q d-1 The key vector is a d-dimensional vector K(k0, k1, k2, ..., kk). j ..., k d-1 ), then q j With q j+d / 2 The ratio between them, and k j With k j+d / 2 The proportions between them are the same, that is, q j and q j+d / 2 The vector formed on the complex plane, and k j With k j+d / 2 The vectors formed on the complex plane are collinear, and j is less than d / 2.

[0078] In step S102, the query vector and the key vector are respectively positionally encoded to obtain the encoding result of the query vector and the encoding result of the key vector.

[0079] Among them, positional encoding is a method used in large language models to represent the relative positions of word vectors.

[0080] In this step, the query vector and key vector can be position-encoded using methods such as sine position encoding, cosine position encoding, and rotation position encoding. It is preferred to use rotation position encoding to encode the query vector and key vector, that is, to perform rotation position encoding on the query vector and the key vector respectively, so as to obtain the encoding result of the query vector and the encoding result of the key vector.

[0081] In step S103, the feature vector of the target data is determined based on the encoding result of the query vector, the encoding result of the key vector, and the value vector.

[0082] For example, this step can be performed as follows:

[0083] First, the attention weight matrix is ​​determined based on the encoding results of the query vector and the key vector. For example, the attention weight matrix is ​​obtained by multiplying the encoding results of the query vector and the key vector. Each element of the query vector encoding result is multiplied by each element of the key vector encoding result to obtain each attention score in the attention weight matrix.

[0084] Next, the feature vector of the target data is determined based on the attention weight matrix and the value vector. For example, the feature vector of the target data is obtained by multiplying the attention weight matrix by the value vector.

[0085] The feature extraction method provided in this specification is used to extract feature vectors from target data. The query vector and key vector determined by this method both have a first dimension. The first index position and the second index position are any two index positions in the first dimension with an interval of 0.5 times the first dimension. Furthermore, the ratio between the elements of the query vector and the key vector at the first index position is the same as the ratio between the elements of the query vector and the key vector at the second index position. That is, the vector formed by the elements at any two index positions of the query vector with an interval of 0.5 times the first dimension in the complex plane is collinear with the vector formed by the elements at the corresponding index positions of the key vector in the complex plane, i.e., the included angle is 0°. Since the attention score between any element of the encoded result of the query vector (hereinafter referred to as the first element) and any element of the encoded result of the key vector (hereinafter referred to as the second element) can be visualized as the vector formed by the elements at any two index positions of the query vector with an interval of 0.5 times the first dimension in the complex plane, and the vector formed by the elements at the corresponding index positions of the key vector, the vector is collinear with the vector formed by the first element. The inner product of the vectors formed by the elements at the corresponding index positions on the complex plane after rotation (i.e., the inner product of the two vectors after rotation), and the rotation angles of the two vectors depend on the index positions of the first and second elements respectively. In this method, the two vectors are collinear before rotation, so the distance between the index positions of the first and second elements conforms to prior knowledge with the attention scores of the first and second elements (i.e., the greater the distance, the lower the attention score; the smaller the distance, the higher the attention score). This makes the attention scores of any two elements determined by this method more accurate than related techniques, because in related techniques, the two vectors have an angle greater than 0 before rotation. Furthermore, the feature vectors extracted by this method can more accurately represent the correlation between different elements of the target data. Moreover, the extrapolation ability of large language data trained using this method can be greatly improved compared to related techniques, and no model fine-tuning is required when training large language data using this method.

[0086] The technical effects of the above embodiments will be explained in more detail below using rotational position encoding.

[0087] Rotational position encoding can be represented by the following formula:

[0088]

[0089] In the above formula, n is the index position, n∈[0, c) and is an integer; x is a d-dimensional input vector (i.e., the vector encoded by the rotation position, such as the query vector, key vector, etc.), x=(x0, x1, x2, …, xd-1); The imaginary unit;

[0090]

[0091] According to the above formula, after the query vector and key vector are rotated and encoded, the attention score between a certain element in the encoded result of the query vector and a certain element in the encoded result of the key vector can be calculated by the following formula:

[0092]

[0093] In the above formula, a(m, n) is the attention score between the element at the m-th index position in the Q-rotary encoding result of the query vector and the element at the n-th index position in the K-rotary encoding result of the key vector; f(q, m) is the element at the m-th index position in the Q-rotary encoding result of the query vector, i.e., the element at the m-th index position in the query vector Q after rotation encoding; f(k, n) is the element at the n-th index position in the K-rotary encoding result of the key vector, i.e., the element at the n-th index position in the key vector K after rotation encoding; d is the dimension of the query vector and the key vector; q j q j+d / 2 These represent the elements at index positions j and j+d / 2 in the query vector Q, respectively; k j k j+d / 2 These are the elements at the j-th and j+d / 2-th index positions in the key vector K, respectively.

[0094] As shown in the above formula, the attention score between an element in the encoding result of the query vector and an element in the encoding result of the key vector depends only on the relative positions of their index positions. Therefore, this application proposes that the attention score between any element of the query vector after rotational encoding (hereinafter referred to as the first element) and any element of the key vector after rotational encoding (hereinafter referred to as the second element) can be visualized as the inner product of the vector formed by the elements at any two index positions of the query vector with an interval of 0.5 times the first dimension in the complex plane, and the vector formed by the corresponding index position of the key vector in the complex plane after a (counterclockwise) rotation (i.e., the inner product of the two vectors after rotation). The rotation angles of these two vectors respectively depend on the index positions of the first and second elements, and are respectively mθ. j nθ j .

[0095] In related technologies, the vector formed by the elements at any two index positions of the query vector with an interval of 0.5 times the first dimension in the complex plane (hereinafter referred to as the first vector) has an initial angle θ0 greater than 0 with the vector formed by the corresponding index positions of the key vector in the complex plane (hereinafter referred to as the second vector). It is understood that in the appendix... Figure 2 In the case shown, the first vector is in the counterclockwise direction of the second vector. Therefore, when m > n, the greater the difference between m and n, the smaller the inner product of the first and second vectors after rotation (the smaller the attention score between the first and second elements). This aligns with prior knowledge (e.g., the monotonicity of rotational position encoding: the greater the distance between m and n, the lower the attention score between the first and second elements; the smaller the distance between m and n, the higher the attention score between the first and second elements). However, in the attached... Figure 3 In the case shown, the first vector is in the counterclockwise direction of the second vector, therefore, when m < n and |mn|θ j When θ < 0, the larger the difference between m and n, the larger the inner product of the first and second vectors after rotation (the smaller the attention score between the first and second elements). This clearly contradicts prior knowledge (e.g., the monotonicity of rotational position encoding: the larger the distance between m and n, the lower the attention score between the first and second elements; the smaller the distance between m and n, the higher the attention score between the first and second elements). However, in the appendix... Figure 4 In the case shown, the second vector is in the counterclockwise direction of the first vector, therefore, when m > n and |mn|θ jWhen θ < 0, the larger the difference between m and n, the larger the inner product of the first and second vectors after rotation. This is obviously inconsistent with prior knowledge (e.g., the monotonicity of rotation position encoding: the larger the distance between m and n, the lower the attention score between the first and second elements; the smaller the distance between m and n, the higher the attention score between the first and second elements).

[0096] It should also be pointed out that, attached Figure 3 , 4 |mn|θ j When the distance between m and n is less than θ0, the correlation between index positions is often stronger, and the index position distance is more important. Therefore, the negative impact of rotation position encoding in related technologies affects the important correlation information between the encoding results of the query vector and the encoding results of the key vector, and its impact cannot be ignored. Conversely, the effect achieved by this application after overcoming this impact is significant and the progress is obvious.

[0097] In this application, the vector formed by the elements at any two index positions of the query vector with an interval of 0.5 times the first dimension in the complex plane (hereinafter referred to as the first vector) and the vector formed by the corresponding index position of the key vector in the complex plane (hereinafter referred to as the second vector) have an angle of 0. Therefore, regardless of whether m > n or m < n, the distance between the index positions of the first element and the second element, and the attention scores of the first element and the second element, conform to the above-mentioned prior knowledge (e.g., the monotonicity of rotational position encoding: the larger the distance, the lower the attention score; the smaller the distance, the higher the attention score), and the situation that violates the monotonicity of rotational position encoding will not occur.

[0098] By comparing related technologies with the present application, it can be seen that the attention scores of any two elements determined by the method are more accurate than those of related technologies; furthermore, the feature vectors extracted by the method can more accurately represent the correlation between different elements of the target data; moreover, the extrapolation ability of large language data trained by the method can be greatly improved compared with related technologies, and no model fine-tuning is required when training large language data using the method.

[0099] In some embodiments of this specification, the appendix can be used as per the appendix. Figure 5 The query vector, key vector, and value vector are determined in the manner shown. Next, we will combine this with the appendix... Figure 5 The above determination method is described in detail, and the method includes steps S501 to S503.

[0100] In step S501, the query vector is determined based on the target data and the first projection matrix. For example, the query vector is determined according to the following formula:

[0101] Q = Wq X

[0102] In the above formula, Q is the query vector, and W... q Let X be the first projection matrix, and let X be the target data.

[0103] In step S502, a folding vector is determined based on the target data and the second projection matrix, and the key vector is determined based on the query vector and the folding vector. In the folding vector, any two elements with a second distance between them are equal, and the second distance is 0.5 times the dimension of the folding vector.

[0104] For example, this step determines the folded vector as follows: First, the target data and the second projection matrix are multiplied to obtain a first vector, wherein the first vector and the folded vector have the same dimension; Next, the average value of every two elements in the first vector that are separated by the second distance is determined as the value of the corresponding two elements in the folded vector.

[0105] For example, each element t in the folded vector is obtained sequentially according to the following formulas. n :

[0106]

[0107]

[0108] In the above formula, W t Let x be the second projection matrix. n For each element in the target data.

[0109] It is understandable that the operation in this step is equivalent to folding along d to T and then copying it again.

[0110] As another example, this step can determine the key vector in the following manner: the key vector is determined by the Hadamard product of the query vector and the folded vector; that is, each element of the query vector is multiplied by the corresponding element of the folded vector to obtain the corresponding element of the key vector.

[0111] Based on the method of determining the key vector in this embodiment, in the appendix Figure 1 In the illustrated embodiment, when determining the attention weight matrix based on the encoding results of the query vector and the key vector, the Einstein summation algorithm can be used to multiply the encoding results of the query vector and the key vector to obtain the attention weight matrix.

[0112] For example, the process of multiplying the encoded result of the query vector and the encoded result of the key vector is transformed as follows:

[0113] QKT =Q(Q°T) T

[0114] In the above formula, Q is the encoding result of the query vector, K is the encoding result of the key vector, T is the folded vector, and ° is the Hadamard product.

[0115] Then, based on the above transformation results, the attention weight matrix is ​​calculated using the Einstein summation algorithm.

[0116] Because the key vector determined according to the method of determining the key vector in this embodiment has a dimension that is at least d times larger than the key vector determined by directly using the projection matrix and target data in related technologies, the amount of computation and memory usage are increased accordingly; while the multiplication form of the above-mentioned transformed encoding result can be calculated using the Einstein summation algorithm, so that high computational efficiency and small memory usage can still be maintained when the amount of computation is large.

[0117] In step S503, the value vector is determined based on the target data and the third projection matrix. For example, the value vector is determined according to the following formula:

[0118] V = W v X

[0119] In the above formula, V is the query vector, and W... v Let X be the third projection matrix, and X be the target data.

[0120] Please refer to the appendix. Figure 6 The diagram illustrates, exemplarily, a flowchart of a feature extraction method combining the aforementioned embodiments, which is executed by a self-attention model. The self-attention model first receives target data X; then, through three projection matrices, it transforms the input target data into a query vector Q, a folded vector T, and a value vector V of the same shape; then, the query vector Q and the folded vector T are multiplied term by term to obtain a key vector K; the query vector Q and the key vector K are then rotated and encoded to obtain Q-rotary and K-rotary, respectively; then, Q-rotary and K-rotary are multiplied to obtain an attention weight matrix; finally, the attention weight matrix and the value vector V are multiplied to obtain a feature vector O.

[0121] Figure 7 This is a schematic structural diagram of a device provided in an exemplary embodiment. Please refer to... Figure 7At the hardware level, the device includes a processor 702, an internal bus 704, a network interface 706, memory 708, and non-volatile memory 710, and may also include other hardware required for tasks. One or more embodiments of this specification can be implemented in software, such as the processor 702 reading the corresponding computer program from the non-volatile memory 710 into memory 708 and then running it. Of course, in addition to software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0122] Please refer to Figure 8 Feature extraction devices can be applied to, for example Figure 7 The device shown is used to implement the technical solution of this specification. The device includes:

[0123] Vector module 801 is used to determine a query vector, a key vector, and a value vector based on target data, wherein the ratio between the elements of the query vector and the key vector at the first index position is the same as the ratio between the elements of the query vector and the key vector at the second index position, the first index position and the second index position are any two index positions in the first dimension with an interval of 0.5 times the first dimension, and the first dimension is the dimension of the query vector and the key vector;

[0124] The encoding module 802 is used to perform position encoding on the query vector and the key vector respectively to obtain the encoding result of the query vector and the encoding result of the key vector;

[0125] The feature module 803 is used to determine the feature vector of the target data based on the encoding result of the query vector, the encoding result of the key vector, and the value vector.

[0126] In at least one embodiment of this specification, the vector module is used for:

[0127] The query vector is determined based on the target data and the first projection matrix;

[0128] The folding vector is determined based on the target data and the second projection matrix, and the key vector is determined based on the query vector and the folding vector, wherein any two elements with a second distance between them are equal, and the second distance is 0.5 times the dimension of the folding vector;

[0129] The value vector is determined based on the target data and the third projection matrix.

[0130] In at least one embodiment of this specification, when the vector module is used to determine the folding vector based on the target data and the second projection matrix, it is used for:

[0131] The target data and the second projection matrix are multiplied to obtain a first vector, wherein the first vector and the folded vector have the same dimension;

[0132] The average value of every two elements in the first vector that are separated by the second distance is determined as the value of the corresponding two elements in the folded vector.

[0133] In at least one embodiment of this specification, when the vector module is used to determine the key vector based on the query vector and the folded vector, it is used to:

[0134] The key vector is determined by the Hadamard product of the query vector and the folded vector.

[0135] In at least one embodiment of this specification, the encoding module is used for:

[0136] The query vector and the key vector are rotated and encoded respectively to obtain the encoding results of the query vector and the key vector.

[0137] In at least one embodiment of this specification, the feature module is used for:

[0138] The attention weight matrix is ​​determined based on the encoding results of the query vector and the encoding results of the key vector;

[0139] The feature vector of the target data is determined based on the attention weight matrix and the value vector.

[0140] In at least one embodiment of this specification, when the feature module is used to determine the attention weight matrix based on the encoding result of the query vector and the encoding result of the key vector, it is used to:

[0141] The attention weight matrix is ​​obtained by multiplying the encoded result of the query vector and the encoded result of the key vector using the Einstein summation algorithm.

[0142] In at least one embodiment of this specification, the apparatus further includes a processing module for:

[0143] The target data is encoded or decoded based on its feature vector.

[0144] In at least one embodiment of this specification, the target data includes training samples during the training process or data to be processed during the application process.

[0145] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0146] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0147] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0148] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0149] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0150] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0151] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of one or more embodiments of this specification. The singular forms “a,” “described,” and “the” used in one or more embodiments of this specification and in the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0152] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0153] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0154] It should be understood that although the terms first, second, third, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of one or more embodiments of this specification, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "in response to a determination," or "when," or "in the event of a determination."

[0155] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit the scope of one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments of this specification should be included within the protection scope of one or more embodiments of this specification.

Claims

1. A feature extraction method, the method comprising: A query vector, a key vector, and a value vector are determined based on the target data. The ratio between the elements of the query vector and the key vector at the first index position is the same as the ratio between the elements of the query vector and the key vector at the second index position. The first index position and the second index position are any two index positions in a first dimension with an interval of 0.5 times the first dimension. The first dimension is the dimension of the query vector and the key vector. The query vector is determined based on the target data and a first projection matrix. A folded vector is determined based on the target data and a second projection matrix, and the key vector is determined based on the query vector and the folded vector. The value vector is determined based on the target data and a third projection matrix. The target data includes at least one corresponding data sequence of text data, image data, and audio data. The query vector and the key vector are respectively positionally encoded to obtain the encoding result of the query vector and the encoding result of the key vector; The feature vector of the target data is determined based on the encoding results of the query vector, the encoding results of the key vector, and the value vector.

2. The feature extraction method according to claim 1, wherein in the folded vector, any two elements with a second distance between them are equal, and the second distance is 0.5 times the dimension of the folded vector.

3. The feature extraction method according to claim 2, wherein determining the folding vector based on the target data and the second projection matrix includes: The target data and the second projection matrix are multiplied to obtain a first vector, wherein the first vector and the folded vector have the same dimension; The average value of every two elements in the first vector that are separated by the second distance is determined as the value of the corresponding two elements in the folded vector.

4. The feature extraction method according to claim 2, wherein determining the key vector based on the query vector and the folded vector comprises: The key vector is determined by the Hadamard product of the query vector and the folded vector.

5. The feature extraction method according to claim 1, wherein performing positional encoding on the query vector and the key vector respectively to obtain the encoding result of the query vector and the encoding result of the key vector includes: The query vector and the key vector are rotated and encoded respectively to obtain the encoding results of the query vector and the key vector.

6. The feature extraction method according to claim 1, wherein determining the feature vector of the target data based on the encoding result of the query vector, the encoding result of the key vector, and the value vector comprises: Based on the encoding results of the query vector and the encoding results of the key vector, determine the attention weight matrix; The feature vector of the target data is determined based on the attention weight matrix and the value vector.

7. The feature extraction method according to claim 6, wherein determining the attention weight matrix based on the encoding result of the query vector and the encoding result of the key vector comprises: The attention weight matrix is ​​obtained by multiplying the encoded result of the query vector and the encoded result of the key vector using the Einstein summation algorithm.

8. The feature extraction method according to claim 1, further comprising: The target data is encoded or decoded based on its feature vector.

9. The feature extraction method according to claim 1, wherein the target data includes training samples during the training process or data to be processed during the application process.

10. A feature extraction apparatus, the apparatus comprising: A vector module is used to determine a query vector, a key vector, and a value vector based on target data. The ratio between the elements of the query vector and the key vector at a first index position is the same as the ratio between the elements of the query vector and the key vector at a second index position. The first index position and the second index position are any two index positions in a first dimension with an interval of 0.5 times the first dimension. The first dimension is the dimension of the query vector and the key vector. The query vector is determined based on the target data and a first projection matrix; a folded vector is determined based on the target data and a second projection matrix; the key vector is determined based on the query vector and the folded vector; and the value vector is determined based on the target data and a third projection matrix. The target data includes at least one corresponding data sequence of text data, image data, and audio data. An encoding module is used to perform position encoding on the query vector and the key vector respectively, to obtain the encoding result of the query vector and the encoding result of the key vector; The feature module is used to determine the feature vector of the target data based on the encoding result of the query vector, the encoding result of the key vector, and the value vector.

11. An electronic device, comprising: processor; Memory used to store processor-executable instructions; The processor implements the method as described in any one of claims 1-9 by executing the executable instructions.

12. A computer-readable storage medium having stored thereon computer instructions that, when executed by a processor, implement the steps of the method as claimed in any one of claims 1-9.

Images