Data processing method, device, electronic device and storage medium

Abstract

The present disclosure provides a data processing method, device, electronic device and storage medium, wherein the method comprises: obtaining a data sequence of data to be processed to be input into a self-attention module, and determining a query feature vector, a key feature vector and a value feature vector of each feature data in the data sequence; the processed data comprises: image data and / or text data; the query feature vector and the key feature vector are respectively converted by a nonlinear conversion function to obtain a first vector and a second vector; the elements in the first vector and the second vector are both greater than or equal to zero; for the current feature data of the data sequence, the second vector and the value feature vector of each feature data are calculated and then multiplied with the first vector of the current feature data to obtain a target product; the attention matrix of the data to be processed is determined according to the target product, and the attention matrix is ​​input into the feedforward module for processing to obtain a data processing result.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a data processing method, device, electronic device and storage medium. Background Art

[0002] With the continuous development of artificial intelligence and big data technology, natural language processing (for example, intelligent question-answering systems, machine translation systems) is rapidly changing people's lifestyles and greatly improving work efficiency and user experience. At present, the Transformer model has achieved great success in natural language processing, computer vision, and audio processing. As one of its core components, the softmax attention mechanism helps to capture long-distance dependencies. However, when calculating the attention matrix, the softmax attention mechanism in the existing Transformer model has quadratic space and time complexity relative to the input data sequence, that is, the total space and time complexity of the attention matrix grows quadratically with the length of the input data sequence. Therefore, due to the quadratic space and time complexity of the data sequence length, the softmax attention mechanism is difficult to expand. Summary of the invention

[0003] The embodiments of the present disclosure at least provide a data processing method, device, electronic device, and storage medium.

[0004] In a first aspect, an embodiment of the present disclosure provides a data processing method, which is applied to a Transformer model including a self-attention module and a feedforward module, comprising: obtaining a data sequence of data to be processed to be input into the self-attention module, and determining a query feature vector, a key feature vector, and a value feature vector of each feature data in the data sequence; the data to be processed includes: image data and / or text data; the query feature vector and the key feature vector are respectively converted by a nonlinear conversion function to obtain a first vector and a second vector; the elements in the first vector and the second vector are both greater than or equal to zero; for the current feature data of the data sequence, the second vector and the value feature vector of each feature data are calculated and then multiplied with the first vector of the current feature data to obtain a target product; according to the target product, an attention matrix of the data to be processed is determined, and the attention matrix is ​​input into the feedforward module for processing to obtain a data processing result.

[0005] Through the above processing method, the processing process of the self-attention model can be linearized, thereby reducing the space and time complexity of the attention matrix, so that the total space and time complexity of the attention matrix will not increase quadratically with the length of the input data sequence, thereby improving the scalability of the softmax attention mechanism.

[0006] In an optional implementation, the nonlinear conversion function is a nonlinear activation function; the query feature vector and the key feature vector are converted by the nonlinear conversion function respectively to obtain a first vector and a second vector, including: converting the query feature vector by the nonlinear activation function to obtain the first vector, and converting the key feature vector by the nonlinear activation function to obtain the second vector.

[0007] After the query feature vector and the key feature vector are transformed by the nonlinear activation function Relu, the elements in the first vector and the second vector are both greater than or equal to 0. By retaining the positive values ​​in the vector, the self-attention module can ignore features with negative correlation, thereby effectively avoiding the aggregation of irrelevant contextual information.

[0008] In an optional implementation, for the current feature data of the data sequence, the second vector and the value feature vector of each feature data are calculated and then multiplied with the first vector of the current feature data to obtain a target product, including: based on the position of the feature data in the data sequence, determining the weight of the first vector of the current feature data to obtain a first weight, and determining the weight of the second vector of each feature data to obtain a second weight; calculating the product of the first vector and the first weight to obtain a first result, and calculating the product of the second vector and the second weight to obtain a second result; and calculating the second result and the value feature vector, and then multiplying them with the first result to obtain the target product.

[0009] In the above implementation, by determining the first weight and the second weight by the position of the feature data in the data sequence, more weights can be applied between adjacent feature data in the data sequence, thereby enhancing the locality between the feature data.

[0010] In an optional implementation, the method of determining the weight of the first vector of the current feature data based on the position of the feature data in the data sequence to obtain a first weight, and determining the weight of the second vector of each feature data to obtain a second weight, includes: determining the cosine distance between each feature data and the current feature data based on the position of each feature data in the data sequence; and determining the first weight and the second weight according to the cosine distance.

[0011] In an optional embodiment, the number of the first weights and the second weights is multiple; the calculating the product of the first vector and the first weight to obtain a first result, and the calculating the product of the second vector and the second weight to obtain a second result, includes: calculating the product of the first vector and each first weight to obtain multiple first results; calculating the product of the second vector and the second weight to obtain multiple second results; the calculating the second result and the value feature vector and then multiplying them with the first result to obtain the first result, includes: calculating each second result and the value feature vector and then multiplying them with the corresponding first result to obtain multiple target sub-products, and summing the multiple target sub-products to obtain the target product.

[0012] In the above implementation, by determining the first weight and the second weight by the position of the feature data in the data sequence, more weights can be applied between adjacent feature data in the data sequence, thereby strengthening the locality between the feature data. By calculating each second result and the value feature vector and then multiplying them with the corresponding first result, the processing process of the self-attention model can be linearized, thereby reducing the space and time complexity of the attention matrix, so that the total space and time complexity of the attention matrix will not increase quadratically with the length of the input data sequence, thereby improving the scalability of the softmax attention mechanism.

[0013] In an optional embodiment, the number of the target products is multiple; determining the attention matrix of the data to be processed based on the target products includes: summing the multiple target products to obtain a first operation result; for the current feature data in the data sequence, summing the calculation results of the first vector of the current feature data and the second vector of each feature data to obtain a second operation result; determining the attention matrix of the data to be processed based on the ratio of the first operation result to the second operation result.

[0014] In an optional implementation, for the current feature data in the data sequence, the calculation results of the first vector of the current feature data and the second vector of each feature data are summed to obtain a second calculation result, including: determining the weight of the first vector of the current feature data to obtain a first weight, and determining the weight of the second vector of each feature data to obtain a second weight; calculating the product of the first vector and the first weight to obtain a first result, and calculating the product of the second vector and the second weight to obtain a second result; summing the product of the first result and the second result to obtain the second calculation result.

[0015] In an optional embodiment, the first weight includes: a first cosine weight and a first sine weight, and the second weight includes: a second cosine weight and a second sine weight; wherein the first cosine weight and the first sine weight are respectively the cosine value and the sine value determined based on the position of the current feature data in the data sequence, and the second cosine weight and the second sine weight are respectively the cosine value and the sine value determined based on the position of each feature data in the data sequence.

[0016] In the above implementation, this processing method can reduce the space and time complexity of the attention matrix, so that the total space and time complexity of the attention matrix will not increase quadratically with the length of the input data sequence, thereby improving the scalability of the softmax attention mechanism. After replacing the self-attention module in the Transformer model in the above-described manner, the Transformer model can be more stable, and at the same time, the speed of the model can be improved and the memory can be reduced while ensuring the effect.

[0017] In a second aspect, an embodiment of the present disclosure provides a data processing device, which is applied to a Transformer model including a self-attention module and a feedforward module, including: an acquisition unit, which is used to acquire a data sequence of data to be processed to be input into the self-attention module, and determine the query feature vector, key feature vector, and value feature vector of each feature data in the data sequence; the data to be processed includes: image data and / or text data; a conversion unit, which is used to convert the query feature vector and the key feature vector respectively through a nonlinear conversion function to obtain a first vector and a second vector; the elements in the first vector and the second vector are both greater than or equal to zero; a calculation unit, which is used to calculate the second vector and the value feature vector of each feature data for the current feature data of the data sequence, and then multiply them with the first vector of the current feature data to obtain a target product; a determination unit, which is used to determine the attention matrix of the data to be processed according to the target product, and input the attention matrix into the feedforward module for processing to obtain a data processing result.

[0018] In a third aspect, an embodiment of the present disclosure further provides an electronic device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the above-mentioned first aspect, or any possible implementation of the first aspect are performed.

[0019] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned first aspect or any possible implementation of the first aspect are executed.

[0020] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following is a brief introduction to the drawings required for use in the embodiments. The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used together with the specification to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can also be obtained based on these drawings without creative work.

[0022] Figure 1 A schematic diagram of the model structure of a Multi-Head Attention model in a Transformer model is shown;

[0023] Figure 2 A flow chart of a data processing method provided by an embodiment of the present disclosure is shown;

[0024] FIG3(a) is a schematic diagram showing the processing process of an existing Self Attention module in a Transformer model;

[0025] FIG3( b ) is a schematic diagram showing a processing process of a linearized self-attention module provided in an embodiment of the present disclosure;

[0026] Figure 4 A flowchart showing a specific method of converting the query feature vector and the key feature vector respectively by a nonlinear conversion function to obtain a first vector and a second vector in the data processing method provided by an embodiment of the present disclosure;

[0027] Figure 5 A flowchart showing a specific method for obtaining a target product by calculating, for each feature data in the data sequence, a second vector and a value feature vector corresponding to other feature data except the feature data and then multiplying the result with the first vector of the feature data in the data processing method provided by an embodiment of the present disclosure;

[0028] FIG6( a ) shows a schematic diagram of image data provided by an embodiment of the present disclosure;

[0029] FIG6( b ) is a schematic diagram showing an image segmentation result of image data provided by an embodiment of the present disclosure;

[0030] FIG6( c ) is a schematic diagram showing a standard segmentation result of image data provided by an embodiment of the present disclosure;

[0031] Figure 7 A schematic diagram of a data processing device provided by an embodiment of the present disclosure is shown;

[0032] Figure 8 A schematic diagram of an electronic device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical scheme and advantages of the embodiments of the present disclosure clearer, the technical scheme in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the present disclosure for protection, but merely represents the selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present disclosure.

[0034] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0035] The term "and / or" herein only describes an association relationship, indicating that three relationships may exist. For example, A and / or B may represent three situations: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C may represent including any one or more elements selected from the set consisting of A, B, and C.

[0036] Research has found that the Transformer model has achieved great success in natural language processing, computer vision, and audio processing. As one of its core components, the softmax attention mechanism helps capture long-distance dependencies. However, when calculating the attention matrix, the softmax attention mechanism in the existing Transformer model has quadratic space and time complexity relative to the input data sequence, that is, the total space and time complexity of the attention matrix grows quadratically with the length of the input data sequence. Therefore, due to the quadratic space and time complexity of the data sequence length, the softmax attention mechanism is difficult to expand.

[0037] Based on the above research, the present disclosure provides a data processing method, device, electronic device and storage medium. In the embodiment of the present disclosure, first, the query feature vector and the key feature vector are respectively converted by a nonlinear conversion function; then, for the current feature data of the data sequence, the conversion vector (i.e., the second vector) of the key feature vector of each feature data and the value feature vector of the feature data can be calculated, and the calculation result can be multiplied with the conversion vector (i.e., the first vector) of the query feature vector of the current feature data to determine the attention matrix of each feature data in the data sequence according to the target product. Through the above processing method, the processing process of the self-attention model can be linearized, thereby reducing the space and time complexity of the attention matrix, so that the total space and time complexity of the attention matrix will not increase quadratically with the length of the input data sequence, thereby improving the scalability of the softmax attention mechanism.

[0038] The Transformer model includes multiple encoders and multiple decoders, where each encoder includes a Feed Forward module and a Self Attention module, and each decoder includes a Feed Forward module, an Encoder-Decoder Attention module, and a Self Attention module.

[0039] The method provided in the embodiment of the present disclosure can be understood as the process of processing the input data sequence by the Self Attention module in the Transformer model. Figure 1 As shown, Figure 1The model structure diagram of the Multi-Head Attention model in the Transformer model is shown. The method provided in the embodiment of the present disclosure can be applied to the Scale Dot-Product Attention part of the Multi-Head Attention model. Compared with the processing process of the existing Self Attention module in the Transformer model, the method provided in the embodiment of the present disclosure can linearize the processing process of the self-attention module, thereby reducing the space and time complexity of the attention matrix, so that the total space and time complexity of the attention matrix will not increase quadratically with the length of the input data sequence.

[0040] To facilitate understanding of this embodiment, a data processing method disclosed in the embodiment of the present disclosure is first introduced in detail. The data processing method provided in the embodiment of the present disclosure is generally executed by an electronic device with certain computing capabilities.

[0041] See also Figure 2 FIG. 2 is a flowchart of a data processing method provided by an embodiment of the present disclosure, wherein the method includes steps S201 to S207, wherein:

[0042] S201: Obtain a data sequence of data to be processed to be input into the self-attention module, and determine the query feature vector, key feature vector, and value feature vector of each feature data in the data sequence; the data to be processed includes: image data and / or text data.

[0043] In the embodiments of the present disclosure, initial data may be obtained and processed to obtain data to be processed. For example, the initial data may be video data, in which case the video frame to be processed may be extracted from the video data as image data, or audio data may be extracted from the video data and the extracted audio data may be converted into text data. In addition, the initial data may also be audio data, in which case the audio data may be converted into corresponding text data to obtain data to be processed.

[0044] The above is specifically introduced below in conjunction with Figure 3(a) and Figure 3(b). Figure 3(a) shows the processing process of the existing Self Attention module in the Transformer model, and Figure 3(a) shows the processing process of the linear self-attention module provided in the embodiment of the present disclosure.

[0045] For each feature data in the data sequence to be processed, the Self Attention module will first calculate three new vectors, which are called Query (query feature vector, hereinafter referred to as vector Q), Key (key feature vector, hereinafter referred to as vector K), and Value (value feature vector, hereinafter referred to as vector V). These three vectors are the result of multiplying the embedding vector with a matrix, where the embedding vector is each feature data in the data sequence.

[0046] As shown in Figure 3(a), the existing Self Attention module in the Transformer model first uses the affinity function S to transform the vectors Q and K into a transposed matrix K T Perform similarity calculation; then, perform softmax normalization on the calculated result and multiply it with the vector V to obtain the attention matrix of the current text.

[0047] If the length of the data sequence is N, and the feature size of each feature data feature in the data sequence is d, as shown in Figure 3(a), the transposed matrix K of Q and K is T After multiplying the similarity result by V, we can get a complexity of O(N 2 )’s attention matrix.

[0048] As shown in FIG. 3( b ), in the embodiment of the present disclosure, first, the transposed matrix K of K is T Multiplying by vector V gives a complexity of O(Nd 2 ) is calculated; then, the result is multiplied by the vector Q to get the attention matrix with a complexity of O(N). By reducing the complexity of the attention matrix in time and space, the data processing efficiency of the Transformer model can be improved.

[0049] S203: Convert the query feature vector and the key feature vector respectively by using a nonlinear conversion function to obtain a first vector and a second vector; elements in the first vector and the second vector are both greater than or equal to zero.

[0050] In the embodiment of the present disclosure, in order to realize the transposed matrix K of K T After multiplying with vector V, it is multiplied with vector Q. It is necessary to transform the affinity function S in the existing self-attention module into a decomposable affinity function. For example, the affinity function S in the existing self-attention module can be transformed into a decomposable affinity function by the following formula:

[0051] S(Q i ,K j )=φ(Q i)φ(K j ) T ; where i represents the i-th feature data in the data sequence, j represents the j-th feature data in the data sequence, and Q i The vector Q, K representing the i-th feature data j The vector K representing the j-th feature data.

[0052] After that, the attention operation under linear complexity can be implemented through the matrix product property, for example, as follows: (φ(Q)φ(K)) T V=φ(Q)(φ(K) T V).

[0053] Based on this, in the embodiment of the present disclosure, it is necessary to transform the query feature vector Q and the key feature vector K respectively through a nonlinear transformation function to obtain a first vector and a second vector. Here, φ(·) is represented by the nonlinear transformation function.

[0054] In the disclosed embodiment, the sequence length of the data sequence is N, that is, the data sequence contains N feature data, and for each feature data, a set of vectors can be obtained, namely: query feature vector, key feature vector, value feature vector. Therefore, the query feature vector and key feature vector in each set of vectors can be converted by a nonlinear conversion function to obtain a first vector and a second vector.

[0055] In the disclosed embodiment, the elements in the first vector and the second vector obtained after converting the query feature vector and the key feature vector are both greater than or equal to 0. By retaining the positive values ​​in the vector, the self-attention module can ignore features with negative correlation, thereby effectively avoiding the aggregation of irrelevant context information.

[0056] S205: For the current feature data of the data sequence, calculate the second vector and the value feature vector of each feature data, and then multiply the result with the first vector of the current feature data to obtain a target product.

[0057] In the embodiment of the present disclosure, after the query feature vector and the key feature vector are converted in the manner described above, the attention matrix of each feature data in the data sequence can be determined.

[0058] In specific implementation, for the current feature data in the data sequence, first, the conversion vector (i.e., the second vector) of the key feature vector of each feature data in the data sequence and the value feature vector of the feature data can be calculated, and then the calculation result can be multiplied with the conversion vector (i.e., the first vector) of the query feature vector of the current feature data to obtain the target product.

[0059] S207: Determine the attention matrix of the data to be processed according to the target product, and input the attention matrix into the feedforward module for processing to obtain a data processing result.

[0060] In the disclosed embodiment, first, the query feature vector and the key feature vector are respectively transformed by a nonlinear transformation function; then, for the current feature data in the data sequence, the transformation vector (i.e., the second vector) of the key feature vector of each feature data and the value feature vector of the feature data can be calculated, and the calculation result can be multiplied with the transformation vector (i.e., the first vector) of the query feature vector of the current feature data to determine the attention matrix of each feature data in the data sequence according to the target product. Through the above processing method, the processing process of the self-attention model can be linearized, thereby reducing the space and time complexity of the attention matrix, so that the total space and time complexity of the attention matrix will not increase quadratically with the length of the input data sequence, thereby improving the scalability of the softmax attention mechanism.

[0061] Here, the processing process of the linearized self-attention model can be understood as follows: the total space and time complexity of the attention matrix is ​​linearly related to the sequence length of the input data sequence, that is, the total space and time complexity of the attention matrix will not increase quadratically with the length of the input data sequence.

[0062] In the embodiments of the present disclosure, the data to be processed may be image data or text data. The data processing method will be described below in different scenarios.

[0063] Scene 1: Image segmentation scene.

[0064] First, obtain the data to be processed, that is, the image data to be segmented. For example, the image data may be the image shown in Figure 6(a). After obtaining the image data, the image data may be processed to obtain a data sequence of the image data, wherein the data sequence contains each feature data in the image data. Next, the query feature vector, key feature vector, and value feature vector of each feature data in the data sequence may be determined. Afterwards, the determined feature vector may be input into the encoder of the Transformer model for processing. For example, it may be input into Figure 1The encoder shown in FIG6( b ) is used for processing. At this time, the Multi-Head Attention model in the Transformer model can process the query feature vector, the key feature vector, and the value feature vector by the method described in steps S201 to S207 above, thereby obtaining the attention matrix of the image data. Next, the attention matrix can be input into the feedforward module for processing, thereby obtaining the image segmentation result (i.e., the data processing result) shown in FIG6( c ), wherein FIG6( c ) is the standard segmentation result of the image data.

[0065] Through the above Figure 6(a) to Figure 6(c) It can be seen that the method provided by the present disclosure can accurately perform image segmentation on the image to be segmented. Therefore, for the image segmentation scenario, the data processing method provided by the present disclosure can reduce the complexity of image segmentation while ensuring the accuracy of image segmentation, thereby improving the efficiency of image segmentation.

[0066] Scenario 2: Text translation scenario.

[0067] First, obtain the data to be processed, that is, the text data to be translated (for example, English to German). For example, the text data may be Nevada has already completed a pilot. After obtaining the text data, the text data may be processed to obtain a data sequence of the text data, wherein the data sequence contains feature data of each text in the text data. Next, the query feature vector, key feature vector, and value feature vector of each feature data in the data sequence may be determined. Afterwards, the determined feature vector may be input into the encoder of the Transformer model for processing. For example, it may be input into Figure 1 In the encoder shown in the figure, at this time, the Multi-Head Attention model in the Transformer model can process the query feature vector, the key feature vector, and the value feature vector by the method described in the above steps S201 to S207, thereby obtaining the attention matrix of the text data. Next, the attention matrix can be input into the feedforward module for processing, thereby obtaining the following translation result: Nevada hat bereits einen Piloten absolviert. Among them, the standard translation result of the text data is: Nevada hat bereits ein Pilotprojekt abgeschlossen.

[0068] It can be seen from the above translation results that the method provided by the present disclosure can accurately translate the text to be translated. Therefore, for the text translation scenario, the data processing method provided by the present disclosure can reduce the complexity of text translation while ensuring the accuracy of text translation, thereby improving the efficiency of text translation.

[0069] The above steps will be described in detail below in conjunction with specific embodiments.

[0070] In an optional embodiment, if Figure 4 As shown, in the case where the nonlinear conversion function is a nonlinear activation function, the above step S203: respectively converting the query feature vector and the key feature vector by the nonlinear conversion function to obtain the first vector and the second vector, specifically includes the following steps:

[0071] Step S401: transforming the query feature vector by a nonlinear activation function to obtain the first vector;

[0072] Step S402: transform the key feature vector through a nonlinear activation function to obtain the second vector.

[0073] In specific implementation, the query feature vector (vector Q) can be transformed by a nonlinear activation function Relu to obtain the first vector. For example, the above transformation process can be expressed by the following formula: Q′ i =Relu(Q i ), where Q′ i represents the first vector, Q i represents the query feature vector Q of the i-th text in the data sequence, and Relu(·) represents the nonlinear activation function.

[0074] In the disclosed embodiment, the key feature vector (vector K) may also be transformed by a nonlinear activation function Relu to obtain a second vector. For example, the above transformation process may be expressed by the following formula: K′ i =Relu(K i ), where K′ i represents the second vector, K i represents the key feature vector K of the i-th feature data in the data sequence, and Relu(·) represents the nonlinear activation function.

[0075] Here, Relu(x) = max(x, 0), so after the query feature vector and the key feature vector are transformed by the nonlinear activation function Relu, the elements in the first vector and the second vector are both greater than or equal to zero. By retaining the positive values ​​in the vector, the self-attention module can ignore features with negative correlation, thereby effectively avoiding the aggregation of irrelevant context information.

[0076] In an optional embodiment, if Figure 5 As shown, the above step S205: for the current feature data of the data sequence, the second vector and the value feature vector of each feature data are calculated and then multiplied with the first vector of the current feature data to obtain the target product, specifically including the following steps:

[0077] Step S501: based on the position of the feature data in the data sequence, determine the weight of the first vector of the current feature data to obtain a first weight, and determine the weight of the second vector of each feature data to obtain a second weight;

[0078] Step S502: Calculate the product of the first vector and the first weight to obtain a first result, and calculate the product of the second vector and the second weight to obtain a second result;

[0079] Step S503: After calculating the second result and the value feature vector, multiply them with the first result to obtain the target product.

[0080] Since there is a local bias in the downstream data processing task, that is, most of the context depends on the adjacent data. Therefore, in order to reduce the distance between adjacent data (e.g., adjacent texts) in the data sequence and expand the distance between distant data (e.g., texts) in the data sequence, the first vector Q′ of each feature data can be calculated based on the position of the feature data in the data sequence (or the cosine distance between the feature data in the data sequence). i Set the corresponding weight, that is, the first weight, and also the second vector K′ of other feature data j Set the corresponding weight, that is, the second weight.

[0081] After the first weight and the second weight are determined, the second vector and the value feature vector of each feature data may be calculated based on the first weight and the second weight, and then multiplied with the first vector of the current feature data to obtain a target product.

[0082] In specific implementation, the first vector Q can be i ' and the first weight are multiplied to obtain the first result. The second vector K' j The second weight is multiplied to obtain a second result.

[0083] Assume that the first weight is represented by W1 and the second weight is represented by W2. At this time, Q′ i W1 calculates the first result above, and passes (K' j W1) T The above second result is obtained by calculation. Then, the second result (K'j W1) T Multiply it with vector V to get the product (K' j W1) T V j ; Next, we can multiply the product (K' j W1) T V j With the first result Q′ i W1 is multiplied to obtain the target product.

[0084] In the above implementation, by determining the first weight and the second weight by the position of the feature data in the data sequence, more weights can be applied between adjacent feature data in the data sequence, thereby enhancing the locality between the feature data.

[0085] In an optional implementation, when there are multiple first weights and multiple second weights, the above step S502: calculating the product of the first vector and the first weight to obtain a first result, and calculating the product of the second vector and the second weight to obtain a second result, specifically includes the following steps:

[0086] Step S11: calculating the product of the first vector and each first weight to obtain a plurality of first results; calculating the product of the second vector and the second weight to obtain a plurality of second results;

[0087] Step S503: Calculating the second result and the value feature vector and then multiplying the result with the first result to obtain the first result, specifically comprising the following steps:

[0088] Step S12: After calculating each second result and the value feature vector, multiply the result with the corresponding first result to obtain a plurality of target sub-products, and sum the plurality of target sub-products to obtain the target product.

[0089] In the embodiment of the present disclosure, the product of the first vector and each first weight can be calculated to obtain multiple first results. Assuming that the first weights include W11 and W12, they can be obtained by Q′ i W11 and Q′ i W12 is calculated to obtain multiple first results. The product of the second vector and the second weight can also be calculated to obtain multiple second results. Assuming that the second weight includes W21 and W22, it can be obtained by (K' j W21) T and (K' j W22) T The above-mentioned multiple second results are obtained by calculation.

[0090] Afterwards, each second result and value feature vector may be calculated and then multiplied with the corresponding first result to obtain a plurality of target sub-products, and the plurality of target sub-products may be summed to obtain the target product.

[0091] For example, the second result (K' j W21) T After multiplying the vector V, the corresponding first result Q' i W11 is multiplied to obtain a target sub-product. And the second result (K' j W22) T After multiplying the vector V, the corresponding first result Q' i W12 is multiplied to obtain another target sub-product. At this time, multiple target sub-products can be summed to obtain the target product.

[0092] In the above implementation, by determining the first weight and the second weight by the position of the feature data in the data sequence, more weights can be applied between adjacent feature data in the data sequence, thereby strengthening the locality between the feature data. By calculating each second result and the value feature vector and then multiplying them with the corresponding first result, the processing process of the self-attention model can be linearized, thereby reducing the space and time complexity of the attention matrix, so that the total space and time complexity of the attention matrix will not increase quadratically with the length of the input data sequence, thereby improving the scalability of the softmax attention mechanism.

[0093] In an optional implementation, the above step S501: determining the weight of the first vector of the current feature data based on the position of the feature data in the data sequence to obtain the first weight, and determining the weight of the second vector of each feature data to obtain the second weight, specifically includes the following steps:

[0094] Step S21: determining the cosine distance between each feature data and the current feature data based on the position of each feature data in the data sequence;

[0095] Step S22: determining the first weight and the second weight according to the cosine distance.

[0096] In specific implementation, in the embodiment of the present disclosure, for the i-th feature data in the data sequence (i.e., the current feature data), and each feature data in the data sequence (i.e., the j-th feature data), it can be determined that the position of the i-th feature data in the data sequence is i, and the position of each feature data in the data sequence is j. At this time, the cosine distance between the i-th feature data and the j-th feature data can be determined based on i and j. For example, the cosine distance can be:

[0097] After the cosine distance is determined, the first weight and the second weight can be determined based on the cosine distance. The specific determination process is described as follows:

[0098] The sum and difference product expansion formula of trigonometric functions can be used to After expansion processing, the following expansion processing results are obtained:

[0099] At this point, you can and Determine as the first weight, and Determined as the second weight. Here, is the first cosine weight, is the first sine weight, is the second cosine weight, is the second sine weight.

[0100] In the embodiment of the present disclosure, the first cosine weight and the first sine weight are respectively the cosine value and the sine value determined based on the position of each feature data in the data sequence, and the second cosine weight and the second sine weight are respectively the cosine value and the sine value determined based on the position of other feature data in the data sequence.

[0101] For example, assuming that the first weight includes a first cosine weight and the first sine weight The second weight contains the second cosine weight and the second sine weight At this time, the calculation process of the above step S11 and step S12 can be described as the following process:

[0102] Calculate the first vector Q′ i With the first cosine weight and the first sine weight The product of and calculate the second vector K' j With the second cosine weight and the second sine weight The second vector K' j With the second cosine weight The product of and vector V j After multiplication, add it to the first vector Q′ i With the first cosine weight Multiply them together to get a target sub-product. j With the second sine weight The product of and vector V j After multiplication, add it to the first vector Q′ iWith the first sine weight Another target sub-product is obtained, and then multiple target sub-products are summed to obtain the target product, which can be specifically expressed by the following formula:

[0103]

[0104] In an optional implementation, when the number of target products is multiple, step S207: determining the attention matrix of the data to be processed according to the target products specifically includes the following steps:

[0105] Step S2071: performing a sum operation on the multiple target products to obtain a first operation result.

[0106] By formula It can be seen that for the i-th feature data and each j-th feature data in the data sequence, a target product can be obtained by the above-described method. If there are multiple j-th feature data, multiple target products can be obtained. At this time, the multiple target products can be summed to obtain the first operation result.

[0107] Step S2072: for the current feature data in the data sequence, perform a sum operation on the calculation results of the first vector of the current feature data and the second vector of each feature data to obtain a second operation result.

[0108] Assume that the first vector is Q′ i , the second vector is K' j At this time, the first vector Q′ of the current feature data can be i And the second vector K' of each feature data j The calculation results are summed up to obtain the second calculation result.

[0109] It can be seen from the above description that the corresponding first weight can be determined for the first vector of the current feature data based on the position of the feature data in the data sequence, and the corresponding second weight can be determined for the second vector of each feature data in the data sequence. In this case, step S2072 can be described as:

[0110] (1) Determine the weight of the first vector of the current feature data to obtain a first weight, and determine the weight of the second vector of each feature data to obtain a second weight.

[0111] (2) Calculate the product of the first vector and the first weight to obtain a first result, and calculate the product of the second vector and the second weight to obtain a second result.

[0112] (3) Performing a sum operation on the product of the first result and the second result to obtain the second operation result.

[0113] From the above description, it can be seen that the first weight includes: the first weight includes: a first cosine weight and a first sine weight, and the second weight includes: a second cosine weight and a second sine weight; wherein the first cosine weight and the first sine weight are respectively the cosine value and the sine value determined based on the position of the current feature data in the data sequence, and the second cosine weight and the second sine weight are respectively the cosine value and the sine value determined based on the position of each feature data in the data sequence.

[0114] At this point, the steps described in (2) and (3) above can be described as the following process:

[0115] The products of the first vector, the first cosine weight and the first sine weight are calculated respectively to obtain two first results (the first sub-result and the second sub-result respectively); and the products of the second vector, the second cosine weight and the second sine weight are calculated respectively to obtain multiple second results (the third sub-result and the fourth sub-result respectively); the products of the first sub-result and the third sub-result are summed to obtain a first sub-operation result, and the products of the second sub-result and the fourth sub-result are summed to obtain a second sub-operation result; the first sub-operation result and the second sub-operation result are added to obtain a second operation result.

[0116] Assume that is the first cosine weight, is the first sine weight, is the second cosine weight, is the second sine weight, the above process can be expressed by the following formula:

[0117] here, is the product of the first vector and the first cosine weight (i.e., the first subresult), is the product of the first vector and the first sine weight (i.e., the second subresult), is the product of the second vector and the second cosine weight (i.e., the third subresult), is the product of the second vector and the second sine weight (ie, the fourth subresult).

[0118] Step S2073: Determine the attention matrix of the data to be processed according to the ratio of the first operation result to the second operation result.

[0119] After the first operation result and the second operation result are calculated, the attention matrix of the data to be processed can be determined according to the ratio of the first operation result to the second operation result, which can be specifically expressed as the following formula:

[0120]

[0121] Here, assuming that Then the above formula can be expressed as the following formula:

[0122]

[0123] In the above implementation, this processing method can reduce the space and time complexity of the attention matrix, so that the total space and time complexity of the attention matrix will not increase quadratically with the length of the input data sequence, thereby improving the scalability of the softmax attention mechanism. After replacing the self-attention module in the Transformer model in the above-described manner, the Transformer model can be more stable, and at the same time, the speed of the model can be improved and the memory can be reduced while ensuring the effect.

[0124] In data processing tasks, new data may have dependencies with data from a long time ago, and the speed of data generation is crucial to user experience. Using the method provided by the embodiment of the present disclosure, the long dependencies between data can be captured, and the data generation speed is much faster than the original method.

[0125] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0126] Based on the same inventive concept, a data processing device corresponding to the data processing method is also provided in the embodiment of the present disclosure. Since the principle of solving the problem by the device in the embodiment of the present disclosure is similar to the above-mentioned data processing method in the embodiment of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0127] Reference Figure 7 FIG. 1 is a schematic diagram of a data processing device provided by an embodiment of the present disclosure, wherein the device comprises: an acquisition unit 10, a conversion unit 20, a calculation unit 30, and a determination unit 40; wherein:

[0128] An acquisition unit 10 is used to acquire a data sequence of data to be processed to be input into the self-attention module, and determine a query feature vector, a key feature vector, and a value feature vector of each feature data in the data sequence; the data to be processed includes: image data and / or text data;

[0129] A conversion unit 20, configured to convert the query feature vector and the key feature vector respectively through a nonlinear conversion function to obtain a first vector and a second vector; elements in the first vector and the second vector are both greater than or equal to zero;

[0130] A calculation unit 30, for calculating the second vector and the value feature vector of each feature data for the current feature data of the data sequence, and then multiplying the result with the first vector of the current feature data to obtain a target product;

[0131] The determination unit 40 is used to determine the attention matrix of the data to be processed according to the target product, and input the attention matrix to the feedforward module for processing to obtain a data processing result.

[0132] In the disclosed embodiment, first, the query feature vector and the key feature vector are respectively transformed by a nonlinear transformation function; then, for the current feature data of the data sequence, the transformation vector (i.e., the second vector) of the key feature vector of each feature data and the value feature vector of the feature data can be calculated, and the calculation result can be multiplied with the transformation vector (i.e., the first vector) of the query feature vector of the current feature data to determine the attention matrix of each feature data in the data sequence according to the target product. Through the above processing method, the processing process of the self-attention model can be linearized, thereby reducing the space and time complexity of the attention matrix, so that the total space and time complexity of the attention matrix will not increase quadratically with the length of the input data sequence, thereby improving the scalability of the softmax attention mechanism.

[0133] In a possible implementation, the conversion unit is further used for: when the nonlinear conversion function is a nonlinear activation function, converting the query feature vector by a nonlinear activation function to obtain the first vector; and converting the key feature vector by the nonlinear activation function to obtain the second vector.

[0134] In a possible implementation, the calculation unit is further used to: determine the weight of the first vector of the current feature data based on the position of the feature data in the data sequence to obtain a first weight, and determine the weight of the second vector of each feature data to obtain a second weight; calculate the product of the first vector and the first weight to obtain a first result, and calculate the product of the second vector and the second weight to obtain a second result; after calculating the second result and the value feature vector, multiply them with the first result to obtain the target product.

[0135] In a possible implementation, the calculation unit is further used to: determine the cosine distance between each feature data and the current feature data based on the position of each feature data in the data sequence; and determine the first weight and the second weight according to the cosine distance.

[0136] In one possible implementation, the calculation unit is further used to: when there are multiple first weights and multiple second weights, calculate the product of the first vector and each first weight to obtain multiple first results; calculate the product of the second vector and the second weight to obtain multiple second results; and calculate the second result and the value feature vector and then multiply them with the first result to obtain the first result, including: calculating each second result and the value feature vector and then multiplying them with the corresponding first result to obtain multiple target sub-products, and summing the multiple target sub-products to obtain the target product.

[0137] In a possible implementation, the determination unit is further used to: when there are multiple target products, sum the multiple target products to obtain a first operation result; for the current feature data in the data sequence, sum the calculation results of the first vector of the current feature data and the second vector of each feature data to obtain a second operation result; and determine the attention matrix of the data to be processed according to the ratio of the first operation result to the second operation result.

[0138] In a possible implementation, the determination unit is also used to: determine the weight of the first vector of the current feature data to obtain a first weight, and determine the weight of the second vector of each feature data to obtain a second weight; calculate the product of the first vector and the first weight to obtain a first result, and calculate the product of the second vector and the second weight to obtain a second result; and sum the product of the first result and the second result to obtain the second calculation result.

[0139] In one possible implementation, the first weight includes: a first cosine weight and a first sine weight, and the second weight includes: a second cosine weight and a second sine weight; wherein the first cosine weight and the first sine weight are respectively the cosine value and the sine value determined based on the position of the current feature data in the data sequence, and the second cosine weight and the second sine weight are respectively the cosine value and the sine value determined based on the position of each feature data in the data sequence.

[0140] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference may be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0141] Corresponds to Figure 1 The data processing method in the present disclosure also provides an electronic device 800, such as Figure 8 FIG. 8 is a schematic diagram of the structure of an electronic device 800 provided in an embodiment of the present disclosure, including:

[0142] Processor 81, memory 82, and bus 83; memory 82 is used to store execution instructions, including internal memory 821 and external memory 822; the internal memory 821 is also called internal memory, which is used to temporarily store the operation data in the processor 81 and the data exchanged with the external memory 822 such as a hard disk. The processor 81 exchanges data with the external memory 822 through the internal memory 821. When the electronic device 800 is running, the processor 81 communicates with the memory 82 through the bus 83, so that the processor 81 executes the following instructions:

[0143] Obtain a data sequence of to-be-processed data to be input into the self-attention module, and determine a query feature vector, a key feature vector, and a value feature vector of each feature data in the data sequence; the to-be-processed data includes: image data and / or text data;

[0144] The query feature vector and the key feature vector are respectively transformed by a nonlinear transformation function to obtain a first vector and a second vector; elements in the first vector and the second vector are both greater than or equal to zero;

[0145] For the current feature data of the data sequence, the second vector and the value feature vector of each feature data are calculated and then multiplied with the first vector of the current feature data to obtain a target product;

[0146] The attention matrix of the data to be processed is determined according to the target product, and the attention matrix is ​​input into the feedforward module for processing to obtain a data processing result.

[0147] The present disclosure also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the data processing method described in the above method embodiment are executed. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0148] The embodiments of the present disclosure also provide a computer program product, which carries a program code. The instructions included in the program code can be used to execute the steps of the data processing method described in the above method embodiment. For details, please refer to the above method embodiment, which will not be repeated here.

[0149] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is implemented as a computer storage medium. In another optional embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0150] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the device or unit can be electrical, mechanical or other forms.

[0151] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0152] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0153] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling an electronic device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0154] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The protection scope of the present disclosure is not limited thereto. Although the present disclosure is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed in the present disclosure, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A data processing method, characterized in that: Applied to the Transformer model containing the self-attention module and the feed-forward module, including: Obtain a data sequence of to-be-processed data to be input into the self-attention module, and determine a query feature vector, a key feature vector, and a value feature vector of each feature data in the data sequence; the to-be-processed data includes: image data and / or text data; The query feature vector and the key feature vector are respectively transformed by a nonlinear transformation function to obtain a first vector and a second vector; elements in the first vector and the second vector are both greater than or equal to zero; For the current feature data of the data sequence, based on the position of each feature data in the data sequence, determine the cosine distance between each feature data and the current feature data; Determine a first weight of a first vector of the current feature data and a second weight of a second vector of the current feature data according to the cosine distance; Calculate the product of the first vector and the first weight to obtain a first result, and calculate the product of the second vector and the second weight to obtain a second result; After calculating the second result and the value feature vector, multiply them with the first result to obtain a target product; The attention matrix of the data to be processed is determined according to the target product, and the attention matrix is ​​input into the feedforward module for processing to obtain a data processing result.

2. The method according to claim 1, characterized in that The nonlinear conversion function is a nonlinear activation function; the query feature vector and the key feature vector are converted by the nonlinear conversion function to obtain a first vector and a second vector, including: The query feature vector is transformed by the non-linear activation function to obtain the first vector, and the key feature vector is transformed by the non-linear activation function to obtain the second vector.

3. The method according to claim 1, characterized in that The number of the first weight and the second weight is multiple; The calculating the product of the first vector and the first weight to obtain a first result, and calculating the product of the second vector and the second weight to obtain a second result, includes: Calculate the product of the first vector and each first weight to obtain a plurality of first results; calculate the product of the second vector and the second weight to obtain a plurality of second results; The step of calculating the second result and the value feature vector and then multiplying the calculated result with the first result includes: calculating each second result and the value feature vector and then multiplying the calculated result with the corresponding first result to obtain a plurality of target sub-products, and summing the plurality of target sub-products to obtain the target product.

4. The method according to any one of claims 1 to 3, characterized in that The number of the target products is multiple; and determining the attention matrix of the data to be processed according to the target products includes: Performing a sum operation on a plurality of the target products to obtain a first operation result; For the current feature data in the data sequence, performing a sum operation on the calculation results of the first vector of the current feature data and the second vector of each feature data to obtain a second operation result; Determine the attention matrix of the data to be processed according to the ratio of the first operation result to the second operation result.

5. The method according to claim 4, characterized in that The step of performing a sum operation on the calculation results of the first vector of the current feature data and the second vector of each feature data for the current feature data in the data sequence to obtain a second operation result includes: Determine the weight of the first vector of the current feature data to obtain a first weight, and determine the weight of the second vector of each feature data to obtain a second weight; Calculate the product of the first vector and the first weight to obtain a first result, and calculate the product of the second vector and the second weight to obtain a second result; A sum operation is performed on the product of the first result and the second result to obtain the second operation result.

6. The method according to claim 5, characterized in that The first weight includes: a first cosine weight and a first sine weight, and the second weight includes: a second cosine weight and a second sine weight; wherein the first cosine weight and the first sine weight are respectively the cosine value and the sine value determined based on the position of the current feature data in the data sequence, and the second cosine weight and the second sine weight are respectively the cosine value and the sine value determined based on the position of each feature data in the data sequence.

7. A data processing device, characterized in that: Applied to the Transformer model containing the self-attention module and the feed-forward module, including: An acquisition unit is used to acquire a data sequence of data to be processed to be input into the self-attention module, and determine a query feature vector, a key feature vector, and a value feature vector of each feature data in the data sequence; the data to be processed includes: image data and / or text data; A conversion unit, used to convert the query feature vector and the key feature vector respectively through a nonlinear conversion function to obtain a first vector and a second vector; elements in the first vector and the second vector are both greater than or equal to zero; A calculation unit is used to determine, for the current feature data of the data sequence, a cosine distance between each feature data and the current feature data based on the position of each feature data in the data sequence; determine a first weight of a first vector of the current feature data and a second weight of a second vector of the current feature data according to the cosine distance; calculate the product of the first vector and the first weight to obtain a first result, and calculate the product of the second vector and the second weight to obtain a second result; after calculating the second result and the value feature vector, multiply the second result with the first result to obtain a target product; A determination unit is used to determine the attention matrix of the data to be processed according to the target product, and input the attention matrix into the feedforward module for processing to obtain a data processing result.

8. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the data processing method as described in any one of claims 1 to 6 are performed.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 6 are executed.

Images