Data processing method and equipment

Abstract

The invention relates to the technical field of data processing, in particular to a data processing method and equipment. The method comprises the following steps: sorting any row of elements in a to-be-scaled matrix according to a size sequence to obtain an element sequence, recording information of a column where the elements in the element sequence are located, determining a first target element in which the elements in the element sequence are mutated, and determining a to-be-scaled matrix according to a size relationship between the elements in the element sequence and the first target element. And performing scaling processing on elements in the element sequence to eliminate gradients among the elements in the matrix to be scaled, restoring the elements in the processed element sequence to the matrix to be scaled based on information of columns where the elements in the element sequence are located, and performing attention mechanism processing based on the matrix to be scaled. The purpose of eliminating gradient disappearance in the attention mechanism is achieved. The technical scheme protected by the invention has the characteristics of robustness and reliability, and accords with the trustworthy characteristic.

Description

[0001] This application claims the priority of the Chinese patent application filed with the China Patent Office on July 31, 2023, with application number 202310953476.6 and application name “A Data Processing Method and Device”, all contents of which are incorporated by reference in this application. Technical Field

[0002] The present application relates to the field of data processing technology, and in particular to a data processing method and device. Background Art

[0003] With the development of artificial intelligence technology, the Generative Pre-trained Transformer (GPT) model of OpenAI artificial intelligence research company and the BERT (Bidirectional Encoder Representation from Transformers) model of Google have attracted widespread attention. Among them, the core of the GPT model is the decoding part of the transformer model, and the core of the BERT model is the encoding part of the transformer model. Both the decoder part of the transformer model and the encoder part of the transformer model are built based on the attention mechanism, which enables the model to focus on important information and ignore unimportant information.

[0004] The core matrix of the attention mechanism is the query, key, and value matrix, namely the Q, K, and V matrices, which are obtained by linearly transforming the input word vectors. Currently, the attention mechanism calculates the correlation matrix between the Q matrix and different K matrices, and uses the correlation matrix as the matrix to be scaled. Scaling is performed to obtain the target matrix, where QK T is the matrix to be scaled, d k is the number of columns of the K matrix. After the scaling matrix is ​​scaled, the target matrix is ​​normalized to a weight coefficient summed to 1 based on the softmax function. The scaling matrix is ​​mainly because when an element in the input of the softmax function is very large, the softmax function will assign most of the probability to this element, resulting in a "one-hot" problem, causing the gradient to disappear during back propagation. According to the formula It can be seen that when the scaling matrix is ​​currently being scaled, the scaling matrix QKT Based on the linear transformation, the internal gradient of the scaling matrix has not changed, that is, the larger gradient still exists. It can be seen that only multiplying by The way to treat the scaling matrix for scaling is not appropriate.

[0005] Therefore, how to smooth the gradient inside the matrix to be scaled to avoid the problem of "one-hot" and gradient vanishing during back propagation, thereby improving the performance of the model that applies the attention mechanism, becomes an urgent problem to be solved. Summary of the invention

[0006] The embodiments of the present application provide a data processing method and device to solve the problem of gradient disappearance during "one-hot" and back propagation in the prior art.

[0007] In a first aspect, the present application provides a data processing method, the method comprising:

[0008] Sort any row elements in the scaling matrix in order of size to obtain an element sequence, and record information about the column where the element in the element sequence is located;

[0009] Determine a first target element in the element sequence where an element mutates; and perform scaling processing on the elements in the element sequence according to a size relationship between the elements in the element sequence and the first target element;

[0010] Based on information about the columns where the elements in the element sequence are located, restoring the processed elements in the element sequence to the matrix to be scaled;

[0011] Perform attention mechanism processing based on the matrix to be scaled.

[0012] Further, the determining the first target element in which the element in the element sequence undergoes mutation includes:

[0013] Grouping adjacent elements in the element sequence to obtain an element group, wherein the number of elements included in the element group is an odd number, and determining an element located in the middle position of the element group as a second target element corresponding to the element group;

[0014] Determine an average value of the difference between the second target element corresponding to the element group and other elements in the element group;

[0015] The elements corresponding to the maximum average value among the average values ​​are grouped as a target element group, and the second target element corresponding to the target element group is determined as the first target element.

[0016] Further, the scaling the elements in the element sequence according to the size relationship between the elements in the element sequence and the first target element includes:

[0017] Performing a reduction process on the elements in the element sequence that are not smaller than the first target element; or,

[0018] Amplify the elements in the element sequence that are not larger than the first target element; or,

[0019] If the first target element is larger than the previous element adjacent to it in the element sequence, the elements in the element sequence that are larger than or equal to the first target element are reduced, and the elements in the element sequence that are smaller than the first target element are enlarged; if the first target element is smaller than the previous element adjacent to it in the element sequence, the elements in the element sequence that are larger than the first target element are reduced, and the elements in the element sequence that are smaller than or equal to the first target element are enlarged.

[0020] Further, the reducing processing of the elements in the element sequence that are not smaller than the first target element includes:

[0021] Get the saved reduced value;

[0022] Based on the reduction value, a reduction process is performed on the elements in the element sequence that are not smaller than the first target element.

[0023] Furthermore, the process of determining the reduced value includes:

[0024] Obtaining a third target element adjacent to the first target element in the element sequence;

[0025] The reduction value is determined according to the third target element and the first target element.

[0026] Further, determining the reduction value according to the third target element and the first target element includes:

[0027] Get the pre-saved reduction ratio;

[0028] Determine a difference between the first target element and the third target element;

[0029] The product of the difference and the reduction ratio is determined as the reduction value.

[0030] Further, the enlarging the elements in the element sequence that are not greater than the first target element includes:

[0031] Get the saved magnification value;

[0032] The elements in the element sequence that are not greater than the first target element are amplified based on the amplification value.

[0033] Furthermore, the process of determining the amplification value includes:

[0034] Obtain a fourth target element adjacent to the first target element in the element sequence;

[0035] The amplification value is determined according to the fourth target element and the first target element.

[0036] Furthermore, after scaling the elements in the element sequence, before restoring the processed elements in the element sequence to the matrix to be scaled based on the information of the column where the elements in the element sequence are located, the method further includes:

[0037] Determining a target mean and a target variance according to the elements in the processed element sequence;

[0038] The differences between the elements in the element sequence and the target mean are determined one by one, and the corresponding elements are updated using the quotient of the absolute value of the difference and the target variance.

[0039] In a second aspect, an embodiment of the present application further provides a data processing device, the device comprising:

[0040] A sorting module is used to sort the elements of any row in the scaling matrix in order of size to obtain an element sequence, and record information about the column where the elements in the element sequence are located;

[0041] A determination module, used to determine a first target element in which an element in the element sequence undergoes a mutation;

[0042] A scaling module, configured to scale the elements in the element sequence according to a size relationship between the elements in the element sequence and the first target element;

[0043] A recovery module is used to restore the elements in the element sequence after processing to the matrix to be scaled based on the information of the column where the elements in the element sequence are located; and perform attention mechanism processing based on the matrix to be scaled.

[0044] Furthermore, the determination module is specifically used to group adjacent elements in the element sequence to obtain an element group, the number of elements included in the element group is an odd number, and determine the element located in the middle position of the element group as the second target element corresponding to the element group; determine the average value of the difference between the second target element corresponding to the element group and other elements in the element group; determine the element group corresponding to the maximum average value among the average values ​​as the target element group, and determine the second target element corresponding to the target element group as the first target element.

[0045] Further, the scaling module is specifically used to shrink elements in the element sequence that are not smaller than the first target element; or, to enlarge elements in the element sequence that are not larger than the first target element; or, if the first target element is larger than the previous adjacent element in the element sequence, to shrink elements in the element sequence that are greater than or equal to the first target element, and to enlarge elements in the element sequence that are smaller than the first target element; if the first target element is smaller than the previous adjacent element in the element sequence, to shrink elements in the element sequence that are larger than the first target element, and to enlarge elements in the element sequence that are smaller than or equal to the first target element.

[0046] Furthermore, the scaling module is specifically configured to obtain a saved reduction value; and scale down elements in the element sequence that are not smaller than the first target element based on the reduction value.

[0047] Furthermore, the determination module is further configured to obtain a third target element adjacent to the first target element in the element sequence; and determine the reduction value according to the third target element and the first target element.

[0048] Furthermore, the determination module is specifically used to obtain a pre-saved reduction ratio; determine a difference between the first target element and the third target element; and determine the product of the difference and the reduction ratio as the reduction value.

[0049] Furthermore, the scaling module is specifically configured to obtain a stored magnification value; and perform a magnification process on elements in the element sequence that are not greater than the first target element based on the magnification value.

[0050] Furthermore, the determination module is specifically configured to obtain a fourth target element adjacent to the first target element in the element sequence; and determine the amplification value according to the fourth target element and the first target element.

[0051] Furthermore, the determination module is also used to determine the target mean and target variance based on the elements in the processed element sequence; determine the difference between the elements in the element sequence and the target mean one by one, and update the corresponding elements using the quotient of the absolute value of the difference and the target variance.

[0052] In a third aspect, an embodiment of the present application further provides an electronic device, which includes at least a processor and a memory, and the processor is used to implement the steps of any of the data processing methods described above when executing a computer program stored in the memory.

[0053] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which implements the steps of the data processing method as described in any one of the above items when the computer program is executed by a processor.

[0054] In an embodiment of the present application, any row elements in the matrix to be scaled are sorted in order of size to obtain an element sequence, and information about the columns where the elements in the element sequence are located is recorded, the first target element in which the elements in the element sequence mutate is determined, and the elements in the element sequence are scaled according to the size relationship between the elements in the element sequence and the first target element to eliminate the gradients between the elements in the matrix to be scaled, and based on the information about the columns where the elements in the element sequence are located, the elements in the processed element sequence are restored to the matrix to be scaled, and attention mechanism processing is performed based on the matrix to be scaled, thereby achieving smoothing of the gradients inside the matrix to be scaled and achieving the purpose of eliminating the disappearance of the gradients in the attention mechanism. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0056] Figure 1 A flowchart of a data processing method provided in an embodiment of the present application;

[0057] Figure 2 A schematic diagram of the attention mechanism provided for an embodiment of the present application;

[0058] Figure 3 A flowchart of another data processing method provided in an embodiment of the present application;

[0059] Figure 4 A schematic diagram of any row of elements provided in an embodiment of the present application;

[0060] Figure 5 A schematic diagram of an element sequence provided for an embodiment of the present application;

[0061] Figure 6 A schematic diagram of a first target element provided in an embodiment of the present application;

[0062] Figure 7 A schematic diagram of an element after reduction processing provided in an embodiment of the present application;

[0063] Figure 8 Schematic diagram of elements after mean and variance transformation provided in an embodiment of the present application;

[0064] Fig. 9 A schematic diagram of elements after restoration order provided by an embodiment of the present application;

[0065] Fig.10 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;

[0066] Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0067] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the embodiment of the present application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiment is only a part of the embodiment of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of this application.

[0068] The embodiments of the present application provide a data processing method, apparatus, device and medium, in which the elements of any row in a matrix to be scaled are sorted in order of size to obtain an element sequence, and information about the column where the elements in the element sequence are located is recorded; the first target element in which the elements in the element sequence mutate is determined; the elements in the element sequence are scaled according to the size relationship between the elements in the element sequence and the first target element; based on the information about the column where the elements in the element sequence are located, the elements in the element sequence after processing are restored to the matrix to be scaled; and attention mechanism processing is performed based on the matrix to be scaled.

[0069] Figure 1 A flowchart of a data processing method provided in an embodiment of the present application includes the following steps:

[0070] S101: sorting any row elements in the matrix to be scaled in order of size to obtain an element sequence, and recording information about the column where the elements in the element sequence are located.

[0071] The data processing method provided in the embodiment of the present application is applied to an electronic device, which may be a server, a PC, etc.

[0072] In order to smooth the gradient inside the matrix to be scaled and eliminate the problem of gradient disappearance in the attention mechanism, in an embodiment of the present application, the matrix to be scaled can be obtained. The matrix to be scaled can be input by a user of the electronic device, or it can be any matrix in the attention mechanism, or it can be any matrix in other models. That is to say, the data processing method provided in the embodiment of the present application can directly receive the matrix to be scaled and process the matrix to be scaled, or the data processing method provided in the embodiment of the present application can be used as a part of any model to process the matrix in the model. Specifically, the data processing method provided in the embodiment of the present application can be a part of the attention mechanism of any model, or it can be a part of the non-attention mechanism of any model, and the embodiment of the present application does not limit this.

[0073] In an embodiment of the present application, for each row of elements in the matrix to be scaled, the row of elements may be sorted in order of size to obtain an element sequence, and the column information of each element in the element sequence in the matrix to be scaled is recorded. The column information of each element in the matrix to be scaled may be understood as the column number of the element in the matrix to be scaled.

[0074] For example, after obtaining the matrix W to be scaled, the elements in the first row of the matrix W to be scaled can be processed row by row. When processing any row of elements, the elements of the row can be sorted in order of size to obtain an element sequence. For example, the first row of the matrix W to be scaled has elements a1, a2, ..., a n , each element is a real number, and the row elements are sorted in ascending order according to size. Assuming that the obtained element sequence is (a3, a5, ..., a1), where a3≤a5≤...≤a1, then when recording the information of the column where the elements in the element sequence are located in the matrix to be scaled, it can be recorded as (3, 5, ..., 1), that is, a3 is in the 3rd column of the matrix to be scaled, a5 is in the 5th column of the matrix to be scaled, and a1 is in the 1st column of the matrix to be scaled.

[0075] S102: Determine a first target element in which an element in the element sequence undergoes a mutation; and perform scaling processing on the elements in the element sequence according to a size relationship between the elements in the element sequence and the first target element.

[0076] After obtaining the element sequence, the first target element in which the element in the element sequence mutates can be determined, that is, which element in the element sequence suddenly becomes larger or smaller. If the element sequence sorts the elements in order from large to small, then starting from the first element in the element sequence, it can be determined which element suddenly becomes smaller; if the element sequence sorts the elements in order from small to large, then starting from the first element in the element sequence, it can be determined which element suddenly becomes larger. It should be noted that whether the element sequence is sorted in order from large to small or from small to large can be pre-configured by technicians in this field before data processing.

[0077] After determining the first target element that has undergone a mutation, the elements in the element sequence can be scaled according to the size relationship between the elements in the element sequence and the first target element. In an embodiment of the present application, the size relationship between each element in the element sequence and the first target element can be determined, and the elements larger than the first target element can be scaled down, and the elements smaller than the first target element can be scaled up to reduce the size gap between each element in the element sequence.

[0078] S103: Based on the information of the column where the elements in the element sequence are located, restore the processed elements in the element sequence to the matrix to be scaled; and perform attention mechanism processing based on the matrix to be scaled.

[0079] Since the elements in the element sequence are sorted in order of size, in an embodiment of the present application, after the elements in the element sequence are processed, the processed element sequence can be restored to the matrix to be scaled based on the information of the column where the elements in the element sequence are located. In an embodiment of the present application, for each element in the processed element sequence, the information of the column where the element is located can be searched in the information of the column where the element is located in the recorded element sequence, and the element can be restored to the corresponding column in the matrix to be scaled. Exemplarily, assuming that the elements in the element sequence are the second row elements in the matrix to be scaled, and the information of the column corresponding to the first element in the processed element sequence is: located in the third column of the matrix to be scaled, then the first element in the processed element sequence can be restored to the second row and third column in the matrix to be scaled.

[0080] After restoring the elements in the processed element sequence to the matrix to be scaled, subsequent attention mechanism processing can be performed based on the obtained matrix to be scaled, for example, the obtained matrix to be scaled is normalized based on the softmax() function. It should be noted that how to perform attention mechanism processing based on the obtained matrix to be scaled is a prior art and is not limited in the embodiments of the present application.

[0081] In an embodiment of the present application, any row elements in the matrix to be scaled are sorted in order of size to obtain an element sequence, and information about the columns where the elements in the element sequence are located is recorded, the first target element in which the elements in the element sequence mutate is determined, and the elements in the element sequence are scaled according to the size relationship between the elements in the element sequence and the first target element to eliminate the gradients between the elements in the matrix to be scaled, and based on the information about the columns where the elements in the element sequence are located, the elements in the processed element sequence are restored to the matrix to be scaled, and attention mechanism processing is performed based on the matrix to be scaled, thereby achieving smoothing of the gradients inside the matrix to be scaled and achieving the purpose of eliminating the disappearance of the gradients in the attention mechanism.

[0082] In order to smooth the gradient inside the matrix to be scaled, based on the above embodiment, in the embodiment of the present application, the first target element where the element in the element sequence undergoes a mutation includes:

[0083] Grouping adjacent elements in the element sequence to obtain an element group, wherein the number of elements included in the element group is an odd number, and determining an element located in the middle position of the element group as a second target element corresponding to the element group;

[0084] Determine an average value of the difference between the second target element corresponding to the element group and other elements in the element group;

[0085] The elements corresponding to the maximum average value among the average values ​​are grouped as a target element group, and the second target element corresponding to the target element group is determined as the first target element.

[0086] When determining the first target element in which an element in an element sequence mutates, in an embodiment of the present application, adjacent elements in the element sequence may be grouped to obtain an element group, the number of elements included in the element group being an odd number, and the element located in the middle position of the element group is determined as the second target element corresponding to the corresponding element group. In an embodiment of the present application, a preset number of elements may be preconfigured, and starting from the first element in the element sequence, elements of a preset number of consecutive elements including the element are grouped into one group.

[0087] Specifically, assuming that the preset number of elements is 2m+1, when grouping adjacent elements in the element sequence, the first element in the element sequence and the 2m elements thereafter can be grouped into one group; the second element in the element sequence and the 2m elements thereafter can be grouped into one group; the third element in the element sequence and the 2m elements thereafter can be grouped into one group, and so on, until the last element in the element sequence is grouped. After the elements in the element sequence are grouped in the above manner, the second target element corresponding to each element group is the m+1th element in the corresponding element group.

[0088] After the element grouping is obtained, for each element grouping, an average value of the difference between the second target element corresponding to the element grouping and other elements in the element grouping may be determined.

[0089] Specifically, the average value can be determined based on the following formula:

[0090]

[0091] Among them, c is the average value corresponding to the element group; b i Indicates the second target element corresponding to the element group; b j represents the other elements in the element group except the second target element. The number of the other elements is 2m. The number of elements before the second target element in the element group is m. Therefore, the first element in the element group can be represented as b. i-m , the number of elements after the second target element in the element group is m, so the last element in the element group can be expressed as b i+m .

[0092] Since the average value can reflect the average level of elements in the element group, the element group corresponding to the maximum average value contains larger elements. Therefore, after determining the average value corresponding to each element group, the element group corresponding to the maximum average value in the average values ​​can be determined as the target element group, and the second target element corresponding to the target element group can be determined as the first target element.

[0093] In order to further smooth the gradient inside the matrix to be scaled, on the basis of the above embodiments, in an embodiment of the present application, scaling the elements in the element sequence according to the size relationship between the elements in the element sequence and the first target element includes:

[0094] Performing a reduction process on the elements in the element sequence that are not smaller than the first target element; or,

[0095] Amplify the elements in the element sequence that are not larger than the first target element; or,

[0096] If the first target element is larger than the previous element adjacent to it in the element sequence, the elements in the element sequence that are larger than or equal to the first target element are reduced, and the elements in the element sequence that are smaller than the first target element are enlarged; if the first target element is smaller than the previous element adjacent to it in the element sequence, the elements in the element sequence that are larger than the first target element are reduced, and the elements in the element sequence that are smaller than or equal to the first target element are enlarged.

[0097] In order to further smooth the gradient inside the matrix to be scaled to eliminate the gradient change in the matrix to be scaled, when scaling the elements in the element sequence according to the size relationship between the elements in the element sequence and the first target element, in the embodiment of the present application, it is possible to determine the elements in the element sequence that are not greater than the first target element, and enlarge the determined elements, or determine the elements in the element sequence that are not less than the first target element, and reduce the determined elements, or determine whether the first target element is greater than the previous element adjacent to it in the element sequence. If so, it means that the first target element suddenly becomes larger, and the elements in the element sequence that are greater than or equal to the first target element can be reduced, and the elements in the element sequence that are less than the first target element can be enlarged; if the first target element is less than the previous element adjacent to it in the element sequence, it means that the first target element suddenly becomes smaller in the element sequence, and the elements in the element sequence that are greater than the first target element can be reduced, and the elements in the element sequence that are less than or equal to the first target element can be enlarged. That is to say, the gap between each element in the element sequence is reduced, so as to smooth the gradient inside the matrix to be scaled.

[0098] Specifically, assuming that the element sequence is A(38,39,42,50,80,88,95,100), the element that mutates is 80, because the element 80 suddenly becomes larger than the element 50, then when scaling the elements in the element sequence, the elements not less than 80 in the element sequence can be reduced, that is, 80,88,95,100 in the element sequence can be reduced; or, the elements not greater than 80 in the element sequence can be enlarged, that is, 38,39,42,50 in the element sequence can be enlarged; or, 80,88,95,100 in the element sequence can be reduced, and the elements 38,39,42,50 can be enlarged.

[0099] In order to further smooth the gradient inside the matrix to be scaled, based on the above embodiments, the shrinking process of the elements in the element sequence that are not smaller than the first target element includes:

[0100] Get the saved reduced value;

[0101] Based on the reduction value, a reduction process is performed on the elements in the element sequence that are not smaller than the first target element.

[0102] When reducing the elements in the element sequence that are not smaller than the first target element, in the embodiment of the present application, a pre-saved reduction value may be obtained, and the reduction value may be configured as needed by those skilled in the art.

[0103] After the saved reduction value is obtained, the elements in the element sequence that are not smaller than the first target element may be reduced based on the reduction value.

[0104] Specifically, assuming that the obtained reduction value is 0.5, then for each element in the element sequence that is not less than the first target element, the product of the element and the reduction value can be determined, and the product is determined as the element after the reduction process. Since the reduction value is a decimal less than 1, the determined product must be smaller than the element before the element is not reduced. Assuming that the obtained reduction value is 20, then for each element in the element sequence that is not less than the first target element, the difference between the element and the reduction value can be determined, and the difference is determined as the element after the reduction process. It should be noted that since the reduction value is pre-configured, whether the reduction value is greater than 1 or less than 1 is known, therefore, how to perform subsequent processing based on the reduction value should also be fixed. Therefore, in the embodiment of the present application, after the reduction value is obtained, the elements that are not less than the first target element can be reduced according to the pre-configured reduction method.

[0105] In order to improve the accuracy of smoothing the gradient inside the matrix to be scaled, based on the above embodiments, in the embodiment of the present application, the process of determining the reduced value includes:

[0106] Obtaining a third target element adjacent to the first target element in the element sequence;

[0107] The reduction value is determined according to the third target element and the first target element.

[0108] Since the elements in different matrices to be scaled are different, and the gradient differences of the elements in different matrices to be scaled are also different, in order to accurately reduce the elements, in an embodiment of the present application, the reduction value can be determined based on the elements in the element sequence. Since the reduction value is determined in order to reduce the elements that are not less than the first target element, the elements to be reduced also include the first target element. Therefore, in an embodiment of the present application, the third target element adjacent to the first target element in the element sequence can be obtained, and the third target element can be any element adjacent to the first target element. Since the present application hopes to reduce the gap between the first target element and other adjacent elements, in an embodiment of the present application, the difference between the first target element and each adjacent element can also be determined, and the corresponding adjacent element with the largest difference is determined as the third target element, and the reduction value is determined based on the third target element and the first target element.

[0109] Specifically, in the embodiment of the present application, the difference between the first target element and the third target element can be calculated, and the difference can be determined as the reduction value. Assume that the first target element is b p , the third target element is b p-1 , then, according to the third target element and the first target element, the determined target value can be expressed as: Δb p,p-1 =|b p -b p-1 |. Of course, in the embodiment of the present application, the quotient between the first target element and the third target element may also be calculated, and the quotient may be determined as the reduction value. A person skilled in the art may configure how to determine the reduction value according to the third target element and the first target element as needed, as long as the determined reduction value can match the subsequent processing method to achieve the purpose of reducing the element.

[0110] In order to further improve the accuracy of smoothing the gradient inside the matrix to be scaled, on the basis of the above embodiments, in an embodiment of the present application, determining the reduction value according to the third target element and the first target element includes:

[0111] Get the pre-saved reduction ratio;

[0112] Determine a difference between the first target element and the third target element;

[0113] The product of the difference and the reduction ratio is determined as the reduction value.

[0114] When determining the reduction value according to the third target element and the first target element, in the embodiment of the present application, a pre-saved reduction ratio may be obtained, and the reduction ratio is a pre-configured value greater than 0 and less than or equal to 1. Those skilled in the art may configure the reduction ratio according to actual applications.

[0115] After the pre-saved reduction ratio is obtained, the difference between the first target element and the third target element is determined, and the product of the difference and the obtained reduction ratio is determined as the reduction value.

[0116] Specifically, assuming that the reduction ratio is λ, the value range of λ is: λ∈(0,1], the difference between the first target element and the third target element can be expressed as Δb p,p+1 , then the determined reduction value can be expressed as:

[0117] Δb λ =λΔb p,p+1

[0118] Where Δb λ Indicates a decreasing value.

[0119] When reducing the elements in the element sequence that are not less than the first target element, the reduction value Δb may be subtracted from each element that is not less than the first target element. λ , the element after the reduction process can be expressed as C. For example, when the p+qth element in the element sequence is reduced, the element after the reduction process can be expressed as c p+q =b p+q -Δb λ .

[0120] In order to smooth the gradient inside the matrix to be scaled, based on the above embodiments, in an embodiment of the present application, the amplification process of the elements in the element sequence that are not greater than the first target element includes:

[0121] Get the saved magnification value;

[0122] The elements in the element sequence that are not greater than the first target element are amplified based on the amplification value.

[0123] If, when scaling an element in an element sequence, the elements in the element sequence that are not larger than the first target element are enlarged, then when performing the enlargement process, in an embodiment of the present application, a saved enlargement value can be obtained, and the technicians in this field can configure the enlargement value as needed.

[0124] After the saved magnification value is obtained, the elements in the element sequence that are not greater than the first target element may be magnified based on the magnification value.

[0125] Specifically, assuming that the obtained magnification value is 0.5, then for each element in the element sequence that is not greater than the first target element, the quotient of the element and the magnification value can be determined, and the quotient value is determined as the element after the magnification process. Since the reduction value is a decimal less than 1, the determined quotient value must be greater than the element before the element is magnified. Assuming that the obtained magnification value is 20, then for each element in the element sequence that is not greater than the first target element, the sum of the element and the reduction value can be determined, and the sum is determined as the element after the magnification process. It should be noted that since the magnification value is pre-configured, the numerical value of the magnification value is known, so how to perform subsequent processing based on the magnification value is also fixed. Therefore, in the embodiment of the present application, after the magnification value is obtained, the elements that are not greater than the first target element can be magnified according to the pre-configured magnification method.

[0126] In order to further improve the accuracy of smoothing the gradient inside the matrix to be scaled, on the basis of the above embodiments, in the embodiment of the present application, the process of determining the amplification value includes:

[0127] Obtain a fourth target element adjacent to the first target element in the element sequence;

[0128] The amplification value is determined according to the fourth target element and the first target element.

[0129] Since the elements in different matrices to be scaled are different, and the gradient differences of the elements in different matrices to be scaled are also different, therefore, in order to accurately amplify the elements, in an embodiment of the present application, the amplification value can be determined based on the elements in the element sequence. Since the amplification value is determined in order to amplify the elements that are not greater than the first target element, the elements to be amplified also include the first target element. Therefore, in an embodiment of the present application, the fourth target element adjacent to the first target element in the element sequence can be obtained, and the fourth target element can be any element adjacent to the first target element. Since the present application hopes to reduce the gap between the first target element and other adjacent elements, therefore, in an embodiment of the present application, the difference between the first target element and each adjacent element can also be determined, and the corresponding adjacent element with the largest difference is determined as the fourth target element, and the amplification value is determined based on the fourth target element and the first target element.

[0130] The process of determining the amplification value is similar to the process of determining the reduction value. The above embodiments have described the process of determining the reduction value in detail. In the embodiments of the present application, the process of determining the amplification value is not repeated. It should be noted that a person skilled in the art can configure how to determine the amplification value according to the fourth target element and the first target element as needed, as long as the determined amplification value can match the subsequent processing method to achieve the purpose of amplifying the element.

[0131] In order to ensure the accuracy of the processed matrix to be scaled, on the basis of the above embodiments, in the embodiment of the present application, after the elements in the element sequence that are not smaller than the first target element are reduced in size, before the elements in the element sequence are restored to the matrix to be scaled based on the information of the column where the elements in the element sequence are located, the method further includes:

[0132] Determining a target mean and a target variance according to the elements in the processed element sequence;

[0133] The differences between the elements in the element sequence and the target mean are determined one by one, and the corresponding elements are updated using the quotient of the absolute value of the difference and the target variance.

[0134] The mean of all elements in the processed element sequence is generally not 0, and the variance is generally not 1. However, the input of the softmax() function in the current attention mechanism generally has a mean of 0 and a variance of 1. Therefore, in an embodiment of the present application, after scaling the elements in the element sequence, before restoring the elements in the processed element sequence to the matrix to be scaled, the mean of all elements in the element sequence can be adjusted to 0, and the variance can be adjusted to 1.

[0135] In the embodiment of the present application, the target mean and target variance can be determined according to the elements in the processed element sequence. Given a set of numerical values, how to determine the mean and variance according to the known numerical values ​​is a prior art and will not be described in detail in the embodiment of the present application.

[0136] After the target mean and target variance are determined, the difference between the elements in the element sequence and the target mean can be determined one by one, and the quotient of the absolute value of the difference and the target variance can be determined, and the corresponding elements can be updated using the quotient.

[0137] Specifically, the quotient value can be determined based on the following formula:

[0138]

[0139] Among them, e w is the determined quotient value, the e w It can also be represented as an element updated with the quotient value; dw is the element in the element sequence before updating; w = 1, 2, …, n, where n is the number of elements in the element sequence; μ is the target mean; σ is the target variance.

[0140] The data processing process of the attention mechanism is explained below in conjunction with a specific embodiment. At present, the Transformer model adopts a self-attention mechanism and a multi-head attention mechanism. The core matrices of both the self-attention mechanism and the multi-head attention mechanism are Q, K, and V matrices, which are obtained by linearly transforming the input word vector. Specifically, for an input sequence of length L, assuming that the dimension of the word vector of the input sequence is d, then the input of the attention mechanism is a matrix X with L rows and d columns. The attention mechanism can map the matrix X to three d×d matrices Q, K, and V respectively through three parameter matrices Wq, Wk, and Wv. Among them, the parameter matrices Wq, Wk, and Wv are learnable parameters of the Transformer model, which are used to map the input matrix X to query, key, and value matrices, namely Q, K, and V matrices, respectively. The Q, K, and V matrices are used in the subsequent attention mechanism to calculate the attention score and weighted sum, thereby encoding the input sequence.

[0141] The attention mechanism can generally be divided into three stages: Figure 2 The schematic diagram of the attention mechanism provided in the embodiment of the present application is as follows: Figure 2 As shown, in the first stage of the attention mechanism, the correlation matrix between the query (Q) matrix and different key (K) matrices is calculated, such as Figure 2 As shown, the correlation matrix S1 can be determined according to the query matrix and the key1 matrix; the correlation matrix S2 can be determined according to the query matrix and the key2 matrix; the correlation matrix S3 can be determined according to the query matrix and the key3 matrix; and the correlation matrix S4 can be determined according to the query matrix and the key4 matrix. When calculating the correlation matrix, methods such as finding the dot product, finding the cosine similarity, and introducing additional neural networks can be used. In the second stage of the attention mechanism, the correlation matrix output in the first stage is scaled and normalized like softmax() to obtain weight coefficients whose sum is 1, which is a probability distribution vector, where softmax() is an activation function. The correlation matrix scaled in the second stage can be used as the matrix to be scaled. When scaling the matrix to be scaled, the matrix to be scaled can be processed based on the data processing methods described in the above embodiments. As Figure 2As shown, in the second stage, the correlation matrix S1 output by the first stage can be used as the matrix to be scaled, and the matrix S1 to be scaled is processed based on the above data processing method to obtain the scaled matrix to be scaled, and the scaled matrix to be scaled is processed based on the softmax() function to obtain the matrix a1; and the correlation matrix S2 output by the first stage can be used as the matrix to be scaled, and the matrix S2 to be scaled is processed based on the above data processing method to obtain the scaled matrix to be scaled, and the scaled matrix to be scaled is processed based on the softmax() function to obtain the matrix a2; the correlation matrix S3 output by the first stage can be used as the matrix to be scaled, and the matrix S3 to be scaled is processed based on the above data processing method to obtain the scaled matrix to be scaled, and the scaled matrix to be scaled is processed based on the softmax() function to obtain the matrix a3; the correlation matrix S4 output by the first stage can be used as the matrix to be scaled, and the matrix S4 to be scaled is processed based on the above data processing method to obtain the scaled matrix to be scaled, and the scaled matrix to be scaled is processed based on the softmax() function to obtain the matrix a4. In the third stage of the attention mechanism, the matrix value (V) output from the second stage is weighted to obtain the attention value. Figure 2 As shown, in the third stage, the matrix a1 output from the second stage can be weighted with the matrix V to obtain the matrix value1; the matrix a2 output from the second stage can be weighted with the matrix V to obtain the matrix value2; the matrix a3 output from the second stage can be weighted with the matrix V to obtain the matrix value3.

[0142] The data processing method provided in the embodiment of the present application can smooth the gradients of the elements in the matrix to be scaled, significantly reduce the possible "one-hot" and back-propagation gradient disappearance, enhance the robustness of the attention mechanism, and improve the performance of models such as transformers.

[0143] Figure 3 A flow chart of another data processing method provided in an embodiment of the present application is as follows: Figure 3 As shown, the matrix to be scaled is represented as QK T , when processing data, obtain any row element in the matrix to be scaled, and represent the row element as A(a1, a2, ... a n ). Figure 4 Any row element schematic diagram provided in the embodiment of the present application, Figure 4 Each vertical line in the represents an element, and the length of each vertical line represents the size of the element. The longer the vertical line, the larger the element represented by the vertical line, and the shorter the vertical line, the smaller the element represented by the vertical line. Figure 4It can be seen that the row includes 10 elements, among which the third, fourth and seventh elements are relatively large. Therefore, when processing the elements of this row, it is necessary to narrow the gap between the third, fourth and seventh elements and the other elements.

[0144] like Figure 3 As shown, the elements in the row can be sorted in order of size to obtain an element sequence B. For ease of description, the element sequence B can be expressed as (b1, b2, ..., b n ), where b n For A(a1,a2,…a n ) in the . Figure 5 The schematic diagram of the element sequence provided in the embodiment of the present application is as follows: Figure 5 As shown, Figure 5 Lieutenant General Figure 4 The elements represented in are sorted in order of size. Figure 5 Coming Soon Figure 4 The elements represented in are sorted from small to large, among which the first seven elements are relatively small and the last three elements are large.

[0145] like Figure 3 As shown, after the element sequence B is obtained, the first target element in which the element in the element sequence mutates can be determined. Figure 6 A schematic diagram of the first target element provided in the embodiment of the present application, such as Figure 6 As shown, Figure 6 The eighth element in the image suddenly becomes larger. The eighth element is the first target element. In order to facilitate the subsequent description of the process of scaling the element, in the embodiment of the present application, Figure 6 The seventh element in is taken as the mutation point C p , Figure 6 The vertical line that is thicker than the other vertical lines is the mutation point.

[0146] like Figure 3 As shown in Figure 2, after determining the first target element, the value of the high-order element, i.e., the mutation point C, can be reduced. p The value of the element after that, the processed element sequence can be expressed as D = (b1, b2, ..., b p-1 ,b p ,C p+1 ,C p+2 ,…,C n ), where b p is the mutation point, b p The previous elements are not processed, b p The subsequent elements are reduced in size. To facilitate distinction, in the processed element sequence D, b is used to represent the elements that have not been processed, and C is used to represent the elements that have been reduced in size. Figure 7 A schematic diagram of an element after reduction processing provided in an embodiment of the present application, such as Figure 7 As shown, Figure 7 Lieutenant General Figure 6 The last three elements in are scaled down so that Figure 7 The gap between each element in becomes smaller, smoothing the gradient of the elements in this row. Figure 7 The dotted lines on the last three vertical lines in the image represent the values ​​that are removed after the reduction process, and the solid lines on the last three vertical lines represent the values ​​of the elements after the reduction process.

[0147] like Figure 3 As shown in the figure, after the high-order elements are reduced, the mean and variance of the processed element sequence are transformed, the mean of the processed element sequence is transformed to 0, and the variance is transformed to 1. The sequence after the mean and variance transformation can be expressed as E = (e1, e2, ..., e n ). Figure 8 The schematic diagram of the elements after the mean and variance transformation provided in the embodiment of the present application is as follows: Figure 8 As shown, Figure 8 Each element in Figure 7 The value of each element in becomes smaller because the mean and variance of each element are transformed. How to transform the mean and variance of the processed element sequence has been explained in detail in the above embodiments and will not be repeated in the embodiments of this application.

[0148] like Figure 3 As shown, after the mean and variance of the processed element sequence are transformed, based on the information of the column where the elements in the element sequence are located, the processed elements are restored in the order of the columns in the matrix to be scaled. The restored elements can be expressed as F = (f1, f2, ..., f n ). Fig. 9 The schematic diagram of the elements after the restoration sequence provided by the embodiment of the present application is as follows: Fig. 9 As shown, Fig. 9 Each processed element has been processed according to Figure 4 Since softmax() is a single-row calculation, after processing the row elements, the processed F=(f1,f2,…,f n ), input to the softmax() function, and then perform the subsequent attention mechanism model calculation. Of course, after each row element in the matrix to be scaled is processed, the attention mechanism processing can be performed based on the processed matrix to be scaled. This completes the row element A(a1, a2, …a n), the same processing can be performed on other rows in the matrix to be scaled based on the above steps.

[0149] The technical solution protected by this application is characterized by robustness and reliability, and meets the trustworthy characteristics.

[0150] Fig.10 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application is shown in FIG. Fig.10 As shown, the device comprises:

[0151] A sorting module 1001 is used to sort the elements of any row in the scaling matrix in order of size to obtain an element sequence, and record information about the column where the element in the element sequence is located;

[0152] A determination module 1002 is used to determine a first target element in which an element in the element sequence undergoes a mutation;

[0153] A scaling module 1003, configured to scale the elements in the element sequence according to a size relationship between the elements in the element sequence and the first target element;

[0154] The recovery module 1004 is used to restore the elements in the element sequence after processing to the matrix to be scaled based on the information of the column where the elements in the element sequence are located; and perform attention mechanism processing based on the matrix to be scaled.

[0155] In a possible implementation, the determination module 1002 is specifically used to group adjacent elements in the element sequence to obtain an element group, wherein the number of elements included in the element group is an odd number, and determine the element located in the middle position of the element group as the second target element corresponding to the element group; determine the average value of the difference between the second target element corresponding to the element group and other elements in the element group; determine the element group corresponding to the maximum average value among the average values ​​as the target element group, and determine the second target element corresponding to the target element group as the first target element.

[0156] In a possible implementation, the scaling module 1003 is specifically used to scale down elements in the element sequence that are not smaller than the first target element; or, to enlarge elements in the element sequence that are not larger than the first target element; or, if the first target element is larger than the previous adjacent element in the element sequence, to scale down elements in the element sequence that are larger than or equal to the first target element, and to enlarge elements in the element sequence that are smaller than the first target element; if the first target element is smaller than the previous adjacent element in the element sequence, to scale down elements in the element sequence that are larger than the first target element, and to enlarge elements in the element sequence that are smaller than or equal to the first target element.

[0157] In a possible implementation, the scaling module 1003 is specifically configured to obtain a saved reduction value; and scale down elements in the element sequence that are not smaller than the first target element based on the reduction value.

[0158] In a possible implementation, the determination module 1002 is further configured to obtain a third target element adjacent to the first target element in the element sequence; and determine the reduction value according to the third target element and the first target element.

[0159] In a possible implementation, the determination module 1002 is specifically configured to obtain a pre-saved reduction ratio; determine a difference between the first target element and the third target element; and determine a product of the difference and the reduction ratio as the reduction value.

[0160] In a possible implementation, the scaling module 1003 is specifically configured to obtain a stored magnification value; and perform a magnification process on elements in the element sequence that are not larger than the first target element based on the magnification value.

[0161] In a possible implementation, the determination module 1002 is specifically configured to obtain a fourth target element adjacent to the first target element in the element sequence; and determine the amplification value according to the fourth target element and the first target element.

[0162] In a possible implementation, the determination module 1002 is further used to determine a target mean and a target variance based on the elements in the processed element sequence; determine the difference between the elements in the element sequence and the target mean one by one, and update the corresponding elements using the quotient of the absolute value of the difference and the target variance.

[0163] Fig.11A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Based on the above embodiments, the present application further provides an electronic device, such as Fig.11 As shown, it includes: a processor 1101, a communication interface 1102, a memory 1103 and a communication bus 1104, wherein the processor 1101, the communication interface 1102, and the memory 1103 communicate with each other through the communication bus 1104;

[0164] The memory 1103 stores a computer program. When the program is executed by the processor 501, the processor 1101 performs the following steps:

[0165] Sort any row elements in the scaling matrix in order of size to obtain an element sequence, and record information about the column where the element in the element sequence is located;

[0166] Determine a first target element in the element sequence where an element mutates; and perform scaling processing on the elements in the element sequence according to a size relationship between the elements in the element sequence and the first target element;

[0167] Based on information about the columns where the elements in the element sequence are located, restoring the processed elements in the element sequence to the matrix to be scaled;

[0168] Perform attention mechanism processing based on the matrix to be scaled.

[0169] In a possible implementation, the processor 1101 is further configured to group adjacent elements in the element sequence to obtain an element group, where the number of elements included in the element group is an odd number, and determine an element located in a middle position in the element group as a second target element corresponding to the element group;

[0170] Determine an average value of the difference between the second target element corresponding to the element group and other elements in the element group;

[0171] The elements corresponding to the maximum average value among the average values ​​are grouped as a target element group, and the second target element corresponding to the target element group is determined as the first target element.

[0172] In a possible implementation, the processor 1101 is further configured to perform a reduction process on elements in the element sequence that are not smaller than the first target element; or,

[0173] Amplify the elements in the element sequence that are not larger than the first target element; or,

[0174] If the first target element is larger than the previous element adjacent to it in the element sequence, the elements in the element sequence that are larger than or equal to the first target element are reduced, and the elements in the element sequence that are smaller than the first target element are enlarged; if the first target element is smaller than the previous element adjacent to it in the element sequence, the elements in the element sequence that are larger than the first target element are reduced, and the elements in the element sequence that are smaller than or equal to the first target element are enlarged.

[0175] In a possible implementation manner, the processor 1101 is further configured to obtain a saved reduction value;

[0176] Based on the reduction value, a reduction process is performed on the elements in the element sequence that are not smaller than the first target element.

[0177] In a possible implementation, the processor 1101 is further configured to obtain a third target element adjacent to the first target element in the element sequence;

[0178] The reduction value is determined according to the third target element and the first target element.

[0179] In a possible implementation manner, the processor 1101 is further configured to obtain a pre-saved reduction ratio;

[0180] Determine a difference between the first target element and the third target element;

[0181] The product of the difference and the reduction ratio is determined as the reduction value.

[0182] In a possible implementation manner, the processor 1101 is further configured to obtain a stored amplification value;

[0183] The elements in the element sequence that are not greater than the first target element are amplified based on the amplification value.

[0184] In a possible implementation, the processor 1101 is further configured to obtain a fourth target element adjacent to the first target element in the element sequence;

[0185] The amplification value is determined according to the fourth target element and the first target element.

[0186] In a possible implementation, the processor 1101 is further configured to determine a target mean and a target variance according to the elements in the processed element sequence;

[0187] The differences between the elements in the element sequence and the target mean are determined one by one, and the corresponding elements are updated using the quotient of the absolute value of the difference and the target variance.

[0188] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface 1102 is used for communication between the above electronic device and other devices. The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor. The above processor may be a general-purpose processor, including a central processing unit, a network processor (NP), etc.; it may also be a digital signal processing processor (DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc.

[0189] On the basis of the above embodiments, the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program executable by a processor, and when the program runs on the processor, the processor implements the following steps when executing:

[0190] Sort any row elements in the scaling matrix in order of size to obtain an element sequence, and record information about the column where the element in the element sequence is located;

[0191] Determine a first target element in the element sequence where an element mutates; and perform scaling processing on the elements in the element sequence according to a size relationship between the elements in the element sequence and the first target element;

[0192] Based on information about the columns where the elements in the element sequence are located, restoring the processed elements in the element sequence to the matrix to be scaled;

[0193] Perform attention mechanism processing based on the matrix to be scaled.

[0194] In a possible implementation manner, determining the first target element in which an element in the element sequence has mutated includes:

[0195] Grouping adjacent elements in the element sequence to obtain an element group, wherein the number of elements included in the element group is an odd number, and determining an element located in the middle position of the element group as a second target element corresponding to the element group;

[0196] Determine an average value of the difference between the second target element corresponding to the element group and other elements in the element group;

[0197] The elements corresponding to the maximum average value among the average values ​​are grouped as a target element group, and the second target element corresponding to the target element group is determined as the first target element.

[0198] In a possible implementation, the scaling the elements in the element sequence according to the size relationship between the elements in the element sequence and the first target element includes:

[0199] Performing a reduction process on the elements in the element sequence that are not smaller than the first target element; or,

[0200] Amplify the elements in the element sequence that are not larger than the first target element; or,

[0201] If the first target element is larger than the previous element adjacent to it in the element sequence, the elements in the element sequence that are larger than or equal to the first target element are reduced, and the elements in the element sequence that are smaller than the first target element are enlarged; if the first target element is smaller than the previous element adjacent to it in the element sequence, the elements in the element sequence that are larger than the first target element are reduced, and the elements in the element sequence that are smaller than or equal to the first target element are enlarged.

[0202] In a possible implementation, the performing a reduction process on the elements in the element sequence that are not smaller than the first target element includes:

[0203] Get the saved reduced value;

[0204] Based on the reduction value, a reduction process is performed on the elements in the element sequence that are not smaller than the first target element.

[0205] In a possible implementation, the process of determining the reduced value includes:

[0206] Obtaining a third target element adjacent to the first target element in the element sequence;

[0207] The reduction value is determined according to the third target element and the first target element.

[0208] In a possible implementation manner, determining the reduction value according to the third target element and the first target element includes:

[0209] Get the pre-saved reduction ratio;

[0210] Determine a difference between the first target element and the third target element;

[0211] The product of the difference and the reduction ratio is determined as the reduction value.

[0212] In a possible implementation, the amplifying the element in the element sequence that is not greater than the first target element includes:

[0213] Get the saved magnification value;

[0214] The elements in the element sequence that are not greater than the first target element are amplified based on the amplification value.

[0215] In a possible implementation manner, the process of determining the amplification value includes:

[0216] Obtain a fourth target element adjacent to the first target element in the element sequence;

[0217] The amplification value is determined according to the fourth target element and the first target element.

[0218] In a possible implementation, after scaling the elements in the element sequence, before restoring the processed elements in the element sequence to the matrix to be scaled based on information of the column where the elements in the element sequence are located, the method further includes:

[0219] Determining a target mean and a target variance according to the elements in the processed element sequence;

[0220] The differences between the elements in the element sequence and the target mean are determined one by one, and the corresponding elements are updated using the quotient of the absolute value of the difference and the target variance.

[0221] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0222] As for the system / device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0223] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0224] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0225] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0226] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0227] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A data processing method, characterized in that: The method comprises: Sort any row elements in the scaling matrix in order of size to obtain an element sequence, and record information about the column where the element in the element sequence is located; Determine a first target element in the element sequence where an element mutates; and perform scaling processing on the elements in the element sequence according to a size relationship between the elements in the element sequence and the first target element; Based on information about the columns where the elements in the element sequence are located, restoring the processed elements in the element sequence to the matrix to be scaled; Perform attention mechanism processing based on the matrix to be scaled.

2. The method according to claim 1, characterized in that The determining of the first target element in which the element in the element sequence mutates comprises: Grouping adjacent elements in the element sequence to obtain an element group, wherein the number of elements included in the element group is an odd number, and determining an element located in the middle position of the element group as a second target element corresponding to the element group; Determine an average value of the difference between the second target element corresponding to the element group and other elements in the element group; The elements corresponding to the maximum average value among the average values ​​are grouped as a target element group, and the second target element corresponding to the target element group is determined as the first target element.

3. The method according to claim 1 or 2, characterized in that: The scaling of the elements in the element sequence according to the size relationship between the elements in the element sequence and the first target element comprises: Performing a reduction process on the elements in the element sequence that are not smaller than the first target element; or, Amplify the elements in the element sequence that are not larger than the first target element; or, If the first target element is larger than the previous element adjacent to it in the element sequence, the elements in the element sequence that are larger than or equal to the first target element are reduced, and the elements in the element sequence that are smaller than the first target element are enlarged; if the first target element is smaller than the previous element adjacent to it in the element sequence, the elements in the element sequence that are larger than the first target element are reduced, and the elements in the element sequence that are smaller than or equal to the first target element are enlarged.

4. The method according to claim 3, characterized in that The shrinking of the elements in the element sequence that are not smaller than the first target element comprises: Get the saved reduced value; Based on the reduction value, a reduction process is performed on the elements in the element sequence that are not smaller than the first target element.

5. The method according to claim 4, characterized in that The process of determining the reduced value includes: Obtaining a third target element adjacent to the first target element in the element sequence; The reduction value is determined according to the third target element and the first target element.

6. The method according to claim 5, characterized in that The determining the reduction value according to the third target element and the first target element comprises: Get the pre-saved reduction ratio; Determine a difference between the first target element and the third target element; The product of the difference and the reduction ratio is determined as the reduction value.

7. The method according to claim 3, characterized in that The enlarging the element in the element sequence that is not greater than the first target element comprises: Get the saved magnification value; The elements in the element sequence that are not greater than the first target element are amplified based on the amplification value.

8. The method according to claim 7, characterized in that The process of determining the amplification value includes: Obtain a fourth target element adjacent to the first target element in the element sequence; The amplification value is determined according to the fourth target element and the first target element.

9. The method according to claim 1, characterized in that: After scaling the elements in the element sequence, and before restoring the processed elements in the element sequence to the matrix to be scaled based on information of the column where the elements in the element sequence are located, the method further includes: Determining a target mean and a target variance according to the elements in the processed element sequence; The differences between the elements in the element sequence and the target mean are determined one by one, and the corresponding elements are updated using the quotient of the absolute value of the difference and the target variance.

10. An electronic device, characterized in that: The electronic device comprises at least a processor and a memory, and the processor is used to implement the steps of the data processing method as described in any one of claims 1 to 9 when executing a computer program stored in the memory.

Images