Data processing method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a data processing method and device, electronic equipment, storage medium and product, and relate to the technical field of data processing and artificial intelligence. The method comprises: obtaining a first feature map and a second feature map by performing feature extraction on to-be-processed data and reference data respectively; determining a first similarity matrix between the first feature map and the second feature map; obtaining a similarity threshold, and converting the first similarity matrix into a third similarity matrix; obtaining an attention matrix by performing at least one self-multiplication operation on the element value of each element in the third similarity matrix; weighting the second feature map based on the attention matrix to obtain a weighted feature map, and obtaining target data after a target operation on the to-be-processed data according to the weighted feature map. The embodiments of the present application improve the efficiency of determining the attention matrix and reduce the consumption of computing resources.

Description

Technical Field

[0001] The present application relates to the fields of data processing technology and artificial intelligence technology. Specifically, the present application relates to a data processing method, device, electronic device, storage medium and product. Background Art

[0002] In deep learning models, especially those involving attention mechanisms, the Softmax operation is used to calculate the weight of each position in the input data. Softmax converts the input data into a corresponding probability distribution, assigning a weight to each position. These weights reflect the relative importance of each element in the input data. In this way, the Softmax operation achieves sparse processing of the input data, allowing the model to focus on the parts that are more relevant to the current input, thereby improving expressiveness and prediction accuracy.

[0003] However, the Softmax operation requires performing exponential operations on each element, which has a high computational complexity. In computationally intensive models such as the Stable Diffusion model, the dimension of the input data of the corresponding Softmax operation is usually large. As the dimension of the input feature increases, the amount of Softmax calculation increases exponentially. Therefore, there are significant time delays and computing resource consumption problems, which in turn affects the inference speed and efficiency of the model. Summary of the Invention

[0004] The present invention provides a data processing method, apparatus, electronic device, computer-readable storage medium, and computer program product to address the computational complexity and resource consumption issues of the Softmax operation when processing high-dimensional data. The technical solution is as follows:

[0005] According to a first aspect of an embodiment of the present application, a data processing method is provided, which is executed by a processor. The method includes:

[0006] By performing feature extraction on the data to be processed and the reference data used for target processing of the data to be processed, a first feature map of the data to be processed and a second feature map of the parameter data are obtained, wherein the data to be processed and the reference data are the same or different; wherein the sizes of the first feature map and the second feature map are , h represents the number of eigenvectors included in the feature map, and m represents the number of eigenvalues ​​included in each eigenvector in the feature map;

[0007] Determine a first similarity matrix between the first feature map and the second feature map, where an element value of an element in the i-th row and j-th column of the first similarity matrix represents a similarity between an i-th eigenvector in the first feature map and a j-th eigenvector in the second feature map;

[0008] Obtaining a similarity threshold, subtracting the elements in the first similarity matrix from the similarity threshold to obtain a second similarity matrix, and setting the elements in the second similarity matrix that are less than 0 to 0 to obtain a third similarity matrix;

[0009] Obtaining an attention matrix between the data to be processed and the reference data by performing at least one self-multiplication operation on the element value of each element in the third similarity matrix;

[0010] The second feature map is weighted based on the attention matrix to obtain a weighted feature map, and target data after the target operation is performed on the data to be processed is obtained according to the weighted feature map.

[0011] According to a second aspect of an embodiment of the present application, there is provided a data processing device, the device comprising:

[0012] The feature extraction module is used to extract features from the data to be processed and the reference data used to perform target processing on the data to be processed, respectively, to obtain a first feature map of the data to be processed and a second feature map of the parameter data, wherein the data to be processed and the reference data are the same or different; wherein the sizes of the first feature map and the second feature map are , h represents the number of eigenvectors included in the feature map, and m represents the number of eigenvalues ​​included in each eigenvector in the feature map;

[0013] A similarity determination module is configured to determine a first similarity matrix between the first feature map and the second feature map, wherein the element value of the element in the i-th row and j-th column in the first similarity matrix represents the similarity between the i-th eigenvector in the first feature map and the j-th eigenvector in the second feature map;

[0014] a similarity threshold obtaining module, configured to obtain a similarity threshold, subtract the elements in the first similarity matrix from the similarity threshold to obtain a second similarity matrix, and set the elements in the second similarity matrix that are less than 0 to 0 to obtain a third similarity matrix;

[0015] an attention matrix obtaining module, configured to obtain an attention matrix between the to-be-processed data and the reference data by performing at least one self-multiplication operation on the element value of each element in the third similarity matrix;

[0016] A weighting module is used to weight the second feature map based on the attention matrix to obtain a weighted feature map, and obtain target data after the target operation is performed on the data to be processed according to the weighted feature map.

[0017] According to a third aspect of an embodiment of the present application, an electronic device is provided, which includes a memory, a processor, and a computer program stored in the memory, and when the processor executes the program, the steps of the method provided in the first aspect are implemented.

[0018] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method provided in the first aspect are implemented.

[0019] According to the fifth aspect of the embodiment of the present application, a computer program product is provided, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. When the processor of a computer device reads the computer instructions from the computer-readable storage medium, the processor executes the computer instructions, so that the computer device performs the steps of implementing the method provided in the first aspect.

[0020] The beneficial effects of the technical solution provided by the embodiments of the present application are:

[0021] By extracting features from the data to be processed and the reference data, a first feature map of the data to be processed and a second feature map of the reference data are obtained, wherein the data to be processed and the reference data may be the same or different, so that the method provided in the embodiment of the present application can be applied to the self-attention mechanism and the cross-attention mechanism. By determining the first similarity matrix between the first feature map and the second feature map, the degree of association between the data to be processed and the reference data in the feature space is obtained; by using a similarity threshold, the elements in the first similarity matrix are subtracted from the similarity threshold to obtain a second similarity matrix, and the elements in the second similarity matrix that are less than 0 are set to 0 to obtain a third similarity matrix, so that the obtained third similarity matrix retains feature pairs with high similarity; by performing at least one self-multiplication operation on the element value of each element in the third similarity matrix, the sparse processing of the third similarity matrix is ​​realized, and the attention matrix between the data to be processed and the reference data is obtained; based on the attention matrix, the second feature map is weighted to obtain a weighted feature map, so that the model can perform prediction and inference based on the weighted feature map, and obtain the target data after the target operation on the data to be processed. In an embodiment of the present application, by obtaining a similarity threshold, the first similarity matrix is ​​converted into a third similarity matrix with more 0 elements. By performing a self-multiplication operation on each element in the third similarity matrix, elements with smaller values ​​are made closer to 0, thereby achieving sparseness of the first similarity matrix. Compared with the exponential operation in the Softmax operation, the self-multiplication operation of the elements has lower computational complexity and better numerical stability, thereby effectively reducing the consumption of computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments of the present application.

[0023] Figure 1 A schematic diagram of the data processing system architecture provided in an embodiment of the present application;

[0024] Figure 2 A flowchart of a data processing method provided in an embodiment of the present application;

[0025] Figure 3 A schematic diagram of determining a third similarity matrix provided in an embodiment of the present application;

[0026] Figure 4 A schematic diagram of obtaining an attention matrix provided in an embodiment of the present application;

[0027] Figure 5 A comparison diagram of data distribution after sparseness provided in an embodiment of the present application;

[0028] Figure 6A flowchart of a data processing method provided in an embodiment of the present application;

[0029] Figure 7 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;

[0030] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] The following describes the embodiments of the present application in conjunction with the accompanying drawings. It should be understood that the embodiments described below in conjunction with the accompanying drawings are exemplary descriptions for explaining the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions of the embodiments of the present application.

[0032] Those skilled in the art will understand that, unless otherwise stated, the singular forms "a", "an", "said", and "the" used herein may also include plural forms. It should be further understood that the terms "including" and "comprising" used in the embodiments of the present application mean that the corresponding features can be implemented as the presented features, information, data, steps, operations, elements, and / or components, but do not exclude implementation as other features, information, data, steps, operations, elements, components, and / or combinations thereof supported by the present technical field. It should be understood that when we say that an element is "connected" or "coupled" to another element, the element can be directly connected or coupled to the other element, or it can refer to the element and the other element establishing a connection relationship through an intermediate element. In addition, the "connection" or "coupling" used here can include wireless connection or wireless coupling. The term "and / or" used here indicates at least one of the items defined by the term, for example, "A and / or B" can be implemented as "A", or as "B", or as "A and B".

[0033] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0034] First, several terms involved in this application are introduced and explained:

[0035] Stable Diffusion is a type of diffusion model that treats the data distribution as the steady state of a diffusion process and generates data using a back-diffusion process. It is currently commonly used in image generation, text generation, audio generation, and other fields. The UNet network is the most commonly used basic network structure in Stable Diffusion. The UNet network can effectively capture multi-scale features while maintaining high-resolution details. The implementation of the attention mechanism in the UNet network helps improve the model's performance in feature extraction and generation. In particular, in image generation tasks, it can more accurately focus on key information, thereby generating high-quality images.

[0036] The attention mechanism is a data processing method in machine learning, widely used in various machine learning tasks such as natural language processing, image recognition, and speech recognition. The attention mechanism's degree of attention to different pieces of information is reflected by weights. The attention mechanism can be viewed as a multilayer perceptron (MLP) composed of a query matrix (Query), a key matrix (Key), and a weighted average. The concept of attention is similar to addressing. Given a Query element in a target, the similarity or correlation between the Query and each Key is calculated to obtain a weight coefficient for each Key's corresponding Value. The weighted sum of the Values ​​is then taken to obtain the final Attention value. Therefore, the Attention mechanism essentially performs a weighted sum of the Values, with the Query and Key used to calculate the corresponding Value weight coefficients.

[0037] The data processing method, device, electronic device, computer-readable storage medium, and computer program product provided in this application are intended to solve the above technical problems in the prior art.

[0038] The following describes several exemplary embodiments to illustrate the technical solutions of the embodiments of the present application and the technical effects produced by the technical solutions of the present application. It should be noted that the following embodiments can refer to, draw on, or combine with each other, and the same terms, similar features, and similar implementation steps in different embodiments will not be repeated.

[0039] Figure 1 The data processing system architecture diagram provided in the embodiment of the present application is as follows: Figure 1 As shown, the data processing system includes a terminal 101 and a server 102. The server 102 can be deployed with any neural network model including an attention mechanism module, and the server can run the neural network model based on a processor NPU.

[0040] The terminal 101 can send the input data required by the neural network model to the server 102, so that the processor processes the input data based on the neural network model. When the neural network model processes the input data, it can obtain the data to be processed and the reference data corresponding to the attention mechanism module based on the input data. The data to be processed and the reference data can be the same or different. When the data to be processed and the reference data are the same, the attention mechanism module is a self-attention mechanism module. When the data to be processed and the reference data are different, the attention mechanism module is a cross-attention mechanism module.

[0041] The server 102 performs the following operations through the attention mechanism module: by extracting features from the data to be processed and the reference data, a first feature map of the data to be processed and a second feature map of the reference data are obtained; further, by similarity analysis, a first similarity matrix between the first feature map and the second feature map is obtained; based on a preset similarity threshold, the similarity thresholds of the elements in the first similarity matrix are subtracted to obtain a second similarity matrix; the elements less than 0 in the second similarity matrix are set to 0 to obtain a third similarity matrix; the element values ​​of each element in the third similarity matrix are self-multiplied to obtain an attention matrix of the image to be processed and the text; the second feature map is weighted based on the attention matrix to obtain a weighted feature map; and according to the weighted feature map. The attention mechanism model in the embodiment of the present application replaces the Softmax operation in the attention mechanism of the prior art by presetting a similarity threshold and performing a self-multiplication operation on the elements, and further improves the processing efficiency of the NPU on the basis of achieving the sparsification of the first similarity matrix.

[0042] The neural network model can perform subsequent early operations based on the weighted feature map obtained by the attention mechanism module to obtain the final target data.

[0043] Taking a specific application scenario as an example, the terminal 101 can be any terminal running the Wenshengtu application / software, such as a user's mobile phone or computer. The user terminal 101 can establish a communication connection with the Wenshengtu application / software server 102 via a wired or wireless method, and the user can use the various services provided by the application based on the user interface of the application displayed on the terminal 101.

[0044] Text is input into the user interface of the application displayed on the terminal 101, and the terminal 101 sends the text to the server 102. A stable diffusion model can be deployed in the server 102. The server runs the stable diffusion model based on the processor NPU, and generates an initial noise image based on the acquired text. The initial noise image is used as the image to be processed. The image to be processed and the text are input to the cross-attention mechanism module. The above operations are performed based on the cross-attention mechanism module to obtain a weighted feature map. The stable diffusion model performs noise prediction based on the obtained weighted feature map. By performing noise prediction on the image to be processed, an image corresponding to the content described by the text is generated.

[0045] Server 102 can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server providing cloud computing services. Terminal 102 (also referred to as a user terminal or user device) can be, but is not limited to, a smartphone, tablet computer, laptop computer, desktop computer, intelligent voice interaction device (e.g., smart speaker), wearable electronic device (e.g., smart watch), in-vehicle terminal, smart home appliance (e.g., smart TV), AR / VR device, etc. The terminal and server can be connected directly or indirectly via wired or wireless communication, which is not limited in this disclosure.

[0046] Optionally, the data processing method provided in the embodiment of the present application can be implemented as an independent application or a functional module / plug-in of an application. For example, the application can be an application with data processing function.

[0047] The present application provides a data processing method, such as Figure 2 As shown, the method includes S201-S205.

[0048] S201 , extracting features from the data to be processed and the reference data used for performing target processing on the data to be processed, respectively, to obtain a first feature map of the data to be processed and a second feature map of the parameter data.

[0049] The data to be processed and the reference data are the same or different; the sizes of the first feature map and the second feature map are , h represents the number of eigenvectors included in the feature map, and m represents the number of eigenvalues ​​included in each eigenvector in the feature map.

[0050] In the embodiments of the present application, the data to be processed and the reference data can be data such as images or text, or feature maps obtained by processing during the intermediate stages of a neural network model. The target processing can be determined based on actual needs. For example, in a text map, the target processing can be denoising a noisy image. In text translation, the target processing can be semantically understanding text data to obtain a translation result. In image classification, the target processing can be determining the category to which an image belongs.

[0051] When the data to be processed is the same as the reference data, the embodiment of the present application determines the attention weights of each feature of the data to be processed relative to each feature of the data itself based on the self-attention mechanism by executing steps S201-S205, and obtains the target data after the target operation is performed on the data to be processed. When the data to be processed is different from the reference data, the embodiment of the present application determines the attention weights of each feature of the data to be processed relative to each feature of the reference data based on the cross-attention mechanism by executing steps S201-S205, and obtains the target data after the target operation is performed on the data to be processed.

[0052] In an embodiment of the present application, a feature extraction module can be used to perform feature extraction on the processed data and the reference data, respectively. Optionally, a first feature extraction module can be used to perform feature extraction on the processed data to obtain a first feature map of the processed data, and a second feature extraction module can be used to perform feature extraction on the reference data to obtain a second feature map of the reference data. When the processed data and the reference data are feature maps, the feature map extraction module can be a linear transformation module.

[0053] Among them, the size of the first feature map and the second feature is , where h is used to represent the number of eigenvectors included in the feature map, and m is used to represent the number of eigenvalues ​​included in each eigenvector, that is, the dimension of the eigenvector.

[0054] As an optional embodiment, feature extraction can be performed on the processed data and the reference data respectively to obtain feature maps corresponding to the processed data and feature maps corresponding to the reference data, wherein the feature maps corresponding to the processed data and the reference data are then matrix transformed so that the feature maps after the matrix transformation have the same size, and the feature map after the matrix transformation of the processed data is used as the first feature map, and the feature map after the matrix transformation of the reference data is used as the second feature map.

[0055] In the embodiments of the present application, the first feature map can be taken as a query matrix, which can be referred to as Qurey or Q for short, and the second feature map can be taken as a key matrix, which can be referred to as Key or K for short. In the embodiments of the present application, the feature vectors corresponding to the respective features of the to-be-processed data and the feature vectors corresponding to the respective features of the reference data are obtained by performing feature extraction on the to-be-processed data and the reference data, so as to determine the attention matrix of the to-be-processed data and the reference data according to the respective feature vectors of the to-be-processed data and the respective feature vectors of the reference data.

[0056] S202, determine a first similarity matrix between the first feature map and the second feature map.

[0057] The element value of the element in the i-th row and the j-th column of the first similarity matrix represents the similarity between the i-th feature vector in the first feature map and the j-th feature vector in the second feature map.

[0058] In the embodiments of the present application, the first similarity matrix between the first feature map and the second feature map can be obtained by performing inner product operation on the transposition of the first feature map and the second feature map, that is, matrix multiplication operation. The first feature map is taken as a first matrix of , and the second feature map is taken as a second matrix of To obtain the first similarity matrix between the first feature map and the second feature map, the second feature map is transposed to obtain a third matrix of , so that the first matrix can be subjected to inner product operation with the third matrix, and the result of the inner product operation of the first matrix and the third matrix is taken as the first similarity matrix.

[0059] It should be noted that the element value of the element in the i-th row and the j-th column of the first similarity matrix is obtained by multiplying the element values of the respective elements in the i-th row of the first matrix and the original values of the respective elements in the j-th column of the third matrix and then adding them up, that is, the result of the inner product operation of the vector represented by the i-th row of the first matrix and the vector represented by the j-th column of the third matrix, the inner product operation of the two vectors is used to calculate the similarity of the two vectors, and the vector represented by the i-th row of the first matrix is the i-th feature vector of the first feature map, and the vector represented by the j-th column of the third matrix is the j-th feature vector of the second feature map, so that the first similarity matrix of the first feature map and the second feature map can be obtained by performing inner product operation on the transposition of the first feature map and the second feature map.

[0060] As an optional embodiment, the cosine similarity, Euclidean distance or Manhattan distance between the vector represented by the i-th row of the first matrix and the vector represented by the j-th column of the third matrix can also be taken as the element value of the element in the i-th row and the j-th column of the first similarity matrix.

[0061] As an optional embodiment, a fourth matrix can be obtained by performing an inner product operation on the transpose of the first feature map and the second feature map, and each element in the fourth matrix can be divided by the scaling factor to obtain a matrix after scaling the fourth matrix as the first similarity matrix. Considering that when the dimension of each feature in the first feature map and the second feature map is large, their inner product may be very large, and the effect of subsequent sparsification is poor, the dimension of each feature can be As the scaling factor, the scaled matrix is ​​used as the first similarity matrix.

[0062] In an embodiment of the present application, the similarity between different features in the first feature map and the second feature map is obtained by determining a first similarity matrix, so as to obtain the attention matrix of the first feature map and the second feature map through subsequent sparsification processing.

[0063] S203 , obtaining a similarity threshold, subtracting the elements in the first similarity matrix from the similarity threshold to obtain a second similarity matrix, and setting the elements in the second similarity matrix that are less than 0 to 0 to obtain a third similarity matrix.

[0064] In an embodiment of the present application, the similarity threshold may be a preset value, and the phase velocity threshold may be used as an adjustable hyperparameter in the neural network model. The similarity threshold may be obtained by obtaining the value preset by the technician for the hyperparameter.

[0065] As an optional embodiment, the similarity threshold may also be obtained according to element values ​​of the elements of the first similarity matrix, and the average value of each element in the first similarity matrix may be used as the similarity threshold.

[0066] In an embodiment of the present application, a second similarity matrix is ​​obtained by subtracting the element values ​​of the elements in the first similarity matrix from the similarity threshold, and the element values ​​of the elements in the second similarity matrix that are less than 0 are set to 0 to obtain a third similarity matrix. The similarity threshold is used to divide the elements in the first similarity matrix into elements with larger values ​​and elements with smaller values. The elements with smaller values ​​have a value less than 0 after subtracting the similarity threshold. By setting the elements less than 0 in the second similarity matrix to 0, the elements with smaller values ​​in the first similarity matrix are set to 0. The amount of computation can be reduced by reducing the number of 0 elements in the similarity matrix, thus achieving preliminary sparsification processing. The amount of computation can be reduced by limiting the focus to only a few positions with higher similarity, avoiding calculation of unimportant elements.

[0067] As an optional embodiment of the present application, subtracting the elements in the first similarity matrix from the similarity threshold to obtain the second similarity matrix includes:

[0068] Obtaining a similarity threshold corresponding to each row in the first similarity matrix;

[0069] For each row of the first similarity matrix, subtract the element value of each element in the row from the similarity threshold corresponding to the row to obtain a second similarity matrix;

[0070] For each row of the first similarity matrix, the similarity threshold corresponding to the row is determined by:

[0071] Determine the maximum element value in the row;

[0072] A preset bias coefficient is obtained, and the maximum element value in the row is adjusted using the bias coefficient to obtain a similarity threshold corresponding to the row, wherein the bias coefficient is a positive number greater than or equal to 0 and less than 1.

[0073] In an embodiment of the present application, for a eigenvector in the first feature map, the numerical range of the similarity between the eigenvector and each eigenvector in the second feature map may be different from the numerical range of the similarity between another eigenvector in the first feature map and each eigenvector in the second feature map. Therefore, in order to accurately distinguish the numerical size of the similarity of each eigenvector in the first feature map with respect to each eigenvector in the second feature map, a corresponding similarity threshold is set for each row of the first similarity matrix.

[0074] In the embodiment of the present application, for each row of the first pixel matrix, the maximum element value max in the row is obtained, the preset bias coefficient α is obtained, and As the similarity threshold of the row, the value range of α is [0,1), that is, α is a positive number greater than or equal to 0 and less than 1. Figure 3 As shown, Figure 3 A schematic diagram of determining a third similarity matrix is ​​provided in an embodiment of the present application, wherein the first similarity matrix is ​​a 4×4 matrix. For each row in the matrix, the maximum value of the row is determined, and the bias coefficient is 0.5 to obtain the similarity threshold of each row. The similarity threshold of the corresponding row is subtracted from each element in the first similarity matrix to obtain the second similarity matrix.

[0075] As an optional embodiment, the ReLU activation function can be used to set the element values ​​less than 0 in the second similarity matrix to 0 to obtain a third similarity matrix. It can be seen from the third similarity matrix that the embodiment of the present application achieves the sparsification of the first similarity matrix.

[0076] As an optional embodiment, the specific value of α can be determined according to the target processing operation. For example, the data processing method provided in the embodiment of the present application can be applied to a stable diffusion model. In this case, the target processing is denoising processing, and the value range of α can be (0.9, 0.95). This value range is obtained based on specific experiments, and the target data obtained based on the bias coefficient within this value range has better effect.

[0077] By As the similarity threshold of the row, the other element values ​​in the row are subtracted from the similarity threshold. If the subtraction result is a negative number, it means that the similarity represented by the element value is small. The similarity can be set to 0 to achieve preliminary sparsification of the first similarity matrix. If the subtraction result is a positive number, it means that the similarity represented by the element value is large. The subtraction result corresponding to the element value can be included to obtain a third similarity matrix. There may still be some smaller element values ​​in the third similarity matrix, and the third similarity matrix can be further sparsified through subsequent operations.

[0078] S204. Obtain an attention matrix between the data to be processed and the reference data by performing at least one self-multiplication operation on the element value of each element in the third similarity matrix.

[0079] In an embodiment of the present application, the element value of each element in the third similarity matrix is ​​subjected to at least one self-multiplication operation, wherein one self-multiplication operation is a square operation, and two self-multiplication operations are cubic operations. By replacing the exponential operation in the Softmax operation with at least one self-multiplication operation, the algorithm complexity is reduced, the efficiency of the attention mechanism processing is improved, and the attention matrix between the data to be processed and the reference data is obtained.

[0080] In the embodiment of the present application, by performing a self-multiplication operation on each element in the third similarity matrix, elements smaller than 1 become smaller, thereby achieving further sparsification of the third similarity matrix.

[0081] As an optional embodiment, at least one self-multiplication operation is a square operation. The square operation is more user-friendly and efficient for hardware such as the end-side device NPU and TPU that are optimized for multiplication and addition operations. That is, the end-side device NPU and TPU are more efficient when processing square operations than processing exponential operations.

[0082] As an optional embodiment of the present invention, an attention matrix between the to-be-processed data and the reference data is obtained by performing at least one self-multiplication operation on the element value of each element in the third similarity matrix, including:

[0083] The fourth similarity matrix is obtained by performing at least one self-multiplication operation on the element value of each element in the third similarity matrix.

[0084] A normalization coefficient is obtained, and a normalization operation is performed on the fourth similarity matrix according to the normalization coefficient to obtain the attention matrix.

[0085] In the embodiment of the present application, after at least one self-multiplication operation is performed on each element in the third similarity matrix, the element less than 1 will become smaller and the element greater than 1 will become larger. In order to scale the result of at least one self-multiplication operation on the element to the range of 0-1, the result of at least one self-multiplication operation on each element in the third similarity matrix is taken as the fourth similarity matrix, and the element in the fourth similarity matrix is normalized.

[0086] It needs to be understood that, in the embodiment of the present application, the attention matrix is obtained by processing the first similarity matrix, which is used to replace the Softmax operation in the attention mechanism, and the value range of the element in the matrix processed by the Softmax operation is 0-1. In order to not change the value range of the input data corresponding to the subsequent operation after the Softmax operation in the attention mechanism, the element in the fourth similarity matrix is scaled to the range of 0-1, that is, the fourth similarity matrix is normalized, and the normalized matrix is taken as the attention matrix.

[0087] The value range of the normalization coefficient β can be (0, 1), which is used to scale the element value of the element greater than 1 in the fourth similarity matrix to the range of 0-1. The normalization coefficient can be used as a hyperparameter in the neural network model. The technician can set the hyperparameter according to actual needs. When the neural network model is running, the value of β set will be read, and the normalization is performed based on the value. Optionally, the value of the normalization coefficient β can be 0.01. Experiments prove that the effect of the target data generated based on the normalization coefficient with the value is better.

[0088] As an optional embodiment, when the normalization operation is performed on the fourth similarity matrix, each element in the fourth similarity matrix can be multiplied by the normalization coefficient to obtain the normalization result of each element, thereby obtaining the attention matrix.

[0089] As an optional embodiment of the present application, the normalization operation is performed on the fourth similarity matrix, comprising:

[0090] For each row of the fourth similarity matrix, a normalization coefficient corresponding to the row is determined according to the element value of at least one element in the row.

[0091] For each element of each row of the fourth similarity matrix, the element is adjusted to the range of 0-1 according to the normalization coefficient corresponding to the row.

[0092] In an embodiment of the present application, the numerical ranges of the element values ​​of the elements in each row in the fourth similarity matrix may be different. By setting a normalization coefficient for each row, the normalized matrix can reflect the difference in similarity of a certain eigenvector in the first feature map with respect to each eigenvector in the second feature map.

[0093] For each row in the fourth similarity matrix, a normalization coefficient for the row is determined based on the value of at least one element in the row. Specifically, the maximum element value (Square max) in the row and a preset adjustment coefficient γ can be obtained, and the ratio of γ to Square max is used as the normalization coefficient for the row, where the value of γ can be 0.1. For each element in each row of the fourth similarity matrix, the element is multiplied by the normalization coefficient for the row to adjust the element to a range of 0-1.

[0094] like Figure 4 As shown, Figure 4 A schematic diagram of obtaining an attention matrix is ​​provided in an embodiment of the present application. For the third similarity matrix, the element value of each element in the third similarity matrix is ​​squared to obtain a fourth similarity matrix. For each row in the fourth similarity matrix, the maximum value of the row is determined, and the adjustment coefficient is taken as 0.1 to obtain the normalization coefficient of each row. For each element of the fourth similarity matrix, the element value of the element is multiplied by the normalization coefficient of the corresponding row (for convenience of representation, four significant digits are retained) to obtain the attention matrix.

[0095] In the embodiment of the present application, a Softmax approximation operation is implemented through steps S203 and S204, which implements the sparsification of the first similarity matrix, and the processor uses the approximate operation for sparse processing more efficiently than the sparse processing through Softmax. Furthermore, for processors such as the end-side NPU or TPU based on optimized multiplication and addition, the approximate operation can greatly improve hardware utilization and increase computing speed. In addition, the approximate solution supports processors that only have fixed-point operations, such as the Zhouyi Z series and X1 NPU processors, where fixed-point operations mean that the representation of numbers is fixed during operations, that is, the accuracy and range of all numerical values ​​are predetermined.

[0096] For an M × N Feature Map A1, the calculation steps of the Softmax operation are:

[0097] S1. For each row of matrix A1, find the maximum value of N data;

[0098] S2. For each data in matrix A1, subtract the maximum value of the corresponding row to obtain matrix A2;

[0099] S3, for each data of the matrix A2 , find , get the matrix A3;

[0100] S4, for each row of the matrix A3, find the sum of all data of the row;

[0101] S5, for each data of the matrix A3, divide the data by the sum of all data of the row.

[0102] For an MxN Feature Map B1, the operation steps of the approximate operation of the Softmax operation in the embodiment of the application are as follows:

[0103] S1, for each row of the matrix B1, find the maximum value of N data;

[0104] S2, obtain the bias coefficient a e [0, 1);

[0105] S3, for each data of the matrix B1, subtract the maximum value of the row multiplied by a from the data to obtain the matrix B2;

[0106] S4, for each data in the matrix B2, execute the ReLU activation function on the data to obtain the matrix B3;

[0107] S5, execute the square operation on each data in B3 to obtain the matrix B4;

[0108] S6, for each row of the matrix B4, find the maximum value of N data;

[0109] S7, for each data of B4, multiply the data by the normalization coefficient β, β = 0.1 / (the maximum value of the row).

[0110] The core of this approximate operation is to replace the sparse process Exp operation with a square operation, and the square multiplication is more friendly and efficient for the end-side NPU, TPU and other hardware optimized for multiplication and addition operations. For fine-tuned diffusion models such as InstaFlow, images can be generated in one step of iteration. Such models avoid the problem of large differences in data distribution caused by multiple iterations of the same operator. When processing, S1, S2, and S3 steps and S6 and S7 steps can be integrated into the offline quantization process and saved in advance to save the time of traversing each row of data during inference. Therefore, the approximate operation process is simplified to four steps:

[0111] S1, for the matrix B1, obtain the maximum value of each row in the B1 for each data of the matrix B1, subtract the maximum value of the row multiplied by a from the data to obtain the matrix B2;

[0112] S2, perform the ReLU activation function on each data in the matrix B2 to obtain the matrix B3;

[0113] S3, perform self-multiplication on each data in matrix B3, that is, square operation, to obtain matrix B4;

[0114] S4. For matrix B4, obtain β for each row in matrix B4, where β = 0.1 / (maximum value of the row). For each data in matrix B4, multiply the data by the corresponding β.

[0115] The principle behind the faster running of the Softmax approximation operation on the client platform in the embodiments of this application is that the number of unit calculations is reduced. Taking the Zhouyi NPU as an example, the floating-point operation process for implementing the Softmax operation is more complex than the additional operation process for implementing the approximate operation.

[0116] In the S3 step of the Softmax operation, each element of the matrix needs to be calculate , exponential calculations are very complex, especially in hardware implementation, and require complex computational processes. For example:

[0117] 1) Element-by-element traversal limits the size of the input: avoid input that is too large or too small to ensure computational stability;

[0118] 2) Look up the table and calculate the approximate value: Implemented by looking up the table Approximate calculation of , and perform two element-by-element subtractions and corrections;

[0119] 3) Multiple element-by-element multiplications and additions: Further approximating the exponential value requires at least two multiplications and one addition.

[0120] 4) Element-by-element multiplication: used to adjust the final result.

[0121] It can be seen that exponential operations require multiple table lookups, multiplications, and additions, and involve complex unit calculations in hardware.

[0122] In the approximate operation, S4 and S5 use ReLU activation function and square operation instead of traditional exponential operation. Specific calculation categories include:

[0123] 1) Calculate activation values ​​with a single table lookup: Implements the ReLU operation (setting negative values ​​to zero), which is computationally simple and hardware-friendly.

[0124] 2) Calculate the square by multiplying each element at a time: The square operation only requires one multiplication unit to complete, which is much simpler than the exponential operation.

[0125] In contrast, the computational complexity of the square approximation algorithm is significantly reduced because it avoids complex exponential calculations and is more efficient in hardware implementation, occupying fewer resources.

[0126] The Softmax approximation operation in the embodiment of the present application can be completed using fewer unit calculations in the Zhouyi NPU. Similarly, in other end-side computing architectures, if exponential calculations require more complex unit calculations than square calculations, the Softmax operation can be replaced based on this approximate operation in the end-side computing architecture to effectively improve the computing performance of the end-side computing architecture.

[0127] To compare the difference between the Softmax approximation operation and the Softmax operation in the embodiment of the present application in terms of the effect of sparse data, as shown in FIG. Figure 5 As shown, Figure 5 A data distribution comparison diagram after sparseness is provided in an embodiment of the present application, wherein the original data distribution diagram is used to represent the data distribution of a normally distributed random number sequence with a data volume of 2048. The random number sequence is used as the first similarity matrix and is processed based on the Softmax operation and the approximate operation in the embodiment of the present application. The data distribution obtained after the Softmax operation and the data distribution obtained after the approximate operation are shown as follows: Figure 5 As shown in the figure, analysis shows that the cosine similarity of the data distributions obtained by the Softmax operation and the approximate operation is 75%, with a mean square error (MSE) of 2.4e-4. Furthermore, the approximate operation effectively sparsifies the data, highlighting features near the maximum value. In terms of performance, the approximate operation requires approximately 40% fewer instructions than the Softmax implementation in the Zhouyi NPU simulator.

[0128] This approximation has the following advantages:

[0129] (1) Fast inference speed: The approximate algorithm reduces the computational complexity of the Softmax operator, effectively improving the running speed while ensuring the model effect.

[0130] (2) Wide range of applications: not limited to a specific end-side chip architecture, the approximate algorithm that reduces complexity is universal

[0131] (3) Easy to implement: Unlike other Softmax alternatives that require re-fine-tuning and retraining of the model, this approximate algorithm can be directly tuned and replaced on the inference side.

[0132] S205. Weight the second feature map based on the attention matrix to obtain a weighted feature map, and obtain target data after the target operation is performed on the data to be processed according to the weighted feature map.

[0133] In the embodiments of the present application, the second feature map can also be used as a value matrix in the attention mechanism, referred to as Value or V for short. By performing an inner product operation, i.e., a matrix multiplication operation, between the second feature map as a value matrix and a weight matrix in the attention mechanism, a feature map required for further processing can be generated.

[0134] As an optional embodiment of the present application, the second feature map includes a third feature map obtained by performing feature extraction on the reference data by the first module, and a fourth feature map obtained by performing feature extraction on the reference data by the second module.

[0135] determining a first similarity matrix between the first feature map and the second feature map, comprising:

[0136] performing an inner product operation on the transposes of the first feature map and the second feature map to obtain the first similarity matrix;

[0137] weighting the second feature map based on the attention matrix to obtain a weighted feature map, comprising:

[0138] weighting the fourth feature map by the attention matrix to obtain a weighted feature map.

[0139] In the embodiments of the present application, when performing feature extraction on the reference data, two different feature extraction modules can be used to obtain the third feature map and the fourth feature map. The first module can be a feature extraction module for obtaining a key matrix, and the second module can be a feature extraction module for obtaining a value matrix. For the to-be-processed data, a query matrix can be obtained by performing feature extraction on the to-be-processed data by a third module.

[0140] In the embodiments of the present application, the inner product operation on the transposes of the first feature map and the second feature map can be an inner product operation, i.e., a matrix multiplication operation, between the first feature map and the transpose of the third feature map to obtain the first similarity matrix, so as to obtain the relevance or similarity between the query matrix and the key matrix.

[0141] In the embodiments of the present application, the fourth feature map can be weighted by the attention matrix as a weighted feature map. By performing a weighted sum operation between the attention matrix and the value matrix, a weighted feature map can be obtained, which can better focus on important regions or features in the reference data, thereby improving the effectiveness of the target operation.

[0142] As an optional embodiment of the present application, the to-be-processed data is a to-be-processed image, the reference data is a content description text corresponding to the to-be-processed image, and the target processing is a denoising processing. The target data obtained by performing a target operation on the to-be-processed data according to the weighted feature map comprises:

[0143] Predicting a first noise in the image to be processed according to the weighted feature map;

[0144] De-noising is performed on the image to be processed according to the first noise to obtain a denoised target image.

[0145] In this embodiment of the present application, the data to be processed can be a noisy image generated in a stable diffusion model, the reference data can be content description text, and the target operation is a denoising operation. In the stable diffusion model, the content description text guides the step-by-step denoising of the noisy image to generate a target image, the content of which is the content described in the content description text.

[0146] Specifically, the stable diffusion model includes a Unet network, which denoises the noisy image through a downsampling structure and an upsampling structure. The Unet network includes a Transformer module, the most important part of which is a cross-attention module. The input of the cross-attention mechanism module is a noisy image and text. The cross-attention mechanism module uses the noisy image and text as the image to be processed and the content description text, respectively. Through steps S201-S204 in the embodiment of the present application, an attention matrix of the image to be processed and the content description text is obtained. After weighting the attention matrix with the fourth feature map of the content description text, a weighted feature map is obtained. Based on the weighted feature map, the Unet network predicts the first noise in the current image to be processed to obtain a prediction result of the first noise.

[0147] After obtaining the prediction result of the current image to be processed, the noise is removed by using the back diffusion process according to the prediction result of the first noise to generate a clearer image. The Unet network can repeatedly perform the denoising process until a target image with a clarity greater than a preset value is obtained.

[0148] In an embodiment of the present application, after denoising is completed, a denoised target image is obtained, the content and quality of which are significantly affected by the weighted feature map, that is, the weighted feature map helps the model focus on important image areas related to the text description.

[0149] In this embodiment of the present application, the Softmax approximation operation is applied to the stable diffusion model, improving the efficiency of the stable diffusion model in generating target images. The stable diffusion model Stable Diffusion 1.5 was used as a model for verification. The five Softmax locations with the largest amount of data to be processed in the model were determined. The approximate operation was used to replace any one of the five Softmax operations to obtain the generated target image. The approximate operation was used to replace any two of the five Softmax operations to obtain the generated target image. Experimental results show that the effect of the target image is similar to that of the Softmax operation, and that replacing Softmax with the approximate operation can significantly improve the model's inference speed on the end-side computing platform.

[0150] In order to more clearly illustrate the process of the data processing method provided in the embodiment of the present application, a complete embodiment of the data processing method of the present application is now given, please refer to Figure 6 , Figure 6 A flowchart of a data processing method provided in an embodiment of the present application includes steps S601-S610.

[0151] S601, extracting features from the data to be processed and the reference data to obtain a first feature map of the data to be processed and a third feature map and a fourth feature map of the parameter data;

[0152] S602: Perform an inner product operation on the first feature map and the third feature map to obtain a first similarity matrix;

[0153] S603: Obtain a similarity threshold corresponding to each row in the first similarity matrix according to the maximum element value and the bias coefficient of each row in the first similarity matrix;

[0154] S604: Subtract each element value in the first similarity matrix from the similarity threshold of the corresponding row to obtain a second similarity matrix;

[0155] S605: Set the elements in the second similarity matrix that are less than 0 to 0 to obtain a third similarity matrix;

[0156] S606: Perform a square operation on each element in the third similarity matrix to obtain a fourth similarity matrix;

[0157] S607: Obtain a normalization coefficient corresponding to each row of the fourth similarity matrix according to the maximum element value and the adjustment coefficient of each row of the fourth similarity matrix;

[0158] S608: Multiply each element in the fourth similarity matrix by the normalization coefficient of the corresponding row to obtain an attention matrix;

[0159] S609: Weighting the fourth feature map using the attention matrix to obtain a weighted feature map;

[0160] S610: Obtain target data after performing a target operation on the data to be processed according to the weighted feature map.

[0161] The present application embodiment provides a data processing device, such as Figure 7 As shown, the data processing device may include: a feature extraction module 701, a similarity determination module 702, a similarity threshold acquisition module 703, an attention matrix acquisition module 704 and a weighting module 705, wherein,

[0162] The feature extraction module 701 is used to extract features from the data to be processed and the reference data used to perform target processing on the data to be processed, respectively, to obtain a first feature map of the data to be processed and a second feature map of the parameter data, wherein the data to be processed and the reference data are the same or different; wherein the sizes of the first feature map and the second feature map are , h represents the number of eigenvectors included in the feature map, and m represents the number of eigenvalues ​​included in each eigenvector in the feature map;

[0163] A similarity determination module 702 is configured to determine a first similarity matrix between the first feature map and the second feature map, wherein the element value of the element in the i-th row and j-th column of the first similarity matrix represents the similarity between the i-th eigenvector in the first feature map and the j-th eigenvector in the second feature map;

[0164] A similarity threshold obtaining module 703 is configured to obtain a similarity threshold, subtract the elements in the first similarity matrix from the similarity threshold to obtain a second similarity matrix, and set the elements in the second similarity matrix that are less than 0 to 0 to obtain a third similarity matrix;

[0165] an attention matrix obtaining module 704, configured to obtain an attention matrix between the to-be-processed data and the reference data by performing at least one self-multiplication operation on the element value of each element in the third similarity matrix;

[0166] The weighting module 705 is used to weight the second feature map based on the attention matrix to obtain a weighted feature map, and obtain the target data after the target operation is performed on the data to be processed according to the weighted feature map.

[0167] The device of the embodiment of the present application can execute the method provided by the embodiment of the present application, and its implementation principle is similar. The actions performed by each module in the device of each embodiment of the present application correspond to the steps in the method of each embodiment of the present application. For the detailed functional description of each module of the device, please refer to the description in the corresponding method shown in the previous text, and will not be repeated here.

[0168] As an optional embodiment of the present application, when the similarity threshold obtaining module is used to obtain the second similarity matrix by subtracting the elements in the first similarity matrix from the similarity threshold, it is specifically used to:

[0169] Obtaining a similarity threshold corresponding to each row in the first similarity matrix;

[0170] For each row of the first similarity matrix, subtract the element value of each element in the row from the similarity threshold corresponding to the row to obtain a second similarity matrix;

[0171] For each row of the first similarity matrix, the similarity threshold corresponding to the row is determined by:

[0172] Determine the maximum element value in the row;

[0173] A preset bias coefficient is obtained, and the maximum element value in the row is adjusted using the bias coefficient to obtain a similarity threshold corresponding to the row, wherein the bias coefficient is a positive number greater than or equal to 0 and less than 1.

[0174] As an optional embodiment of the present application, when the attention matrix obtaining module is used to obtain the attention matrix between the to-be-processed data and the reference data by performing at least one self-multiplication operation on the element value of each element in the third similarity matrix, the module is specifically used to:

[0175] Obtaining a fourth similarity matrix by performing at least one self-multiplication operation on the element value of each element in the third similarity matrix;

[0176] Obtain a normalization coefficient, and perform a normalization operation on the fourth similarity matrix according to the normalization coefficient to obtain the attention matrix.

[0177] As an optional embodiment of the present application, when the attention matrix obtaining module is used to perform a normalization operation on the fourth similarity matrix, it is specifically used to:

[0178] For each row of the fourth similarity matrix, determining a normalization coefficient corresponding to the row according to an element value of at least one element in the row;

[0179] For each element of each row of the fourth similarity matrix, the element is adjusted to a range of 0-1 according to the normalization coefficient corresponding to the row.

[0180] As an optional embodiment of the present application, the second feature map includes a third feature map obtained by extracting features from the reference data through the first module, and a fourth feature map obtained by extracting features from the reference data through the second module;

[0181] When the similarity determination module is used to determine the first similarity matrix between the first feature map and the second feature map, it is specifically used to:

[0182] Performing an inner product operation on the transpose of the first feature map and the third feature map to obtain the first similarity matrix;

[0183] The weighting module, when used to weight the second feature map based on the attention matrix to obtain a weighted feature map, is specifically used to:

[0184] The fourth feature map is weighted by the attention matrix to obtain the weighted feature map.

[0185] As an optional embodiment of the present application, the data to be processed is an image to be processed, the reference data is a content description text corresponding to the image to be processed, and the target processing is denoising processing; when the weighting module is used to obtain target data after the target operation is performed on the data to be processed based on the weighted feature map, it is specifically used to:

[0186] Predicting a first noise in the image to be processed according to the weighted feature map;

[0187] According to the first noise, denoising is performed on the image to be processed to obtain a denoised target image.

[0188] An embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored on the memory. The processor executes the above-mentioned computer program to implement the steps of the data processing method. Compared with the related art, it can achieve: by extracting features from the data to be processed and the reference data, a first feature map of the data to be processed and a second feature map of the reference data are obtained, wherein the data to be processed and the reference data can be the same or different, so that the method provided in the embodiment of the present application can be applied to the self-attention mechanism and the cross-attention mechanism. By determining a first similarity matrix between the first feature map and the second feature map, the degree of association between the data to be processed and the reference data in the feature space is obtained; through a similarity threshold, the elements in the first similarity matrix are subtracted from the similarity threshold to obtain a second similarity matrix, and the elements less than 0 in the second similarity matrix are set to 0 to obtain a third similarity matrix, so that the obtained third similarity matrix retains feature pairs with high similarity; by performing at least one self-multiplication operation on the element value of each element in the third similarity matrix, the third similarity matrix is ​​sparsely processed to obtain an attention matrix between the data to be processed and the reference data; based on the attention matrix, the second feature map is weighted to obtain a weighted feature map, so that the model can perform prediction and inference based on the weighted feature map to obtain the target data after the target operation is performed on the data to be processed. In an embodiment of the present application, by obtaining a similarity threshold, the first similarity matrix is ​​converted into a third similarity matrix with more 0 elements. By performing a self-multiplication operation on each element in the third similarity matrix, elements with smaller values ​​are made closer to 0, thereby achieving sparseness of the first similarity matrix. Compared with the exponential operation in the Softmax operation, the self-multiplication operation of the elements has lower computational complexity and better numerical stability, thereby effectively reducing the consumption of computing resources.

[0189] In an alternative embodiment, an electronic device is provided, such as Figure 8 As shown, Figure 8 The electronic device 4000 shown includes: a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, via a bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, which may be used for data exchange between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the number of transceivers 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present application.

[0190] Processor 4001 can be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 4001 can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0191] Bus 4002 may include a path for transmitting information between the above components. Bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. Bus 4002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0192] The memory 4003 may be a ROM (Read Only Memory) or other type of static storage device that can store static information and instructions, a RAM (Random Access Memory) or other type of dynamic storage device that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium that can be used to carry or store computer programs and can be read by a computer, without limitation herein.

[0193] The memory 4003 is used to store the computer program for executing the embodiment of the present application, and the execution is controlled by the processor 4001. The processor 4001 is used to execute the computer program stored in the memory 4003 to implement the steps shown in the above method embodiment.

[0194] Among them, electronic devices may include but are not limited to mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., as well as fixed terminals such as digital TVs, desktop computers, etc. Figure 8 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0195] The present invention provides a computer-readable storage medium having a computer program stored thereon. When executed by a processor, the computer program can implement the steps and corresponding contents of the aforementioned method embodiment. Compared with the prior art, the present invention can achieve the following: by extracting features from the to-be-processed data and the reference data, a first feature map of the to-be-processed data and a second feature map of the reference data are obtained. The to-be-processed data and the reference data can be the same or different, so that the method provided in the present invention can be applied to both the self-attention mechanism and the cross-attention mechanism. By determining a first similarity matrix between the first feature map and the second feature map, the degree of association between the data to be processed and the reference data in the feature space is obtained; through a similarity threshold, the elements in the first similarity matrix are subtracted from the similarity threshold to obtain a second similarity matrix, and the elements less than 0 in the second similarity matrix are set to 0 to obtain a third similarity matrix, so that the obtained third similarity matrix retains feature pairs with high similarity; by performing at least one self-multiplication operation on the element value of each element in the third similarity matrix, the third similarity matrix is ​​sparsely processed to obtain an attention matrix between the data to be processed and the reference data; based on the attention matrix, the second feature map is weighted to obtain a weighted feature map, so that the model can perform prediction and inference based on the weighted feature map to obtain the target data after the target operation is performed on the data to be processed. In an embodiment of the present application, by obtaining a similarity threshold, the first similarity matrix is ​​converted into a third similarity matrix with more 0 elements. By performing a self-multiplication operation on each element in the third similarity matrix, elements with smaller values ​​are made closer to 0, thereby achieving sparseness of the first similarity matrix. Compared with the exponential operation in the Softmax operation, the self-multiplication operation of the elements has lower computational complexity and better numerical stability, thereby effectively reducing the consumption of computing resources.

[0196] It should be noted that the computer-readable medium mentioned in the present disclosure may be a computer-readable signal medium or a computer-readable medium, or any combination thereof. Computer-readable storage media may include, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or components, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such a propagated data signal may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wire, optical cable, RF (radio frequency), or any suitable combination thereof.

[0197] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps and corresponding contents of the aforementioned method embodiment. Compared with the prior art, the present application can achieve: by extracting features from the to-be-processed data and the reference data, a first feature map of the to-be-processed data and a second feature map of the reference data are obtained, wherein the to-be-processed data and the reference data can be the same or different, so that the method provided in the present application embodiment can be applied to both the self-attention mechanism and the cross-attention mechanism. By determining a first similarity matrix between the first feature map and the second feature map, the degree of association between the data to be processed and the reference data in the feature space is obtained; through a similarity threshold, the elements in the first similarity matrix are subtracted from the similarity threshold to obtain a second similarity matrix, and the elements less than 0 in the second similarity matrix are set to 0 to obtain a third similarity matrix, so that the obtained third similarity matrix retains feature pairs with high similarity; by performing at least one self-multiplication operation on the element value of each element in the third similarity matrix, the third similarity matrix is ​​sparsely processed to obtain an attention matrix between the data to be processed and the reference data; based on the attention matrix, the second feature map is weighted to obtain a weighted feature map, so that the model can perform prediction and inference based on the weighted feature map to obtain the target data after the target operation is performed on the data to be processed. In an embodiment of the present application, by obtaining a similarity threshold, the first similarity matrix is ​​converted into a third similarity matrix with more 0 elements. By performing a self-multiplication operation on each element in the third similarity matrix, elements with smaller values ​​are made closer to 0, thereby achieving sparseness of the first similarity matrix. Compared with the exponential operation in the Softmax operation, the self-multiplication operation of the elements has lower computational complexity and better numerical stability, thereby effectively reducing the consumption of computing resources.

[0198] In the specification and claims of this application and the accompanying drawings, the terms "first," "second," "third," "fourth," "1," "2," and so on (if any) are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that the terms used in this manner are interchangeable where appropriate, such that the embodiments of the present application described herein can be practiced in an order other than that shown or described.

[0199] It should be understood that, although each operation step is indicated by arrows in the flowchart of the embodiment of the present application, the order of implementation of these steps is not limited to the order indicated by the arrows. Unless otherwise clearly stated herein, in some implementation scenarios of the embodiment of the present application, the implementation steps in each flowchart can be performed in other orders according to demand. In addition, some or all of the steps in each flowchart can include multiple sub-steps or multiple stages based on actual implementation scenarios. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage in these sub-steps or stages can also be executed at different times respectively. Under different scenarios at the execution time, the execution order of these sub-steps or stages can be flexibly configured according to demand, and the embodiment of the present application does not limit this.

[0200] The above description is only an optional implementation method for some implementation scenarios of this application. It should be pointed out that for ordinary technicians in this technical field, without departing from the technical concept of the solution of this application, the use of other similar implementation methods based on the technical ideas of this application also falls within the protection scope of the embodiments of this application.

Claims

1. A data processing method, characterized in that: Executed by a processor, the method includes: By performing feature extraction on the data to be processed and the reference data used for target processing of the data to be processed, a first feature map of the data to be processed and a second feature map of the reference data are obtained, wherein the data to be processed and the reference data are the same or different; wherein the sizes of the first feature map and the second feature map are , Indicates the number of eigenvectors included in the feature map, Indicates the number of eigenvalues ​​included in each eigenvector in the feature map; the data to be processed is an image or text; the reference data is an image or text; Determine a first similarity matrix between the first feature map and the second feature map, where an element value of an element in the i-th row and j-th column of the first similarity matrix represents a similarity between an i-th eigenvector in the first feature map and a j-th eigenvector in the second feature map; Obtaining a similarity threshold, subtracting the elements in the first similarity matrix from the similarity threshold to obtain a second similarity matrix, and setting the elements in the second similarity matrix that are less than 0 to 0 to obtain a third similarity matrix; Obtaining an attention matrix between the data to be processed and the reference data by performing at least one self-multiplication operation on the element value of each element in the third similarity matrix; The second feature map is weighted based on the attention matrix to obtain a weighted feature map, and target data after the target operation is performed on the data to be processed is obtained according to the weighted feature map.

2. The method according to claim 1, characterized in that Subtracting the elements in the first similarity matrix from the similarity threshold to obtain a second similarity matrix includes: Obtaining a similarity threshold corresponding to each row in the first similarity matrix; For each row of the first similarity matrix, subtract the element value of each element in the row from the similarity threshold corresponding to the row to obtain a second similarity matrix; For each row of the first similarity matrix, the similarity threshold corresponding to the row is determined by: Determine the maximum element value in the row; A preset bias coefficient is obtained, and the maximum element value in the row is adjusted using the bias coefficient to obtain a similarity threshold corresponding to the row, wherein the bias coefficient is a positive number greater than or equal to 0 and less than 1.

3. The method according to claim 1, characterized in that The step of performing at least one self-multiplication operation on the element value of each element in the third similarity matrix to obtain an attention matrix between the data to be processed and the reference data includes: Obtaining a fourth similarity matrix by performing at least one self-multiplication operation on the element value of each element in the third similarity matrix; Obtain a normalization coefficient, and perform a normalization operation on the fourth similarity matrix according to the normalization coefficient to obtain the attention matrix.

4. The method according to claim 3, characterized in that The normalizing operation on the fourth similarity matrix includes: For each row of the fourth similarity matrix, determining a normalization coefficient corresponding to the row according to an element value of at least one element in the row; For each element of each row of the fourth similarity matrix, the element is adjusted to a range of 0-1 according to a normalization coefficient corresponding to the row.

5. The method according to claim 1, wherein The second feature map includes a third feature map obtained by extracting features from the reference data through the first module, and a fourth feature map obtained by extracting features from the reference data through the second module; The determining a first similarity matrix between the first feature map and the second feature map includes: Performing an inner product operation on the transpose of the first feature map and the third feature map to obtain the first similarity matrix; The weighting the second feature map based on the attention matrix to obtain a weighted feature map includes: The fourth feature map is weighted by the attention matrix to obtain the weighted feature map.

6. The method according to any one of claims 1 to 5, characterized in that The data to be processed is an image to be processed, the reference data is a content description text corresponding to the image to be processed, the target processing is denoising processing, and obtaining target data after performing a target operation on the data to be processed based on the weighted feature map includes: Predicting a first noise in the image to be processed according to the weighted feature map; De-noising is performed on the image to be processed according to the first noise to obtain a denoised target image.

7. A data processing device, characterized in that: The device comprises: The feature extraction module is used to extract features from the data to be processed and the reference data used to perform target processing on the data to be processed, respectively, to obtain a first feature map of the data to be processed and a second feature map of the reference data, wherein the data to be processed and the reference data are the same or different; wherein the sizes of the first feature map and the second feature map are , h represents the number of eigenvectors included in the feature map, m represents the number of eigenvalues ​​included in each eigenvector in the feature map; the data to be processed is an image or text; the reference data is an image or text; A similarity determination module is configured to determine a first similarity matrix between the first feature map and the second feature map, wherein the element value of the element in the i-th row and j-th column in the first similarity matrix represents the similarity between the i-th eigenvector in the first feature map and the j-th eigenvector in the second feature map; a similarity threshold obtaining module, configured to obtain a similarity threshold, subtract the elements in the first similarity matrix from the similarity threshold to obtain a second similarity matrix, and set the elements in the second similarity matrix that are less than 0 to 0 to obtain a third similarity matrix; an attention matrix obtaining module, configured to obtain an attention matrix between the to-be-processed data and the reference data by performing at least one self-multiplication operation on the element value of each element in the third similarity matrix; A weighting module is used to weight the second feature map based on the attention matrix to obtain a weighted feature map, and obtain target data after the target operation is performed on the data to be processed according to the weighted feature map.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images