Processing method, device, and convolutional neural network model for self-attention mechanism

Abstract

The present application relates to the field of artificial intelligence and provides a method, device, and convolutional neural network model for processing a self-attention mechanism, which obtains an input feature matrix; processes the input feature matrix to obtain a corresponding query matrix, key matrix, and value matrix; multiplies the query matrix and the key matrix to obtain a first matrix; performs a spectral convolution operation on the first matrix to obtain a second matrix; multiplies the second matrix and the value matrix to obtain a third matrix; and determines the target output of the self-attention mechanism based on the third matrix. The method, device, and convolutional neural network model for processing the self-attention mechanism provided in the present application can improve the contribution of sub-probability scores and avoid erroneous prediction results caused by deviations in the calculation of the region obtained by the maximum probability.

Description

Technical Field

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a processing method, device, and convolutional neural network model for a self-attention mechanism. Background Art

[0002] The self-attention mechanism has been extensively studied in the image domain and has achieved advantages in several tasks, such as non-local bloc keys and augmented attention. The classic self-attention mechanism generates three matrices: query, key, and value, derived from the same input. The dot product between the query and key matrices is first calculated. To prevent the result from being too large, the result is divided by a scale, such as the dimensions of the query and key vectors. The result is then normalized to a probability distribution using the softmax (normalized exponential) function and multiplied by the matrix to obtain a weighted summation representation. The matrix constructed by the dot product of the query and key matrices represents the distance relationship between spatial regions in the image, effectively modeling the spatial dependencies between regions. The dependencies between regions are probabilistically expressed using the softmax function, which maximizes the probability score of a region by compressing the probability scores of other regions. This function undoubtedly emphasizes high probability scores, with the region with the highest probability score determining the final attention score. However, this also reduces the contribution of sub-probability scores. Furthermore, any deviation in the calculation of the region with the highest probability score can directly lead to erroneous predictions. Summary of the Invention

[0003] The main purpose of this application is to provide a processing method, device, and convolutional neural network model for a self-attention mechanism, aiming to solve the technical problem that the softmax function in the self-attention mechanism reduces the contribution of sub-probability scores, and at the same time, the regional calculation of the maximum probability score is biased, resulting in erroneous prediction results.

[0004] To achieve the above objectives, this application provides a processing method for the self-attention mechanism, which is applied to the preset self-attention mechanism and includes the following steps:

[0005] Get the input feature matrix;

[0006] Processing the input feature matrix to obtain corresponding query matrix, key matrix and value matrix;

[0007] Multiplying the query matrix and the key matrix to obtain a first matrix;

[0008] Performing a spectral convolution operation on the first matrix to obtain a second matrix;

[0009] Multiplying the second matrix by the value matrix to obtain a third matrix;

[0010] A target output of the self-attention mechanism is determined according to the third matrix.

[0011] Furthermore, the step of performing a convolution operation on the first matrix through spectral convolution to obtain a second matrix includes:

[0012] Get the parameter matrix of spectral convolution;

[0013] Performing a convolution operation on the first matrix and the parameter matrix to obtain the second matrix.

[0014] Furthermore, the step of obtaining a parameter matrix of spectral convolution includes:

[0015] Get the convolution parameter matrix;

[0016] The convolution parameter matrix is ​​spectrally normalized by using the spectral norm to obtain the parameter matrix of the spectral convolution.

[0017] Furthermore, the step of determining the target output of the self-attention mechanism according to the third matrix includes:

[0018] The third matrix is ​​residually added to the input feature matrix to obtain the target output of the self-attention mechanism.

[0019] This application also provides a processing device for a self-attention mechanism, including:

[0020] An acquisition unit, used to obtain an input feature matrix;

[0021] An input unit, configured to process the input feature matrix to obtain a corresponding query matrix, a key matrix, and a value matrix;

[0022] A first multiplication unit, configured to multiply the query matrix and the key matrix to obtain a first matrix;

[0023] a convolution operation unit, configured to perform a convolution operation on the first matrix through spectral convolution to obtain a second matrix;

[0024] A second multiplication unit, configured to multiply the second matrix by the value matrix to obtain a third matrix;

[0025] A determination unit is used to determine a target output of the self-attention mechanism according to the third matrix.

[0026] The present application also provides a convolutional neural network model, which includes several network units, at least one of which is connected to a preset self-attention mechanism as described in any one of the preceding items, wherein the output of the adjacent previous network unit of the preset self-attention mechanism is input as an input feature matrix into the preset self-attention mechanism for processing to obtain a target output, and the target output is used as the input of the adjacent next network unit of the preset self-attention mechanism.

[0027] Furthermore, the network unit includes multiple convolutional layers and pooling layers, and at least one of the pooling layers is connected to a preset self-attention mechanism.

[0028] Furthermore, the convolutional neural network model includes a UNet model, the network unit includes a downsampling layer and an upsampling layer, the downsampling layer includes a convolution operation layer, and a preset self-attention mechanism is connected after the convolution operation layer.

[0029] The present application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the processing method of the self-attention mechanism described in any one of the above items are implemented.

[0030] The present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the processing method of the self-attention mechanism described in any one of the above items.

[0031] The processing method, device, and convolutional neural network model of the self-attention mechanism provided in this application perform spectral convolution on the first matrix after modeling. Instead of using softmax to calculate the attention probability score of each region, the spatial attention score is calculated by comprehensively weighting the relationship between adjacent regions through spectral convolution, thereby improving the contribution of the secondary probability score and avoiding the maximum probability score deviation caused by the softmax function, which in turn affects the regional attention score. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 1 is a schematic diagram of the steps of a processing method of the self-attention mechanism in one embodiment of the present application;

[0033] Figure 2 Schematic diagram of the structure of the preset self-attention mechanism in one embodiment of the present application;

[0034] Figure 3 Schematic diagram of the structure of spectral convolution in one embodiment of the present application;

[0035] Figure 4 This is a structural block diagram of a processing device for a self-attention mechanism in one embodiment of the present application;

[0036] Figure 5 Schematic diagram of the structure of a convolutional neural network model in one embodiment of the present application;

[0037] Figure 6 It is a structural diagram of the UNet model in one embodiment of the present application;

[0038] Figure 7 This is a schematic block diagram of the structure of a computer device according to an embodiment of the present application.

[0039] The realization of the objectives, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0041] Reference Figure 1 and Figure 2 An embodiment of the present application provides a method for processing a self-attention mechanism, which is applied to a preset self-attention mechanism and includes the following steps:

[0042] Step S1, obtaining an input feature matrix;

[0043] Step S12, processing the input feature matrix to obtain corresponding query matrix, key matrix and value matrix;

[0044] Step S13, multiplying the query matrix and the key matrix to obtain a first matrix;

[0045] Step S14, performing a convolution operation on the first matrix through spectral convolution to obtain a second matrix;

[0046] Step S15, multiplying the second matrix by the value matrix to obtain a third matrix;

[0047] Step S16: Determine the target output of the self-attention mechanism according to the third matrix.

[0048] In this embodiment, as described in steps S1-S2 above, an input feature matrix is ​​obtained. The input feature matrix is ​​a matrix of size C×H×W, and is mapped into a query matrix, a key matrix, and a value matrix respectively through a preset self-attention mechanism (Spectral Self-Attention, SSA). The matrix sizes of the query matrix, key matrix, and value matrix are C×HW, HW×C, and C×HW, respectively.

[0049] As described in step S3 above, the query matrix and the key matrix are matrix multiplied to obtain the first matrix M of the distance relationship between the spatial regions. d ,This first matrix models the dependencies between regions and has a size of 1×HW×HW, where 1 represents a channel.

[0050] As described in step S4 above, based on the first matrix M d , use spectral convolution (SpecConv2-k, 2 represents two-dimensional convolution, k represents the convolution kernel size, such as 3x3) to convolve the first matrix M of one channel d Perform a two-dimensional convolution operation to further capture the dependencies between neighbors and obtain the second matrix M s .

[0051] See also Figure 3 Specifically, after obtaining the first matrix, the general self-attention mechanism can obtain the normalized probability scores of the i-th region and all regions through the softmax function. If the softmax operation is performed on the i-th row, there is a region j that obtains the highest probability score, and the sum of all probability scores is 1. In this case, the probability score may be biased, and the probability score of region j determines the final attention score. The preset self-attention mechanism in this embodiment weights the score of the i-th region by integrating the relationship between neighboring regions through spectral convolution. Based on the 3x3 spectral convolution, the j region and its two neighbors and the i region and its two neighbors, a total of 9 relationships are comprehensively considered and weighted into the relationship between j and i. In other words, the relationship between i and j is not only the relationship between their respective regions, but also the relationship between their respective neighboring regions. Based on the softmax function, the relationship between regions i and j depends on themselves and their relationship with other regions other than j. Through spectral convolution, the relationship between their respective neighbors is comprehensively considered, which alleviates the regional contribution bias to a certain extent, and also avoids the region with the maximum probability score having the greatest decision-making power on the final attention score.

[0052] As described in steps S4-S5 above, the value matrix and the second matrix M s Multiplying them together yields a third matrix of size C×H×W. This third matrix represents the attention score for each region. The output of the pre-set self-attention matrix is ​​then determined based on the third matrix. Specifically, the third matrix can be directly used as the output.

[0053] In this embodiment, spectral convolution is performed on the first matrix after modeling. Softmax is no longer used to calculate the attention probability score of each region. Instead, the spatial attention score is calculated by comprehensively weighting the relationship between adjacent regions through spectral convolution, which improves the contribution of the secondary probability score and avoids the maximum probability score deviation caused by the softmax function, which in turn affects the regional attention score.

[0054] In one embodiment, the step S4 of performing a convolution operation on the first matrix to obtain the second matrix by spectral convolution includes:

[0055] Step S41, obtaining a parameter matrix of spectral convolution;

[0056] Step S42: performing a convolution operation on the first matrix and the parameter matrix to obtain the second matrix.

[0057] In this embodiment, SpecConv2-k can be used for spectral convolution, such as further weighting the area within the neighborhood through a 3x3 convolution kernel to obtain the parameter matrix of the spectral convolution Convolve the parameter matrix with the first matrix to obtain the second matrix M s ,Right now in, Represents the convolution operation.

[0058] In one embodiment, the step S41 of obtaining a parameter matrix of spectral convolution includes:

[0059] Step S411, obtaining a convolution parameter matrix;

[0060] Step S412: performing spectral normalization on the convolution parameter matrix using a spectral norm to obtain a parameter matrix of the spectral convolution.

[0061] In this embodiment, the parameter matrix of spectral convolution is The difference between the spectral convolution parameter matrix W and the general convolution parameter matrix is ​​that the spectral convolution parameter matrix undergoes spectral normalization. The purpose of this operation is to prevent a neighboring area from having a large contribution due to excessive convolution parameters. Therefore, this embodiment can use the spectral norm to constrain the convolution kernel parameters to avoid high regional sensitivity, that is, the ability of a specific area to have a large influence on the final result.

[0062] The spectral norm is the maximum singular value of the convolution parameter matrix W. Introducing the spectral norm to constrain convolution parameters ensures the flatness of local minima, improving generalization. Flatness near the local minimum leads to better generalization performance. Conversely, sharpness can lead to significant changes even with slight variations, resulting in poor generalization.

[0063] Specifically, we first calculate the spectral norm of the convolution parameter matrix W. The spectral norm is the maximum singular value of the matrix, which involves a lot of calculations. Therefore, we use the "power iteration" method to approximate the solution of the convolution arithmetic matrix W.

[0064] u n ←W·v n-1

[0065] vn ←W·u n

[0066]

[0067] Where n is the number of power iterations, v0 can be a random vector (such as sampled from a Gaussian distribution), and through iteration, the approximate value σ of the spectral norm, that is, the maximum singular value, can be obtained.

[0068] Use the spectral norm to normalize the convolution parameter matrix W to obtain the parameter matrix of spectral convolution Right now:

[0069] The values ​​in the convolution parameter matrix are spectrally normalized to penalize any region's high contribution to the final attention score due to these parameters, which can lead to high sensitivity of the entire attention mechanism to specific regions. This avoids excessive sensitivity to regional contributions and prevents biased attention scores. Furthermore, the convolution parameter matrix is ​​constrained by a spectral norm to suppress local sharpness, providing regularization and reducing the sensitivity of the convolutional layer.

[0070] In one embodiment, the step S6 of determining the target output of the self-attention mechanism according to the third matrix includes:

[0071] Step S61: Perform residual addition on the third matrix and the input feature matrix to obtain the target output of the self-attention mechanism.

[0072] In this embodiment, the residual addition of the third matrix and the input features is used as the output of the preset self-attention mechanism, which can enrich the features to a certain extent.

[0073] See also Figure 4 , an embodiment of the present application provides a processing device for a self-attention mechanism, comprising the following steps:

[0074] An acquisition unit 10 is used to acquire an input feature matrix;

[0075] An input unit 20 is used to process the input feature matrix to obtain a corresponding query matrix, a key matrix and a value matrix;

[0076] A first multiplication unit 30, configured to multiply the query matrix and the key matrix to obtain a first matrix;

[0077] a convolution operation unit 40, configured to perform a convolution operation on the first matrix through spectral convolution to obtain a second matrix;

[0078] A second multiplication unit 50 is configured to multiply the second matrix by the value matrix to obtain a third matrix;

[0079] A determination unit 60 is configured to determine a target output of the self-attention mechanism based on the third matrix.

[0080] In one embodiment, the convolution operation 40 includes:

[0081] Get subunit, used to obtain the parameter matrix of spectral convolution;

[0082] A convolution operation subunit is used to perform a convolution operation on the first matrix and the parameter matrix to obtain the second matrix.

[0083] In one embodiment, the acquisition subunit includes:

[0084] Acquisition module, used to obtain the convolution parameter matrix;

[0085] The spectral normalization module is used to perform spectral normalization on the convolution parameter matrix through the spectral norm to obtain the parameter matrix of the spectral convolution.

[0086] In one embodiment, the determining unit 60 includes:

[0087] A subunit is obtained, which is used to perform residual addition on the third matrix and the input feature matrix to obtain a target output of the self-attention mechanism.

[0088] In this embodiment, the specific implementation of the above-mentioned units, sub-units, and modules can be referred to the above-mentioned method embodiments, which will not be repeated here.

[0089] See also Figure 5 An embodiment of the present application provides a convolutional neural network model, wherein the convolutional neural network includes a plurality of network units, at least one of which is connected to a preset self-attention mechanism as described in any one of the preceding items, wherein the output of the adjacent previous network unit of the preset self-attention mechanism is input as an input feature matrix into the preset self-attention mechanism for processing to obtain a target output, and the target output is used as the input of the adjacent next network unit of the preset self-attention mechanism.

[0090] In this embodiment, the convolutional neural network model includes several network units. A preset self-attention mechanism can be embedded anywhere within the convolutional neural network model, with the output of the previous network unit serving as the input to the next. The preset self-attention mechanism calculates spatial attention scores by comprehensively weighting the relationships between neighboring regions through spectral convolution, preventing specific regions from having a greater influence on the overall outcome. Incorporating the preset self-attention mechanism into the convolutional neural network improves the model's generalization performance.

[0091] In one embodiment, the network unit includes multiple convolutional layers and pooling layers, and at least one of the pooling layers is connected to a preset self-attention mechanism.

[0092] In this embodiment, see Figure 5 , Figure 5 This is a common convolutional neural network model. Conv3-32 / 64 / 128 represents a convolutional layer with a 3x3 filter, 32 / 65 / 128 represents the number of channels, Max-pool represents a maximum pooling layer, FC-512 represents a fully connected layer with a length of 512, and SSA is the default self-attention mechanism. A default self-attention mechanism is inserted after each maximum pooling layer, and the output of the maximum pooling layer serves as the input feature matrix of the default self-attention mechanism.

[0093] In one embodiment, the convolutional neural network model includes a UNet model, the network unit includes a downsampling layer and an upsampling layer, the downsampling layer includes a convolution operation layer, and a preset self-attention mechanism is connected after the convolution operation layer.

[0094] In this embodiment, the UNet model is a semantic segmentation model. Its main execution process is similar to other semantic segmentation models. First, it uses convolution to perform downsampling, then extracts layer by layer of features, and then uses these layer by layer of features to perform upsampling, and finally obtains an image in which each pixel corresponds to its type. The specific structure can be as follows: Figure 6 As shown, the left side is the downsampling layer, which propagates in a top-down order, and the right side is the upsampling layer, which propagates in a bottom-up order. During the downsampling process, a preset self-attention mechanism is inserted after each convolution operation layer. In another embodiment, a preset self-attention mechanism can also be inserted before the convolution operation layer.

[0095] Reference Figure 7 In the embodiment of the present application, a computer device is also provided. The computer device may be a server, and its internal structure may be as follows: Figure 7 As shown. The computer device includes a processor, memory, network interface and database connected via a system bus. The processor of the computer design is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store data, etc. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a processing method of a self-attention mechanism is implemented.

[0096] Those skilled in the art will understand that Figure 7The structure shown in is merely a block diagram of a portion of the structure related to the present application solution and does not constitute a limitation on the computer device to which the present application solution is applied.

[0097] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, which implements a processing method of a self-attention mechanism when the computer program is executed by a processor.

[0098] In summary, the processing method, device, and convolutional neural network model of the self-attention mechanism provided in the embodiments of the present application obtain an input feature matrix; process the input feature matrix to obtain a corresponding query matrix, key matrix, and value matrix; multiply the query matrix and the key matrix to obtain a first matrix; perform a convolution operation on the first matrix through spectral convolution to obtain a second matrix; multiply the second matrix and the value matrix to obtain a third matrix; and determine the target output of the self-attention mechanism based on the third matrix. The present application performs spectral convolution on the first matrix after modeling, and no longer uses softmax to calculate the attention probability score of each region. Instead, spectral convolution is used to comprehensively weight the relationship between adjacent regions to calculate the spatial attention score, thereby improving the contribution of the secondary probability score and avoiding the maximum probability score deviation caused by the softmax function, which in turn affects the regional attention score.

[0099] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided in this application and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM).

[0100] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, apparatus, article, or method comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, apparatus, article, or method. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, apparatus, article, or method comprising the element.

[0101] The above description is only a preferred embodiment of the present application and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A processing method for a self-attention mechanism, characterized in that: The method is applied to the preset self-attention unit and the UNet model, where the output of the UNet model is an image, and includes the following steps: Get the input feature matrix; Processing the input feature matrix to obtain corresponding query matrix, key matrix and value matrix; Multiplying the query matrix and the key matrix to obtain a first matrix; Performing a spectral convolution operation on the first matrix to obtain a second matrix; Multiplying the second matrix by the value matrix to obtain a third matrix; Determining a target output of the self-attention unit according to the third matrix; The step of performing a convolution operation on the first matrix through spectral convolution to obtain a second matrix includes: Get the parameter matrix of spectral convolution; Performing a convolution operation on the first matrix and the parameter matrix to obtain the second matrix; The step of obtaining the parameter matrix of the spectral convolution comprises: Get the convolution parameter matrix; The convolution parameter matrix is ​​spectrally normalized by using the spectral norm to obtain the parameter matrix of the spectral convolution.

2. The method for processing the self-attention mechanism according to claim 1, wherein: The step of determining the target output of the self-attention unit according to the third matrix includes: The third matrix is ​​residually added to the input feature matrix to obtain the target output of the self-attention unit.

3. A processing device for a self-attention mechanism, configured to execute the method according to any one of claims 1 to 2, characterized in that: include: An acquisition unit, used to obtain an input feature matrix; An input unit, configured to process the input feature matrix to obtain a corresponding query matrix, a key matrix, and a value matrix; A first multiplication unit, configured to multiply the query matrix and the key matrix to obtain a first matrix; a convolution operation unit, configured to perform a convolution operation on the first matrix through spectral convolution to obtain a second matrix; A second multiplication unit, configured to multiply the second matrix by the value matrix to obtain a third matrix; A determination unit is used to determine the target output of the self-attention unit according to the third matrix.

4. A method for processing a convolutional neural network model, characterized in that: The convolutional neural network model is a UNet model, and the output of the UNet model is an image. The convolutional neural network includes several network units, and at least one of the network units is connected to a preset self-attention unit, wherein the output of the adjacent previous network unit of the preset self-attention unit is input into the preset self-attention unit as an input feature matrix for processing to obtain a target output, and the target output is used as the input of the next adjacent network unit of the preset self-attention unit. The preset self-attention unit is used to implement the method as described in any one of claims 1-2.

5. The method for processing a convolutional neural network model according to claim 4, wherein: The network unit includes multiple convolutional layers and pooling layers, and at least one of the pooling layers is connected to a preset self-attention unit.

6. The method for processing a convolutional neural network model according to claim 4, wherein: The network unit includes a downsampling layer and an upsampling layer, the downsampling layer includes a convolution operation layer, and a preset self-attention unit is connected after the convolution operation layer.

7. A computer device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 2 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 2 are implemented.

Images