Data processing method, device, equipment, and medium

Abstract

The present disclosure provides a data processing method, apparatus, device, and medium, the method comprising: inputting data to be processed into a target neural network for processing, obtaining a processing result of the data to be processed, wherein at least one convolution layer of the target neural network is an attention convolution layer based on a first attention mechanism, and / or, feature fusion is performed between at least two levels of convolution layers of the target neural network based on a second attention mechanism, wherein the first attention mechanism includes a self-attention mechanism for local regions of features, and the second attention mechanism includes an attention mechanism for local regions of output features between output features of different scales. According to the embodiments of the present disclosure, features with better expression effects can be obtained, thereby improving data processing effects.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a data processing method and device, an electronic device, and a computer-readable storage medium. Background Art

[0002] The attention mechanism is derived from the human visual attention mechanism. Given limited computing power, it allocates computing resources to more important tasks rather than processing them globally. The self-attention (SA) mechanism is an improvement on the attention mechanism, reducing its reliance on external information and better capturing the internal correlations of data or features. The computational complexity of self-attention is typically proportional to the square of the feature map size, making it quite computationally intensive. Summary of the Invention

[0003] The present disclosure provides a data processing method and device based on a many-core system, an electronic device, and a computer-readable storage medium.

[0004] In a first aspect, the present disclosure provides a data processing method, the data processing method comprising: inputting to-be-processed data into a target neural network for processing, obtaining a processing result of the to-be-processed data, wherein at least one convolutional layer of the target neural network is an attention convolutional layer based on a first attention mechanism, and / or, feature fusion is performed between at least two levels of convolutional layers of the target neural network based on a second attention mechanism,

[0005] Among them, the first attention mechanism includes a self-attention mechanism for the local area of ​​the feature, and the second attention mechanism includes an attention mechanism for the local area of ​​the output feature between output features of different scales.

[0006] In a second aspect, the present disclosure provides a data processing device, comprising: a data processing module for inputting to-be-processed data into a target neural network for processing, and for obtaining a processing result of the to-be-processed data, wherein at least one convolutional layer of the target neural network is an attention convolutional layer based on a first attention mechanism, and / or, feature fusion is performed between at least two levels of convolutional layers of the target neural network based on a second attention mechanism,

[0007] Among them, the first attention mechanism includes a self-attention mechanism for the local area of ​​the feature, and the second attention mechanism includes an attention mechanism between output features of different scales.

[0008] In a third aspect, the present disclosure provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores one or more computer programs executable by the at least one processor, and one or more of the computer programs are executed by the at least one processor to enable the at least one processor to perform the above-mentioned data processing method.

[0009] In a fourth aspect, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the above-mentioned data processing method when executed by a processor.

[0010] The embodiment provided by the present disclosure determines the attention features of the feature point only for the local area of ​​the feature point, which can effectively reduce the amount of calculation compared to determining the attention features of the feature point for the global features. Moreover, since the determination range of the attention mechanism is narrowed down to the local area, the obtained attention features have a certain inductive bias capability relative to the global features, and also have rotation invariance characteristics, thereby improving the data processing effect.

[0011] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. The above and other features and advantages will become more apparent to those skilled in the art by describing detailed example embodiments with reference to the accompanying drawings. In the accompanying drawings:

[0013] Figure 1 A schematic diagram of a processing process of a self-attention mechanism provided in an embodiment of the present disclosure;

[0014] Figure 2 A schematic diagram of a feature pyramid network provided in an embodiment of the present disclosure;

[0015] Figure 3 A flowchart of a data processing method provided in an embodiment of the present disclosure;

[0016] Figure 4 A flowchart of the working process of a data processing method provided in an embodiment of the present disclosure;

[0017] Figure 5 A schematic diagram of a target neural network provided by an embodiment of the present disclosure;

[0018] Figure 6 A schematic diagram of the working process of an attention convolution layer provided in an embodiment of the present disclosure;

[0019] Figure 7 A flowchart of the working process of a data processing method provided in an embodiment of the present disclosure;

[0020] Figure 8 A schematic diagram of a feature fusion process provided in an embodiment of the present disclosure;

[0021] Figure 9 A schematic diagram of a process for obtaining intermediate features provided in an embodiment of the present disclosure;

[0022] Figure 10 A schematic diagram of a mapping relationship of fusion features provided in an embodiment of the present disclosure;

[0023] Figure 11 A schematic diagram of a process for acquiring fusion features provided in an embodiment of the present disclosure;

[0024] Figure 12 A schematic diagram of a mapping relationship of fusion features provided in an embodiment of the present disclosure;

[0025] Figure 13 A block diagram of a data processing device provided in an embodiment of the present disclosure;

[0026] Figure 14 A block diagram of a data processing device provided in an embodiment of the present disclosure;

[0027] Figure 15 A block diagram of a data processing device provided in an embodiment of the present disclosure;

[0028] Figure 16 A block diagram of an electronic device provided in an embodiment of the present disclosure;

[0029] Figure 17 A block diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] To enable those skilled in the art to better understand the technical solutions of the present disclosure, exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0031] In the absence of conflict, the various embodiments of the present disclosure and the various features therein may be combined with each other.

[0032] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0033] The terms used herein are only used to describe specific embodiments and are not intended to limit the present disclosure. As used herein, the singular forms "a" and "the" are also intended to include the plural forms, unless the context clearly indicates otherwise. It will also be understood that when the terms "comprising" and / or "made of" are used in this specification, the presence of the features, wholes, steps, operations, elements and / or components is specified, but the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof is not excluded. Similar words such as "connected" or "connected" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect.

[0034] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted as having an idealized or overly formal meaning unless expressly defined as such herein.

[0035] Neural Network (NN) is a model that imitates the structure and function of biological neural networks and has been widely used in image processing, speech recognition, natural language processing and other fields. Convolution is an important concept in neural networks, and feature extraction can be achieved through convolution operations. In related technologies, sliding convolution kernels can be used for filtering to obtain filter responses, thereby extracting features. Since the receptive field of the convolution kernel is usually a local area of ​​the feature map, convolution has the advantage of inductive bias. Accordingly, when extracting features based on the convolution kernel, it is necessary to continuously stack them in layers to achieve feature extraction in a larger range, thereby associating different areas of the entire feature map.

[0036] The attention mechanism is another important concept in neural networks. Its essence is to derive new feature representations by linearly weighting relationships between objects. The self-attention mechanism (SA) is a variant of the attention mechanism that reduces reliance on external information and is better at capturing internal correlations within data or features. The self-attention mechanism draws on the query-key-value (QKV) concept from the Transformer, a mainstream network in natural language processing (NLP). It treats each feature point in the feature map as an embedding and then performs a QKV self-attention operation.

[0037] Figure 1 A schematic diagram of the processing process of a self-attention mechanism provided in an embodiment of the present disclosure. Figure 1 , which combines the self-attention mechanism with the visual backbone network and realizes the self-attention processing through the qkv mechanism.

[0038] Here, X represents the input feature map. First, a linear mapping is performed on X (i.e., a 1×1×1 convolution is used to compress the number of channels) to obtain three features: θ, φ, and g. Secondly, a transposition operation is performed to merge the dimensions of the three features except the number of channels. Then, a matrix dot multiplication operation is performed on θ and φ to obtain the autocorrelation feature. This step aims to calculate the feature autocorrelation, that is, to obtain the relationship between each pixel (or superpixel) in each frame of the image and all pixels (or superpixels) in all other frames. Thirdly, the autocorrelation feature is normalized (for example, Softmax) to obtain a weight coefficient (Weights) with a value range of [0, 1]. This weight coefficient is the autocorrelation coefficient. Finally, the autocorrelation coefficient is multiplied by the feature g to perform channel expansion, and the expansion result is subjected to a residual operation with X to obtain the final output Z.

[0039] In summary, compared to convolution operations, the biggest advantage of the self-attention mechanism is that it can associate any two points in the entire image with just one layer (i.e., modeling feature patterns of any range). Because of this, the biggest disadvantage of the self-attention mechanism is its excessive computational complexity (the computational complexity is usually proportional to the square of the feature map size). Especially in the field of visual processing, if the self-attention mechanism is not mixed with convolution (i.e., introducing the convolutional Vision Transformer (ViT)), the feature patterns will be difficult to converge due to the lack of inductive bias, or training will be required on very large-scale datasets such as JFT (Google's internal image classification dataset).

[0040] In order to reduce the computational complexity of the self-attention mechanism, the industry has proposed improved self-attention models such as Criss Cross Network (CCNet) and Cross-Shaped Window (CSWin). Among them, the self-attention model based on CCNet calculates the relationship between the target feature pixel and the pixels in the cross-intersection area of ​​its row and column, and uses this relationship to weight the features of the target pixel, so as to obtain more effective target features. The self-attention model based on CSWin calculates the relationship between the target feature pixel and the pixels in the cross-intersection window area of ​​its row and column, and uses this relationship to weight the features of the target pixel, so as to obtain more effective target features.

[0041] The data processing method provided by the disclosed embodiments is not solely focused on reducing the computational complexity of the self-attention model. Rather, it considers that when extracting features based on convolution operations, the filter response is maximized, and the corresponding feature extraction results are better, when the feature pattern to be extracted completely corresponds to the convolution kernel and there is no rotation angle. However, in practical applications, the same feature pattern may appear in an image at different rotation angles, and using a single convolution kernel cannot effectively extract all features corresponding to the same feature pattern. In other words, the convolution operation does not support rotation invariance when extracting features. Furthermore, considering that the self-attention mechanism reweights the feature map based on the similarity between any two points to establish an association between them, and this weighting is independent of distance and relative position, the self-attention mechanism lacks inductive bias properties. However, when the scope of the self-attention mechanism is narrowed to a local area, it will have a certain inductive bias capability. Moreover, because the self-attention calculation is based on the similarity between feature points, it is rotation invariant. In view of this, an embodiment of the present disclosure provides a neural network including a self-attention convolutional layer, which performs feature extraction based on the self-attention convolutional layer and obtains more effective output features.

[0042] In addition, when extracting features, to ensure that features of different scales or levels can be extracted, convolution kernels of different sizes are usually required. Smaller convolution kernels can extract lower-level features, while larger convolution kernels can extract higher-level features. Among them, low-level features have less semantic information, but accurate target locations and higher resolution, while high-level features have richer semantic information, but the target locations are coarse, the resolution is lower, and they are more abstract.

[0043] In related technologies, feature fusion is often used to combine the high resolution of low-level features with the high semantic information of high-level features, thereby enhancing feature expression. Feature Pyramid Network (FPN) is a commonly used feature fusion model. It uses a method of channel splicing or point-by-point addition after resampling to achieve feature fusion at different levels, which is relatively simple to implement.

[0044] Figure 2 A schematic diagram of a feature pyramid network provided by an embodiment of the present disclosure. Figure 2 The FPN network uses a standard feature extraction network to extract features from multiple spatial locations. It then adds a lightweight top-down pathway and connects it laterally to the feature extraction network. Each level of features extracted by the feature extraction network is first upsampled by a factor of two to obtain the upsampled features. These features are then superimposed with the next level of features processed by a 1×1 convolution to obtain the corresponding fused features. Subsequent data processing operations are then performed based on the fused features.

[0045] To further enhance feature fusion, various improved models have been proposed for FPN, such as bidirectional FPN and path aggregation network (PA-Net). However, neither FPN nor its improved models fully consider the spatial correlation between features at two different levels.

[0046] Considering that the attention mechanism can establish the correlation relationship between any two pixels in the feature map, in the embodiment of the present disclosure, the attention mechanism is applied to the feature fusion process to fuse the correlation relationship between features at different levels in the spatial dimension into the features.

[0047] The data processing method according to the embodiments of the present disclosure can be executed by an electronic device such as a terminal device or a server. The terminal device can be an in-vehicle device, a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, an in-vehicle device, a wearable device, etc. The method can be implemented by a processor calling computer-readable program instructions stored in a memory. Alternatively, the method can be executed by a server.

[0048] Figure 3 A flowchart of a data processing method provided by an embodiment of the present disclosure. Figure 3 , the method comprising:

[0049] In step S31, the data to be processed is input into the target neural network for processing to obtain the processing result of the data to be processed.

[0050] In which, at least one convolution layer of the target neural network is an attention convolution layer based on the first attention mechanism, and / or, feature fusion is performed between at least two levels of convolution layers of the target neural network based on the second attention mechanism, the first attention mechanism includes a self-attention mechanism for the local area of ​​the feature, and the second attention mechanism includes an attention mechanism for the local area of ​​the output feature between output features of different scales.

[0051] For example, the data to be processed includes any one of image data, voice data, text data, and video data. The embodiment of the present disclosure does not limit the type and content of the data to be processed.

[0052] In some possible implementations, after the data to be processed is input into the target neural network, the target neural network uses the attention convolution layer to perform self-attention operation on the data input into the layer, and obtains output features for other network layers in the target neural network to process data based on the output features and obtain processing results; and, the target neural network can also fuse the output features (including but not limited to the output features of the attention convolution layer) based on the second attention mechanism to obtain fused features, so as to provide other network layers with further data processing based on the fused features to obtain processing results.

[0053] In some possible implementations, the target neural network can be used to perform any one of image processing tasks, speech processing tasks, text processing tasks, and video processing tasks. Correspondingly, the processing result of the data to be processed can be any one of image processing results, speech processing results, text processing results, and video processing results (wherein the processing may include operations such as recognition, classification, and labeling), which is related to the type of data to be processed, the content of the data to be processed, and the execution task of the target neural network. The embodiments of this disclosure do not limit the specific task types and processing result types performed by the target neural network.

[0054] As mentioned above, the target neural network includes at least one layer of attention convolution layer based on the first attention mechanism, and / or, feature fusion is performed between at least two levels of convolution layers of the target neural network based on the second attention mechanism. Therefore, in step S31, the processing process of the target neural network includes at least feature extraction, and / or feature fusion. It should be noted that, whether it is feature extraction or feature fusion, it utilizes the local attention mechanism, that is, attention operation is performed on the local area of ​​the feature map, and the original features are updated according to the results of the attention operation. The feature extraction working process and feature fusion working process of the target neural network will be explained in detail later.

[0055] In some possible implementations, the target neural network includes an attention convolution layer, which can be used to perform feature extraction. Figure 4 Describe the working process of feature extraction of the target neural network.

[0056] Figure 4 A flowchart of the working process of a data processing method provided by an embodiment of the present disclosure. Figure 4 , the method comprising:

[0057] Step S41: For any attention convolution layer, perform a linear transformation on the input data of the attention convolution layer to obtain the first query feature, the first key feature, and the first value feature corresponding to the input data.

[0058] Step S42: determining a first attention feature corresponding to a plurality of target feature points of the first query feature according to the first query feature, the first key feature, and the first value feature.

[0059] Among them, the first attention feature includes a first attention value corresponding to each target feature point. The first attention value is determined for the local area corresponding to the target feature point. The local area corresponding to the target feature point is the area in the first query feature with the target feature point as the center and determined according to a preset size. The preset size is smaller than the size of the first query feature. The first attention value is used to characterize the association relationship between each feature point in the local area and the target feature point.

[0060] Step S43: Determine the output feature corresponding to the attention convolution layer based on the first attention feature and the input data.

[0061] Among them, the processing result is obtained after the output features are processed by the network layer after the attention convolution layer.

[0062] In some possible implementations, the input data of the attention convolution layer is the data to be processed after being processed by the network layer before the attention convolution layer. In step S41, the input data can be linearly transformed by multiplying the input data with a preset transformation matrix to obtain the first query feature, the first key feature, and the first value feature. In addition, the input data can also be convolved based on a preset convolution kernel, and the convolution result is the first query feature, the first key feature, or the first value feature. The embodiment of the present disclosure does not limit the linear transformation method of the input data.

[0063] It should be noted that, in some possible implementations, the first query feature, the first key feature, and the first value feature can be obtained by multiplying the input data with the same transformation matrix or performing a linear transformation on the input data. In other words, in this case, the first query feature, the first key feature, and the first value feature are exactly the same features. In one example, the input data is the matrix F (h1*w1) , and compare it with the preset transformation matrix w (h2*w2) Multiply and get the multiplication result F' (h1*w2) and F' (h1*w2) As the first query feature, the first key feature and the first value feature, wherein w1=h2.

[0064] In some possible implementations, different first query features, first key features, and first value features may be obtained based on different transformation matrices or different linear transformation methods.

[0065] After obtaining the first query feature, the first key feature, and the first value feature, the first attention feature can be determined by self-attention operation in step S42. In some possible implementations, in step S42, the process of determining the first attention feature includes: for each target feature point, determining multiple first key feature points corresponding to the local area in the first key feature, and first value feature points corresponding to the local area in the first value feature; determining the similarity between the target feature point and each first key feature point, and obtaining a first similarity feature corresponding to the target feature point; obtaining a first attention value corresponding to the target feature point based on the first similarity feature and the first value feature point; and obtaining the first attention feature based on the first attention values ​​of the multiple target feature points.

[0066] In some possible implementations, the similarity between feature points may be calculated based on cosine similarity, Pearson correlation coefficient, etc. The embodiment of the present disclosure does not limit the method for determining the similarity.

[0067] In some possible implementations, when the first query feature is the same as the first key feature, determining the similarity between the target feature point and each first key feature point includes: when obtaining the similarity Sij between the i-th target feature point and the j-th first key feature point, determining the similarity Sji between the j-th first query feature point and the i-th target feature point based on Sij; wherein i and j are both integers greater than or equal to 1 and less than or equal to M, M is the total number of feature points in the first query feature or the first key feature (M is an integer greater than or equal to 1), and i≥j or i≤j. Determining the similarity Sji based on Sij can effectively reduce the amount of computation, thereby reducing data processing pressure.

[0068] It should be noted that in the disclosed embodiment, for each target feature point in the first query feature, self-attention calculation is performed only in a local region for that target feature point to obtain a first attention value, and a first attention feature is obtained based on the first attention values ​​of multiple target feature points. Compared to performing self-attention calculation based on the entire feature region, self-attention calculation based on the local region can effectively reduce the amount of computation. Moreover, because the local region has a certain inductive bias relative to the global region and is also rotationally invariant, it can obtain better features.

[0069] It should also be noted that the target feature point is a feature point belonging to the first query feature, which has a corresponding relationship with the feature point in the input data. The self-attention operation is performed on the target feature point based on the local area. Its essence is to determine the corresponding association relationship based on the local area for the input data.

[0070] In some possible implementations, the range of target feature points can be flexibly set as needed, and can include all feature points in the first query feature or several feature points specified in the first query feature. This is not limited in the embodiments of the present disclosure.

[0071] In some possible implementations, before step S42 , the method further includes: selecting multiple feature points from the first query feature as target feature points; and determining a local area corresponding to each target feature point according to a preset size.

[0072] In some possible implementations, the local area corresponding to the target feature point can be in vector form (for example, text processing scenarios) or a rectangle (including a square) (for example, image or video processing scenarios), which is not limited in the embodiments of the present disclosure.

[0073] In some possible implementations, the first query feature includes either a vector or a matrix. When the first query feature is a vector, the preset size includes a preset number of feature points, and the preset number of feature points is less than the total number of feature points of the first query vector. The local region is a vector centered on the target feature point and having a number of feature points equal to the preset number of feature points. When the first query feature is a matrix, the preset size includes a preset number of rows and a preset number of columns, and the preset number of rows is less than the total number of rows of the first query feature, and the preset number of columns is less than the total number of columns of the first query feature. The local region is a rectangular region centered on the target feature point, with a height of the preset number of rows and a width of the preset number of columns.

[0074] In one example, assuming the first query feature is a 5*5 matrix, all feature points in the first query feature are determined as target feature points, and a region centered on the target feature point and with a side length equal to three feature points is set as the local region corresponding to the target feature point. For target feature points located at the edge of the first query feature, their local regions cannot form a 3*3 feature region. During the processing process, the local regions of these target feature points can be supplemented into a 3*3 feature region by zero padding to facilitate calculation.

[0075] In the above content, the process of obtaining the first attention feature in step S42 includes: first determining the target feature point, then determining the local area, and then determining the first attention feature. In some other possible implementations, the first attention feature can be obtained by using a sliding window method.

[0076] In some possible implementations, in step S42, the process of obtaining the first attention feature based on the sliding window method includes: setting the sliding window and step size according to a preset size of the local area; starting from a preset initial sliding position, sliding the sliding window along the first query feature with a step size, and determining the target feature point corresponding to the sliding window in each sliding operation, multiple first key feature points corresponding to the sliding window in the first key feature, and the first value feature point corresponding to the sliding window in the first value feature; determining the similarity between the target feature point and each first key feature point, and obtaining a first similarity feature corresponding to the target feature point; obtaining a first attention value corresponding to the target feature point based on the first similarity feature and the first value feature point; obtaining the first attention feature based on the first attention values ​​of the multiple target feature points.

[0077] In other words, the method of obtaining the first attention feature based on the sliding window is similar to the method of extracting features based on the convolution kernel. The difference is that the first attention feature determines the feature value based on the self-attention operation, and the convolution kernel determines the feature value through the convolution operation.

[0078] After obtaining the first attention feature, the output feature can be determined based on the first attention feature and the input data in step S43. There are at least two ways to determine the output feature: the first way is to perform a linear transformation on the first attention feature so that it has the same size as the input data, and superimpose the transformed first attention feature on the input data to obtain the output feature; the second way is to establish a position mapping relationship between the first attention feature and the feature points of the input data, and use the position mapping relationship to generate the output feature based on the first attention feature and the input data. In the first way, the size of the output feature remains the same as the input data. In the second way, if the target feature point is not all the feature points in the first query feature, the size of the output feature is different from the input data, and it only includes the feature components corresponding to the target feature point.

[0079] In some possible implementations, in step S43, the output feature corresponding to the attention convolution layer is determined based on the first attention feature and the input data, including: performing a linear transformation on the first attention feature to obtain a first matching attention feature of the same size as the input data; and superimposing the first matching attention feature with the input data to obtain an output feature corresponding to the input data.

[0080] In some possible implementations, in step S43, the output feature corresponding to the attention convolution layer is determined based on the first attention feature and the input data, including: rearranging the first attention value according to the position information of the target feature point corresponding to the first attention value in the first attention feature in the input data to obtain a second matching attention feature; and obtaining the output feature corresponding to the input data based on the second matching attention feature and the feature point corresponding to the target feature point in the input data.

[0081] In summary, the first attention mechanism in the embodiment of the present disclosure is essentially in the category of self-attention, which has the characteristics of "local area", "inductive bias" and "rotation invariance". Among them, "local area" means that when obtaining the first attention feature, the self-attention operation is only performed on the local area of ​​the feature, rather than the self-attention operation on the feature as a whole, which can effectively reduce the amount of calculation; the "inductive bias" characteristic is an additional characteristic generated by the self-attention operation on the local area. Compared with the global self-attention operation, which has no inductive bias ability, the self-attention operation only on the local area has a certain inductive bias ability; the "rotation invariance" characteristic is because the self-attention operation itself focuses on the correlation between feature points. This correlation has nothing to do with the distance and relative position between the features, making it insensitive to the rotation angle.

[0082] The following combination Figure 5 and Figure 6The data processing method according to the embodiment of the present disclosure is described in detail.

[0083] Figure 5 A schematic diagram of a target neural network provided in an embodiment of the present disclosure.

[0084] Reference Figure 5 The target neural network includes a first network layer structure, an attention convolution layer, and a second network layer structure. The first network layer structure is located before the attention convolution layer and may include one or more network layers (the network layer may be a convolution layer). The second network layer structure is located after the attention convolution layer and may also include one or more network layers (the network layer may include a batch normalization layer and an activation layer, etc.).

[0085] In some possible implementations, after the data to be processed is input into the target neural network, the first network layer structure first processes the data to be processed, obtains intermediate data, and inputs the intermediate data into the attention convolution layer. This intermediate data is the input data of the attention convolution layer. The attention convolution layer processes the input data according to any implementation method of the disclosed embodiments to obtain output features, and inputs the output features into the second network layer structure. The second network layer structure processes the output features to obtain a processing result, and the target neural network outputs the processing result.

[0086] Figure 6 A schematic diagram of the working process of an attention convolution layer provided in an embodiment of the present disclosure.

[0087] Reference Figure 6 , the input data F of the attention convolution layer is an h f *w f *c tensor, where h f represents the height of F, w f represents the width of F, and c represents the number of channels of F.

[0088] In some possible implementations, first, F is multiplied by the first transformation matrix wq, the second transformation matrix wk, and the third transformation matrix wv, respectively, to implement a cubic linear transformation of F, and obtain the first query feature Q, the first key feature K, and the first value feature V, where the size of wq is w f *h q *1, correspondingly, Q is h q *w q *c is a tensor, wk has the same size as w f *h k *1, correspondingly, K is h k *w k *c is a tensor, wk has the same size as wf *h v *1, correspondingly, V is h v *w v After obtaining Q, K, and V, calculate the cosine similarity between each target feature point in Q and the feature point in the local area corresponding to the target feature point in K to obtain the similarity feature S, where S is {h q *w q}*{h k *w k} matrix.

[0089] In one example, the process of obtaining the similarity feature S includes: first, transforming Q into {h q *w q}*c matrix form, and then transform K into c*{h k *w k}, and finally, perform matrix multiplication based on the transformed matrix form to obtain a matrix with size {h q *w q}*{h k *w k} similarity feature S. Its significance lies in that the element at position (i, j) in S represents the influence of the jth element on the ith element, or the similarity between the jth and ith elements, thus realizing the dependency relationship between any two elements in the global context. It should be noted that S is sparse. Among its elements, only the similarity between the target feature point and the local region feature point has a non-zero value. The values ​​of other elements that have not undergone self-attention operation are all zero.

[0090] Further, transform V to obtain V′, which is {h v *w v}*c matrix. Perform the inner product operation on S and V′ to obtain {h q *w q}*c matrix, and transform the matrix to obtain h q *w q *c tensor, which is the first attention feature P(h q =h p , w q *w p Finally, P is linearly transformed to make it the same size as F, and then added to F to obtain the final output feature F′. The size of F′ is h f′ *w f′ *c, and h f′ =h f , w f′ =w f .

[0091] It should be noted that in some possible implementations, after obtaining the similarity feature S, the elements therein may be normalized by methods such as Softmax so that the data are at the same order of magnitude, facilitating comparative analysis.

[0092] It should also be noted that in some possible implementations, in order to reduce the amount of computation, the number of channels c can be reduced during the linear transformation process, that is, the number of channels of Q, K and V can be smaller than the number of channels of F, and the number of channels of Q, K and V can be different (the number of channels of Q and K are usually the same).

[0093] The output features are determined based on the sliding window method, which is similar to the calculation method of the above process and will not be repeated here.

[0094] The above content describes how to obtain output features based on the attention convolution layer. In practical applications, after obtaining the output features, in order to enhance the feature expression effect, features at different levels can also be fused to obtain fused features.

[0095] The following combination Figure 7 Describe the working process of feature fusion of the target neural network.

[0096] Figure 7 A flowchart of the working process of a data processing method provided by an embodiment of the present disclosure. In which, feature fusion is performed between N levels of convolutional layers of the target neural network based on the second attention mechanism, and the scales of the features output by each level of convolutional layer are different, and N is an integer greater than or equal to 2. The second attention mechanism includes an attention mechanism for the local area of ​​the output features between output features of different scales. Figure 7 , the method comprising:

[0097] Step S71: For the n-th level convolution layer, determine the n-th level second attention feature based on the n-1-th level intermediate feature of the n-1-th level convolution layer and the n-th level initial feature output by the n-th level convolution layer.

[0098] Here, n represents the number of convolutional layers, and n is an integer with 2≤n≤N-1.

[0099] Step S72: Update the n-th level initial feature according to the n-th level second attention feature to obtain the n-th level intermediate feature.

[0100] Step S73: Determine the n-th level third attention feature based on the n+1-th level fusion feature of the n+1-th level convolutional layer and the n-th level intermediate feature.

[0101] Step S74: Update the n-th level intermediate feature according to the n-th level third attention feature to obtain the n-th level fusion feature.

[0102] In some possible implementations, initial features are the features of the data in its initial state after being processed by the convolutional layer of the target neural network. Intermediate features are intermediate features obtained by combining the initial features with feature information from other layers. Fused features are features obtained by further fusing intermediate features with features from other layers. Accordingly, the processing result is the result obtained after the fused features are processed by the network layer following the convolutional layer.

[0103] In some possible implementations, for the first-level convolutional layer, since there is no convolutional layer with a lower feature level, the first-level intermediate features are set equal to the first-level initial features. Accordingly, the first-level fusion features are obtained by updating the first-level intermediate features based on the first-level third attention features, and the first-level third attention features are obtained based on the second-level fusion features and the first-level intermediate features.

[0104] In some possible implementations, for the Nth level convolution layer, since there is no convolution layer with a higher feature level, the Nth level fusion feature is set equal to the Nth level intermediate feature.

[0105] The above feature fusion process can be summarized as follows: first, the n-th level initial features are updated based on the n-th level second attention features to obtain the n-th level intermediate features; second, the n-th level intermediate features are updated based on the n-th level third attention features to obtain the n-th level fused features. Both the n-th level second attention features and the n-th level third attention features are features obtained based on the second attention mechanism. The n-th level second attention features are obtained by combining the n-1-th level intermediate features and the n-th level initial features. They reflect the correlation between the feature points of the n-1-th level intermediate features and the feature points of the n-th level initial features. The n-th level third attention features are obtained by combining the n+1-th level fused features and the n-th level intermediate features. They reflect the correlation between the feature points of the n+1-th level fused features and the n-th level intermediate features.

[0106] In some possible implementations, in step S71, for the n-th level convolution layer, according to the n-1th level intermediate feature of the n-1th level convolution layer and the n-th level initial feature output by the n-th level convolution layer, the n-th level second attention feature is determined, including: performing a linear transformation on the n-th level initial feature to obtain the n-th level second query feature corresponding to the n-th level initial feature; performing a linear transformation on the n-1th level intermediate feature to obtain the n-1th level second key feature and the n-1th level second value feature corresponding to the n-1th level intermediate feature; determining the correspondence between each feature point of the n-th level second query feature and the n-1th level second key feature. The method comprises the following steps: determining a mapping relationship between each feature point of the n-th level second query feature; determining a feature fusion area corresponding to each feature point of the n-th level second key feature according to the mapping relationship; determining the similarity between each feature point in the n-th level second query feature and each feature point in the n-1-th level second key feature in the feature fusion area to obtain the n-th level second similarity feature; determining the n-1-th level second value feature point corresponding to the feature fusion area in the n-1-th level second value feature; determining the inner product between the n-th level second similarity feature and the n-1-th level second value feature point to obtain the n-th level second attention feature.

[0107] In some possible implementations, in step S72, the n-th level initial feature is updated according to the n-th level second attention feature to obtain the n-th level intermediate feature, including: superimposing the n-th level second attention feature with the n-th level initial feature to obtain the n-th level intermediate feature.

[0108] In some possible implementations, in step S73, the n-th level third attention feature is determined based on the n+1-th level fusion feature and the n-th level intermediate feature of the n+1-th level convolutional layer, including: performing a linear transformation on the n-th level intermediate feature to obtain the n-th level third query feature corresponding to the n-th level intermediate feature; performing a linear transformation on the n+1-th level fusion feature to obtain the n+1-th level third key feature and the n+1-th level third value feature corresponding to the n+1-th level fusion feature; determining the relationship between each feature point of the n-th level third query feature and each feature point of the n+1-th level third key feature. a mapping relationship; according to the mapping relationship, determine that each feature point of the n-th level third query feature corresponds to the feature fusion area of ​​the n+1-th level third key feature; determine the similarity between each feature point in the n-th level third query feature and each feature point in the n+1-th level third key feature in the feature fusion area, and obtain the n-th level third similarity feature; determine the n+1-th level third value feature point corresponding to the feature fusion area in the n+1-th level third value feature; determine the inner product between the n-th level third similarity feature and the n+1-th level third value feature point, and obtain the n-th level third attention feature.

[0109] In some possible implementations, in step S74, the n-th level intermediate feature is updated according to the n-th level third attention feature to obtain the n-th level fusion feature, including: superimposing the n-th level third attention feature and the n-th level intermediate feature to obtain the n-th level fusion feature.

[0110] In summary, the second attention mechanism in the disclosed embodiment has both "other attention" and "local area" characteristics. The "other attention" characteristic means that both the second and third attention features perform attention operations between different features (i.e., output features of different scales), rather than performing attention operations on different feature points of the same feature; the "local area" characteristic means that when performing attention operations between different features, attention operations are not performed on all feature points, but only on a number of feature points with a mapping relationship between the two features.

[0111] The following combination Figures 8-12 The data processing method according to the embodiment of the present disclosure is described in detail.

[0112] Figure 8 A schematic diagram of a feature fusion process provided by an embodiment of the present disclosure. Figure 8 , the first-level initial features, the second-level initial features, ... the Nth-level initial features are the features obtained by sampling the input data using the first convolution layer to the Nth-level convolution layer. Among them, the first convolution layer to the Nth-level convolution layer correspond to sampling rates X1 to X N , and follow the order from X1 to X N Correspondingly, the resolution of the first-level initial features is the highest, and the resolution of the N-level initial features is the lowest.

[0113] For the first-level convolution layer, considering that there is no convolution layer with a lower feature level, it cannot obtain the second attention feature transmitted from the low-level feature layer. Therefore, the first-level intermediate feature is set equal to the first-level initial feature. For the second-level convolution layer, the second-level second attention feature is determined based on the first-level intermediate feature of the first-level convolution layer and the second-level initial feature output by the second-level convolution layer, and the second-level initial feature is updated based on the second-level second attention feature to obtain the second-level intermediate feature. Similarly, for the third-level convolution layer to the N-th-level convolution layer, the third-level intermediate feature to the N-th-level intermediate feature can be obtained by processing in the above manner.

[0114] After obtaining the intermediate features of all convolutional layers, the fusion features of each convolutional layer can be further determined.

[0115] For the N-th level convolution layer, considering that there is no convolution layer with a higher feature level, it cannot obtain the third attention feature transmitted from the high-level feature layer. Therefore, the N-th level intermediate feature is set equal to the N-th level fusion feature. For the N-1-th level convolution layer, the N-1-th level third attention feature is determined based on the N-th level fusion feature and the N-1-th level intermediate feature, and the N-1-th level intermediate feature is updated based on the N-1-th level third attention feature to obtain the N-1-th level fusion feature. Similarly, for the N-2-th level intermediate feature to the 1-th level intermediate feature, the N-2-th level fusion feature to the 1-th level fusion feature can be obtained by processing in the above manner.

[0116] Figure 9 A schematic diagram of a process for acquiring intermediate features provided in an embodiment of the present disclosure.

[0117] Reference Figure 9 , F1′ represents the n-1th level intermediate feature, which is a tensor of h1*w1*c1; F2 represents the nth level initial feature, which is a tensor of h2*w2*c1.

[0118] First, perform a linear transformation on F2 to generate the n-th level second query feature Q2 corresponding to F2. Q2 is an h q1 *w q1 *c1 tensor. And, perform linear transformation on F1′ to obtain the n-1th level second key feature K2 and n-1th level second value feature V2 corresponding to F1′, K2 is an h k1 *w k1 *c1 is a tensor, V2 is an h v1 *w v1 * c1 A tensor of . Among them, h k1 =h v1 , w k1 =w v1 , and h q1 Less than h k1 / h v1 , w q1 Less than w k1 / w v1 .

[0119] Secondly, determine the mapping relationship between each feature point of Q2 and each feature point of K2, and according to the mapping relationship, determine that each feature point of Q2 corresponds to the feature fusion area of ​​K2; and determine the similarity between each feature point in Q2 and each feature point in K2 in the feature fusion area, and obtain the n-th level second similarity feature S2 (the value corresponding to the feature point in S2 that has not calculated similarity is set to 0), where the size of S2 is (h q1 *w q1 )*(h k1*w k1 ); Then, expand V2 into (h v1 *w v1 )*c1 matrix V2′, and calculate the inner product of S2 and V2′ (that is, calculate {(h q1 *w q1 )*(h k1 *w k1 )}·{(h v1 *w v1 )*c1}), the size is (h q1 *w q1 )*c1 matrix, and (h q1 *w q1 )*c1 matrix is ​​rearranged into h q1 *w q1 *c1 tensor, which is the second attention feature P2 of the nth level. p1 =h q1 , w p1 =w q1 .

[0120] Finally, P2 is linearly transformed to make it the same size as F2, and then superimposed with F2 to obtain the n-th level intermediate feature F2′, where the size of F2′ is the same as F2.

[0121] In some possible implementations, the mapping relationship between each feature point of Q2 and each feature point of K2, that is, the corresponding relationship between the position of the same feature in Q2 and the position of the feature in K2, when calculating the second similarity, only the feature fusion area needs to be calculated, which can effectively reduce the amount of calculation compared to calculating the second similarity for the entire feature area.

[0122] Figure 10 A schematic diagram of a mapping relationship of fusion features provided in an embodiment of the present disclosure. Figure 10 The shaded area in the feature map corresponding to Q2 has a mapping relationship with the shaded area in the feature map corresponding to K2 (i.e., they correspond to the same feature pattern). For any feature point in the shaded area of ​​Q2, the calculation range of the second similarity or the influence range of the second attention mechanism is limited to the shaded area of ​​the feature map of K2.

[0123] Figure 11 A schematic diagram of a process for acquiring fusion features provided in an embodiment of the present disclosure.

[0124] Reference Figure 11 , F3′ represents the nth level intermediate feature, which is a tensor of h3*w3*c2; F4″ represents the n+1th level fusion feature, which is a tensor of h4*w4*c2.

[0125] First, perform a linear transformation on F3′ to obtain the nth level third query feature Q3 corresponding to F3′. Q3 ​​is an h q2 *w q2 *c2 tensor; and, F4″ is also linearly transformed to obtain the n+1th level third key feature K3 and the n+1th level third value feature V3 corresponding to F4″, K3 is an h k2 *w k2 *c2 tensor, V3 is a h v2 *w v2 *c2 tensor. Among them, h k2 =h v2 , w k2 =w v2 , and h q2 Greater than h k2 / h v2 , w q2 Greater than w k2 / w v2 .

[0126] Secondly, determine the mapping relationship between each feature point of Q3 and each feature point of K3, and according to the mapping relationship, determine that each feature point of Q3 corresponds to the feature fusion area of ​​K3; determine the similarity between each feature point in Q3 and each feature point in K3 in the feature fusion area, and obtain the n-th level third similarity feature S3 (set the value corresponding to the feature point in S3 that has not calculated similarity to 0), and the size of S3 is (h q2 *w q2 )*(h k2 *w k2 ); Then, expand V3 to (h v2 *w v2 )*c2 matrix V3′, and calculate the inner product of S3 and V3′, and the calculation result is (h q2 *w q2 )*c2 matrix, the (h q2 *w q2 )*c2 matrix is ​​rearranged to h q2 *w q2 *c2 in tensor form, thereby obtaining the n-th level third attention feature P3.

[0127] Finally, P3 is linearly transformed to make it the same size as F3′, and then superimposed with F3′ to obtain the n-th level fusion feature F3″. The size of F3″ is the same as F3′.

[0128] In some possible implementations, the mapping relationship between each feature point of Q3 and each feature point of K3, that is, the corresponding relationship between the position of the same feature in Q3 and the position of the feature in K3, when calculating the third similarity, only the feature fusion area needs to be calculated, which can effectively reduce the amount of calculation compared to calculating the third similarity for the entire feature area.

[0129] Figure 12 A schematic diagram of a mapping relationship of fusion features provided in an embodiment of the present disclosure. Figure 12 The shaded area in the feature map corresponding to Q3 is mapped to the shaded area in the feature map corresponding to K3 (i.e., they correspond to the same feature pattern). For any feature point in the shaded area of ​​Q3, the calculation range of the second similarity or the influence range of the second attention mechanism is limited to the shaded area of ​​the K3 feature map.

[0130] It is understood that the above-mentioned various method embodiments mentioned in this disclosure can be combined with each other to form combined embodiments without violating the principle logic. Due to space limitations, this disclosure will not go into details. It is understood by those skilled in the art that in the above-mentioned methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0131] In addition, the present disclosure also provides a data processing device, an electronic device, and a computer-readable storage medium, all of which can be used to implement any data processing method provided by the present disclosure. The corresponding technical solutions and descriptions are referred to the corresponding records in the method section and will not be repeated here.

[0132] Figure 13 A block diagram of a data processing device provided in an embodiment of the present disclosure.

[0133] Reference Figure 13 , an embodiment of the present disclosure provides a data processing device, the data processing device comprising:

[0134] The data processing module 13 is used to input the data to be processed into the target neural network for processing, and to obtain the processing results of the data to be processed.

[0135] In which, at least one convolution layer of the target neural network is an attention convolution layer based on a first attention mechanism, and / or, feature fusion is performed between at least two levels of convolution layers of the target neural network based on a second attention mechanism, the first attention mechanism includes a self-attention mechanism for the local area of ​​the feature, and the second attention mechanism includes an attention mechanism between output features of different scales.

[0136] In some possible implementations, the data processing device further includes an input module for inputting the data to be processed into the target neural network.

[0137] For example, the data to be processed includes any one of image data, voice data, text data, and video data. The embodiment of the present disclosure does not limit the type and content of the data to be processed.

[0138] In some possible implementations, after the data to be processed is input into the target neural network through the input module, the data processing module uses the attention convolution layer in the target neural network to perform self-attention operation on the data input into the layer, and obtains output features for other network layers in the target neural network to perform data processing based on the output features to obtain processing results; and, the data processing module can also fuse the output features (including but not limited to the output features of the attention convolution layer) based on the second attention mechanism in the target neural network to obtain fused features, so that other network layers can further perform data processing based on the fused features to obtain processing results.

[0139] In some possible implementations, the target neural network can be used to perform any one of image processing tasks, speech processing tasks, text processing tasks, and video processing tasks. Correspondingly, the processing result of the data to be processed can be any one of image processing results, speech processing results, text processing results, and video processing results (wherein the processing may include operations such as recognition, classification, and labeling), which is related to the type of data to be processed, the content of the data to be processed, and the execution task of the target neural network. The embodiments of this disclosure do not limit the specific task types and processing result types performed by the target neural network.

[0140] In some possible implementations, the target neural network includes an attention convolution layer, and accordingly, the data processing module can be used to implement feature extraction based on the first attention mechanism.

[0141] Figure 14 A block diagram of a data processing device provided by an embodiment of the present disclosure. Figure 14The data processing device includes: a transformation submodule 141, a first attention processing submodule 142 and an output feature determination submodule 143. Among them, the transformation submodule 141 is used to perform a linear transformation on the input data of the attention convolution layer for any attention convolution layer to obtain the first query feature, the first key feature and the first value feature corresponding to the input data; the first attention processing submodule 142 is used to determine the first attention feature corresponding to multiple target feature points of the first query feature based on the first query feature, the first key feature and the first value feature, wherein the first attention feature includes a first attention value corresponding to each target feature point, and the first attention value is determined for the local area corresponding to the target feature point. The local area corresponding to the target feature point is the area in the first query feature centered on the target feature point and determined according to a preset size, and the preset size is smaller than the size of the first query feature. The first attention value is used to characterize the association relationship between each feature point in the local area and the target feature point; the output feature determination submodule 143 is used to determine the output feature corresponding to the attention convolution layer based on the first attention feature and the input data, wherein the input data is the data to be processed after the network layer before the attention convolution layer processes the data, and the processing result is the output feature obtained after the network layer after the attention convolution layer processes the output feature.

[0142] In some possible implementations, the first attention processing submodule includes: an area mapping unit, a similarity determination unit, a first attention value acquisition unit, and a first attention feature acquisition unit. The area mapping unit is used to determine, for each target feature point, multiple first key feature points corresponding to the local area in the first key feature, and first value feature points corresponding to the local area in the first value feature; the similarity determination unit is used to determine the similarity between the target feature point and each first key feature point, and obtain the first similarity feature corresponding to the target feature point; the first attention value acquisition unit is used to obtain the first attention value corresponding to the target feature point based on the first similarity feature and the first value feature point; the first attention feature acquisition unit is used to obtain the first attention feature based on the first attention values ​​of multiple target feature points.

[0143] In some possible implementations, the data processing module further includes: a selection submodule for selecting a plurality of feature points from the first query feature as target feature points; and a region determination submodule for determining, based on a preset size, a local region corresponding to each target feature point.

[0144] In some possible implementations, the first query feature includes any one of a vector and a matrix; when the first query feature is a vector, the preset size includes a preset number of feature points, and the preset number of feature points is less than the total number of feature points of the first query vector, and the local area is a vector centered on the target feature point and having a number of feature points equal to the preset number of feature points; when the first query feature is a matrix, the preset size includes a preset number of rows and a preset number of columns, and the preset number of rows is less than the total number of rows of the first query feature, and the preset number of columns is less than the total number of columns of the first query feature, and the local area is a rectangular area centered on the target feature point, with a preset number of rows as a height, and a preset number of columns as a width.

[0145] In some possible implementations, a sliding window method is used to obtain the first attention feature. Accordingly, the first attention processing submodule includes, in addition to the first similarity determination unit, the first attention value acquisition unit, and the first attention feature acquisition unit, a sliding setting unit and a sliding unit. The sliding setting unit is used to set the sliding window and step size according to the preset size of the local area; the sliding unit is used to slide the sliding window along the first query feature with a step size starting from a preset initial sliding position, and determine the target feature point corresponding to the sliding window in each sliding operation, the multiple first key feature points corresponding to the sliding window in the first key feature, and the first value feature point corresponding to the sliding window in the first value feature; the first similarity determination unit is used to determine the similarity between the target feature point and each first key feature point, and obtain the first similarity feature corresponding to the target feature point; the first attention value acquisition unit is used to obtain the first attention value corresponding to the target feature point based on the first similarity feature and the first value feature point; the first attention feature acquisition unit is used to obtain the first attention feature based on the first attention values ​​of the multiple target feature points.

[0146] In some possible implementations, the first query feature is the same as the first key feature. When determining the similarity between the target feature point and each first key feature point, the first similarity determination unit includes: upon obtaining the similarity Sij between the i-th target feature point and the j-th first key feature point, determining the similarity Sji between the j-th first query feature point and the i-th target feature point based on Sij; wherein i and j are both integers greater than or equal to 1 and less than or equal to M, M is the total number of feature points in the first query feature or the first key feature, and i≥j or i≤j.

[0147] In some possible implementations, the output feature determination submodule includes: a first transformation unit and a first superposition unit. The first transformation unit is configured to perform a linear transformation on the first attention feature to obtain a first matching attention feature of the same size as the input data; and the first superposition unit is configured to superimpose the first matching attention feature with the input data to obtain an output feature corresponding to the input data.

[0148] In some possible implementations, the output feature determination submodule includes: a rearrangement unit and a feature acquisition unit. The rearrangement unit is configured to rearrange the first attention value based on the position information of the target feature point corresponding to the first attention value in the first attention feature in the input data to obtain a second matching attention feature; and the feature acquisition unit is configured to obtain an output feature corresponding to the input data based on the second matching attention feature and the feature point in the input data corresponding to the target feature point.

[0149] In some possible implementations, feature fusion is performed between N levels of convolutional layers of the target neural network based on the second attention mechanism. The scales of the features output by each level of convolutional layer are different, and N is an integer greater than or equal to 2. Accordingly, the data processing module can be used to implement feature fusion based on the second attention mechanism.

[0150] Figure 15 A block diagram of a data processing device provided by an embodiment of the present disclosure. Figure 15 The data processing device includes: a second attention processing submodule 151, a first updating submodule 152, a third attention processing submodule 153 and a second updating submodule 154. Among them, the second attention processing submodule 151 is used to determine the n-th level second attention feature for the n-th level convolution layer according to the n-1-th level intermediate feature of the n-1-th level convolution layer and the n-th level initial feature output by the n-th level convolution layer, where n is an integer and 2≤n≤N-1; the first updating submodule 152 is used to update the n-th level initial feature according to the n-th level second attention feature to obtain the n-th level intermediate feature; the third attention processing submodule 153 is used to determine the n-th level third attention feature according to the n+1-th level fusion feature and the n-th level intermediate feature of the n+1-th level convolution layer; the second updating submodule 154 is used to update the n-th level intermediate feature according to the n-th level third attention feature to obtain the n-th level fusion feature, wherein the initial feature is the feature of the data to be processed after the convolution layer of the target neural network processes the feature, and the processing result is obtained after the fusion feature is processed by the network layer after the convolution layer.

[0151] In some possible implementations, for the first-level convolutional layer, since there is no convolutional layer with a lower feature level, the first-level intermediate features are set equal to the first-level initial features. Accordingly, the first-level fusion features are obtained by updating the first-level intermediate features based on the first-level third attention features, and the first-level third attention features are obtained based on the second-level fusion features and the first-level intermediate features.

[0152] In some possible implementations, for the Nth level convolution layer, since there is no convolution layer with a higher feature level, the Nth level fusion feature is set equal to the Nth level intermediate feature.

[0153] In some possible implementations, the second attention processing submodule includes: a second transformation unit, a third transformation unit, a first mapping unit, a first fusion region determination unit, a second similarity determination unit, a first feature point determination unit, and a second attention feature acquisition unit. The second transformation unit is used to perform a linear transformation on the n-th level initial feature to obtain the n-th level second query feature corresponding to the n-th level initial feature; the third transformation unit is used to perform a linear transformation on the n-1th level intermediate feature to obtain the n-1th level second key feature and the n-1th level second value feature corresponding to the n-1th level intermediate feature; the first mapping unit is used to determine the mapping relationship between each feature point of the n-th level second query feature and each feature point of the n-1th level second key feature; the first fusion region determination unit is used to determine each feature point of the n-th level second query feature according to the mapping relationship. A feature fusion area corresponding to the n-1th level second key feature; a second similarity determination unit, used to determine the similarity between each feature point in the n-th level second query feature and each feature point in the n-1th level second key feature in the feature fusion area, to obtain the n-th level second similarity feature; a first feature point determination unit, used to determine the n-1th level second value feature point corresponding to the feature fusion area in the n-1th level second value feature; a second attention feature acquisition unit, used to determine the inner product between the n-th level second similarity feature and the n-1th level second value feature point, to obtain the n-th level second attention feature.

[0154] In some possible implementations, the first updating submodule includes a second superposition unit, configured to superimpose the n-th level second attention feature with the n-th level initial feature to obtain the n-th level intermediate feature.

[0155] In some possible implementations, the third attention processing submodule includes: a fourth transformation unit, a fifth transformation unit, a second mapping unit, a second fusion region determination unit, a third similarity determination unit, a second feature point determination unit, and a third attention feature acquisition unit. Among them, the fourth transformation unit is used to perform a linear transformation on the n-th level intermediate feature to obtain the n-th level third query feature corresponding to the n-th level intermediate feature; the fifth transformation unit is used to perform a linear transformation on the n+1-th level fusion feature to obtain the n+1-th level third key feature and the n+1-th level third value feature corresponding to the n+1-th level fusion feature; the second mapping unit is used to determine the mapping relationship between each feature point of the n-th level third query feature and each feature point of the n+1-th level third key feature; the second fusion region determination unit is used to determine each feature point of the n-th level third query feature according to the mapping relationship. A feature fusion area corresponding to the n+1th level third key feature; a third similarity determination unit, used to determine the similarity between each feature point in the nth level third query feature and each feature point in the n+1th level third key feature in the feature fusion area, to obtain the nth level third similarity feature; a second feature point determination unit, used to determine the n+1th level third value feature point in the n+1th level third value feature corresponding to the feature fusion area; a third attention feature acquisition unit, used to determine the inner product between the nth level third similarity feature and the n+1th level third value feature point, to obtain the nth level third attention feature.

[0156] In some possible implementations, the second updating submodule includes a third superposition unit, configured to superimpose the n-th level third attention feature with the n-th level intermediate feature to obtain the n-th level fusion feature.

[0157] Figure 16 A block diagram of an electronic device provided in an embodiment of the present disclosure.

[0158] Reference Figure 16 An embodiment of the present disclosure provides an electronic device, which includes: at least one processor 1601; at least one memory 1602, and one or more I / O interfaces 1603 connected between the processor 1601 and the memory 1602; wherein the memory 1602 stores one or more computer programs that can be executed by the at least one processor 1601, and the one or more computer programs are executed by the at least one processor 1601 to enable the at least one processor 1601 to perform the above-mentioned data processing method.

[0159] It should be noted that the data processing method provided in the embodiments of the present disclosure can also be applied to electronic devices based on many-core systems. Figure 17 A block diagram of an electronic device provided in an embodiment of the present disclosure.

[0160] Reference Figure 17An embodiment of the present disclosure provides an electronic device, which includes multiple processing cores 1701 and an on-chip network 1702, wherein the multiple processing cores 1701 are connected to the on-chip network 1702, and the on-chip network 1702 is used to exchange data between the multiple processing cores and external data.

[0161] One or more instructions are stored in one or more processing cores 1701 , and the one or more instructions are executed by one or more processing cores 1701 , so that the one or more processing cores 1701 can execute the above-mentioned data processing method.

[0162] In some embodiments, the electronic device may be a brain-inspired chip. Because brain-inspired chips can use vectorized computing and require external memory, such as Double Data Rate (DDR) synchronous dynamic random access memory, to load parameters such as weight information of the neural network model, the disclosed embodiments utilize batch processing for higher computational efficiency.

[0163] The present disclosure also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the above-mentioned data processing method when executed by a processor / processing core. The computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium.

[0164] An embodiment of the present disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above-mentioned data processing method.

[0165] It will be understood by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware implementation, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable storage medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium).

[0166] As is well known to those skilled in the art, the term computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information (such as computer-readable program instructions, data structures, program modules or other data). Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technology, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, as is well known to those skilled in the art, communication media typically contains computer-readable program instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0167] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0168] The computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and conventional procedural programming languages ​​such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., utilizing an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions. The electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0169] The computer program product described herein may be implemented in hardware, software, or a combination thereof. In one embodiment, the computer program product is implemented as a computer storage medium. In another embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0170] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0171] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0172] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0173] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and the part of the module, program segment or instruction contains one or more executable instructions for realizing the prescribed logical function. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the prescribed function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0174] Example embodiments have been disclosed herein, and although specific terms are employed, they are used and should be interpreted only in a general illustrative sense and not for purposes of limitation. In some instances, it will be apparent to those skilled in the art that, unless otherwise expressly indicated, features, characteristics, and / or elements described in conjunction with a particular embodiment may be used alone or in combination with features, characteristics, and / or elements described in conjunction with other embodiments. Therefore, it will be understood by those skilled in the art that various changes in form and detail may be made without departing from the scope of the present disclosure as set forth in the appended claims.

Claims

1. A data processing method, characterized in that: include: Inputting the data to be processed into the target neural network for processing to obtain a processing result of the data to be processed, wherein at least one convolution layer of the target neural network is an attention convolution layer based on the first attention mechanism, and / or, feature fusion is performed between at least two levels of convolution layers of the target neural network based on the second attention mechanism, The first attention mechanism includes a self-attention mechanism for a local region of a feature, and the second attention mechanism includes an attention mechanism for a local region of the output feature between output features of different scales; The data to be processed includes a feature map, the feature map includes a plurality of pixels, the first attention mechanism is used to perform self-attention mechanism processing on the pixels in a local area of ​​the feature map, and the second attention mechanism is used to perform feature fusion based on the correlation relationship between pixels in the spatial dimension of output features of different scales; the target neural network is used to perform an image processing task, and the processing result of the data to be processed includes an image processing result; The target neural network includes an attention convolution layer, and the inputting of the data to be processed into the target neural network for processing to obtain a processing result of the data to be processed includes: For any attention convolution layer, perform a linear transformation on the input data of the attention convolution layer to obtain a first query feature, a first key feature, and a first value feature corresponding to the input data; determining, based on the first query feature, the first key feature, and the first value feature, a first attention feature corresponding to a plurality of target feature points of the first query feature; Determine an output feature corresponding to the attention convolution layer based on the first attention feature and the input data.

2. The method according to claim 1, characterized in that The first attention feature includes a first attention value corresponding to each target feature point, the first attention value is determined for a local area corresponding to the target feature point, the local area corresponding to the target feature point is an area in the first query feature centered on the target feature point and determined according to a preset size, the preset size being smaller than the size of the first query feature, and the first attention value is used to characterize an association relationship between each feature point in the local area and the target feature point; The input data is data after the data to be processed is processed by the network layer before the attention convolution layer, and the processing result is obtained after the output feature is processed by the network layer after the attention convolution layer.

3. The method according to claim 2, characterized in that The determining, based on the first query feature, the first key feature, and the first value feature, a first attention feature corresponding to a plurality of target feature points of the first query feature includes: For each of the target feature points, determining a plurality of first key feature points corresponding to the local area in the first key feature, and a first value feature point corresponding to the local area in the first value feature; Determine the similarity between the target feature point and each of the first key feature points, and obtain a first similarity feature corresponding to the target feature point; Obtaining a first attention value corresponding to the target feature point according to the first similarity feature and the first value feature point; The first attention feature is obtained according to the first attention values ​​of the plurality of target feature points.

4. The method according to claim 3, characterized in that Before determining the first attention feature corresponding to the plurality of target feature points of the first query feature according to the first query feature, the first key feature, and the first value feature, the method further includes: Selecting multiple feature points from the first query feature as the target feature points; According to the preset size, a local area corresponding to each target feature point is determined.

5. The method according to claim 4, characterized in that The first query feature includes any one of a vector and a matrix; In the case where the first query feature is a vector, the preset size includes a preset number of feature points, and the preset number of feature points is less than the total number of feature points of the first query vector, and the local area is a vector centered on the target feature point and having a number of feature points equal to the preset number of feature points; In the case where the first query feature is a matrix, the preset size includes a preset number of rows and a preset number of columns, and the preset number of rows is smaller than the total number of rows of the first query feature, and the preset number of columns is smaller than the total number of columns of the first query feature. The local area is a rectangular area centered on the target feature point, with the preset number of rows as the height and the preset number of columns as the width.

6. The method according to claim 2, characterized in that The determining, based on the first query feature, the first key feature, and the first value feature, a first attention feature corresponding to a plurality of target feature points of the first query feature includes: Setting a sliding window and a step size according to a preset size of the local area; Starting from a preset initial sliding position, sliding the sliding window along the first query feature with the step size, and determining a target feature point corresponding to the sliding window in each sliding operation, a plurality of first key feature points corresponding to the sliding window in the first key feature, and a first value feature point corresponding to the sliding window in the first value feature; Determine the similarity between the target feature point and each of the first key feature points, and obtain a first similarity feature corresponding to the target feature point; Obtaining a first attention value corresponding to the target feature point according to the first similarity feature and the first value feature point; The first attention feature is obtained according to the first attention values ​​of the plurality of target feature points.

7. The method according to claim 3 or 6, characterized in that The first query feature is the same as the first key feature; The determining the similarity between the target feature point and each of the first key feature points includes: In the case of obtaining the similarity Sij between the i-th target feature point and the j-th first key feature point, determining the similarity Sji between the j-th first query feature point and the i-th target feature point according to the Sij; Wherein, i and j are both integers greater than or equal to 1 and less than or equal to M, M is the total number of feature points in the first query feature or the first key feature, and i≥j or i≤j.

8. The method according to claim 2, characterized in that The determining, according to the first attention feature and the input data, an output feature corresponding to the attention convolution layer includes: Performing a linear transformation on the first attention feature to obtain a first matching attention feature having the same size as the input data; The first matching attention feature is superimposed on the input data to obtain an output feature corresponding to the input data.

9. The method according to claim 2, characterized in that The determining, according to the first attention feature and the input data, an output feature corresponding to the attention convolution layer includes: Rearranging the first attention value according to position information of the target feature point corresponding to the first attention value in the first attention feature in the input data to obtain a second matching attention feature; Based on the second matching attention feature and the feature point in the input data corresponding to the target feature point, an output feature corresponding to the input data is obtained.

10. The method according to claim 1, characterized in that The target neural network performs feature fusion between N levels of convolutional layers based on the second attention mechanism. The scales of the features output by each level of convolutional layer are different. N is an integer greater than or equal to 2. The step of inputting the data to be processed into the target neural network for processing to obtain a processing result of the data to be processed includes: For the n-th convolutional layer, determine the n-th level second attention feature according to the n-1-th level intermediate feature of the n-1-th level convolutional layer and the n-th level initial feature output by the n-th level convolutional layer, where n is an integer and 2≤n≤N-1; According to the n-th level second attention feature, the n-th level initial feature is updated to obtain the n-th level intermediate feature; Determine the n-th level third attention feature according to the n+1-th level fusion feature of the n+1-th level convolutional layer and the n-th level intermediate feature; According to the third attention feature of the nth level, the intermediate feature of the nth level is updated to obtain the fusion feature of the nth level, The initial features are features of the data to be processed after being processed by the convolution layer of the target neural network, and the processing results are obtained after the fusion features are processed by the network layer after the convolution layer.

11. The method according to claim 10, characterized in that The method of determining, for the n-th level convolutional layer, the n-th level second attention feature according to the n-1-th level intermediate feature of the n-1-th level convolutional layer and the n-th level initial feature output by the n-th level convolutional layer, includes: Performing a linear transformation on the n-th level initial feature to obtain an n-th level second query feature corresponding to the n-th level initial feature; Performing a linear transformation on the n-1th level intermediate feature to obtain an n-1th level second key feature and an n-1th level second value feature corresponding to the n-1th level intermediate feature; Determining a mapping relationship between each feature point of the n-th level second query feature and each feature point of the n-1-th level second key feature; Determining, according to the mapping relationship, a feature fusion region where each feature point of the n-th level second query feature corresponds to the n-1-th level second key feature; Determine the similarity between each feature point in the n-th level second query feature and each feature point in the n-1-th level second key feature in the feature fusion region to obtain an n-th level second similarity feature; Determining an n-1th level second value feature point corresponding to the feature fusion area in the n-1th level second value feature; Determine the inner product between the n-th level second similarity feature and the n-1-th level second value feature point to obtain the n-th level second attention feature.

12. The method according to claim 10, characterized in that The determining of the n-level third attention feature according to the n+1-level fusion feature of the n+1-level convolutional layer and the n-level intermediate feature includes: Performing a linear transformation on the n-th level intermediate feature to obtain an n-th level third query feature corresponding to the n-th level intermediate feature; Performing a linear transformation on the n+1th level fusion feature to obtain an n+1th level third key feature and an n+1th level third value feature corresponding to the n+1th level fusion feature; Determining a mapping relationship between each feature point of the nth-level third query feature and each feature point of the n+1th-level third key feature; Determining, according to the mapping relationship, that each feature point of the third query feature of the nth level corresponds to a feature fusion region of the third key feature of the n+1th level; Determine the similarity between each feature point in the n-th level third query feature and each feature point in the (n+1)-th level third key feature located in the feature fusion region, to obtain an n-th level third similarity feature; Determining an n+1th level third-value feature point corresponding to the feature fusion area in the n+1th level third-value feature; Determine the inner product between the n-th level third similarity feature and the n+1-th level third value feature point to obtain the n-th level third attention feature.

13. The method according to claim 10, characterized in that The updating of the n-level initial feature according to the n-level second attention feature to obtain the n-level intermediate feature includes: The n-th level second attention feature is superimposed on the n-th level initial feature to obtain the n-th level intermediate feature, The updating of the n-level intermediate features according to the n-level third attention features to obtain the n-level fusion features includes: The n-th level third attention feature is superimposed on the n-th level intermediate feature to obtain the n-th level fusion feature.

14. The method according to claim 10, characterized in that The first-level intermediate feature is equal to the first-level initial feature, the first-level fusion feature is obtained by updating the first-level intermediate feature based on the first-level third attention feature, and the first-level third attention feature is obtained according to the second-level fusion feature and the first-level intermediate feature; The Nth level fusion feature is equal to the Nth level intermediate feature.

15. A data processing device, characterized in that: include: A data processing module is used to input the data to be processed into the target neural network for processing, and to obtain the processing result of the data to be processed, wherein at least one convolution layer of the target neural network is an attention convolution layer based on the first attention mechanism, and / or, feature fusion is performed between at least two levels of convolution layers of the target neural network based on the second attention mechanism, The first attention mechanism includes a self-attention mechanism for a local area of ​​a feature, and the second attention mechanism includes an attention mechanism between output features of different scales; The data to be processed includes a feature map, the feature map includes a plurality of pixels, the first attention mechanism is used to perform self-attention mechanism processing on the pixels in a local area of ​​the feature map, and the second attention mechanism is used to perform feature fusion based on the correlation relationship between pixels in the spatial dimension of output features of different scales; the target neural network is used to perform an image processing task, and the processing result of the data to be processed includes an image processing result; The target neural network includes an attention convolution layer, and the inputting of the data to be processed into the target neural network for processing to obtain a processing result of the data to be processed includes: For any attention convolution layer, perform a linear transformation on the input data of the attention convolution layer to obtain a first query feature, a first key feature, and a first value feature corresponding to the input data; determining, based on the first query feature, the first key feature, and the first value feature, a first attention feature corresponding to a plurality of target feature points of the first query feature; Determine an output feature corresponding to the attention convolution layer based on the first attention feature and the input data.

16. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs that can be executed by the at least one processor. The one or more computer programs are executed by the at least one processor to enable the at least one processor to perform the data processing method according to any one of claims 1 to 14.

17. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the computer program implements the data processing method according to any one of claims 1 to 14.

Images