Image processing method, device, chip, equipment and computer readable storage medium

Abstract

The present disclosure provides an image processing method, device, chip, equipment and computer readable storage medium, the method comprises: obtaining the input feature of the image to be processed; split the input feature into multiple sub-features; determine the first attention feature and the second attention feature corresponding to the multiple sub-features, wherein the first attention feature is obtained by self-attention calculation based on multiple element features contained in the sub-feature, and the second attention feature is obtained by attention calculation between the sub-feature and other sub-features; the feature fusion processing is carried out on the first attention feature and the second attention feature to obtain the output feature; based on the output feature, the image to be processed is processed. According to the embodiment of the present disclosure, the operation complexity of self-attention calculation can be reduced and the global modeling ability of self-attention calculation can be retained, which can improve the accuracy of image processing and the efficiency of image processing.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to an image processing method, apparatus, chip, device, and computer-readable storage medium. Background Technology

[0002] Attention is a crucial concept in neural networks (NNs), derived from human visual attention. When computational resources are limited, attention allows for the allocation of resources to more important tasks, rather than processing tasks globally. Self-attention (SA) is an improvement on the attention mechanism, reducing reliance on external information and excelling at capturing the internal correlations of data or features.

[0003] When performing self-attention computation, computational complexity and global modeling capability are important indicators for evaluating the results of self-attention computation. Summary of the Invention

[0004] This disclosure provides an image processing method, apparatus, chip, device, and computer-readable storage medium, which can reduce the computational complexity of self-attention calculation in image processing while retaining the global modeling capability of self-attention calculation, thereby improving the accuracy and efficiency of image processing.

[0005] In a first aspect, this disclosure provides an image processing method, which includes: acquiring input features of an image to be processed; splitting the input features into multiple sub-features; determining a first attention feature and a second attention feature corresponding to the multiple sub-features, wherein the first attention feature is obtained by self-attention calculation based on multiple element features contained in the sub-features, and the second attention feature is obtained by attention calculation between the sub-features and other sub-features; performing feature fusion processing on the first attention feature and the second attention feature to obtain an output feature; and processing the image to be processed based on the output feature.

[0006] Secondly, this disclosure provides an image processing apparatus, comprising: an acquisition module for acquiring input features of an image to be processed; a splitting module for splitting the input features into multiple sub-features; a determination module for determining a first attention feature and a second attention feature corresponding to the multiple sub-features, wherein the first attention feature is obtained by self-attention calculation based on multiple element features contained in the sub-features, and the second attention feature is obtained by attention calculation between the sub-features and other sub-features; a fusion module for performing feature fusion processing on the first attention feature and the second attention feature to obtain an output feature; and a processing module for processing the image to be processed based on the output feature.

[0007] Thirdly, this disclosure provides an artificial intelligence chip, which includes at least an attention computing array and other computing arrays; wherein the attention computing array includes multiple first computing arrays and one second computing array; the other computing arrays include: a decomposition computing unit, a convergence computing unit, and a feature fusion computing unit; the decomposition computing unit is used to decompose the input features of the image to be processed into multiple sub-features; the convergence computing unit is used to perform downsampling processing on each of the sub-features to obtain a downsampled feature of each sub-feature; the first computing array is used to perform self-attention calculation based on multiple element features contained in the sub-features to obtain a first attention feature corresponding to each sub-feature; the second computing array is used to perform attention calculation on the sub-features and other sub-features to obtain a second attention feature corresponding to each sub-feature; the feature fusion computing unit is used to perform feature fusion processing on the first attention feature and the second attention feature to obtain an output feature.

[0008] Fourthly, this disclosure provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores one or more computer programs executable by the at least one processor, the one or more computer programs being executed by the at least one processor to enable the at least one processor to perform the image processing method described above.

[0009] Fifthly, this disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor / processor core, implements the above-described image processing method.

[0010] The embodiments provided in this disclosure, after decomposing the input features of the image to be processed into multiple sub-features, each sub-feature can represent a local region of the input features. Self-attention calculation is performed based on multiple element features contained in the sub-features to obtain a first attention feature with local modeling capabilities. Then, attention calculation is performed between the aforementioned sub-features and other sub-features to obtain a second attention feature, resulting in a self-attention calculation result with global modeling capabilities. Feature fusion of the self-attention calculation result with local modeling capabilities and the self-attention calculation result with global modeling capabilities can reduce the computational complexity of self-attention calculation while retaining its global modeling capabilities. According to the embodiments provided in this disclosure, in the process of performing data processing tasks in the field of image processing, using the attention mechanism provided by this image processing method can associate global information and detailed information of the image, making the image processing results more accurate and thus effectively improving the accuracy of image recognition. Furthermore, since the above scheme can reduce the computational complexity of self-attention calculation during the self-attention calculation process, it effectively improves the efficiency of image processing.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0012] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the embodiments of the present disclosure to explain the disclosure and do not constitute a limitation thereof. The above and other features and advantages will become more apparent to those skilled in the art from the detailed description of exemplary embodiments with reference to the accompanying drawings, in which:

[0013] Figure 1 A schematic diagram of a processing procedure based on a self-attention mechanism is shown;

[0014] Figure 2 A schematic diagram of another processing procedure based on a self-attention mechanism is shown;

[0015] Figure 3 A flowchart of an image processing method provided in an embodiment of this disclosure;

[0016] Figure 4 A schematic flowchart illustrating an image processing method according to another embodiment of this disclosure is shown;

[0017] Figure 5 A block diagram of an image processing apparatus provided in an embodiment of this disclosure;

[0018] Figure 6This is a schematic block diagram of an image processing apparatus according to another embodiment of the present disclosure;

[0019] Figure 7 This diagram illustrates the structure of the artificial intelligence chip in an embodiment of the present disclosure.

[0020] Figure 8 This is a block diagram of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0021] To enable those skilled in the art to better understand the technical solutions of this disclosure, exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments of this disclosure to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0022] Where there is no conflict, the various embodiments of this disclosure and the features thereof in the embodiments may be combined with each other.

[0023] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0024] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, they specify the presence of features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof. Words such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0025] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this disclosure, and will not be interpreted as having an idealized or overly formal meaning, unless expressly so defined herein.

[0026] In the field of machine learning, a neural network (NN) is a computational system that processes data by mimicking the structure of biological neural networks. Neural networks are generated by a large number of interconnected neurons and have powerful nonlinear and adaptive data processing capabilities. They are widely used in fields such as image processing, speech recognition, and natural language processing (NLP).

[0027] Attention is an important concept in neural networks. Its essence is to obtain new feature representations by linearly weighting the relationships between things. Self-attention (SA) is a variant of attention, which reduces the dependence on external information and is better at capturing the internal correlations of data or features.

[0028] Specifically, the self-attention mechanism can be implemented through self-attention computation performed by a self-attention model. This model employs a query-key-value (QKV) pattern to perform attention computation on the input features of the image to be processed, thereby reducing reliance on external information and capturing the internal correlations of data or features. The query vector, key vector, and value vector all originate from the input features themselves, and are vectors obtained by linearly transforming the input feature vectors. This linear transformation can improve the model's fitting ability.

[0029] The specific calculation process of the self-attention mechanism can be summarized into two steps: the first step is to calculate the weight coefficients based on the query and the key, and the second step is to perform a weighted summation of the values ​​based on the weight coefficients. The first step can be further divided into two stages: the first stage calculates the similarity or relevance between the query and the key; the second stage normalizes the similarity or relevance calculated in the first stage to obtain the weight coefficients.

[0030] To facilitate understanding, the following will be combined with... Figures 1 to 4 Describe the self-attention calculation process for different examples.

[0031] Figure 1 A schematic diagram of a processing procedure based on a self-attention mechanism is shown. Figure 1 In this context, the input feature is X, which represents the feature map of the image to be processed. For example, the data to be processed can be the feature map of an image or the feature map of a video. The size of X is T×H×W×d. v T represents the number of frames in the image to be processed, and the feature map obtained from each frame has dimensions H×W×d. v H, W and dv These represent the height, width, and number of channels (i.e., feature dimensions) of the feature map for each frame, for example, using... Figure 1 d v =1024, thus X represents an eigenvector of size T×H×W×1024. In Figure 1 middle, Represents matrix multiplication. This indicates pixel-by-pixel addition.

[0032] The following uses d v Taking 1024 as an example, this describes the process of calculating attention for the input feature X using a non-local model. First, a linear transformation is performed on the input feature X (e.g., using a 1×1×1 convolution to compress the number of channels, resulting in an output feature map with d channels). v / 2), to obtain θ, g consists of three feature maps, each with a size of (T×H×W×512); secondly, for θ, g performs a shape transformation (Reshape operation) to obtain matrix representations of the three feature maps, thereby merging the dimensions of the three feature maps except for the number of channels. After the shape transformation, the dimensions of the three feature maps are THW×512, 512×THW, and THW×512, respectively. Then, θ undergoes a shape transformation and... The shape transformation results are multiplied by a matrix to obtain autocorrelation features (with dimensions THW×THW). This step aims to calculate the autocorrelation of the features, that is, to obtain the relationship between each pixel (or superpixel) in each frame and all pixels (or superpixels) in all other frames. Then, the autocorrelation features are normalized (e.g., Softmax) to obtain attention weights with a value range of [0,1]. These attention weights are the autocorrelation coefficients. Finally, the autocorrelation coefficients are multiplied by the shape transformation results of feature g to obtain a THW×512 feature matrix. This feature matrix is ​​then shape transformed to obtain a feature vector with dimensions T×H×W×512. This feature vector is then subjected to a 1×1×1 convolution to expand the channels, resulting in an expanded result with dimensions T×H×W×1024. The expanded result is then added pixel by pixel to obtain the final output result Z.

[0033] In some embodiments, in the field of visual processing, self-attention mechanisms can be used in combination with convolution, i.e., a convolutional Vision-Transformer (ViT) is introduced to avoid feature patterns from being difficult to converge due to a lack of inductive bias.

[0034] As seen from the above processing steps, the core idea of ​​the Non-local model is that when calculating the output of each pixel position, it no longer calculates only with its neighborhood, but also with all positions in the image. This correlation is then used as a weight to represent the similarity between other positions and the current position to be calculated. Image processing based on self-attention using the Non-local model considers the relationship between a pixel in a given frame and all pixels in all frames, thus performing QKV-based self-attention calculation globally on the feature map, achieving a global receptive field for each pixel. Based on the above processing steps, it can be seen that the Self-Attention mechanism can achieve global self-attention feature fusion within the global space of the feature map, i.e., fusing global information. However, when the input feature map is large, the intermediate processing consumes a lot of GPU memory, resulting in high computational complexity and a large computational load.

[0035] Figure 2 This diagram illustrates another processing procedure based on a self-attention mechanism. (For example...) Figure 2 As shown, feature encoding is performed on each element in the input sequence x1, x2, x3, ..., xr to obtain the feature vector I (i.e., Input: I), where I = [a1, a2, a3, ..., ar]; a1 to ar are input vectors with different elements.

[0036] Continue to refer to Figure 2 After the feature vector I is processed by the self-attention layer, the feature vector O (i.e., Output: O) is obtained, where O = [b1, b2, b3, ..., br], and b1 to br are output vectors with different elements.

[0037] The following is combined with Figure 2 This describes the computational steps for calculating attention to input feature I using a Self-Attention Layer.

[0038] First, perform a linear transformation on each input vector in feature vector I to obtain the Query vector, Key vector, and Value vector corresponding to each input vector.

[0039] Specifically, the eigenvectors I, a1, a2, a3, ..., ar, are respectively transformed by the linear transformation matrix w. Q Multiplying these results in multiple query vectors (i.e., Q: q1, q2, q3, ..., qr); then, the a1, a2, a3, ..., ar features in the feature vector I are multiplied by the linear transformation matrix w. KMultiplying them yields multiple key vectors (i.e., K: k1, k2, k3, ..., kr); the a1, a2, a3, ..., ar in the eigenvector I are then multiplied by the linear transformation matrix w. V Multiplying them yields multiple Value vectors (i.e., V: v1, v2, v3, ..., vr).

[0040] Secondly, similarity is calculated based on the query vector and the key vector.

[0041] Specifically, for any Query vector, the inner product of the matrix transpose (T) of the Query vector and all Key vectors is calculated to obtain the correlation coefficient between the Query vector and each Key vector; where the correlation coefficient is used to characterize feature similarity.

[0042] For example, in Figure 2 In the query vector, for q1, the matrix transpose of q1 is multiplied by the inner product of all key vectors to calculate the similarity between q1 and all keys. This similarity value is then divided by... Obtain the correlation coefficient α1 between q1 and all keys (i.e., α). 1,1 α 1,2 α 1,3 , ..., α 1,r );in, For q2, q2, q3, ..., qr in the Query vector, perform the same operation as q1 to obtain the relevance coefficients of q2, q2, q3, ..., qr with all keys respectively.

[0043] In this step, d is the dimension of each query vector, calculated by dividing by The operation can narrow the range of the dot product and also keep the gradient value stable during the model training process;

[0044] Next, the correlation coefficients are normalized (e.g., using Softmax) to obtain attention weights with a value range of [0,1]. In this step, the Softmax operation not only normalizes the correlation coefficients but also highlights the weights of important features through the inherent processing mechanism of SoftMax.

[0045] Finally, the Value vector is multiplied by the transpose of the correlation coefficient matrix α to obtain the output eigenvector O.

[0046] As described above, the eigenvector O can be represented as [b1, b2, b3, ..., br]; where, The methods for obtaining b2, b3, ..., br are similar to those for obtaining b1, and will not be repeated here.

[0047] pass Figure 2 As can be seen from the described attention mechanism processing process, this self-attention operation process needs to consider the correlation between each query vector and all key vectors, and then perform a weighted summation of the value vector according to the weight coefficients corresponding to the correlation. That is, QKV self-attention calculation also needs to be performed in the global scope to realize the feature fusion of the global self-attention mechanism.

[0048] pass Figure 2 The processing procedure shows that, with Figure 1 The processing is similar. The Self Attention mechanism can realize feature fusion of global self attention mechanism in the global space of feature map. However, when the input feature map is large, the intermediate processing will also occupy a lot of GPU memory, and the computational complexity and amount of computation are high.

[0049] This disclosure provides an image processing method, apparatus, artificial intelligence chip, electronic device, and computer-readable storage medium. The image processing method according to the embodiments of this disclosure can reduce the computational complexity of self-attention calculation while retaining the global modeling capability of self-attention calculation, thereby improving the accuracy and efficiency of image processing.

[0050] The image processing method according to embodiments of this disclosure can be executed by an electronic device such as a terminal device or a server. The terminal device can be an in-vehicle device, user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, wearable device, etc. The method can be implemented by a processor calling computer-readable program instructions stored in memory. Alternatively, the method can be executed by a server.

[0051] Figure 3 A flowchart illustrating an image processing method provided in an embodiment of this disclosure. (Refer to...) Figure 3 The image processing method may include the following steps.

[0052] S310, Obtain the input features of the image to be processed.

[0053] In this step, the input features of the image to be processed can be image features obtained by feature extraction of the image to be processed. Image features can be basic information that distinguishes one image from another, used to identify and label image content. Image features may include at least one of color features, texture features, and spatial relationship features, etc. The embodiments in this specification do not specifically limit the specific content of the above-mentioned input features.

[0054] In step S310, the input features can be represented as a tensor (b, c, h, w), where the tensor represents multidimensional data, b represents the number of feature maps (batch), c represents the number of channels in the feature map, and h and w represent the height and width of the feature map. For example, taking image processing as an example, batch can be used to represent the number of images processed at one time. For a RED Green Blue (RGB) color image, the number of channels in the input features can be 3; height represents the image height, and width represents the image width.

[0055] S320 splits the input features into multiple sub-features.

[0056] In this step, the sub-features can be local features of the input image; after the input features are split into multiple sub-features, each sub-feature can represent a local region of the input features.

[0057] In some embodiments, a fold function can be used to extract multiple feature patches from the input features. Each patch can represent a sub-feature of size N×M, where N and M are the height and width of the sub-feature obtained by folding, respectively. The specific values ​​of N and M can be set according to actual needs, and this embodiment does not impose specific limitations.

[0058] In this step, when there are multiple input features that need to be split, the operation of splitting each input feature into multiple sub-features can be performed in parallel for the multiple input features. In this embodiment of the disclosure, the multiple representations are greater than one.

[0059] S330, determine the first attention feature and the second attention feature corresponding to multiple sub-features, wherein the first attention feature is obtained by self-attention calculation based on the element features contained in the sub-features, and the second attention feature is obtained by attention calculation between the sub-features and other sub-features.

[0060] For example, multiple feature patches in the input features, each patch representing a sub-feature of size N×M, self-attention calculation is performed based on each element contained in each N×M sub-feature to obtain the attention feature of each N×M sub-feature, and each attention feature is the result of self-attention calculation with local modeling capability.

[0061] For example, multiple feature patches in the input features, each patch representing a sub-feature of size N×M, are used to perform attention calculations between each N×M sub-feature and other N×M sub-features to obtain attention features between multiple N×M sub-features. These attention features are the result of self-attention calculations with global modeling capabilities.

[0062] In this embodiment of the disclosure, attention calculation can be based on a self-attention mechanism, that is, attention calculation is performed on the input features of the image to be processed using the QKV mode. It should be understood that attention calculation can also be performed on the input features of the image to be processed using other modes to reduce the dependence of image features on external information, capture the internal correlation of image data or image features, and improve the fitting ability of image data or image features.

[0063] S340 performs feature fusion processing on the first attention feature and the second attention feature to obtain the output feature.

[0064] In some embodiments, the purpose of feature fusion is to combine features extracted from an image into a single feature that is more discriminative than the input features. For example, feature fusion can be performed by concatenating or combining features, or by other linear operations between features. This disclosure does not impose specific limitations; in practical applications, the feature fusion method can be selected based on the image processing requirements.

[0065] The S350 processes the image to be processed based on the output features.

[0066] In this embodiment, image classification, image recognition, and image enhancement processing can be performed based on the output features, i.e., the output image features. The specific processing method can be determined according to actual image processing needs, and this embodiment does not impose any specific limitations.

[0067] In this embodiment, local modeling reduces the computational complexity of self-attention calculation, while global modeling enhances the ability to capture contextual information. Feature fusion of the self-attention calculation results with local and global modeling capabilities reduces computational complexity while preserving global modeling capabilities. According to the embodiments provided in this disclosure, in performing data processing tasks in the field of image processing, the attention mechanism provided by this image processing method can correlate global and detailed information of the image, making the image processing results more accurate and effectively improving the accuracy of image recognition. Furthermore, since the above scheme reduces the computational complexity of self-attention calculation, it effectively improves the efficiency of image processing.

[0068] In some embodiments, step S330, which determines the first attention feature and the second attention feature corresponding to the multiple sub-features, may specifically include:

[0069] S01, perform attention calculation based on the element features contained in each sub-feature to obtain the first attention feature corresponding to each sub-feature.

[0070] S02, perform downsampling on each sub-feature to obtain the downsampled feature of each sub-feature.

[0071] In the steps, each element in the downsampled feature corresponds to a sub-feature.

[0072] In some optional implementations, if there are multiple sub-features, the downsampling operation for each sub-feature can be performed in parallel to obtain the downsampled feature corresponding to each sub-feature.

[0073] S03, perform attention calculation based on the downsampled features of multiple sub-features to obtain the second attention feature corresponding to each sub-feature.

[0074] According to the image processing method of this disclosure, after the input features are split into multiple sub-features, each sub-feature can represent a local region of the input features. Attention calculation is performed on the multiple element features contained in each sub-feature to obtain a self-attention calculation result with local modeling capability, thereby reducing the complexity of self-attention calculation. Furthermore, attention calculation is performed on downsampled features, which represent features after compression and dimensionality reduction of the input feature vector. Attention calculation based on downsampled features can obtain a self-attention calculation result with global modeling capability within the global range of the size of the downsampled features. Feature fusion is performed on the self-attention calculation result with local modeling capability and the self-attention calculation result with global modeling capability, which can reduce the computational complexity of self-attention calculation while retaining the global modeling capability of self-attention calculation.

[0075] In some embodiments, step S320 may specifically include: splitting the input feature into multiple N×M sub-features; where N and M are both integers greater than or equal to 1; if any sub-feature obtained by splitting has missing values, then zero-padding is performed on the missing values.

[0076] In this embodiment of the disclosure, the input features of the image to be processed are split into multiple sub-features of a specified size, providing a data foundation for subsequent self-attention calculation with local modeling capability within the size range of the split sub-features.

[0077] In some embodiments, step S01 may specifically include the following steps.

[0078] S11, perform a preset number of linear transformations on each element feature contained in each sub-feature to obtain the query feature, key feature and value feature corresponding to each element feature of the sub-feature.

[0079] In step S11, each sub-feature undergoes a predetermined number of linear transformations to obtain the query feature, key feature, and value feature corresponding to each sub-feature. Specifically, the query feature corresponding to any sub-feature can be represented by its Query vector, the key feature by its Key vector, and the value feature by its Value vector.

[0080] In this step, the preset number of transformations can be determined based on the number of linear transformations performed. For example, if a sub-feature is transformed at least once to obtain the corresponding query feature, at least once to obtain the corresponding key feature, and at least once to obtain the corresponding value feature, then the preset number of transformations is greater than or equal to 3.

[0081] S12, perform self-attention calculation on the query feature, key feature and value feature corresponding to each element feature of each sub-feature to obtain the first attention feature corresponding to each sub-feature.

[0082] In step S12, the QKV self-attention calculation process based on the Query vector, Key vector, and Value vector of any sub-feature can be referred to the above-described combination. Figures 1-2 Any processing procedure based on a self-attention mechanism described herein. This disclosure does not limit the specific processing procedure for QKV self-attention computation.

[0083] In this embodiment, self-attention calculation is performed on each sub-feature separately to obtain the attention feature corresponding to each sub-feature. That is, when the feature map is split into different patches, self-attention calculation is performed on each patch separately to obtain the internal correlation of features within the patch, thereby reducing the complexity of QKV self-attention calculation by leveraging the correlation of features within the patch. Furthermore, since QKV self-attention calculation is performed based on sub-features, the scope of the attention mechanism can be narrowed to the local region of the feature map, and the obtained attention features have a certain inductive bias ability relative to the global features, thereby improving the data processing effect.

[0084] In some embodiments, step S02, which involves downsampling each sub-feature to obtain the downsampled features of each sub-feature, may specifically include the following steps.

[0085] S21, extract features from the input features according to a predetermined step size, where the predetermined size can be the size of a sub-feature; S22, perform average pooling on each extracted feature to obtain multiple average pooling results; S23, concatenate the multiple average pooling results to obtain the downsampled features of the input features.

[0086] As an example, the predetermined size in step S21 can be N×M, where N and M are both integers greater than 1; in step S21, the step size for feature extraction of the input features is the same as the size for splitting the input features, so as to extract features from the patch corresponding to each sub-feature; in step S22, average pooling can refer to: when sliding a window with a step size of N×M on the feature map of the input features, taking the average value of the feature values ​​in each sliding window as the feature value of the extracted feature corresponding to the sliding window; in step S23, the feature value of each feature point in the downsampled features is the feature value of the feature corresponding to a sub-feature (or a sliding window) in the input features.

[0087] As an example, after step S21, the downsampled features can also be obtained based on max pooling. Specifically: max pooling is performed on each extracted feature to obtain multiple max pooling results; the multiple max pooling results are concatenated to obtain the downsampled features of the input features.

[0088] In this embodiment, in the downsampled features of the input features, a feature point can correspond to a sub-feature in the feature map, representing a region or a patch in the feature map. The receptive field of a feature point in the downsampled features is an N×M region, thereby increasing the receptive field of each feature point. Furthermore, when performing QKV self-attention calculation based on the downsampled features, it is beneficial to establish the internal correlation between feature points in the downsampled features, that is, to establish the internal correlation between the sub-features of the input features, and realize the information exchange between the sub-features of the input features.

[0089] In some embodiments, step S03 may specifically include: S31, performing a predetermined number of linear transformations on the downsampled features of multiple sub-features to obtain the query features, key features, and value features corresponding to the downsampled features of each sub-feature; S32, performing self-attention calculation on the query features, key features, and value features corresponding to the downsampled features of multiple sub-features to obtain the second attention features corresponding to each sub-feature.

[0090] In step S32, the QKV self-attention calculation process based on the Query vector, Key vector, and Value vector of the downsampled features can also be referred to the above combined with Figures 1-2 This describes any processing procedure based on a self-attention mechanism. The embodiments disclosed herein do not limit the specific processing procedure for QKV self-attention computation.

[0091] In this embodiment, by performing self-attention calculation on the downsampled features, the attention features corresponding to the downsampled features can be obtained, thereby realizing the acquisition of the internal correlation between inter-slice features, establishing the internal association between the sub-features of the input features, and realizing the information exchange between the sub-features of the input features.

[0092] In some embodiments, step S340 may specifically include the following steps.

[0093] S41, the first attention features corresponding to multiple sub-features are concatenated to obtain the first feature; wherein the size of the first feature is the same as the size of the input feature.

[0094] In step S41, the attention features corresponding to each sub-feature can be spliced ​​together by the tiling (unfold) operation to obtain the first feature.

[0095] S42, upsample the second attention features corresponding to multiple sub-features to obtain the second features; the size of the second features is the same as the size of the input features.

[0096] In step S42, upsampling can be performed through unpooling to obtain the second feature. Unpooling is the inverse operation of pooling. When performing unpooling, the size of the downsampled feature is first restored to the size of the input feature, and then each value in the pooling result is filled into the corresponding position in the region where the corresponding input feature is located.

[0097] S43, perform feature fusion based on the first feature and the second feature to obtain the output feature.

[0098] In this embodiment, the internal correlation of intra-slice features is obtained through the first feature, which can reduce the amount of computation and computational complexity; the internal correlation of inter-slice features is obtained through the second feature, thereby retaining the modeling ability to capture global context information. The first feature and the second feature are fused together to obtain the input feature that combines the advantages of both. This reduces the computational complexity of self-attention computation while retaining the global modeling ability of self-attention computation, thereby achieving highly efficient QKV self-attention computation.

[0099] In some embodiments, S43 may specifically include: S51, determining the average value of elements with the same position in the first feature and the second feature to obtain the third feature; S52, summing the elements with the same position in the third feature and the input feature to obtain the fourth feature; S53, using the fourth feature as the output feature.

[0100] In step S51, the third feature can also be a feature obtained by performing other operations on the elements with the same position in the first feature and the second feature. Such other operations can be any one of summation, maximum value calculation, and multiplication. In step S52, the fourth feature can also be a feature obtained by performing other operations on the elements with the same position in the third feature and the input feature. Such other operations can be any one of mean calculation, maximum value calculation, and multiplication.

[0101] It should be understood that in practical applications, feature fusion of the first feature and the second feature can also be performed in other ways, as can feature fusion of the third feature and the input feature. The specific feature fusion method can be selected according to the actual computational needs.

[0102] In some embodiments, step S52 may specifically include: summing the elements with the same position in the third feature and the input feature to obtain the fourth feature; and determining the fourth feature as the output feature of the image to be processed.

[0103] In this embodiment, the feature fusion result of the first feature and the second feature is combined with the input feature by adding them element by element. Adding elements by element means summing the elements of corresponding points at each position. This element-by-element addition achieves the effect of noise reduction of the feature map, making the details of the data processing result richer, thereby optimizing the data processing result and improving the data processing efficiency.

[0104] In this embodiment of the disclosure, if there are multiple input features, step S43 can be executed in parallel for different input features. Specifically, at least one of steps S51 and S52 can be executed in parallel for multiple input features.

[0105] According to the image processing method of this disclosure, after decomposing the input features of the image to be processed into multiple sub-features, each sub-feature can represent a local region of the input features. Self-attention calculation is performed based on the multiple element features contained in the sub-features to obtain a first attention feature with local modeling capability; and a second attention feature is obtained by performing attention calculation between the sub-features and other sub-features to obtain a self-attention calculation result with global modeling capability; the image features of the image to be processed are output based on the fusion of the first attention feature and the second attention feature. Image processing based on these image features can associate the global information and the detailed information of the image, making the image processing result more accurate. In addition, since the above scheme can reduce the computational complexity of self-attention calculation during the self-attention calculation process, the efficiency of image processing is effectively improved.

[0106] In some embodiments, the image processing method of this disclosure can be applied to an artificial intelligence chip, which includes at least a plurality of attention computing arrays.

[0107] In some embodiments, step S330, which determines the first attention feature and the second attention feature corresponding to multiple sub-features, may specifically include: S61, assigning a first attention calculation array to each sub-feature, and using each first attention calculation array to perform attention calculation based on the element features contained in the corresponding sub-feature to obtain the first attention feature corresponding to each sub-feature; S62, assigning a second attention calculation array to multiple sub-features, and using the second attention calculation array to perform attention calculation on the sub-features and other sub-features to obtain the second attention feature corresponding to each sub-feature.

[0108] In this embodiment, different first attention calculation arrays can be assigned to different sub-features. Each attention calculation array performs attention calculations based on the element features contained in the corresponding sub-feature to obtain the first attention feature corresponding to the sub-feature. Furthermore, a second attention calculation array can be assigned to multiple sub-features, and this second attention calculation array performs attention calculations on the sub-features and other sub-features to obtain the second attention feature corresponding to each sub-feature. Calculating the first and second attention features using different calculation arrays facilitates parallel calculation of attention features, thereby effectively improving data processing efficiency.

[0109] In this embodiment, for multiple sub-features obtained by splitting the input features, when performing QKV self-attention calculation on each sub-feature, since the QKV self-attention calculations between sub-features are relatively independent, a first attention calculation array can be allocated for the QKV self-attention calculation of each sub-feature. The first attention calculation arrays can perform the QKV self-attention calculation of each element within their respective sub-features in parallel. If the downsampled features obtained by downsampling the input features cannot be further split, a second attention array can be exclusively used for QKV self-attention calculation between the sub-features and other sub-features. The attention calculation of the second attention array can be performed in parallel with the attention calculation of the first attention array, thereby improving computational efficiency.

[0110] Figure 4 A schematic flowchart illustrating an image processing method according to another embodiment of this disclosure is shown. Figure 4 In this context, the image processing method includes the following steps.

[0111] like Figure 4 As shown in S401 "Splitting into multiple sub-features", the input features of a predetermined image are split into multiple sub-features.

[0112] In this step, the input features can be represented as a tensor (b, c, h, w), where b represents the number of feature maps, c represents the number of channels in the feature map, and h and w represent the height and width of the feature map. For example, if the input features have h = 6 and w = 12, then the input features can also be represented as a tensor (b, c, 6, 12).

[0113] For example, the input features of a predetermined image are split using the Fold function. Figure 4The example shows that the parameters of the Fold function are k=3 and s=3, which means that the input features are split into 8 sub-features of size 3×3. The multiple sub-features obtained after splitting the input features can be represented as tensors (8b, c, 3, 3), where 8b means that the input features are split into 8 sub-features (the number of feature maps is increased by 8 times), the number of channels c remains unchanged, and the height and width of the feature map are both 3.

[0114] like Figure 4 As shown in S402 "In-chip Attention Calculation", attention calculation is performed on multiple element features contained in each sub-feature to obtain the attention feature corresponding to each sub-feature.

[0115] In this step, Intra-Patch QKV SelfAttention is performed on each sub-feature to obtain the attention feature corresponding to each sub-feature, thereby realizing the acquisition of the internal correlation of the intra-feature.

[0116] Since each sub-feature can represent a region of the input feature, and there is no information exchange between regions or patches of different sub-features, performing attention calculations on multiple element features contained in each sub-feature separately can yield self-attention calculation results with local modeling capabilities, thus limiting the modeling mechanism to a fixed size range. This calculation process can be called intra-chip QKV self-attention calculation.

[0117] Furthermore, by performing intra-slice attention calculations on each sub-feature, the scope of the attention mechanism can be narrowed down to a local region of the feature map. The resulting attention features have a certain inductive bias capability relative to the global features, thus improving the data processing effect.

[0118] like Figure 4 As shown in S403 "Synthesizing Sub-features", the attention features corresponding to each sub-feature are concatenated through a tiling operation to obtain the first feature.

[0119] In this embodiment, for the input feature tensor (b, c, h, w), after being split into multiple sub-feature tensors (8b, c, 3, 3) according to a 3×3 size, after calculating the attention feature corresponding to each sub-feature, the attention features corresponding to each sub-feature are concatenated by a tiling (UnFold) operation to obtain the tensor of the first feature as (b, c, h, w); that is, after performing intra-chip attention calculation, the output size is the same as the size of the input feature.

[0120] In this embodiment of the disclosure, the computational complexity can be expressed as O((N×M)). 2×L), where L represents the number of sub-features obtained from the split. Taking a sub-feature size of 3×3 and L=8 as an example, the computational complexity of the in-chip attention calculation performed through the above steps S401-S403 is O. Intra ((3×3) 2 ×8).

[0121] like Figure 4 As shown in S404 "Get Downsampled Features", the input features are downsampled to obtain downsampled features, and each element in the downsampled features corresponds to a sub-feature.

[0122] In this step, the average pooling operation can be performed on the input feature tensor (b, c, h, w) using the AvgPool function. Figure 4 The example illustrates the AvgPool function with parameters k and s, indicating that it can slide across the input feature tensor (b, c, h, w) in the form of a window (window size k×s). Each window corresponds to one extracted feature, and the feature value of the corresponding feature is obtained by calculating the average value within the window, resulting in a downsampled feature tensor of (b, c, h / k, w / s). Here, k and s are both integers greater than or equal to 1, and k and s are not both equal to 1.

[0123] As an example, when the parameters k=3 and s=3 of the average pooling function, it slides in the form of a 3×3 window on the tensor (b, c, 6, 12) of the input feature. Each window corresponds to a feature point of the downsampled feature. The average value of the feature value in each window is calculated to obtain the feature value of each feature point in the downsampled feature. The tensor of the downsampled feature is (b, c, 2, 4).

[0124] like Figure 4 As shown in S405 "Inter-slice attention calculation", attention is calculated based on the downsampled features to obtain the attention features corresponding to the downsampled features.

[0125] In this step, inter-patch QKV self-attention is calculated for each sub-feature to obtain the attention feature corresponding to the downsampled feature. For example, in... Figure 4 In the example, the tensor of the downsampled features is (b, c, 2, 4). The attention computation performed on the downsampled features can realize the QKV self-attention mechanism operation with global modeling capability within a "global" range of 2×4, and the computational complexity is O(n log n). Inter ((2×4) 2 ).

[0126] like Figure 4As shown in S406 "Obtain downsampled features", the attention features corresponding to the downsampled features are upsampled to obtain the second feature.

[0127] In this step, an upsampling process can be performed using an unpooling operation to obtain a second feature; wherein the size of the second feature is the same as the size of the input feature.

[0128] In this embodiment of the disclosure, the attention calculation performed on the downsampled features is performed by using each feature point in the downsampled features to represent a local region or a piece in the input features, and then performing attention calculation on the downsampled features to perform global modeling, thereby establishing the intrinsic relationship between the isolated pieces.

[0129] In some embodiments, steps S401-S403 correspond to intra-chip attention calculation, and steps S404-S406 correspond to inter-chip attention calculation. The intra-chip attention calculation process and the inter-chip attention calculation process can be executed synchronously, thereby improving computational efficiency.

[0130] like Figure 4 S407 "Calculate the average value of elements" determines the average value of elements in the same position in the first and second features, thus obtaining the third feature.

[0131] like Figure 4 S408 in the algorithm "Perform element summation operation" sums the elements in the third feature that are in the same position as the input feature to obtain the fourth feature, and determines the fourth feature as the output feature of the predetermined image.

[0132] The image processing method described in steps S401-S408 above has a computational complexity of O. Intra ((3×3) 2 ×8)+O Inter ((2×4) 2 )=O(648)+O(64)=O(712); Since the size of the input feature is 6×12, if combined as described above Figure 1 and Figure 2 The standard QKV self-attention computation described has a computational complexity of O. std ((6×12) 2 = O(5184). Therefore, it can be seen that the self-attention operation performed on the input features using the method of this embodiment can reduce the computational complexity of self-attention calculation while retaining the global modeling capability of self-attention calculation.

[0133] It is understood that the various method embodiments mentioned above in this disclosure can be combined with each other to form combined embodiments without violating the principle and logic. Due to space limitations, this disclosure will not elaborate further. Those skilled in the art will understand that in the above methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0134] In addition, this disclosure also provides an image processing apparatus, an artificial intelligence chip, an electronic device, and a computer-readable storage medium, all of which can be used to implement any of the image processing methods provided in this disclosure. The corresponding technical solutions and descriptions are described in the corresponding section of the method and will not be repeated here.

[0135] Figure 5 This is a block diagram of an image processing apparatus provided according to an embodiment of the present disclosure. (Refer to...) Figure 5 This disclosure provides an image processing apparatus 500, which may include the following modules.

[0136] The acquisition module 510 is used to acquire the input features of the image to be processed;

[0137] The splitting module 520 is used to split the input features into multiple sub-features;

[0138] The determination module 530 is used to determine the first attention feature and the second attention feature corresponding to multiple sub-features, wherein the first attention feature is obtained by self-attention calculation based on multiple element features contained in the sub-feature, and the second attention feature is obtained by attention calculation between the sub-feature and other sub-features;

[0139] The fusion module 540 is used to perform feature fusion processing on the first attention feature and the second attention feature to obtain the output feature;

[0140] The processing module 550 is used to process the image to be processed based on the output features.

[0141] According to the image processing apparatus of this disclosure, feature fusion can be performed on the self-attention calculation result with local modeling capability and the self-attention calculation result with global modeling capability, thereby reducing the computational complexity of self-attention calculation while retaining its global modeling capability. Furthermore, in the process of performing data processing tasks in the field of image processing, this image processing apparatus can correlate global information and detailed information of the image, making the image processing result more accurate and effectively improving the accuracy of image recognition. In addition, since the above-mentioned apparatus can reduce the computational complexity of self-attention calculation during the self-attention calculation process, the efficiency of image processing is effectively improved.

[0142] Figure 6 This is a schematic block diagram of an image processing apparatus according to another embodiment of the present disclosure. Figure 6 and Figure 5 Identical or equivalent components use the same label. (See below for details.) Figure 5 and Figure 6 The image processing apparatus according to another embodiment of the present invention will be described in detail.

[0143] In some embodiments, the determining module 530 is specifically configured to: perform attention calculation based on the element features contained in each sub-feature to obtain a first attention feature corresponding to each sub-feature; perform downsampling processing on each sub-feature to obtain a downsampled feature of each sub-feature; and perform attention calculation based on the downsampled features of multiple sub-features to obtain a second attention feature corresponding to each sub-feature.

[0144] In some embodiments, the determining module 530 includes a decomposition calculation unit 531; the decomposition calculation unit 531 is used to: decompose the input feature according to the size of N×M to obtain multiple sub-features of size N×M; wherein N and M are both integers greater than or equal to 1; if any sub-feature obtained by decomposition has missing values, then zero-padding is performed on the missing values.

[0145] In some embodiments, the determining module 530 includes a convergence calculation unit 532; the convergence calculation unit 532 is used to: obtain multiple sub-features obtained by splitting the input features, perform downsampling processing on each sub-feature, and obtain downsampled features of each sub-feature.

[0146] In some embodiments, the determining module 530 includes an attention calculation array 533; the attention calculation array 533 is used to perform a preset number of linear transformations on each element feature contained in each sub-feature to obtain the query feature, key feature and value feature corresponding to each element feature of the sub-feature respectively; and to perform self-attention calculation on the query feature, key feature and value feature corresponding to each element feature of each sub-feature to obtain the first attention feature corresponding to each sub-feature.

[0147] In some embodiments, the attention calculation array 533 is further configured to perform a predetermined number of linear transformations on the downsampled features of multiple sub-features to obtain the query features, key features, and value features corresponding to the downsampled features of each sub-feature; and to perform self-attention calculation on the query features, key features, and value features corresponding to the downsampled features of multiple sub-features to obtain the second attention features corresponding to each sub-feature.

[0148] In some embodiments, the fusion module 540 includes a reverse disassembly unit 541, a reverse convergence unit 542, an average value calculation unit 543, and a summation calculation unit 544.

[0149] The inverse decomposition unit 541 is used to concatenate the first attention features corresponding to multiple sub-features to obtain the first feature; wherein the size of the first feature is the same as the size of the input feature; the inverse convergence unit 542 is used to upsample the second attention features corresponding to multiple sub-features to obtain the second feature; the size of the second feature is the same as the size of the input feature; the average value calculation unit 543 and the summation calculation unit 544 are used to perform feature fusion based on the first feature and the second feature to obtain the output feature.

[0150] In some embodiments, the average value calculation unit 543 is used to determine the average value of the elements with the same position in the first feature and the second feature to obtain the third feature; the summation calculation unit 544 is used to perform summation calculation on the elements with the same position in the third feature and the input feature to obtain the fourth feature, and the fourth feature is used as the output feature.

[0151] In some embodiments, the image processing method is applied to an artificial intelligence chip, which includes at least a plurality of attention computing arrays; the determining module 530 is specifically configured to: allocate a first attention computing array to each sub-feature, perform attention calculations using each first attention computing array based on the element features contained in the corresponding sub-feature, and obtain a first attention feature corresponding to each sub-feature; allocate a second attention computing array to a plurality of sub-features, and perform attention calculations using the second attention computing arrays on the sub-features and other sub-features, and obtain a second attention feature corresponding to each sub-feature.

[0152] According to the image processing apparatus of this disclosure, after decomposing the input features into multiple sub-features, each sub-feature can represent a local region of the input features. Self-attention calculation is performed based on the multiple element features contained in the sub-features to obtain a first attention feature with local modeling capabilities. Then, a second attention feature is obtained by performing attention calculation between the sub-features and other sub-features, resulting in a self-attention calculation result with global modeling capabilities. Feature fusion is performed on the self-attention calculation result with local modeling capabilities and the self-attention calculation result with global modeling capabilities, which can reduce the computational complexity of self-attention calculation while retaining its global modeling capabilities. According to the embodiments provided in this disclosure, in the process of performing data processing tasks in the field of image processing, this image processing apparatus can associate global information and detailed information of the image, making the image processing results more accurate, thereby effectively improving the accuracy of image recognition. Furthermore, since the above apparatus can reduce the computational complexity of self-attention calculation during the self-attention calculation process, it effectively improves the efficiency of image processing.

[0153] Figure 7 A schematic diagram of the structure of the artificial intelligence chip in an embodiment of this disclosure is shown.

[0154] like Figure 7 As shown, the artificial intelligence chip includes at least an attention computing array 710 and other computing arrays (not shown in the figure); wherein, the attention computing array 710 includes multiple first computing arrays 711 and one second computing array 712; the other computing arrays include: a decomposition computing unit 721, a convergence computing unit 722, and a feature fusion computing unit 723; the decomposition computing unit 721 is used to decompose the input features of the image to be processed into multiple sub-features; the convergence computing unit 722 is used to perform downsampling processing on each sub-feature to obtain a downsampled feature of each sub-feature; the first computing array 711 is used to perform self-attention calculation based on multiple element features contained in the sub-features to obtain a first attention feature corresponding to each sub-feature; the second computing array 712 is used to perform attention calculation on the sub-features and other sub-features to obtain a second attention feature corresponding to each sub-feature; the feature fusion computing unit 723 is used to perform feature fusion processing on the first attention feature and the second attention feature to obtain the output feature.

[0155] The chip also includes an image processing unit; the image processing unit is used to process the image to be processed based on the output features.

[0156] In this embodiment of the disclosure, the main computational load lies in the QKV self-attention computation, and the attention computation array 1010 is used to accelerate the QKV self-attention computation process. As mentioned above, the intra-chip attention computation process is executed within each chip, and there is no dependency between chips. Therefore, multiple chips obtained by decomposing the input features can be assigned to different QKV attention arrays (i.e., the first computation array) for parallel computation. However, the inter-chip attention computation cannot be further decomposed, so it can be computed exclusively in a QKV attention array (i.e., the second computation array).

[0157] According to the embodiments of the present disclosure, the artificial intelligence chip reduces the complexity of self-attention computation through local modeling and enhances its ability to capture contextual information through global modeling. Feature fusion of the self-attention computation results with local and global modeling capabilities reduces the computational complexity of self-attention computation while preserving its global modeling capabilities. According to the embodiments provided in this disclosure, during image processing tasks, the artificial intelligence chip can correlate global and detailed information of the image, resulting in more accurate image processing results and effectively improving image recognition accuracy. Furthermore, the reduced computational complexity of self-attention computation through this artificial intelligence chip effectively improves image processing efficiency.

[0158] In this embodiment of the disclosure, in Figure 7 In the middle, the disassembly calculation unit 721 and Figure 6 The disassembly calculation unit 531 in the middle has the same structure. Figure 7 The convergence computing unit 722 in the middle and Figure 6 The aggregation computing unit 532 in the middle has the same structure, and, Figure 7 Feature fusion computing unit 723 and Figure 5 The fusion module 540 in the middle has the same structure; wherein, the same structure is used to implement the same function.

[0159] It should be clarified that the present invention is not limited to the specific configurations and processes described in the above embodiments and shown in the figures. For the sake of convenience and brevity, detailed descriptions of known methods are omitted here, and the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0160] Figure 8 This is a block diagram of an electronic device provided in an embodiment of the present disclosure.

[0161] Reference Figure 8 This disclosure provides an electronic device, which includes: at least one processor 801; at least one memory 802; and one or more I / O interfaces 803 connected between the processor 801 and the memory 802; wherein the memory 802 stores one or more computer programs that can be executed by the at least one processor 801, and the one or more computer programs are executed by the at least one processor 801 to enable the at least one processor 801 to perform the above-described image processing method.

[0162] This disclosure also provides a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor / processing core, implements the image processing method described above. The computer-readable storage medium may be volatile or non-volatile.

[0163] This disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in the processor of an electronic device, the processor in the electronic device executes the above-described image processing method.

[0164] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0165] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable program instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0166] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0167] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0168] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0169] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0170] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable image processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable image processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable image processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0171] Computer-readable program instructions may also be loaded onto a computer, other programmable image processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable image processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable image processing apparatus, or other device to perform the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0172] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0173] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in connection with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in connection with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of this disclosure as set forth by the appended claims.

Claims

1. An image processing method, characterized in that, include: Obtain the input features of the image to be processed; The input features are split into multiple sub-features; A first attention feature and a second attention feature are determined for multiple sub-features, wherein the first attention feature is obtained by self-attention calculation based on multiple element features contained in the sub-feature, and the second attention feature is obtained by attention calculation between the sub-feature and other sub-features; The first attention feature and the second attention feature are fused to obtain the output feature. Based on the output features, the image to be processed is processed; The feature fusion process of the first attention feature and the second attention feature to obtain the output feature includes: The first attention features corresponding to multiple sub-features are concatenated to obtain a first feature; wherein the size of the first feature is the same as the size of the input feature. Upsampling is performed on the second attention features corresponding to multiple sub-features to obtain second features; the size of the obtained second features is the same as the size of the input features. The first feature and the second feature are fused to obtain the output feature.

2. The method according to claim 1, characterized in that, Determining the first attention feature and the second attention feature corresponding to the multiple sub-features includes: Attention is calculated based on the element features contained in each sub-feature to obtain the first attention feature corresponding to each sub-feature; Each of the sub-features is downsampled to obtain the downsampled feature of each sub-feature; Attention is calculated based on the downsampled features of multiple sub-features to obtain the second attention feature corresponding to each sub-feature.

3. The method according to claim 1, characterized in that, The step of splitting the input features into multiple sub-features includes: The input features are split into multiple N×M sub-features; where N and M are both integers greater than or equal to 1. If any of the sub-features obtained from the splitting has missing values, then the missing values ​​are padded with zeros.

4. The method according to claim 2, characterized in that, The step of performing attention calculation based on the element features contained in each sub-feature to obtain the first attention feature corresponding to each sub-feature includes: Based on the element features contained in each sub-feature, a predetermined number of linear transformations are performed to obtain the query feature, key feature, and value feature corresponding to each element feature of the sub-feature; Self-attention is calculated for each element feature of each sub-feature, corresponding to the query feature, key feature, and value feature, to obtain the first attention feature corresponding to each sub-feature.

5. The method according to claim 2, characterized in that, The step of performing attention calculation based on the downsampled features of multiple sub-features to obtain the second attention feature corresponding to each sub-feature includes: Perform a predetermined number of linear transformations on the downsampled features of multiple sub-features to obtain the query features, key features, and value features corresponding to the downsampled features of each sub-feature; Self-attention is calculated for the query features, key features, and value features corresponding to the downsampled features of multiple sub-features to obtain the second attention features corresponding to each sub-feature.

6. The method according to claim 1, characterized in that, The step of fusing features based on the first feature and the second feature to obtain output features includes: The third feature is obtained by determining the average value of the elements with the same position in the first and second features; The fourth feature is obtained by summing the elements in the same position in the third feature and the input feature; The fourth feature is used as the output feature.

7. The method according to claim 1, characterized in that, The image processing method is applied to an artificial intelligence chip, which includes at least multiple attention computing arrays. Determining the first attention feature and the second attention feature corresponding to the multiple sub-features includes: A first attention calculation array is assigned to each of the sub-features, and attention calculation is performed using each first attention calculation array based on the element features contained in the corresponding sub-feature to obtain the first attention feature corresponding to each sub-feature; A second attention calculation array is assigned to each of the sub-features, and attention calculation is performed on the sub-features and other sub-features using the second attention calculation array to obtain the second attention feature corresponding to each sub-feature.

8. An image processing apparatus, characterized in that, An image processing method for implementing any one of claims 1-7 includes: The acquisition module is used to acquire the input features of the image to be processed; The splitting module is used to split the input features into multiple sub-features; The determining module is used to determine a first attention feature and a second attention feature corresponding to multiple sub-features, wherein the first attention feature is obtained by self-attention calculation based on multiple element features contained in the sub-feature, and the second attention feature is obtained by attention calculation between the sub-feature and other sub-features; The fusion module is used to perform feature fusion processing on the first attention feature and the second attention feature to obtain the output feature; The processing module is used to process the image to be processed based on the output features.

9. An artificial intelligence chip, characterized in that, The chip is used to implement the image processing method as described in any one of claims 1-7, and includes at least an attention computing array and other computing arrays; wherein the attention computing array includes a plurality of first computing arrays and a second computing array; the other computing arrays include: a decomposition computing unit, a convergence computing unit, and a feature fusion computing unit; The decomposition calculation unit is used to decompose the input features of the image to be processed into multiple sub-features; The aggregation calculation unit is used to perform downsampling processing on each of the sub-features to obtain the downsampled features of each of the sub-features; The first computing array is used to perform self-attention calculation based on multiple element features contained in the sub-feature to obtain a first attention feature corresponding to each sub-feature; The second computation array is used to perform attention calculations on the sub-features and other sub-features to obtain the second attention feature corresponding to each sub-feature; The feature fusion calculation unit is used to perform feature fusion processing on the first attention feature and the second attention feature to obtain the output feature.

10. The chip according to claim 9, characterized in that, The chip also includes an image processing unit; The image processing unit is used to process the image to be processed based on the output features.

11. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs that can be executed by the at least one processor, the one or more of the computer programs being executed by the at least one processor to enable the at least one processor to perform the image processing method as described in any one of claims 1-7.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the image processing method as described in any one of claims 1-7.

Images